How to fetch URLs in parallell with EventMachine and Ruby

How to fetch URLs in parallell with EventMachine and Ruby

Ruby posted 10 months ago by christian

Save time by doing things in parallell:

   1  require 'rubygems'
   2  require 'eventmachine'
   3  require 'open-uri'
   4  require 'pp'
   5  require 'thread'
   6  require 'benchmark'
   7  
   8  class Worker
   9    include EM::Deferrable
  10  
  11    def run
  12      get_google
  13      set_deferred_status :succeeded
  14    end
  15  end
  16  
  17  def get_google
  18    # sorry for spamming you
  19    open('http://www.google.com/') do |f|
  20      #pp f.meta
  21    end
  22  end
  23  
  24  def asynchronous(i)
  25    worker = Worker.new
  26    # on success
  27    worker.callback do
  28      p "#{Thread.current} done #{i}!"
  29    end 
  30    worker.errback do 
  31      p "Unexpected error"    
  32      EM.stop  
  33    end
  34    #
  35    Thread.new do
  36      worker.run
  37      EM.stop
  38    end 
  39    #puts "scheduling done!"
  40  end
  41  
  42  def synchronous(i)
  43    get_google
  44  end
  45  
  46  # on error
  47  EM.error_handler do |e|  
  48    p "Unexpected error: #{e}" 
  49  end
  50  
  51  EM.run do
  52    seconds = Benchmark.realtime do
  53      50.times do |i|
  54        asynchronous i
  55      end
  56    end
  57    p "With EventMachine: #{seconds} elapsed..."
  58  
  59    seconds = Benchmark.realtime do
  60      50.times do |i|
  61        synchronous i
  62      end
  63    end
  64    p "Without EventMachine: #{seconds} elapsed..."
  65  end

Output:

 

   1  With EventMachine: 9.05974316596985 elapsed...
   2  Without EventMachine: 19.1381118297577 elapsed...

Conclusion

• Speeds up blocking operations.
• EventMachine is currently limited to one CPU core (native thread) per process.

References

• http://ujihisa.blogspot.com/2009/08/try-eventmachine.html
• http://www.scribd.com/doc/21017871/Event-Machine-Presentation
• http://www.scribd.com/doc/25939580/Event-Machine