20Mar/10Off
Hydra attacks Riak cluster
The great thing about querying a Riak cluster is that it's all just HTTP, so the huge universe of tools that are available to work with HTTP resources are at your disposal. For example, lets say we want to quickly request 20 keys from a Riak cluster. We could use curl or curb and request them one at a time, or we could use the multi-headed hydra that is Typhoeus to get them concurrently! Let's see this monster in action with a little Ruby code:
require 'rubygems' require 'typhoeus' require 'yajl' HOST = "riak.cluster.com" PORT = "8098" # bucket is the name of a Riak bucket # keys is an array of keys to get # returns array of response hashes with the keys, :key :code :headers :body def multi_get(bucket, keys) hydra = Typhoeus::Hydra.new @responses = [] keys.each do |key| url = "http://#{HOST}:#{PORT}/riak/#{bucket}/#{key}" request = Typhoeus::Request.new(url) # When the request completes, this block gets run request.on_complete do |response| result = {} result[:key] = key result[:code] = response.code result[:headers] = response.headers result[:body] = response.body @responses << result end # queue up the request to run later hydra.queue request end # This is a blocking call that executes all queued requests concurrently, # and returns when all requests have completed hydra.run @responses end # OK, lets try this baby out now, # first, set up the list of keys we want to get keys = YAML::load(<<EOT) --- - 1244idS1NaricUO2RtXJrjcfzr8 - 12ktMhh8KZOCYzMIRTLlqf5JeGA - 129Izkjd6Fh2i1zqxCE2acT6iju - 129AomedyZIa3gjudjCxpke5kU9 - 12BEfyiKWPqwZqiqcKGmizVN34i - 12EZagKQHnakkIChE3ruLUu4TrA - 12RBVtZV0EwQyTixXFLwHHqwLuK EOT # Now let the Hydra out! rs = multi_get("my_bucket", keys) # Or, if you are storing JSON objects, get the results as JSON using the speedy Yajl gem... rs = multi_get("my_bucket", keys).map{|r| Yajl::Parser.parse(r[:body])}
As you can see, it's pretty darn easy to leverage all the great work by the Typhoeus folks to speed up your queries.
March 23rd, 2010 - 00:01
You might want to check out my gem Riaktor, a Riak client and object mapper that uses Typhoeus. Caveat is that it’s new and probably has some bugs still. http://rubygems.org/gems/riaktor