I really like Redis. Lately, I’ve been working on real-time spatial processing systems and Redis’ combination of distributed data structures and message queues fits that workload perfectly. Storm was another framework I evaluated, but it didn’t fit as well. Storm looks good if you have fixed processing pipelines, but my processing topology changes based on client-defined filters and preferences. I don’t get the automatic scaling and management that Storm provides, but I get a richer set of primitives to build upon and better support for dynamic pipelines.

One of the intriguing aspects in the beta version of Redis is the ability to define Lua scripts that run on the server. I’ll explore some of the scripts I generated and share what I think some of the benefits and challenges are to using them.

We’ll start off really easy. The following method gets a key and expires it at the same time (note we are using Ruby with the redis-rb gem).

1
2
3

def getex(key, seconds)
  self.eval('local result = redis.call("get", KEYS[1]); redis.call("expire", KEYS[1], ARGV[1]); return result', [key], [seconds])
end

With Lua scripting, you need to differentiate key arguments from any other arguments. The eval documentation explains why. On the Ruby side, these are passed in as arrays to eval. On the Lua side, they exist as arrays with the names KEYS and ARGS, respectively. This example simply calls the two separate Redis commands and returns the get result. In this case, all we are gaining from this script is reducing the redundancy of repeating the same key for the two different commands. And by doing so, we are losing the result of the expire command. If we really cared, we could use a Lua table to return both, but this would be a little strange on the Ruby side - we don’t really expect a get to return an array.

1
2
3

def getex(key, seconds)
  self.eval('local result = redis.call("get", KEYS[1]); return {result, redis.call("expire", KEYS[1], ARGV[1])};', [key], [seconds])
end

We can use here documents to make the examples a little more readable. Here is the same expire behavior for mget.

1
2
3
4
5
6
7
8
9
10
11

def mgetex(*keys, seconds)
  return if keys.empty?
  self.eval(<<-SCRIPT, keys, [seconds])
    local result = {};
    for i=1, #KEYS do
      result[i] = redis.call("get",KEYS[i])
      redis.call("expire",KEYS[i], ARGV[1])
    end
    return result
  SCRIPT
end

This shows how you can handle arrays. Again, our main benefit of doing this is avoiding the need to repeat the keys. And in this case, there is no list version of expire so this would require N+1 “plain” Redis commands.

One of the other main benefits of Lua scripts is that they run atomically within the server. That means we can implement commands that read or modify multiple keys and ensure they always operate on a consistent state. Here are some examples of scripts using conditional logic.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

def rpush_if_exists(list_key, *list_values, conditional_key)
  self.eval(<<-SCRIPT, [list_key, conditional_key], list_values)
    local result = nil
    if redis.call("exists", KEYS[2]) == 1 then
      result = redis.call("rpush", KEYS[1], unpack(ARGV, 1, #ARGV))
    end
    return result
  SCRIPT
end

def ltrim_unless_exists(list_key, n, conditional_key)
  self.eval(<<-SCRIPT, [list_key, conditional_key], [n])
    local result = {ok="OK"}
    if redis.call("exists", KEYS[2]) == 0 then
      result = redis.call("ltrim", KEYS[1], -ARGV[1], -1)
    end
    return result
  SCRIPT
end

# kind of opposite of hsetnx
# hsetnx will set field only if field does not exist
# hmset_if_hexists will set other fields only if conditional field exists (or the hash key does not yet exist), then clears the conditional field
def hmset_if_hexists(key, *values, field)
  self.eval(<<-SCRIPT, [key], values+[field])
    if redis.call("exists", KEYS[1]) == 0 or redis.call("hexists", KEYS[1], ARGV[#ARGV]) == 1 then
      redis.call("hmset", KEYS[1], unpack(ARGV, 1, #ARGV-1))
      redis.call("hdel", KEYS[1], ARGV[#ARGV])
      return 1
    end
    return 0
  SCRIPT
end

You can use this technique on almost any combination of operators. One key thing to watch out for, as in the case of ltrim_unless_exists, is to make sure the return value matches the expected value of the underlying command under all cases, including when the desired keys don’t exist in the database.

Another thing I grappled with is the ordering, naming and cardinality of arguments. The underlying Redis and Ruby library restrictions require you to pass in all the keys in one array and all the non-keys in another array. You need to map your Ruby arguments to the right eval call arguments, then map those to the right Lua redis.call arguments. The here documents allow me to put the ruby arguments and eval arguments right next to each other so I only have to look at one place to understand the order of KEYS and ARGV.

I would prefer to have the Lua commands in separate files (e.g. mgetex.lua). This would allow use of these extensions in any Redis client, not just a Ruby one. But the Ruby method signatures are so much more expressive. Lua doesn’t support slicing for me to be able to do something like the following, for example.

1
2
3
4
5
6
7
8
9

local key = KEYS[1]
local values = slice(ARGV, 1, #ARGV-1) -- not valid
local field =  ARGV[#ARGV]
if redis.call("exists", key) == 0 or redis.call("hexists", key, field) == 1 then
  redis.call("hmset", key, unpack(values, 1, #values))
  redis.call("hdel", key, field)
  return 1
end
return 0

That is really my only quibble - that I haven’t found a way to be able to write reusable Lua functions that are easily mapped (possibly automatically mapped) to their Ruby wrapper.

Like I mentioned before, Redis already has a great set of primitives to build distributed systems. The scripting extension lets you make your own primitives that run atomically on the server to preserve data consistency. Check them out.