How we cut down memory usage by 82%

RAM memory in comparison to disk storage is not so cheap. Having big Redis cluster could lead to some expenses on such memory. Thanks to lessons we learned on our previous projects we can share with you some optimizing techniques. A Nice fact is, they are based on application-side so they won’t affect the performance of your database servers.

 

Compress your data!

It’s a big win if you compress data before saving to Redis.

In one of our Redis instance we store mid-large json objects. Thanks to PHP gzcompress function which internally uses ZLIB library we were able to reduce memory usage by 82% – from about 340 GB to 60 GB. Due to that, now we need only one server with 64GB of RAM instead of having an instance with 512 GB which would be likely 3 times more expensive. It was the way we could scale up and avoid premature horizontally scaling.

Test

Before using compression we’ve made some tests which let us to chose the best algorithm for us. We prepared test cases for gzcompress and bzcompress algorithms with different compression levels.

They made 500 iterations of the test with ~30KB JSON objects and:

• calculated compression ratios
• counted overhead time of the algorithms

Results

Gzcompress proved to be about 6 times faster than bzcompress when compressing and decompressing data.

Compress/Uncompress times in comparison with different algorithms

Compression ratio was similar for both algorithms.

Compression ratio in comparision with different algorithms

So decision was easy – we used gzcompress because of its speed and good compression ratios.

The CPU overhead in comparison to memory saving is small. Also compression operations are running on application server, so they aren’t affecting in any way Redis performance.

Remember that using this approach is not always appropriate. You shouldn’t do it when your data doesn’t compress well:

• Isn’t well-structured – JSON, XML is good at compression because of repeated characters, tags
• You have millions of small keys – which e.g. stores some numbers, short words

Of course you should make tests on your own using data stubs which you would store in the database.

Use short keys

Easy trick to implement but can also reduce amount of required memory.

Think of it: if you’ve 100,000,000 keys named like

my/fancy/key-name/... (18 characters)

the overhead for just storing names is rather big in comparision to possible another format like:

m/f/kn/...   (7 characters)

you save 11 characters (11 bytes), which at big scale could let you to save

100,000,000 * 11 bytes = 1 GB of RAM memory!

Using hasmaps

The cool thing in Redis is that, it can handle data types above then standard key-value memcached model.

You must be aware that every key in Redis comes with some additional metadata. So when you’ll store objects like “key=value” that doesn’t mean you will use 8 bytes per key.

redis redis:6379> DEBUG OBJECT my/key
Value at:0x7f36f2980900 refcount:1 encoding:raw serializedlength:4 lru:463740 lru_seconds_idle:1215660

Redis has to e.g. store some info for LRU algorithm.

Thanks to hashes you can save some space. The idea was nicely described by Instagram engineers. Mainly you can create your own, simple memcache in a Redis using hashes 😉 Values stored in hashes don’t come with additional metadata. Internally Redis uses simple dictionary structure for storing them.

More info about this technique can be find at newly added memory optimization tips page on Redis.io.

Last thoughts…

As you can see, some micro optimizations could let you to save much amount of RAM memory which is not so cheap. But as shown on examples – they will be noticeable at larger scale.

In fact, such tips can be used not only in Redis, also other databases can be affected. For e.g. MongoDB need to store duplicated attribute names for every object- using short names will also be effective.