Redis architecture for production

Redis is a very fast cache server. But in production, there is something to consider.

Redis works in single thread

To be exact, Redis is a multi threaded server. But the main logic runs in a single thread.

For the following single Redis process

[tkstone@localhost bin]$ ps -p 23073 -f
UID        PID  PPID  C STIME TTY          TIME CMD
tkstone  23073     1  0 Nov22 ?        00:09:36 /home/tkstone/redis-4.0.6/bin/redis-server *:6379

when I check it’s threads

[tkstone@localhost bin]$ ps -p 23073 -L -f
UID        PID  PPID   LWP  C NLWP STIME TTY          TIME CMD
tkstone  23073     1 23073  0    4 Nov22 ?        00:09:36 /home/tkstone/redis-4.0.6/bin/redis-server *:6379
tkstone  23073     1 23077  0    4 Nov22 ?        00:00:00 /home/tkstone/redis-4.0.6/bin/redis-server *:6379
tkstone  23073     1 23078  0    4 Nov22 ?        00:00:00 /home/tkstone/redis-4.0.6/bin/redis-server *:6379
tkstone  23073     1 23079  0    4 Nov22 ?        00:00:00 /home/tkstone/redis-4.0.6/bin/redis-server *:6379

The result shows that a Redis process is composed of 4 threads.

  • Thread 1 : main thread which works for servicing almost every command
  • Thread 2 : AOF file close thread. It is active even when AOF is disabled
  • Thread 3 : AOF write thread which runs every 1 second. It’s enabled when AOF is disabled
  • Thread 4 : UNLINK, FLUSHALL or FLUSHDB command processor. (Those commands run asynchronosly) Added from version 4

Therefore, one Redis process usually use 1 cpu core. (by thread 1)

Recommended architecture for Redis

To make full use of a production system, I recommend the following architecture.

redis_architecture

  • Run multi Redis processes per a machine (process count should be equal to that of cpu core)
  • For HA, run slave Redis on another machine
  • Slave machine should have the same computing power with the master
  • For automatic failover, run several Sentinels (each Sentinel should run on a separate machine.) To avoid split brain issue, odd number is recommended
  • Sentinel is a lightweight process. Therefore Sentinel server don’t need high computing power
  • To distribute traffic to multi Redis processes evenly, client needs to implement data sharding

More on data sharding

  • Redis is a key-value store
  • Sharding rule based on key is recommended
  • For the same key, sharding target must be consistent
  • All clients must have the same sharding rule
  • Some client library (i.e. Jedis) supports data sharding
  • When data sharding is enabled, secondary index should be avoided. (More on secondary index)

Limitation on the above architecture

  • If a master and the matching slave die, 1/n of data is lost
  • Sentinel supporting client library is needed. From my experience, Redis and Lettuce support Sentinel
  • Supporting Sentinel and data sharding together makes client development difficult. Next time, I’m explaining how to do it with Jedis

 

 

 

 

One thought on “Redis architecture for production

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.