Redis Overview and Benchmark

Image Courtesy Morioh

ReDiS which stands for Remote Directory Server, is an open source in-memory data store, used as a database and as a cache. Redis provides data structures such as strings, hashes, lists, sets and sorted sets. Redis has built-in replication, Lua scripting, LRU eviction, transactions, and different levels of on-disk persistence and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster.

Redis is an open source, advanced key-value store and an apt solution for building highperformance, scalable web applications.

Redis has three main features that sets it apart from others:

• Redis holds its database entirely in the memory, using the disk only for persistence.
• Redis has a relatively rich set of data types when compared to many key-value data stores.
• Redis can replicate data to any number of slaves.
  

Following are certain advantages of Redis:

• Exceptionally fast − Redis is very fast and can perform about 110000 SETs per second, about 81000 GETs per second.
• Supports rich data types − Redis natively supports most of the datatypes that developers already know such as list, set, sorted set, and hashes. This makes it easy to solve a variety of problems as we know which problem can be handled better by which data type.
• Operations are atomic − All Redis operations are atomic, which ensures that if two clients concurrently access, Redis server will receive the updated value.
• Multi-utility tool − Redis is a multi-utility tool and can be used in a number of use cases such as caching, messaging-queues (Redis natively supports Publish/Subscribe), any short-lived data in your application, such as web application sessions, web page hit counts, etc.
  
  

Redis Monitoring

Availability, the redis server will respond to the PING command when it's running smoothly.

$ redis-cli -h 127.0.0.1 ping
PONG

Cache Hit Rate

This information can be calculated with the help of INFO command.

$ redis-cli -h 127.0.0.1 info stats | grep keyspace
keyspace_hits:1069963628
keyspace_misses:2243422165

Workload Statistics

The first two stats talks about connections and commands processed where last two stats talk about bytes received and sent from the redis server.

$ redis-cli -h 127.0.0.1 info stats | grep "^total"
total_connections_received:1687889
total_commands_processed:5602955422
total_net_input_bytes:198210899161
total_net_output_bytes:309040592973

Key Space

Anytime to know number of keys in the database, use this command. The size of the keyspace with a quick drop or spike in the number of keys is a good indicator of issues.

$ redis-cli -h 127.0.0.1 info keyspace
# Keyspace
db0:keys=3857884,expires=277,avg_ttl=259237

Clear Keys

We can clear all the keys from the Redis, using the below command.

$ redis-cli -h 127.0.0.1
127.0.0.1:6379> flushall

How to Perform Redis Benchmark?

Redis benchmark is the utility to check the performance of Redis by running n commands simultaneously.

redis-benchmark [option] [option value]

Option Description
-h Specifies server host name 127.0.0.1
-p Specifies server port 6379
-c Specifies number of parallel connections, default is 50
-n Specifies total number of requests, default is 100000
-d Specifies data size of SET/GET value in bytes, default is 3
-r Use random keys for SET/GET/INCR
-q Forces Quiet to Redis. Just shows query/sec values
-l Generates loop, Run the tests forever
-t Only runs the comma-separated list of tests
--csv Output in CSV format

$ redis-benchmark -h 127.0.0.1 -n 100000 -q
PING_INLINE: 57306.59 requests per second
PING_BULK: 57273.77 requests per second
SET: 56657.22 requests per second
GET: 57012.54 requests per second
INCR: 57240.98 requests per second
LPUSH: 57045.07 requests per second
RPUSH: 56657.22 requests per second
LPOP: 57142.86 requests per second
RPOP: 57175.53 requests per second
SADD: 56369.79 requests per second
HSET: 55679.29 requests per second
SPOP: 54704.60 requests per second
LPUSH (needed to benchmark LRANGE): 52798.31 requests per second
LRANGE_100 (first 100 elements): 35448.42 requests per second
LRANGE_300 (first 300 elements): 17618.04 requests per second
LRANGE_500 (first 450 elements): 12812.30 requests per second
LRANGE_600 (first 600 elements): 10036.13 requests per second
MSET (10 keys): 47281.32 requests per second

References

• Documentation
• Command Reference
• Benchmark

Read More

Materialized Views in RDBMS - Is it a View or Table?

 

Google uses structured data to understand the content on the page and use that data to display in richer features in the search results.

Below you can see the difference in the search result data provided by Google for the OnePlus 8T product from official site vs amazon site. The amazon one has additional data rendered in the same search results.

Image Courtesy - Shushma

As we started to build the feature to feed the product details to Google, the data has to be retrieved from various tables from the catalog schema based on multiple  conditions.

Once we had the API ready, we did load testing, having 25 threads with a ramp-up time of 5 seconds and test duration of 30 seconds. Though we know the search engine bots won't put so much load on the site but the performance of the API was a concern to check.

If you look at the 95 percentile of time, it took 10 seconds to respond and the number of requests reached 430 only. Team members came up with the idea of caching the data on redis, so that next call will be faster. Caching at application layer for this use case will not be of that great help as we have a TTL of 600 seconds on Redis.

I recalled in one of my previous assignments, where we had a similar use case and we had utilised the caching of the data on the database (materialized view) side instead on the application layer which had given better results.

Understand Terminologies

Let's start with TABLE, it's basically an organized storage for your data in the form of rows and columns. You can easily query the table using predicates on the columns. 

To simplify the queries or maybe to apply different security mechanisms on data being accessed you can use VIEWs. Think of a view as glasses through which you can look at your data without knowing the actual tables and realtionship details etc.

If the table is a storage, a view is just a way of looking at it or a projection of the data as view doesn't store the data physically. If you query a table, you fetch its data directly. On the other hand, when you query a view, you are basically querying another query that is stored in the view's definition. But the query planner is aware of that and build a plan to merge the two together and give the results.

Between the table and view, we have the MATERIALIZED VIEW. It's a view that has a query in its definition and uses this query to fetch the data directly from the storage, but it also has it's own storage that basically acts as a cache in between the underlying table(s). 

Now you might have a question, if any data is updated in the underlying table(s) whether they will reflect in materialized view. The simple answer is NO as materialized view acts like a cache, it wont reflect the changes. We need to refresh the materialized view, through a process that would cause it's definition's query to be executed again on actual data and rebuilds the cache.

Materialized View Approach

We created a materialized view to cache the data on the database side, as we were retrieving the data from multiple tables. The required data is available in it, so we can avoid lot of computation at the real-time and also the fetched data won't change very frequently.

With the code changes, we did same load testing having 25 threads with a ramp-up time of 5 seconds and test duration of 30 seconds.

If you look at the 95 percentile of time it took 75 milli seconds to respond and the number of requests reached to 8376. The performance gain with respect to throughput is 19x and 95 percentile of time is 134x time faster. The results looks wonderful and it's without caching on the application layer side.

Query Plan

• The original query which used to join multiple tables with various conditions had the query planning time of 0.852 ms and execution time of 5.511 ms.
• The query which we fire on materialized view from the application layer had the query planning time of 0.087 ms and execution time of 0.052 ms.

Conclusion

• The Materialized View is supported by all relational databases like Oracle, MySQL, Postgres etc.
• The  Materialized View is a powerful tool enabling many performance improvements while providing another way of ensuring data consistency.

References

• Materialized Views by Oracle
• Materialized Views by PostgreSQL

Read More