Key Value Benchmark FAQ

written by John Hugg on June 1, 2010 with one Comment

This is a follow up to a previous post on Benchmarking VoltDB against Cassandra on Key-Value-like workloads.
What’s the point of this benchmark?

Point 1: Demonstrate SQL can be fast. Say what you want about our numbers and benchmark, but the language used to manipulate data was SQL and it clearly wasn’t bottlenecking VoltDB performance. We wanted to show the assumption that dropping SQL is a precondition for performance and scalability is false.

Point 2: VoltDB competes well on simple workloads like the key-value puts and gets, as well as complex workloads like our TPC-C like benchmark.

How can I verify what you did without code or details?

As of today, we’ve made the code used to run the benchmark and additional details available in our forums. We welcome efforts to reproduce or refute our results. We welcome feedback on the implementation of the Cassandra driver. We also welcome additional or modified benchmarks based on this code. You may freely distribute this code, but please respect the licenses of VoltDB, the Apache Foundation and other copyright holders of the supporting code.

Benchmark Details Forum Post

Benchmarking is hard. Do you think this benchmark is fair?

We do not intend this benchmark to be used as a tool to help users decide between VoltDB and Cassandra. There are many reasons why someone might choose one system or the other; raw speed on these synthetic workloads is probably not at the top of the list. For the rare case where both choices make sense and speed is an issue, a more workload-specific comparison should still be performed by the deciding user.

Why did you choose to benchmark against Cassandra?

If imitation is the sincerest form of flattery, then choosing a system to benchmark against is probably a close second.

First, we wanted to pick a system without SQL. This was necessary to support our argument that SQL isn’t the problem. This ruled out in-memory SQL databases and clustered SQL databases.

Among the NoSQL leaders, Cassandra seemed to have the most in common with VoltDB. It focuses on horizontal scaling of write-heavy workloads. That’s not to say we think it solves the same problem as VoltDB, just that it’s the most similar of a host of dissimilar options.

Main memory vs disk? 

In these benchmarks, VoltDB is using main memory. Cassandra is keeping the data set in main memory, but periodically flushing blocks to disk. Cassandra can be set to hold back responses to the client for a period, batch their disk writes all together for a sync, then respond to the clients, trading some latency for security. We did not run it in this mode. In either case, the disk bottleneck is for writes only, and it is limited by the sequential write speed of the disk. In our case, the sequential throughput of our 15K RPM disks was substantial.

Still, not using the disk at all is an advantage, but disk isn’t always the bottleneck.

Network can be the bottleneck.

Perhaps what’s more interesting is how often disk isn’t the bottleneck in these situations. A gigabit ethernet connection has a theoretical bandwidth of 125MB per second. Each of the three drives in our test cluster can sequentially write faster than that.

Furthermore, for both VoltDB and Cassandra, the data often moves on the network more times than it is written to disk. And there are network messages that don’t involve data bound for disks, such as agreement, or control messages. Because VoltDB has a more formal partitioning model, stored procedure access, and a native serialization of messages, we believe that VoltDB uses the network less per unit of work. Cassandra offers simpler repartitioning and access from many client languages at the cost of this additional network usage.

In some of the benchmarks we did, I believe the network was the limiting factor for both systems. Also, as mentioned, we gave Cassandra a private channel for intra-cluster traffic that we didn’t give VoltDB. This is a huge advantage for Cassandra.

Lock contention can be the bottleneck.

Another limiting factor is the overhead of multithreaded access to data. VoltDB offers a serialized execution of transactions, the strongest consistency model available. Running a traditional RDBMS that is using serialized consistency in main memory is asking for a lot of contention on locks. This is one of the reasons systems use MVCC or eventual consistency. Yes, these techniques improve the partition tolerance of these systems, but they’re also important to reduce the multithreading overhead.

Before VoltDB, many would say that our level of scaling isn’t possible with our strict consistency. Benchmarks of NoSQL vs SQL would be considered unfair to SQL-based systems because the SQL systems offer more transactional guarantees.

Does VoltDB skip the D in ACID?

D (for Durable) means that once the client has received notice that a transaction has committed, it won’t ever be rolled back. VoltDB doesn’t report to a client application that a transaction has committed until all replicas have acknowledged the transaction completed and the changes are permanent.

As for the safety of the data, VoltDB allows the user to specify the number of copies of data stored in a cluster. If there are two or more copies and any single node fails, then your cluster will continue to operate. In the case of a cluster-wide, immediate power failure, you will lose a record of the transactions that have committed since the last snapshot. Depending on your snapshot policy, data size and disk speed, this could be under a minute, but will usually be several minutes.

Additional protection can be gained by using redundant power supplies and a rack-local UPS that can power the machines during a final snapshot. Further safety including multiple data center operation is planned for future releases.

Can you benchmark against product X?

We’re very busy at VoltDB working on our next release and benchmarking takes time. One of the great things about the open source business model is that anyone from anywhere is free to benchmark our software. The released Key-Value benchmark driver could be extended to support other systems. We also welcome more complex, transactional benchmarks. If you do do any benchmarking of VoltDB, we’d love to hear about it. Just keep in mind that VoltDB was built for partitionable, OLTP-style workloads.

John Hugg
Software Engineer
VoltDB