Performance Battle: Quarkus vs Phoenix

At my job, I am evaluating some microservices platform options. A good way to test them is doing some benchmarks. Everybody likes benchmarks. So, why not?

Disclaimer: although I tried to make conditions fair and "scientific", it's an amateur benchmark and I'm making this for fun. Please, don't take it too seriously.

It's the "supersonic subatomic Java" framework, sponsored by Red Hat. My work place is a Red Hat shop and we have many subscriptions from them. So, Quarkus is not only a good fit, but we already use it on some projects. It promises a better startup time and memory footprint. Tested version: 1.6.1.

Well, personal preference. I like Elixir, it is a powerful and fun language. When you discover Erlang and what problems it was made to solve, you will find it amazing. WhatsApp, Discord and many others use it. The framework of choice for the Elixir language is Phoenix. Achieving 2 million connections in a single machine is really impressive too. Tested version: 1.5.4.

I couldn't find any real benchmarks on these technologies. There is this benchmark on Netty and Cowboy, which are the technologies behind these frameworks, but using a "sleep" to simulate load is not ok. These frameworks handle databases and connection pools differently. Acquiring a database connection from a pool can timeout and throw an exception. A "sleep" command won't timeout. Serializing data to JSON also impose memory and CPU usage.

Techempower does something different, but it uses good hardware and low load. Its focus is on how many requests per second a framework can handle, and this is not ok too. I need to know what framework can handle more requests within constrained hardware, like a Heroku dyno. So, speed isn't a concern at this point.

In this benchmark, I decided to use my work development database and data. It's an Oracle database and my test data is on a table with 5 columns and 6000 rows. Yes, it is a lot of JSON, so it can force some timeouts, CPU and memory usage. Although Oracle JDBC driver is well tested and supported, in Phoenix we must rely on jamdb_oracle package. It works well and it is maintained, but it's not "official". Oracle database driver is not a first class citizen on Quarkus either, and we can not compile it to a native image using GraalVM out of the box.

We are going to use the siege tool. Quarkus will run on OpenJDK 11 by Red Hat and Phoenix on Erlang OTP 22, both on a Red Hat Linux 8 with 4 cores and 8 GB of RAM. But, we are pretending to be inside a Heroku dyno or docker container, so the memory limit will be 512 MB. On Heroku or docker, any memory allocation beyond this will use disk swap, reducing perfomance. Keep that in mind!

Before we start, we have to remember this. We can not use these frameworks as they come out of the box. To compare apples to apples (and not to oranges) and make things fair, I will configure both frameworks with the same connection pool size and timeout.

In Phoenix, the default connection timeout and pool size are very low, so it can fail fast and use less memory. In Quarkus, the connection timeout is longer and pool size bigger, so it can serve more requests at the cost of memory. Well, that is what I think, but I might be wrong. For the test, we had to impose some limits to the JVM, as suggested by Heroku:

java -Xss512k -Xmx300m -jar <my-app>.jar

We will use siege to hit the REST endpoint with 25, 50, 100 e 250 concurrent connections, 5 times each.

siege -d1 -r5 -c250 http://<my-server>

Winner: Quarkus

Both frameworks have a similar CPU usage profile and "warmup" peak. Quarkus and JVM peaked at 56% of CPU usage while Phoenix and Erlang VM peaked at 79% of CPU usage. OpenJDK uses less CPU over time and is the clear winner.

Winner: Phoenix (x2)

Phoenix peaked at 364 MB and Quarkus peaked at 807 MB. Here we can see Quarkus and JVM problem on microservices. Java refuses to stay within the 512 MB limit. Actually, it uses more than twice the memory Phoenix used. As stated above, at Heroku or docker with a 512 MB of RAM limit, any memory beyond that will use disk swap, reducing performance. In this benchmark, I could not simulate this, because I had memory available.

Why Phoenix won 2 times? I could not show it on the graph, but another thing to notice is that Erlang/Elixir gives back unused memory to the operating system. It peaked at 364 MB, but after the test, it was using only 152 MB. JVM, on the other hand, never return unused memory back to the operating system, it peaked at 807 MB and stayed there. So, Quarkus lost twice here. If you run your own cloud, like me, this is an important and desired behaviour.

Winner: Quarkus

Quarkus took, at most, 34 secs to respond to all requests. Phoenix took 60 secs. Not much to argue here. Quarkus and JVM are faster. And in this benchmark, they are almost twice as fast than Phoenix. Quarkus is the clear winner. If your microservice have any speed requirement, Quarkus is a better fit than Phoenix.

Winner: Draw

What? Why a draw? Well, handling 250 simultaneous connections, Quarkus timed out 110 times while Phoenix timed out 155 times. Quarkus should be the winner. The problem is the difference is not so big. As stated above, Quarkus used more memory than it was available. If it was at Heroku or docker, it would be using disk swap, having perfomance penalties and leading to more timeouts. But, I could not simulate that and Quarkus had this "advantage". So, I will consider it a draw.

As I said in the introduction, I was looking for a framework that could handle more simultaneous connections using less memory. If you are like me and need to deploy your microservices on premise hardware or on Heroku, memory usage comes first. And the ability to return unused memory back to the cluster is desirable. Requirements like speed and CPU usage comes after. Phoenix is the clear winner under this circuntances.

Quarkus, on the other hand, shines where speed is a requirement and memory consumption is acceptable. The cool thing about a microservices archtecture is that you can use the best tool for the job. So, use both frameworks. They are new, comparable and capable.