Virtual threads

The basic unit of execution in Java has long been the thread, now called platform thread to distinguish it from virtual thread. A JVM thread is a thin wrapper around an OS thread–that is, there is a one-to-one relationship between JVM and OS threads.

The problem is that threads are expensive to create. They share code, data, and heap, but have separate stacks. The default stack size for my JVM is 2048 KB, or 2 MB, so a few thousand threads will exhaust memory. When trying to create 10,000 threads, we get:

{:type java.lang.OutOfMemoryError
 :message unable to create native thread: possibly out of memory or process/resource limits reached
 :at [java.lang.Thread start0 Thread.java -2]}

So we have to be careful not to create too many at once. This can be done by reusing threads with a ThreadPoolExecutor.

A major source of performance issues for applications is blocking IO. When a thread services a request by talking to some external system, it may need to wait a long time for a response. While the thread is waiting, it cannot do anything else, such as service another request.

An alternative is asynchronous IO. Instead of waiting, the thread returns immediately. When the response is ready, it's fed to a callback which is then scheduled for execution. This style of programming is especially popular with languages such as JavaScript, which is single-threaded and thus really can't afford to block.

Let's take a look at blocking IO, in this case an HTTP GET request. First, we see that there are 10 cores for execution:

(.availableProcessors (Runtime/getRuntime))
;; => 10

We launch 1,000 threads to hit a test endpoint that delays responses by 1 second.

(require '[clj-http.client :as http]
         '[taoensso.timbre :as log])

(import java.lang.Thread
        (java.util.concurrent CountDownLatch Executors))

(let [n 1000
      div (/ n 10)
      latch (CountDownLatch. n)
      threads (for [i (range n)
                    :let [f #(do
                               (http/get "https://httpstat.us/200?sleep=1000")
                               (when (zero? (mod i div))
                                 (log/info i))
                               (.countDown latch))]]
                (Thread. f))]
  (time (do
          (doseq [th threads]
            (.start th))
          (.await latch))))

How long should this take? Well, 10 cores can execute 10 threads at once. Due to the specified delay, each call takes at least 1 second¹, so in total this should take at least 100 seconds, right?

Wrong. This actually takes less than 5 seconds. Why? Because although each JVM thread does blocking IO on the HTTP call, the corresponding OS threads are not run continuously until their JVM threads finish. Instead, the CPU scheduler uses preemptive multitasking to make sure each thread receives an opportunity to start running early on. This means the 1,000 threads all get to make the HTTP call, wait, and return, resulting in significant time savings.

The issue here is that we're creating 1,000 threads. As mentioned, threads are expensive, and we may not want to create so many. Using a thread pool, we can limit ourselves to 100 threads:

(let [ex (Executors/newFixedThreadPool 100)
      n 1000
      div (/ n 10)
      latch (CountDownLatch. n)]
  (time (do
          (doseq [i (range n)
                  :let [f #(do
                             (http/get "https://httpstat.us/200?sleep=1000")
                             (when (zero? (mod i div))
                               (log/info i))
                             (.countDown latch))]]
            (.submit ex ^Runnable f))
          (.await latch))))

This now takes around 20 seconds, or 4x longer.

So what happens if we use virtual threads?

(let [n 1000
      div (/ n 10)
      latch (CountDownLatch. n)]
  (time (do
          (doseq [i (range n)
                  :let [f #(do
                             (http/get "https://httpstat.us/200?sleep=1000")
                             (when (zero? (mod i div))
                               (log/info i))
                             (.countDown latch))]]
            (Thread/startVirtualThread f))
          (.await latch))))

This takes around 5 seconds, the same as using platform threads, albeit it's much more resource efficient due to using a ForkJoinPool to schedule the virtual threads instead of creating an OS thread per JVM thread. The pool, by default, has as many platform threads as available processors (in our case, 10).

The parallelism of the scheduler is the number of platform threads available for the purpose of scheduling virtual threads. By default it is equal to the number of available processors, but it can be tuned with the system property jdk.virtualThreadScheduler.parallelism.

So why aren't we seeing the slowdown that we saw when using a threadpool with platform threads? Well, that's the key feature of virtual threads–the runtime automatically parks them when a blocking IO call is made.

When code running in a virtual thread calls a blocking I/O operation in the java.* API, the runtime performs a non-blocking OS call and automatically suspends the virtual thread until it can be resumed later.

This means that while one virtual thread is awaiting the HTTP response, others can be run by the threadpool. This is what allows for a thread-based programming style without the penalty of blocking.

Since virtual threads are cheap, we should be able to increase the number of threads to 10,000, right? Well, no. We get the following:

Exception in thread "" java.net.SocketException: Too many open files
    at java.base/sun.nio.ch.Net.socket0(Native Method)
    at java.base/sun.nio.ch.Net.socket(Net.java:534)
    at java.base/sun.nio.ch.Net.socket(Net.java:528)

The problem is that HTTP requests create socket connections, which are treated as files and use file descriptors, of which each process is given a limited number. On Mac OSX, the limit is 10240, which can by altered with the -XX:-MaxFDLimit JVM option.

There are still some issues with virtual threads, however. One is pinning, the binding of a virtual thread to a platform thread, which undermines the M-to-N advantage. This can occur when a virtual thread tries to park inside of a synchonized block, or when it invokes native code. It's not always clear when this can happen, especially with code you didn't write. The JEP suggests that the issue with synchonized may be resolved soon, so it's probably better to wait until that happens before moving to production.

Though virtual threads are still new, the promise of writing code in a linear style using the familiar construct of threads instead of breaking it into async tasks along occurrences of blocking IO is very attractive. For years developers have relied on an assortment of frameworks to deal with concurrency, and it's encouraging to see the JVM team addressing longstanding pain points.