Introducing structured concurrency 2 – Concurrency – Virtual Threads, Structured Concurrency

We skip the rest of the code since you can find it in the bundled code.Of course, we can implement code-answers to each of these questions via error handling, tasks abandon and abortion, ExecutorService, and so on, but this means a lot of work for the developer. Writing failsafe solutions that carefully cover all possible scenarios across multiple tasks/subtasks while tracking their progress in a concurrent environment is not an easy job. Not to mention how hard is to understand and maintain the resulting code by another developer or even the same developer after 1-2 years or even months.It is time to add some structure to this code, so let’s introduce structured concurrency (or, Project Loom).Structured concurrency relies on several pillars meant to bring lightweight concurrency in Java. The fundamental pillar or principle of structured concurrency is highlighted next.

The fundamental principle of structured concurrency: When a task has to be solved concurrently then all the threads needed to solve it are spun and rejoined in the same block of code. In other words, all these threads’ lifetime is bound to the block’s scope, so we have clear and explicit entry-exit points for each concurrent code block.

Based on this principle, the thread that initiates a concurrent context is the parent-thread or the owner-thread. All threads started by the parent-thread are children-threads or forks, so between them, these threads are siblings. Together, the parent-thread and the children-threads, define a parent-child hierarchy.Putting the structured concurrency principle in a diagram can be seen as follows:

Figure 10.2 – Parent-child hierarchy in structured concurrency

In the context of parent-child hierarchy, we have support for error/exception handling with short-circuiting, cancellation propagation, and monitoring/observability.Error/exception handling with short-circuiting: If a child-thread fails then all child-threads are canceled unless they are complete. For instance, if futureTester(1) fails, then futureTester(2) and futureTester(3) are automatically canceled.Cancellation propagation: If the parent-thread is interrupted until joining the child-threads is over then these forks (the child-threads/subtasks) are canceled automatically. For instance, if the thread executing buildTestingTeam() gets interrupted then its three forks are automatically canceled.Monitoring/observability: A thread dump reveals crystal-clear the entire parent-child hierarchy no matter how many levels have been spawned. Moreover, in structured concurrency, we take advantage of scheduling and memory management of threads.While these are purely concepts, writing code that respects and follows these concepts requires the proper API and the following awesome callout:

Figure 10.3 – Don’t reuse virtual threads

Cut this out and stick it somewhere to see it every day! So, in structured concurrency, don’t reuse virtual threads. I know what you are thinking: hey dude, threads are expensive and limited, so we have to reuse them. A quick hint: we are talking about virtual threads (massive throughput), not classical threads, but this topic is covered in the next problem.

Introducing structured concurrency – Concurrency – Virtual Threads, Structured Concurrency

210. Introducing structured concurrency

If you are as old as I am then most probably you’ve started programming with a language such as BASIC or a similar unstructured programming language. At that time, an application was just a sequence of lines that defined a sequential logic/behavior via a bunch of GOTO statements that drive the flow by jumping as a kangaroo back and forward between the code lines. Well, in Java, the building blocks of a typical concurrent code are so primitive that the code looks somehow like unstructured programming because is hard to follow and understand what’s going on. Moreover, a thread dump of a concurrent task doesn’t provide the needed answers.Let’s follow a snippet of Java concurrent code and let’s stop every time we have a question (always check the code below the question). The task is to concurrently load three testers by id and team them up in a testing team. First, let’s list here the server code (we will use this simple code to serve us in this problem and subsequent problems):

public static String fetchTester(int id)
      throws IOException, InterruptedException {
  HttpClient client = HttpClient.newHttpClient();
  HttpRequest requestGet = HttpRequest.newBuilder()
    .GET()
    .uri(URI.create(“https://reqres.in/api/users/” + id))
    .build();
  HttpResponse<String> responseGet = client.send(
    requestGet, HttpResponse.BodyHandlers.ofString());
  if (responseGet.statusCode() == 200) {
    return responseGet.body();
  }
  throw new UserNotFoundException(“Code: “
    + responseGet.statusCode());
}

Next, the code that we are especially interested in starts as follows:

private static final ExecutorService executor
  = Executors.newFixedThreadPool(2);
public static TestingTeam buildTestingTeam()
   throws InterruptedException {
  …

First stop: As you can see buildTestingTeam() throws an InterruptedException. So, if the thread executing buildTestingTeam() gets interrupted then how can we easily interrupt the following threads:

  Future<String> future1 = futureTester(1);
  Future<String> future2 = futureTester(2);
  Future<String> future3 = futureTester(3);
 
  try {
    …

Second stop: Here we have three get() calls. So, the current thread waits for other threads to complete. Can we easily observe those threads?

    String tester1 = future1.get();
    String tester2 = future2.get();
    String tester3 = future3.get();
    logger.info(tester1);
    logger.info(tester2);
    logger.info(tester3);
    return new TestingTeam(tester1, tester2, tester3);
  } catch (ExecutionException ex) {
    …

Third stop: If an ExecutionException is caught then we know that one of these three Future has failed. Can we easily cancel the remaining two or they will just hang on there? Probably future1 will fail while future2 and future3 will complete successfully or maybe future2 will complete successfully while future3 will just run forever (a so-called orphan thread). This may lead to serious mismatches in the expected results, memory leaks, and so on.

    throw new RuntimeException(ex);
  } finally {
    …

Fourth stop: Next line of code is used to shut down the executor, but is so easy to overlook it. Is this the proper place to do this?
    shutdownExecutor(executor);
  }
}

Fifth stop: If you didn’t spot the previous line of code then is legitimate to ask yourself how/where this executor get shut down.

public static Future<String> futureTester(int id) {
  return executor.submit(() -> fetchTester(id));
}

Technical requirements – Fundamentals of Cloud Architecture

Technical requirements

To fully engage with the content in this chapter on cloud computing architecture, you should have a basic understanding of computer systems, networking concepts, and information technology.

Additionally, the following technical requirements are recommended:

  • Internet access: You should have a reliable internet connection to access online resources, references, and examples related to cloud computing.
  • A computing device: A desktop computer, laptop, tablet, or smartphone with a modern web browser is necessary to read this chapter’s content and access any online materials.
  • A web browser: The latest version of a modern web browser such as Google Chrome, Mozilla Firefox, Microsoft Edge, or Safari is recommended. This ensures compatibility and optimal viewing experience of web-based resources and interactive content.
  • Familiarity with cloud services: Some familiarity with cloud services and their basic functionalities will enhance your understanding of this chapter. This includes knowledge of cloud computing models such as Infrastructure-as-a-Service (IaaS), Platform-as-a-Service (PaaS), and Software-as-a-Service (SaaS).

The history of cloud computing

Cloud computing has a rich history that has evolved over several decades. The concept of cloud computing dates back to the 1960s when computer scientists at MIT and Dartmouth College proposed the idea of a “utility computing” system that would allow users to access computing resources on demand.

In the 1970s, IBM introduced virtualization technology, which allowed multiple operating systems to run on a single mainframe computer. This technology enabled companies to consolidate their IT resources and reduce costs.

In the 1990s, the development of the World Wide Web and the rise of e-commerce led to the creation of web-based applications and services. This led to the development of early cloud computing platforms such as Salesforce, which provided customer relationship management (CRM) services over the internet.

In 2002, Amazon launched its web services division, offering cloud-based infrastructure services such as storage and computing power. This was followed by the launch of Amazon Elastic Compute Cloud (EC2) in 2006, which allowed users to rent computing capacity on demand.

In 2008, Google launched its cloud computing platform, Google App Engine, which allowed developers to build and run web applications on Google’s infrastructure.

Microsoft followed suit in 2010 with the launch of Windows Azure, which provided cloud-based services for building and deploying applications.

The growth of cloud computing has been fueled by advances in virtualization technology, which allows computing resources to be shared and used more efficiently. The development of cloud-based services and infrastructure has also made it easier for businesses to scale their IT resources up or down based on demand.

Today, cloud computing has become an integral part of many businesses, offering a range of benefits such as cost savings, scalability, flexibility, and improved collaboration. Cloud computing has also enabled the development of new technologies such as serverless computing, which allows developers to build and run applications without managing servers or infrastructure.

The main idea behind cloud computing was to provide a flexible and cost-effective way for users to access computing resources on demand. In the early days of computing, businesses and organizations had to invest in their IT infrastructure, including hardware, software, and networking equipment. This was expensive and often required a large upfront investment, which made it difficult for small and medium-sized businesses to compete with larger organizations.

Cloud computing was envisioned as a way to address this challenge by providing a shared pool of computing resources that could be accessed over the internet. This allowed businesses to pay only for the resources they needed, and to scale up or down as needed to meet changing demand.

In addition to cost savings, cloud computing was also seen as a way to improve the flexibility and agility of IT operations. By providing access to a shared pool of resources, cloud computing could enable businesses to quickly deploy new applications, scale up or down as needed, and respond to changing business needs more quickly than traditional IT infrastructure.

The thought behind cloud computing was to provide a more efficient, flexible, and cost-effective way for businesses to access the computing resources they need to operate and compete in today’s fast-paced digital economy.

The benefits of cloud architecture 2 – Fundamentals of Cloud Architecture

Cloud services provide a range of collaboration tools that enable teams to work together more efficiently and productively. Some of the key collaboration features provided by cloud services are as follows:

  • Real-time collaboration: Cloud services provide real-time collaboration features such as co-authoring, commenting, and chat, allowing teams to work on the same document or project simultaneously and communicate with each other in real time
  • Shared storage: Cloud services provide shared storage, making it easier for teams to access and share files and documents, regardless of their location or device
  • Version control: Cloud services offer version control features that allow teams to track changes made to documents and restore previous versions if necessary
  • Integration with other tools: Cloud services integrate with a wide range of other collaboration tools such as project management tools, instant messaging, and video conferencing, providing a seamless collaboration experience
  • Access control: Cloud services provide access control features that enable teams to control who has access to their files and documents, ensuring that sensitive data is protected
  • Mobile access: Cloud services are accessible from anywhere, on any device, making it easy for teams to collaborate even when they are not in the same location

Cloud-based collaboration tools provided by cloud architecture can help organizations improve their productivity, streamline their workflows, and foster better collaboration among their teams. In today’s fast-paced business environment, the increasing prevalence of remote work and distributed teams has elevated the significance of cloud-based collaboration. By embracing cloud services, organizations can effectively adapt to these changes and gain a competitive edge.

Integration with other tools in cloud architecture allows cloud services to seamlessly integrate with other collaboration and productivity tools used by an organization. This integration helps create a more efficient and streamlined workflow by allowing users to access all their tools and data from a single location.

Some examples of tools that can be integrated with cloud services include project management software, communication and collaboration tools, CRM systems, and email clients. Here are some benefits of integrating cloud services with other tools:

  • Improved productivity: Integration with other tools enables users to access all their data and tools in one place, reducing the need to switch between different applications and improving productivity
  • Better collaboration: Integration with collaboration tools such as instant messaging and video conferencing can improve communication and collaboration among team members.
  • Automation: Integration with other tools can enable automation of repetitive tasks, such as data entry and reporting, saving time and reducing the risk of errors
  • Data consistency: Integration with other tools can help ensure data consistency across different systems, reducing the risk of errors and improving data quality
  • Real-time updates: Integration with other tools can enable real-time updates, ensuring that all team members have access to the latest data and information

Cloud computing systems are designed to seamlessly connect and collaborate with a wide range of existing tools and technologies. This integration enables organizations to leverage their existing infrastructure, applications, and data seamlessly within the cloud environment. By integrating with other tools, cloud architecture allows for smooth data transfer, streamlined workflows, and improved interoperability between different systems. This integration capability enhances productivity, efficiency, and the overall effectiveness of cloud-based solutions by providing a unified and cohesive ecosystem for organizations to leverage their existing tools and resources alongside cloud services. Integration with other tools is an important aspect of cloud architecture because it helps organizations create a more efficient and streamlined workflow, improving productivity, collaboration, and data quality. By integrating cloud services with other tools, organizations can create a more cohesive and effective technology ecosystem that supports their business objectives.

The following section provides a concise overview of the essential guidelines for designing and implementing effective cloud architectures. It emphasizes key practices such as scalability, high availability, performance optimization, security implementation, cost optimization, automation, and monitoring.

The benefits of cloud architecture – Fundamentals of Cloud Architecture

The benefits of cloud architecture

Cloud architecture provides a wide range of benefits that make it a compelling choice for organizations of all sizes. Firstly, it offers scalability, allowing businesses to easily adjust their resource allocation based on demand, ensuring optimal utilization and cost efficiency. Secondly, cloud architecture promotes cost savings by eliminating the need for upfront investments in hardware and infrastructure, while also reducing maintenance and upgrade expenses. Thirdly, cloud services provide high reliability and availability, minimizing downtime and ensuring seamless operations. Additionally, cloud providers prioritize security measures, protecting data and infrastructure with advanced technologies and stringent protocols. Lastly, cloud architecture enables collaboration and remote access, facilitating seamless teamwork and enhancing productivity. These benefits collectively empower organizations to leverage the advantages of cloud computing and drive their digital transformation initiatives:

  • Scalability: Cloud architecture provides scalability, allowing organizations to rapidly scale up or down their computing resources to meet changing business needs. This means that they can easily add more computing power, storage capacity, or network bandwidth as their workload increases.
  • Cost-effective: Cloud architecture allows organizations to reduce their upfront infrastructure costs as they don’t have to invest in expensive hardware and software. Instead, they pay for what they use on a subscription or pay-as-you-go basis, allowing them to avoid over-provisioning and reduce their overall IT costs.
  • Flexibility: Cloud architecture enables organizations to access their data and applications from anywhere, at any time, and on any device, providing greater flexibility and mobility for their employees.
  • Disaster recovery: Cloud architecture provides built-in disaster recovery and business continuity capabilities, making it easier for organizations to recover their data and systems in the event of a disaster or outage.
  • Security: Cloud architecture offers advanced security features such as encryption, authentication, and access control, helping organizations to protect their data and applications from cyber threats and unauthorized access.
  • Collaboration: Cloud architecture provides easy collaboration tools and integration with other cloud-based services, enabling teams to work together more efficiently and productively.

The following figure presents a comprehensive overview of the benefits of cloud computing:

Figure 1.6 – Cloud benefits

The preceding figure depicts the key benefits of cloud computing, highlighting various aspects such as cost reduction, scalability, hardware refresh, new business opportunities, business continuity, and increased collaboration.

Overall, cloud architecture offers numerous benefits that can help organizations improve their productivity, reduce costs, and provide better services to their customers. By leveraging cloud architecture, organizations can focus on their core business objectives and leave the IT infrastructure management to cloud service providers.

Introducing virtual threads – Concurrency – Virtual Threads, Structured Concurrency

211. Introducing virtual threads

Java allows us to write multithreaded applications via the java.lang.Thread class. These are classical Java threads that are basically just thin wrappers of OS (kernel) threads. As you’ll see, these classical Java threads are referred to as platform threads and they are available for quite a lot of time (from JDK 1.1, as the following diagram reveals):

Figure 10.4 – JDK multithreading evolution

Next, let’s hit the road to JDK 19 virtual threads.

What’s the problem with platform (OS) threads?

OS threads are expensive in every single way, or more clearly, they are costly in time and space. Creating OS threads is a costly operation that requires a lot of stack space (around 20 megabytes) for storing their context, Java call stacks, and additional resources. Moreover, the OS thread scheduler is responsible to schedule Java threads and this is another costly operation that requires moving around a significant amount of data. This is referred to as thread context switching and it requires a lot of resources to take place.In the following figure, you can see the one-to-one relationship between a Java thread and an OS thread:

Figure 10.5 – JVM to OS threads

For decades, our multithreaded application runs in this context. This long time and experience taught us that we can create a limited number of Java threads (because of low throughput) and that we should reuse them wisely. The number of Java threads is a limiting factor that usually is exhausted before other resources such as network connections, CPU, and so on. Java doesn’t make any difference between threads that perform intensive-computational tasks (so, threads that are really exploiting the CPU) or they just wait for data (they just hang on the CPU).Let’s have a quick exercise. Let’s assume that our machine has 8 GB of memory and a single Java thread needs 20 MB. This means that we can have room for around 400 Java threads (8 GB = 8000 MB / 20 MB = 400 threads). Next, let’s assume that these threads perform I/O operations over a network. Each such I/O operation needs around 100 ms to complete, while the request preparation and response processing needs around 500 ns. So, a thread work for 1000 ns (0.001 ms) and just waits for 100 ms (100,000,000 ns) for the I/O operation to complete. This means that at 8 GB of memory, the 400 threads will use 0.4% of CPU, under 1% which is very low. We can conclude that a thread is idle for 99.99% of the time.Based on this exercise, it is quite obvious that Java threads become a bottleneck in throughput that doesn’t allow us to solicit hardware at full capacity. Of course, we can sweeten the situation a little bit by using thread pools for minimizing the costs but it still does not solve the major issues of dealing with resources. You have to go for CompletableFuture, reactive programming (for instance, Spring Mono and Flux) and so on.But, how many classical Java threads we can create? We can easily find out by running a simple snippet of code as follows:

AtomicLong counterOSThreads = new AtomicLong();
      
while (true) {
  new Thread(() -> {
    long currentOSThreadNr
      = counterOSThreads.incrementAndGet();
    System.out.println(“Thread: ” + currentOSThreadNr);              
    LockSupport.park();              
  }).start();
}

Or, if we want to taste from the new concurrent API, we can call the new Thread.ofPlatform() method as follows (OfPlatform is a sealed interface introduced in JDK 19):

AtomicLong counterOSThreads = new AtomicLong();
while (true) {
  Thread.ofPlatform().start(() -> {
    long currentOSThreadNr
      = counterOSThreads.incrementAndGet();
    System.out.println(“Thread: ” + currentOSThreadNr);              
    LockSupport.park();              
  });
}

On my machine, I got an OutOfMemoryError after around 40,000 Java threads. Depending on your OS and hardware this number may vary.The Thread.ofPlatform() method was added in JDK 19 to easily distinguish between Java threads (classical Java threads as we know them for decades – thin wrappers of OS threads) and the new kids in town, the virtual threads.

Using the ExecutorService for virtual threads – Concurrency – Virtual Threads, Structured Concurrency

212. Using the ExecutorService for virtual threads

Virtual threads allow us to write more expressive and straightforward concurrent code. Thanks to the massive throughput obtained via virtual threads we can easily adopt the task-per-thread model (for an HTTP server, this means a request per thread, for a database, this means a transaction per thread, and so on). In other words, we can assign a new virtual thread for each concurrent task. Trying to use the task-per-thread model with platform threads will result in a throughput limited by the number of hardware’s cores – this is explained by Little’s law (https://en.wikipedia.org/wiki/Little%27s_law), L = λW, or throughput equals average concurrency divided by latency.Whenever possible it is recommended to avoid interacting with threads directly. JDK sustains this via ExecutorService/Executor API. More precisely, we are used to submitting a task (Runnable/Callable) to an ExecutorService/Executor and working with the returned Future. This pattern is valid for virtual threads as well.So, we don’t have to write ourselves all the plumbing code for adopting the task-per-thread for virtual threads because, starting with JDK 19, this model is available via the Executors class. More precisely, via the newVirtualThreadPerTaskExecutor() method which creates an ExecutorService capable to create an unbounded number of virtual threads that follows the task-per-thread model. This ExecutorService exposes methods that allow us to give the tasks such as the submit() (as you’ll see next) and invokeAll/Any() (as you’ll see later) methods, and return a Future containing an exception or a result.

Starting with JDK 19, the ExecutorService extends the AutoCloseable interface. In other words, we can use ExecutorService in a try-with-resources pattern.

Consider the following simple Runnable and Callable:

Runnable taskr = () -> logger.info(
  Thread.currentThread().toString());
      
Callable<Boolean> taskc = () -> {
  logger.info(Thread.currentThread().toString());
  return true;
};

Executing the Runnable/Callable can be done as follows (here, we submit 15 tasks (NUMBER_OF_TASKS = 15)):

try (ExecutorService executor
      = Executors.newVirtualThreadPerTaskExecutor()) {
  for (int i = 0; i < NUMBER_OF_TASKS; i++) {
    executor.submit(taskr); // executing Runnable
    executor.submit(taskc); // executing Callable
  }
}

Of course, in the case of Runnable/Callable we can capture a Future and act accordingly via the blocking get() method or whatever we want to do.

Future<?> future = executor.submit(taskr);
Future<Boolean> future = executor.submit(taskc);

A possible output looks as follows:

VirtualThread[#28]/runnable@ForkJoinPool-1-worker-6
VirtualThread[#31]/runnable@ForkJoinPool-1-worker-5
VirtualThread[#29]/runnable@ForkJoinPool-1-worker-7
VirtualThread[#25]/runnable@ForkJoinPool-1-worker-3
VirtualThread[#24]/runnable@ForkJoinPool-1-worker-2
VirtualThread[#27]/runnable@ForkJoinPool-1-worker-5
VirtualThread[#26]/runnable@ForkJoinPool-1-worker-4
VirtualThread[#22]/runnable@ForkJoinPool-1-worker-1
VirtualThread[#36]/runnable@ForkJoinPool-1-worker-1
VirtualThread[#37]/runnable@ForkJoinPool-1-worker-2
VirtualThread[#35]/runnable@ForkJoinPool-1-worker-7
VirtualThread[#34]/runnable@ForkJoinPool-1-worker-4
VirtualThread[#32]/runnable@ForkJoinPool-1-worker-3
VirtualThread[#33]/runnable@ForkJoinPool-1-worker-2
VirtualThread[#30]/runnable@ForkJoinPool-1-worker-1

Check out the virtual threads’ ids. They range between #22-#37 without repetition. Each task is executed by its own virtual thread.The task-per-thread model is also available for classical threads via newThreadPerTaskExecutor(ThreadFactory threadFactory). Here is an example:

static class SimpleThreadFactory implements ThreadFactory {
  @Override
  public Thread newThread(Runnable r) {
    return new Thread(r);                     // classic
   // return Thread.ofVirtual().unstarted(r); // virtual
  }
}
try (ExecutorService executor =
      Executors.newThreadPerTaskExecutor(
          new SimpleThreadFactory())) {
  for (int i = 0; i < NUMBER_OF_TASKS; i++) {
    executor.submit(taskr); // executing Runnable
    executor.submit(taskc); // executing Callable
  }
}

As you can see, newThreadPerTaskExecutor() can be used for classic threads or for virtual threads. The number of created threads is unbounded. By simply modifying the thread factory we can switch between virtual/classic threads.A possible output looks as follows:

Thread[#75,Thread-15,5,main]
Thread[#77,Thread-17,5,main]
Thread[#76,Thread-16,5,main]
Thread[#83,Thread-23,5,main]
Thread[#82,Thread-22,5,main]
Thread[#80,Thread-20,5,main]
Thread[#81,Thread-21,5,main]
Thread[#79,Thread-19,5,main]
Thread[#78,Thread-18,5,main]
Thread[#89,Thread-29,5,main]
Thread[#88,Thread-28,5,main]
Thread[#87,Thread-27,5,main]
Thread[#86,Thread-26,5,main]
Thread[#85,Thread-25,5,main]
Thread[#84,Thread-24,5,main]

Check out the threads’ ids. They range between #75-#89 without repetition. Each task is executed by its own thread.

Explaining how virtual threads work 2 – Concurrency – Virtual Threads, Structured Concurrency

Capturing virtual threads

So far we learned that a virtual thread is mounted by JVM to a platform thread which becomes its carrier thread. Moreover, the carrier thread runs the virtual thread until it hit a blocking (I/O) operation. At that point, the virtual thread is unmounted from the carrier thread and it will be rescheduled after the blocking (I/O) operation is done.While this scenario is true for most of the blocking operations resulting in unmounting the virtual threads and freeing the platform thread (and the underlying OS thread), there are a few exceptional cases when the virtual threads are not unmounted. There are two main causes for this behavior:

Limitations on OS (for instance, a significant number of filesystem operations)

Limitations on JDK (for instance, Object#wait())

When the virtual thread cannot be unmounted from its carrier thread it means that the carrier thread and the underlying OS thread are blocked. This may affect the scalability of the application, so, if the platform threads pool allows it, JVM can take the decision of adding one more platform thread. So, for a period of time, the number of platform threads may exceed the number of available cores.

Pinning virtual threads

There are also two other use cases when a virtual thread cannot be unmounted:

When the virtual thread runs code inside a synchronized method/block

When the virtual thread invokes a foreign function or native method (topic covered in Chapter 7)

In this scenario, we say that the virtual thread is pinned to the carrier thread. This may affect the scalability of the application, but JVM will not increase the number of platform threads. Instead of this, we should take action and refactor the synchronized blocks to ensure that the locking code is simple, clear, and short. Whenever possible prefer java.util.concurrent locks instead of synchronized blocks. If we manage to avoid long and frequent locking periods then we will not face any significant scalability issues. In future releases, the JDK team aims to eliminate the pinning inside synchronized blocks.

Explaining how virtual threads work – Concurrency – Virtual Threads, Structured Concurrency

213. Explaining how virtual threads work

Now that we know how to create and start a virtual thread, let’s see how it actually works.Let’s start with a meaningful diagram:

Figure 10.7 – How virtual threads works

As you can see, figure 10.7 is similar to 10.6 only that we have added a few more elements.First of all, notice that the platform threads run under a ForkJoinPool umbrella. This is a First-In-First-Out (FIFO) dedicated fork/join pool dedicated to scheduling and orchestrating the relationships between virtual threads and platform threads (a detailed coverage of Java fork/join framework is available in Java Coding Problems, First Edition, Chapter 11).

This dedicated ForkJoinPool is controlled by the JVM and it acts as the virtual thread scheduler based on a FIFO queue. Its initial capacity (number of threads) is equal to the number of available cores and it can be increased up to 256. The default virtual thread scheduler is implemented in the java.lang.VirtualThread class:

private static ForkJoinPool createDefaultScheduler() {…}

Do not confuse this ForkJoinPool with the one used for parallel streams (Common Fork Join Pool – ForkJoinPool.commonPool()).

Between the virtual threads and the platform threads, there is a one-to-many association. Nevertheless, the JVM schedules virtual threads to run on platform threads in such a way that only one virtual thread run on a platform thread at a time. When the JVM assigns a virtual thread to a platform thread, the so-called stack chunk object of the virtual thread is copied from the heap memory on the platform thread. If the code running on a virtual thread encounters a blocking (I/O) operation that should be handled by the JVM then the virtual thread is released by copying its stack chunk object back into the heap (this operation of copying the stack chunk between the heap memory and platform thread is the cost of blocking a virtual thread – this is much cheaper than blocking a platform thread). Meanwhile, the platform thread can run other virtual threads. When the blocking (I/O) of the released virtual thread is done, JVM rescheduled the virtual thread for execution on a platform thread. This can be the same platform thread or another one.

The operation of assigning a virtual thread to a platform thread is called mounting. The operation of unassigning a virtual thread from the platform thread is called unmounting. The platform thread running the assigned virtual thread is called a carrier thread.

Let’s have an example that reveals how the virtual threads are mounted:

private static final int NUMBER_OF_TASKS
  = Runtime.getRuntime().availableProcessors();
Runnable taskr = () ->
  logger.info(Thread.currentThread().toString());      
try (ExecutorService executor
     = Executors.newVirtualThreadPerTaskExecutor()) {
  for (int i = 0; i < NUMBER_OF_TASKS + 1; i++) {
    executor.submit(taskr);
  }
}

In this snippet of code, we create a number of virtual threads equal to the number of available cores + 1. On my machine, I have 8 cores (so, 8 carriers) and each of them carries a virtual thread. Since we have + 1, a carrier will work twice. The output reveals this scenario (check out the workers, here worker-8 run virtual threads #30 and #31):

VirtualThread[#25]/runnable@ForkJoinPool-1-worker-3
VirtualThread[#30]/runnable@ForkJoinPool-1-worker-8
VirtualThread[#28]/runnable@ForkJoinPool-1-worker-6
VirtualThread[#22]/runnable@ForkJoinPool-1-worker-1
VirtualThread[#24]/runnable@ForkJoinPool-1-worker-2
VirtualThread[#29]/runnable@ForkJoinPool-1-worker-7
VirtualThread[#26]/runnable@ForkJoinPool-1-worker-4
VirtualThread[#27]/runnable@ForkJoinPool-1-worker-5
VirtualThread[#31]/runnable@ForkJoinPool-1-worker-8

But, we can configure the ForkJoinPool via three system properties as follows:

jdk.virtualThreadScheduler.parallelism – number of CPU cores

jdk.virtualThreadScheduler.maxPoolSize – maximum pool size (256)

jdk.virtualThreadScheduler.minRunnable – minimum number of running threads (half the pool size)

In a subsequent problem, we will use these properties to better shape virtual thread context switching (mounting/unmounting) details.

Introducing scope object (StructuredTaskScope) 2 – Concurrency – Virtual Threads, Structured Concurrency

First, we create a StructuredTaskScope in a try-with-resources pattern. StructuredTaskScope implements AutoCloseable:

try (StructuredTaskScope scope
      = new StructuredTaskScope<String>()) {
    …
}

The scope is a wrapper for the virtual threads’ lifetime. We use the scope to fork as many virtual threads (subtasks) as needed via the fork(Callable task) method. Here, we fork only one virtual thread and get back a Subtask (forking is a non-blocking operation):

Subtask<String> subtask = scope.fork(() -> fetchTester(1));

Next, we have to call the join() method (or joinUntil(Instant deadline)). This method waits for all threads (all Subtask instances) forked from this scope (or, all threads that have been submitted to this scope) to complete, so it is a blocking operation. A scope should block only for waiting its subtasks to complete, and this is happening via join() or joinUntil().

scope.join();          

When the execution passes this line, we know that all threads (all forked Subtask) forked from this scope are complete with a result or an exception (each subtask run indenpendently, therefore each of them can complete with a result or an exception). Here, we call the non-blocking get() mehod to get the result but pay attention that calling get() for a task that did not complete will rise an exception as IllegalStateException(“Owner did not join after forking subtask”):

String result = subtask.get();

On the other hand, we can obtain the exception of a failed task via exception(). However, if we call exception() for a subtask (Subtask) that is completed with a result then we will get back an exception as IllegalStateException(“Subtask not completed or did not complete with exception”). So, if you are not sure that your task(s) is always complete with a result or an exception, it is better to call get() or exception() only after you test the state of the corresponding Subtask. A state of SUCCESS will safely allow you to call get(), while a state of FAILED will safely allow you to call exception(). So, in our case, we may prefer it this way:

String result = “”;
if (subtask.state().equals(Subtask.State.SUCCESS)) {
  result = subtask.get();
}

Beside Subtask.State.SUCCESS and Subtask.State.FAILED, we also have Subtask.State.UNAVAILABLE which means that the subtask is not available (for instance, if the subtask is still running then its state is UNAVAILABLE, but it could be other cause as well).That’s all!

ExecutorService vs. StructuredTaskScope

The previous code looks like the code that we write via a classical ExecutorService but, there are two big differences between these solutions. First of all, an ExecutorService holds the precious platform threads and allows us to pool them. On the other hand, a StructuredTaskScope is just a thin launcher for virtual threads that are cheap and shouldn’t be pooled. So, once we’ve done our job, a StructuredTaskScope can be destroyed and garbage collected. Second, an ExecutorService holds a single queue for all the tasks and the threads take from this queue whenever they have the chance to do it. A StructuredTaskScope relies on a fork/join pool and each virtual thread has it own wait queue. However, a virtual thread can steal a task from another queue as well. This is known as the work-stealing pattern and it was covered in Java Coding Problem, First Edition, Chapter 11.

Copyright © 2025 Concurrency - Virtual Threads, Structured Concurrency | Powered by washingtonred.org