Introducing structured concurrency 2 – Concurrency – Virtual Threads, Structured Concurrency

We skip the rest of the code since you can find it in the bundled code.Of course, we can implement code-answers to each of these questions via error handling, tasks abandon and abortion, ExecutorService, and so on, but this means a lot of work for the developer. Writing failsafe solutions that carefully cover all possible scenarios across multiple tasks/subtasks while tracking their progress in a concurrent environment is not an easy job. Not to mention how hard is to understand and maintain the resulting code by another developer or even the same developer after 1-2 years or even months.It is time to add some structure to this code, so let’s introduce structured concurrency (or, Project Loom).Structured concurrency relies on several pillars meant to bring lightweight concurrency in Java. The fundamental pillar or principle of structured concurrency is highlighted next.

The fundamental principle of structured concurrency: When a task has to be solved concurrently then all the threads needed to solve it are spun and rejoined in the same block of code. In other words, all these threads’ lifetime is bound to the block’s scope, so we have clear and explicit entry-exit points for each concurrent code block.

Based on this principle, the thread that initiates a concurrent context is the parent-thread or the owner-thread. All threads started by the parent-thread are children-threads or forks, so between them, these threads are siblings. Together, the parent-thread and the children-threads, define a parent-child hierarchy.Putting the structured concurrency principle in a diagram can be seen as follows:

Figure 10.2 – Parent-child hierarchy in structured concurrency

In the context of parent-child hierarchy, we have support for error/exception handling with short-circuiting, cancellation propagation, and monitoring/observability.Error/exception handling with short-circuiting: If a child-thread fails then all child-threads are canceled unless they are complete. For instance, if futureTester(1) fails, then futureTester(2) and futureTester(3) are automatically canceled.Cancellation propagation: If the parent-thread is interrupted until joining the child-threads is over then these forks (the child-threads/subtasks) are canceled automatically. For instance, if the thread executing buildTestingTeam() gets interrupted then its three forks are automatically canceled.Monitoring/observability: A thread dump reveals crystal-clear the entire parent-child hierarchy no matter how many levels have been spawned. Moreover, in structured concurrency, we take advantage of scheduling and memory management of threads.While these are purely concepts, writing code that respects and follows these concepts requires the proper API and the following awesome callout:

Figure 10.3 – Don’t reuse virtual threads

Cut this out and stick it somewhere to see it every day! So, in structured concurrency, don’t reuse virtual threads. I know what you are thinking: hey dude, threads are expensive and limited, so we have to reuse them. A quick hint: we are talking about virtual threads (massive throughput), not classical threads, but this topic is covered in the next problem.

Introducing structured concurrency – Concurrency – Virtual Threads, Structured Concurrency

210. Introducing structured concurrency

If you are as old as I am then most probably you’ve started programming with a language such as BASIC or a similar unstructured programming language. At that time, an application was just a sequence of lines that defined a sequential logic/behavior via a bunch of GOTO statements that drive the flow by jumping as a kangaroo back and forward between the code lines. Well, in Java, the building blocks of a typical concurrent code are so primitive that the code looks somehow like unstructured programming because is hard to follow and understand what’s going on. Moreover, a thread dump of a concurrent task doesn’t provide the needed answers.Let’s follow a snippet of Java concurrent code and let’s stop every time we have a question (always check the code below the question). The task is to concurrently load three testers by id and team them up in a testing team. First, let’s list here the server code (we will use this simple code to serve us in this problem and subsequent problems):

public static String fetchTester(int id)
      throws IOException, InterruptedException {
  HttpClient client = HttpClient.newHttpClient();
  HttpRequest requestGet = HttpRequest.newBuilder()
    .GET()
    .uri(URI.create(“https://reqres.in/api/users/” + id))
    .build();
  HttpResponse<String> responseGet = client.send(
    requestGet, HttpResponse.BodyHandlers.ofString());
  if (responseGet.statusCode() == 200) {
    return responseGet.body();
  }
  throw new UserNotFoundException(“Code: “
    + responseGet.statusCode());
}

Next, the code that we are especially interested in starts as follows:

private static final ExecutorService executor
  = Executors.newFixedThreadPool(2);
public static TestingTeam buildTestingTeam()
   throws InterruptedException {
  …

First stop: As you can see buildTestingTeam() throws an InterruptedException. So, if the thread executing buildTestingTeam() gets interrupted then how can we easily interrupt the following threads:

  Future<String> future1 = futureTester(1);
  Future<String> future2 = futureTester(2);
  Future<String> future3 = futureTester(3);
 
  try {
    …

Second stop: Here we have three get() calls. So, the current thread waits for other threads to complete. Can we easily observe those threads?

    String tester1 = future1.get();
    String tester2 = future2.get();
    String tester3 = future3.get();
    logger.info(tester1);
    logger.info(tester2);
    logger.info(tester3);
    return new TestingTeam(tester1, tester2, tester3);
  } catch (ExecutionException ex) {
    …

Third stop: If an ExecutionException is caught then we know that one of these three Future has failed. Can we easily cancel the remaining two or they will just hang on there? Probably future1 will fail while future2 and future3 will complete successfully or maybe future2 will complete successfully while future3 will just run forever (a so-called orphan thread). This may lead to serious mismatches in the expected results, memory leaks, and so on.

    throw new RuntimeException(ex);
  } finally {
    …

Fourth stop: Next line of code is used to shut down the executor, but is so easy to overlook it. Is this the proper place to do this?
    shutdownExecutor(executor);
  }
}

Fifth stop: If you didn’t spot the previous line of code then is legitimate to ask yourself how/where this executor get shut down.

public static Future<String> futureTester(int id) {
  return executor.submit(() -> fetchTester(id));
}

Technical requirements – Fundamentals of Cloud Architecture

Technical requirements

To fully engage with the content in this chapter on cloud computing architecture, you should have a basic understanding of computer systems, networking concepts, and information technology.

Additionally, the following technical requirements are recommended:

  • Internet access: You should have a reliable internet connection to access online resources, references, and examples related to cloud computing.
  • A computing device: A desktop computer, laptop, tablet, or smartphone with a modern web browser is necessary to read this chapter’s content and access any online materials.
  • A web browser: The latest version of a modern web browser such as Google Chrome, Mozilla Firefox, Microsoft Edge, or Safari is recommended. This ensures compatibility and optimal viewing experience of web-based resources and interactive content.
  • Familiarity with cloud services: Some familiarity with cloud services and their basic functionalities will enhance your understanding of this chapter. This includes knowledge of cloud computing models such as Infrastructure-as-a-Service (IaaS), Platform-as-a-Service (PaaS), and Software-as-a-Service (SaaS).

The history of cloud computing

Cloud computing has a rich history that has evolved over several decades. The concept of cloud computing dates back to the 1960s when computer scientists at MIT and Dartmouth College proposed the idea of a “utility computing” system that would allow users to access computing resources on demand.

In the 1970s, IBM introduced virtualization technology, which allowed multiple operating systems to run on a single mainframe computer. This technology enabled companies to consolidate their IT resources and reduce costs.

In the 1990s, the development of the World Wide Web and the rise of e-commerce led to the creation of web-based applications and services. This led to the development of early cloud computing platforms such as Salesforce, which provided customer relationship management (CRM) services over the internet.

In 2002, Amazon launched its web services division, offering cloud-based infrastructure services such as storage and computing power. This was followed by the launch of Amazon Elastic Compute Cloud (EC2) in 2006, which allowed users to rent computing capacity on demand.

In 2008, Google launched its cloud computing platform, Google App Engine, which allowed developers to build and run web applications on Google’s infrastructure.

Microsoft followed suit in 2010 with the launch of Windows Azure, which provided cloud-based services for building and deploying applications.

The growth of cloud computing has been fueled by advances in virtualization technology, which allows computing resources to be shared and used more efficiently. The development of cloud-based services and infrastructure has also made it easier for businesses to scale their IT resources up or down based on demand.

Today, cloud computing has become an integral part of many businesses, offering a range of benefits such as cost savings, scalability, flexibility, and improved collaboration. Cloud computing has also enabled the development of new technologies such as serverless computing, which allows developers to build and run applications without managing servers or infrastructure.

The main idea behind cloud computing was to provide a flexible and cost-effective way for users to access computing resources on demand. In the early days of computing, businesses and organizations had to invest in their IT infrastructure, including hardware, software, and networking equipment. This was expensive and often required a large upfront investment, which made it difficult for small and medium-sized businesses to compete with larger organizations.

Cloud computing was envisioned as a way to address this challenge by providing a shared pool of computing resources that could be accessed over the internet. This allowed businesses to pay only for the resources they needed, and to scale up or down as needed to meet changing demand.

In addition to cost savings, cloud computing was also seen as a way to improve the flexibility and agility of IT operations. By providing access to a shared pool of resources, cloud computing could enable businesses to quickly deploy new applications, scale up or down as needed, and respond to changing business needs more quickly than traditional IT infrastructure.

The thought behind cloud computing was to provide a more efficient, flexible, and cost-effective way for businesses to access the computing resources they need to operate and compete in today’s fast-paced digital economy.

Cloud computing today 2 – Fundamentals of Cloud Architecture

While cloud computing offers many benefits, it also comes with several challenges that must be addressed. One of the primary challenges is security, as cloud providers must ensure that users’ data is protected from unauthorized access or disclosure. Another challenge is vendor lock-in, as users may find it difficult to switch between cloud providers due to differences in technologies and architectures. Finally, there is the challenge of managing cloud costs, as users must carefully monitor and optimize their resource usage to avoid unexpected expenses.

Despite these challenges, cloud computing has become an essential part of the modern technology landscape, enabling businesses and individuals to access and use technology more efficiently and effectively than ever before.

The following figure depicts the general idea behind cloud computing:

Figure 1.1 – The versatility and flexibility of cloud computing

This figure provides a concise overview of cloud computing, featuring key components such as databases, applications, compute, mobile devices, servers, and storage. It also highlights different cloud deployment models: public, private, and hybrid clouds. This figure visually represents these components and models, showcasing the interconnected nature of cloud computing.

Cloud computing has become an essential part of the modern technology landscape, enabling businesses and individuals to access and use technology more efficiently and effectively than ever before. With cloud computing, organizations can access technology resources as needed, without having to invest in and manage on-premises infrastructure. This allows companies to focus on their core business, while the cloud service provider manages the underlying technology. There are three main types of cloud computing: public cloud, private cloud, and hybrid cloud. The following figure depicts the basic design of cloud technology:

Figure 1.2 – Basic cloud design

The preceding figure depicts how basic cloud components reside within the cloud.

In this section, you learned about the origins and evolution of cloud computing, from time-sharing to the commercialization of services. You gained insights into key milestones, such as the development of virtualization technologies and the rise of utility computing.

Next, you explored the current state of cloud computing, including its models (IaaS, PaaS, and SaaS).

The next section dives into the foundational aspects of cloud architecture and provides you with a comprehensive understanding of its key components and design principles. It explores the fundamental building blocks of cloud architecture, including virtualization, resource pooling, and on-demand self-service.

Cloud computing today – Fundamentals of Cloud Architecture

Cloud computing today

This section provides an up-to-date snapshot of the current state of cloud computing and its impact on businesses and individuals. It explores the widespread adoption of cloud computing across various industries and the benefits it offers, such as scalability, cost-efficiency, and enhanced flexibility. The section also delves into the different types of cloud services available today, including IaaS, PaaS, and SaaS, highlighting their respective features and use cases.

In recent years, cloud computing has transformed the way businesses and individuals access and use technology. It has revolutionized the way we store, process, and share data, enabling greater flexibility, scalability, and cost-efficiency than ever before. With the cloud computing market projected to reach $1 trillion by 2024, it is clear that cloud computing has become an essential part of the modern technology landscape. But what exactly is cloud computing, and how does it work? In this book, we will explore the fundamental concepts of cloud computing, from its history and evolution to its various types and deployment models. We will delve into the benefits and challenges of cloud computing and examine real-world examples of how organizations are leveraging this technology to drive innovation, growth, and success. Whether you are a seasoned IT professional or simply curious about the cloud, this book will provide you with the insights and knowledge you need to navigate this exciting and rapidly changing field.

Cloud computing has become a pervasive technology that has transformed the way businesses and individuals access and use computing resources. At its core, cloud computing is about delivering computing resources over the internet, rather than owning and managing physical infrastructure. This enables greater flexibility and scalability as users can easily scale up or down their resource usage based on their needs. It also offers cost-efficiency as users only pay for what they use and can avoid upfront capital expenses. Additionally, cloud computing offers greater resilience and reliability, as cloud providers typically offer redundancy and failover capabilities to ensure that services remain available even in the event of hardware failure or other issues.

Cloud computing is a paradigm that enables the provisioning of computing resources, encompassing servers, storage, applications, and services through the internet. Instead of possessing and overseeing physical infrastructure, individuals and businesses have the option to lease these resources from cloud providers, paying only for what they consume. This approach presents numerous benefits compared to conventional on-site infrastructure, including enhanced adaptability, scalability, cost-effectiveness, and dependability.

There are several different types of cloud computing services, each offering varying levels of abstraction and control. At the lowest level of abstraction is IaaS, which provides users with access to virtualized computing resources, such as VMs, storage, and networking, that they can use to build and deploy their applications. At a higher level of abstraction is PaaS, which provides a platform on top of which users can build and deploy applications, without having to worry about the underlying infrastructure. Finally, at the highest level of abstraction is SaaS, which provides complete applications that are accessed over the internet, without the need for any installation or maintenance on the user’s part.

Understanding cloud architecture 2 – Fundamentals of Cloud Architecture
  • Cloud components: Cloud computing involves several components, such as VMs, containers, storage, networking, security, databases, and middleware. A cloud architect must have a clear understanding of each component’s capabilities and limitations to design and implement efficient and secure cloud solutions. Cloud computing encompasses various components that contribute to its functionality and infrastructure. Examples of these components include VMs, which allow you to run multiple operating systems on a single physical server, enabling efficient resource utilization. Containers, such as Docker and Kubernetes, offer lightweight, isolated environments for deploying and managing applications across different cloud environments. Storage services, such as Amazon S3 and Google Cloud Storage, provide scalable and reliable storage for data and files. Networking services, such as Amazon Virtual Private Cloud (VPC) and Azure Virtual Network, enable the creation of virtual networks to connect resources securely. Security services such as encryption, access control, and firewalls help protect data and applications. Cloud databases, such as Amazon RDS and Microsoft Azure SQL Database, provide scalable and managed database solutions. Middleware tools facilitate communication and integration between different software components and services in the cloud. These components collectively form the infrastructure and services that power cloud computing, offering organizations the flexibility, scalability, and convenience of cloud-based solutions.
  • Cloud providers: Many cloud providers offer various cloud services and tools to build and deploy cloud solutions such as AWS, Microsoft Azure, and Google Cloud Platform (GCP). A cloud architect must have a deep understanding of these providers and their services to choose the right provider and services for their project. There are several prominent cloud providers in the market, each offering a wide range of services. AWS is a leading cloud provider, offering services such as Amazon EC2 for virtual servers, Amazon S3 for scalable storage, and Amazon RDS for managed databases. Microsoft Azure provides services such as Azure Virtual Machines, Azure Blob Storage, and Azure SQL Database. GCP offers services such as Google Compute Engine, Google Cloud Storage, and Google Cloud Spanner for distributed databases. Other notable cloud providers include IBM Cloud, with services such as IBM Cloud Virtual Servers and IBM Cloud Object Storage, and Oracle Cloud, offering services such as Oracle Compute and Oracle Database Cloud. These cloud providers offer a comprehensive suite of services, including compute, storage, databases, machine learning (ML), networking, and security, enabling organizations to build, deploy, and scale applications and infrastructure in the cloud. Figure 1.5 depicts the basic cloud architecture in AWS with key services such as VPC, EC2 (Compute), DynamoDB, and others:

Figure 1.5 – Basic cloud architecture in AWS

  • Cloud security: Cloud security is a critical component of cloud architecture. A cloud architect must design and implement security measures to protect the cloud infrastructure, data, and applications from unauthorized access, data breaches, and other security threats. Cloud security is a critical aspect of cloud computing, and several providers offer robust security services and solutions. One prominent cloud security provider is Cloudflare, which offers a range of security services such as DDoS protection, web application firewalls (WAFs), and content delivery networks (CDNs) to protect against malicious attacks. Another notable provider is Palo Alto Networks, which offers cloud security solutions such as Prisma Cloud, providing visibility, compliance, and threat protection across multi-cloud environments. Microsoft Azure also provides a comprehensive set of security services, including Azure Security Center, Azure Active Directory, and Azure Sentinel, offering identity management, threat detection, and security monitoring capabilities. AWS offers services such as AWS Identity and Access Management (IAM), AWS WAF, and AWS GuardDuty to help secure cloud environments. These cloud security providers and services play a crucial role in safeguarding data, applications, and infrastructure in the cloud, ensuring confidentiality, integrity, and availability of resources.

Overall, cloud architecture involves designing and managing cloud solutions that are scalable, reliable, secure, and cost-effective. A successful cloud architect must have a strong understanding of cloud technologies, architecture principles, and business needs to design and implement efficient and effective cloud solutions. In the upcoming section, we’ll explore the significant advantages and benefits that cloud architecture offers to organizations and individuals. Cloud computing has revolutionized the way we store, access, and process data, providing numerous advantages over traditional on-premises infrastructure.

Understanding cloud architecture – Fundamentals of Cloud Architecture

Understanding cloud architecture

To comprehend the inner workings of cloud computing, it is crucial to understand its underlying architecture. This section provides a comprehensive overview of cloud architecture, elucidating the key components and their interconnections. It explains the concepts of virtualization, distributed computing, and load balancing, which form the building blocks of cloud infrastructure.

Cloud architecture is a term that’s used to describe the design and organization of a cloud computing system. A cloud computing system typically consists of various components, including computing resources, storage, network infrastructure, security measures, and software applications. Cloud architecture refers to the way these components are organized and integrated to provide a seamless and efficient cloud computing environment. The following figure depicts a basic cloud architecture design. It covers the end user connection, backend/database, memory cache, middleware, and frontend in Google Cloud:

Figure 1.3 – A basic cloud architecture

Cloud architecture involves making critical decisions regarding the cloud deployment model, cloud service model, and cloud providers, among others. These decisions will affect the performance, scalability, security, and cost-effectiveness of the cloud computing system. A well-designed cloud architecture should enable an organization to leverage the benefits of cloud computing, such as cost savings, scalability, and flexibility, while minimizing the potential risks and drawbacks.

Cloud architecture is an essential aspect of any cloud computing project, and it requires a deep understanding of cloud computing technologies, business requirements, and architecture principles. A successful cloud architect must be able to design and implement cloud solutions that meet the specific needs of their organization, whether it is a small business, a large enterprise, or a government agency.

Cloud architecture can also be described as a set of principles, guidelines, and best practices that are used to design and manage cloud computing systems. It involves planning, designing, implementing, and managing cloud-based solutions that meet specific business needs and requirements.

The following figure showcases a visual representation of cloud computing, highlighting the different deployment models and service models:

Figure 1.4 – A visual representation of cloud computing

At a high level, cloud architecture involves several key components, including the following:

  • Cloud service models: Cloud computing provides three distinct service models: IaaS, PaaS, and SaaS. Each model offers users different levels of control, flexibility, and customization. For instance, IaaS examples include Amazon Web Services (AWS) EC2 and Microsoft Azure Virtual Machines, which grant users access to virtual servers and infrastructure resources. PaaS examples encompass Google Cloud Platform’s App Engine and Heroku, which provide managed platforms for application development and deployment. Lastly, SaaS examples encompass Salesforce, a cloud-based CRM platform, and Google Workspace, a suite of productivity and collaboration tools. These examples demonstrate how IaaS empowers users to provision and oversee virtual infrastructure, PaaS abstracts the underlying platform for application development, and SaaS grants access to fully functional software over the internet. By utilizing these distinct service models, organizations can leverage cloud-based resources and software without the need to manage infrastructure or install software locally.
  • Cloud deployment models: Cloud computing deployment models encompass public cloud, private cloud, hybrid cloud, and multi-cloud, each presenting unique advantages and challenges. Examples of these deployment models include well-known providers such as AWS, Microsoft Azure, and Google Cloud Platform. In a public cloud, computing resources are shared among multiple organizations and accessible over the internet. Private cloud, on the other hand, involves dedicated cloud infrastructure that can be deployed on-premises or hosted by a single organization, offering greater control and privacy. Hybrid cloud combines both public and private cloud environments, enabling organizations to leverage scalability and flexibility. Multi-cloud refers to utilizing multiple cloud service providers concurrently, allowing for workload distribution, redundancy, cost optimization, and access to specialized services. These deployment models grant varying levels of control, flexibility, and scalability, enabling organizations to tailor their cloud strategies to their specific needs and leverage the full benefits of cloud computing.
Combining StructuredTaskScope and streams 2 – Concurrency – Virtual Threads, Structured Concurrency

And, in the next figure, you can see the event recorded for ending this virtual thread:

Figure 10.12 – JFR event for ending a virtual thread

Besides the JFR CLI, you can use more powerful tools for consuming the virtual thread events such as JDK Mission Control (https://www.oracle.com/java/technologies/jdk-mission-control.html) and the well-known Advanced Management Console (https://www.oracle.com/java/technologies/advancedmanagementconsole.html).For getting a stack trace for threads that block while pinned we can set the system property jdk.tracePinnedThreads. A complete (verbose) stack trace is available via -Djdk.tracePinnedThreads=full. Or if all you need is a brief/short stack trace then rely on -Djdk.tracePinnedThreads=short.In our example, we can easily get a pinned virtual thread by marking the fetchTester() method as synchronized (remember that a virtual thread cannot be unmounted if it runs code inside a synchronized method/block):

public static synchronized String fetchTester(int id)
    throws IOException, InterruptedException {
  …
}

In this context, JFR will record a pinned virtual thread as in the following figure:

Figure 10.13 – JFR event for a pinned virtual thread

If we run the application with -Djdk.tracePinnedThreads=full then your IDE will print a detailed stack trace that starts as follows:

Thread[#26,ForkJoinPool-1-worker-1,5,CarrierThreads]    java.base/java.lang.VirtualThread$VThreadContinuation.onPinned(VirtualThread.java:183)
…

You can see the complete output by executing the bundled code. Of course, you can get a thread dump and analyze it via several other tools. You may prefer any of jstack, Java Mission Control (JMC), jvisualvm, or jcmd. For instance, we can obtain a thread dump in plain text or JSON format via jcmd as follows:

jcmd <PID> Thread.dump_to_file -format=text <file>
jcmd <PID> Thread.dump_to_file -format=json <file>

Next, let’s play with jconsole (JMX) to quickly analyze the performance of virtual threads.

Using Java Management Extensions (JMX)

Until JDK 20 (inclusive), JMX provide support for monitoring only the platform and threads. But, we can still use JMX to observe the performance brought by virtual threads in comparison with platform threads.For instance, we can use JMX to monitor platform threads at each 500 ms via the following snippet of code:

ScheduledExecutorService scheduledExecutor
      = Executors.newScheduledThreadPool(1);
scheduledExecutor.scheduleAtFixedRate(() -> {
  ThreadMXBean threadBean
    = ManagementFactory.getThreadMXBean();
  ThreadInfo[] threadInfo
    = threadBean.dumpAllThreads(false, false);
  logger.info(() -> “Platform threads: ” + threadInfo.length);
}, 500, 500, TimeUnit.MILLISECONDS);

We rely on this code in the following three scenarios.

Running 10000 tasks via cached thread pool executor

Next, let’s add a snippet of code that run 10000 tasks via newCachedThreadPool() and platform threads. We also measure the time elapsed to execute these tasks:

long start = System.currentTimeMillis();
      
try (ExecutorService executorCached
    = Executors.newCachedThreadPool()) {
  IntStream.range(0, 10_000).forEach(i -> {
    executorCached.submit(() -> {
      Thread.sleep(Duration.ofSeconds(1));
      logger.info(() -> “Task: ” + i);
      return i;
    });
  });
}
      
logger.info(() -> “Time (ms): “
  + (System.currentTimeMillis() – start));

On my machine, it took 8147 ms (8 seconds) to run these 10000 tasks using at peak 7729 platform threads. The following screenshot from jconsole (JMX) reveals this information:

Figure 10.14 – Running 10000 tasks via cached thred pool executor

Next, let’s repeat this test via a fixed thread pool.

Copyright © 2025 Concurrency - Virtual Threads, Structured Concurrency | Powered by washingtonred.org