If you’ve ever visited a reasonably large airport, you must have noticed multiple flights taking off or landing at any given hour of the day.
Now imagine how things would be if an airport could handle only one flight on any given day. Global travel would come to a screeching halt.
Handling multiple flights at the same time makes air travel possible. This ability is also known as concurrency and it is the key to doing more work with limited time and resources.
If you apply the concept to software development, an application is concurrent if it can handle multiple tasks simultaneously.
How would you react if you visit the Amazon home page and get a message saying that a maximum number of users are already accessing the website and you need to wait for your turn?
Chances are that you’ll never visit the website again and Amazon would lose a potential customer.
As developers, when we use a platform (such as a web server) for running an application, we expect the platform to support concurrent behavior. Often, the concurrency model of platforms or frameworks is quite easy to understand.
But I have seen that developers often get confused about concurrency in Node.js. They know about the Event Loop but don’t know how it actually works.
In this post, my goal is to clear all the confusion about concurrency in Node.js.
The biggest hurdle to Concurrency
Why is it difficult to achieve concurrency?
Is there any fundamental hurdle to concurrency in software systems?
Yes.
The main hurdle to concurrency is slow and blocking input-output operations.
I/O operations are one of the fundamental operations of a computer system. But they are also the slowest.
Accessing the RAM is in the order of nanoseconds. Accessing a disk is even slower (in the order of milliseconds).
The bandwidth shares the same story.
RAM has a transfer rate in the order of GB/second while disk and network can vary from MB/s to GB/s.
While I/O is not computation heavy, it introduces a delay between the moment of request and the moment the operation completes.
Also, I/O is not just a software concern. The human factor is probably even more important. The biggest delay in an application comes from waiting for user input (for example, mouse clicks and keystrokes). These delays are many orders of magnitude greater than those caused by disk access or network latency.
Ultimately, slow and blocking I/O pushes you towards synchronous code that is not concurrent.
How can platforms overcome blocking I/O?
There’s no doubt that slow and blocking I/O is the biggest hurdle to concurrency.
So, how do some platforms get around this issue?
Let us look at a couple of popular methods to enable concurrency.
Multi-Threading
Multi-threading is a common approach to support concurrency. It is used by popular languages such as Java.
In traditional blocking I/O programming, the function call that makes an I/O request will block the execution until the operation completes. The block can go from a few milliseconds for disk access to even minutes if we are waiting on the user to press a particular key.
A web server that uses blocking I/O will not be able to handle multiple connections within the same thread. Each I/O operation will block the processing of any other connection.
Therefore, to achieve concurrency, web servers start off a new thread for each concurrent connection.
When a thread ‘blocks’ for an I/O operation, other requests will not be impacted as those requests have their own separate threads.
Hence, the name multi-threading.
See below illustration
Unfortunately, a thread is not cheap in terms of system resources.
Threads consume memory and cause context switches. Many times, a thread may sit in idle mode waiting for some I/O operation to complete. This leads to the wastage of resources.
Suffice it to say, having a long-running thread for each connection is not the best approach to increase efficiency.
Busy-Waiting
Most modern operating systems support a mechanism known as non-blocking I/O.
This type of I/O does not block the execution. The system’s call to access the resource returns immediately. If no results are available, the function returns a predefined constant. There is no waiting for data to be read or written.
So, how is the data actually processed?
The answer is busy-waiting.
Busy-waiting is nothing but actively polling a resource in a loop till it returns some actual data.
With busy-waiting, you can use the same thread to handle multiple I/O requests.
But polling algorithms are also inefficient. A large portion of the precious CPU time is wasted in iterating over resources that are mostly unavailable.
Why Node.js cannot use Multi-Threading
When compared to busy-waiting or polling, multi-threading is a better approach to achieve concurrency. Many programming languages and frameworks use multi-threading successfully.
However, Node.js cannot use multi-threading. This is because Node.js uses JavaScript and JavaScript is single-threaded.
So is there no hope?
Actually, there is another mechanism to enable non-blocking resources. This mechanism is known as Event Demultiplexing. You can also call it the Event Notification Service.
The below illustration shows the Event Demultiplexing concept.
There are three important steps in the Event Demultiplexing process:
-
First, the resources are added to a watchlist. Each resource has an associated operation such as read or write.
-
This watchlist of resources along with the callbacks is assigned to the Event Demultiplexer. The demultiplexer makes a synchronous and blocking call for any events generated from the watched resources. When the demultiplexer eventually returns from the blocking call, it has a new set of events available for processing.
-
Each event returned by the Event Demultiplexer is processed by the application callback methods. At this point, the resource is guaranteed to be ready to read and it will not block during the operation. When all the events are processed, the demultiplexer will again make a blocking request for the next set of events.
This process of checking for events, processing them, and checking again is known as the Event Loop and forms the basis of concurrency in Node.js. Moreover, the Event Demultiplexer makes it possible to handle the entire process in a single thread.
There is no need for multi-threading. No need for busy waiting and polling. A single thread with very little idle time is enough. Tasks are spread over time instead of being divided between multiple threads.
The Node.js Event Loop
Having gone through all the explanations, we can now construct a model of how the Node.js event loop works.
Check out the below diagram that shows all the steps in the event loop.
Let us go through all the steps one by one:
-
STEP A - The application generates a new I/O request and submits it to the Event Demultiplexer. Along with the request, the application also provides a handler or a callback. This is like telling the demultiplexer whom to contact once the I/O request is complete. The call to the demultiplexer is non-blocking and the control returns back to the application immediately.
-
STEP B - The Event Demultiplexer waits for events from I/O resources in its watchlist. When a bunch of I/O operations is completed, the Event Demultiplexer pushes the corresponding events into the Event Queue.
-
STEP C - The Event Loop iterates over the items in the Event Queue. Technically, the Event Queue is not just one queue but multiple queues that are processed in different stages. If interested, you can more about the different Event Loop queues.
-
STEP D - For each event, the Event Loop triggers the handler. The information about the handler was given to Event Loop by the demultiplexer in STEP B.
-
STEP E - The handlers are part of the application code. We also know them as callbacks. Once a callback runs, it may give the control back to the Event Loop. This is denoted by path E1. However, the handler might also request for new async operations i.e. path E2. The E2 path results in new operations being sent to the Event Demultiplexer.
-
STEP 6 - When the Event Queue is empty, the loop will block again on the Event Demultiplexer and wait for another cycle.
To describe things briefly, the Event Loop handles I/O by blocking until new events are emitted from a set of watched resources. When events become available, it dispatches each event to its corresponding handler using the Event Queue.
When the Event Demultiplexer will run out of pending operations and there are no events left in the Event Queue, the Node.js application automatically exits.
Conclusion
Node.js is concurrent because of the Event Loop. And the Event Loop is made possible by the demultiplexer.
However, very few developers know about the foundation of the Event Loop. To give credit where it’s due, the complexities of the Event Demultiplexer are so nicely abstracted that developers can build useful applications with Node.js even if they haven’t heard about the Event Loop.
Nonetheless, it is good to know about the role of Event Demultiplexer in the long run. It helps you make better choices about your application.
If you found this post to be useful, consider sharing it with friends and colleagues.
You can also connect with me on other platforms: