Thread Based Asynchronous Programming
Different strategies for reducing I/O overhead in IO heavy environment
In our previous post, we introduced & discussed different strategies for reducing I/O overhead in IO heavy environment. In this post, we will take a deeper look at the thread-based asynchronous programming approach.
What is Thread-Based Asynchronous Programming?
The thread-based asynchronous programming approach, allows one thread pool to hand over a task to another thread pool i.e a worker thread pool, and then get notified to handle the result when the worker thread pool is done with the task.
From the perspective of the caller thread, the system now becomes asynchronous as all of its work on a single call path is not being done sequentially it does something then hands over IO related tasks to one or more worker thread pools, and then comes back to resume execution from that point onwards having done some completely independent task in between.
Threads on the worker pool still get blocked for IO, but now only the threads of this pool get blocked, thereby limiting the cost to the system. Other code paths of the system which do not involve IO activity are scaled-up by the freed caller thread. Service throughput increases considerably because the caller thread doesn’t sit around waiting for IO to complete, it can perform other computations.
This is the reason why we allot a fixed number of database and TCP connection pool threads at the application layer. In a microservice, we can have all individual threads that want to call another microservice by creating their own TCP connection firing the API and then waiting for a response. However, there are other things the system can do while we wait. We can create a smaller worker pool of TCP connections and funnel all API calls through it and free the caller threads from this blockage. By multiplexing the calls over this smaller thread pool, we can free up a lot of other threads for doing non-IO-related work.
A real-life example of understanding this behavior is that of a checkout counter in a shop. A small number of checkout counters are able to handle a large number of customers so long as not everyone comes for checkout at the same time. Only a small number of workers at the checkout counter are blocked on the checkout function other workers are free to assist other customers.
Handling Task Completion
By off-loading the task to a worker thread, the caller thread needs to know where it was in its execution path when it receives the result of the task from the worker pool thread.
This problem is typically solved by introducing callbacks which are methods to be invoked by the worker pool thread on completion of the task given to it. The caller thread registers these callbacks to the Future object returned by the worker pool and now it’s the language/framework's responsibility to invoke them on the caller thread by issuing an interrupt to it (to get it to stop whatever it was doing) and instructing it to execute the relevant callback.
The execution pattern looks something like this → Thread 1 calls the worker pool to give it a task. Thread 2 in the worker pool executes the task and invokes a callback. Thread 1 gets interrupted and switches to executing the callback.
Execution of callback is decided by the language/framework. Different languages can implement this execution pattern in a different fashion. For example in languages like java/scala ExecutorService is the framework that has the responsibility to execute the callback after future completion, the callback can be executed either in the caller thread or from any thread from the pool itself.
Caveats for using Thread-Based Asynchronous Programming
- Asynchronous code can be a little complicated to understand and debug once you have some parallel and serial task handoffs creating a callback hell which is one of the biggest problems in all asynchronous programming languages.
- ThreadLocal variables no longer work. Since the caller thread hands off the work to another thread and moves to other tasks, any context stored as a ThreadLocal for eg: request context for HTTP request being made is lost. The only way to propagate it is to explicitly pass it along as a parameter in the task handoff.
Even though the worker thread model gives us significant improvements but it is more focused on isolating and containing the problem of thread-blocking to a limited no. of thread. The caller thread is merely pretending to be unblocked but essentially its block has been pushed across to some other thread wasting resources. IO is still blocking and threads are still needed for each IO task.
A true non-blocking IO paradigm aka event-based asynchronous programming like in node js can be used, where threads are never blocked on IO and we have no need of creating and maintaining worker thread pools. In the next blog, we will be looking at this paradigm.
I would love to hear from you. You can reach me for any query, feedback, or just want to have a discussion through the following channels:
Drop me a message on Linkedin or email me on akash.codingforliving@gmail.com
Build composable web applications
Don’t build web monoliths. Use Bit to create and compose decoupled software components — in your favorite frameworks like React or Node. Build scalable and modular applications with a powerful and enjoyable dev experience.
Bring your team to Bit Cloud to host and collaborate on components together, and speed up, scale, and standardize development as a team. Try composable frontends with a Design System or Micro Frontends, or explore the composable backend with serverside components.
