# The Vert.x Worker Model
source: https://developer.mastercard.com/blog/the-vertx-worker-model/index.md

Authors: Sébastien Le Callonnec. Payment gateways such as the Mastercard Payment Gateway process hundreds of millions of payments every day. Building and maintaining complex platforms such as gateways requires a team of engineers to study traffic patterns and ensure stability. Our white label small business platform is a reactive system designed to be highly available: a reactive platform makes our platform responsive, resilient, elastic and message-driven. To build this platform, we have used Vert.x to call the various services required to process a payment -- all in an asynchronous fashion. Reactive Systems and Vert.x Reactive systems have emerged in response to the increasing need of highly-available elastic applications which have shown the limits of classic threaded systems. Those systems show resilience when faced with failure and favor loose coupling through asynchronous message-passing. More generally speaking, reactive has become synonymous with working with non-blocking streams of data. Vert.x is a JVM toolkit that can be used to build reactive systems. It is event-driven and uses message passing for processing work; those messages are by default handled by the event loop which is (usually) one thread responsible to find the right handler for an incoming message, and process it. As a result, the "golden rule" is: never block the event loop! But what if you need to process long-running tasks? Those long tasks will prevent any other task from being performed until they are complete, resulting in severe performance degradations, or even catastrophic failures. To solve that problem, Vert.x introduces the concept of worker verticles. A Vert.x application is typically composed of one or several verticles, which are similar to modules, executing in their own context on top of a Vert.x instance. Worker verticles are special ones that can be used to process tasks that would block the event loop, and the purpose of this post is to look at them and see how they fit in reactive systems built with Vert.x. All code for this article can be found on the Payment Gateway Github . Let's Get Reactive Coffee! Blocking the event loop Following the long-standing tradition of using a coffee shop, we are going to use a simple API that allows customers to order different types of coffee. The coffee is then prepared by a barista in the background, in an asynchronous fashion. Our application has two simple verticles: A HTTP server verticle, that receives the incoming orders, logs them and sends them to the event bus for processing A verticle that picks up the message from the event bus and processes it. A long-running task is simulated by a Thread.sleep for a random length of time Our first attempt is straightforward. The HTTP server verticle sends a request to the order processing verticle by sending a message on the event bus: Java def eb = vertx.eventBus() def message = new JsonObject() def correlationId = UUID.randomUUID().toString() message.put('id', correlationId) message.put('customer', requestBody.customer) message.put('coffee', requestBody.coffee) message.put('size', requestBody.size) eb.send('process.coffee.order', message) The worker is listening to messages sent to the process.coffee.order address, and simulates as a barista prepares the coffee: Java eb.consumer('process.coffee.order', { message -\> def body = message.body() log.info("Processing coffee order ${body\['id'\]} for ${body\['customer'\]}: ${body\['coffee'\]}, ${body\['size'\]}, instance:${this}") Thread.sleep(random.nextInt(10) \* 1000) log.info("Order ${body\['id'\]} complete! Calling ${body\['customer'\]} instance:${this}") }) Finally, the two verticles are simply deployed: Java vertx.deployVerticle('groovy:com.example.vertx.MainVerticle', new DeploymentOptions().setConfig(config)) vertx.deployVerticle('groovy:com.example.vertx.WorkerVerticle', new DeploymentOptions().setConfig(config)) We can now start our application. The vertx.logger-delegate-factory-class-name system variable ensures that we use SLF4J for logging, which will be handy to log the thread names for our tests: Bash java -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory -jar target/vertx-workers.jar To simulate our various customers ordering coffee, we use a simple Ruby script that sends several orders to the HTTP endpoint we expose: Bash ruby process_coffee.rb We are ready to run our first test, here is what happens: Log \[vert.x-eventloop-thread-4\] INFO WorkerVerticle - Processing coffee order 1 for aine: mocha, medium \[vertx-blocked-thread-checker\] WARN BlockedThreadChecker - Thread Thread\[vert.x-eventloop-thread-4,5,main\] has been blocked for 2768 ms, time limit is 2000 \[vertx-blocked-thread-checker\] WARN BlockedThreadChecker - Thread Thread\[vert.x-eventloop-thread-4,5,main\] has been blocked for 3771 ms, time limit is 2000 \[vertx-blocked-thread-checker\] WARN BlockedThreadChecker - Thread Thread\[vert.x-eventloop-thread-4,5,main\] has been blocked for 4773 ms, time limit is 2000 \[vertx-blocked-thread-checker\] WARN BlockedThreadChecker - Thread Thread\[vert.x-eventloop-thread-4,5,main\] has been blocked for 5772 ms, time limit is 2000 Vert.x complains that we are blocking the event loop thread. This is bad, as this means that we cannot process any other incoming request, and our customers are waiting in line. After a while, our shop will be so overloaded that the customers will simply go and order their coffee somewhere else. We are losing business! Enter the worker verticle To solve that issue, we are going to make the verticle responsible for making coffees a "worker verticle." Worker verticles do not execute on the event loop: they are executed by worker threads living in a thread pool. To do this, we now deploy the verticle using the setWorker(true) method call. We set up a specific thread pool for this verticle, with a size of 5 -- with that many baristas, we should be able to cope with high demand at peak time! Java vertx.deployVerticle('groovy:com.example.vertx.WorkerVerticle', new DeploymentOptions() .setConfig(config) .setWorkerPoolName("coffee-making-pool") .setWorkerPoolSize(5) .setWorker(true)) Let's run our test again: Log \[vert.x-eventloop-thread-4\] INFO MainVerticle - Processing request... \[customer:bernard, coffee:latte, size:medium\] \[coffee-making-pool-1\] INFO WorkerVerticle - Processing coffee order 1 for bernard: latte, medium, instance:WorkerVerticle@5680476f \[vert.x-eventloop-thread-4\] INFO MainVerticle - Processing request... \[customer:geraldine, coffee:americano, size:large\] \[vert.x-eventloop-thread-4\] INFO MainVerticle - Processing request... \[customer:bernard, coffee:cappucino, size:large\] \[vert.x-eventloop-thread-4\] INFO MainVerticle - Processing request... \[customer:herbert, coffee:americano, size:small\] \[vert.x-eventloop-thread-4\] INFO MainVerticle - Processing request... \[customer:esteban, coffee:mocha, size:small\] \[vert.x-eventloop-thread-4\] INFO MainVerticle - Processing request... \[customer:esteban, coffee:americano, size:medium\] \[coffee-making-pool-1\] INFO WorkerVerticle - order 1 complete! Calling bernard instance:WorkerVerticle@5680476f \[coffee-making-pool-1\] INFO WorkerVerticle - Processing coffee order 2 for geraldine: americano, large, instance:WorkerVerticle@5680476f \[coffee-making-pool-1\] INFO WorkerVerticle - order 2 complete! Calling geraldine instance:WorkerVerticle@5680476f \[coffee-making-pool-1\] INFO WorkerVerticle - Processing coffee order 3 for bernard: cappucino, large, instance:WorkerVerticle@5680476f \[coffee-making-pool-1\] INFO WorkerVerticle - order 3 complete! Calling bernard instance:WorkerVerticle@5680476f \[coffee-making-pool-1\] INFO WorkerVerticle - Processing coffee order 4 for herbert: americano, small, instance:WorkerVerticle@5680476f \[coffee-making-pool-1\] INFO WorkerVerticle - order 4 complete! Calling herbert instance:WorkerVerticle@5680476f \[coffee-making-pool-1\] INFO WorkerVerticle - Processing coffee order 5 for esteban: mocha, small, instance:WorkerVerticle@5680476f Wait! What is going on, here? It looks like we are still processing one coffee at a time, despite setting the size to our worker pool to 5! The reason for this is that we have only created one worker verticle instance, and that instance can only run on one given thread from the pool at a time. This means that we have 5 coffee machines, but only 1 barista who can operate one machine at a time! This time, let's create 5 instances of our worker instance: Java vertx.deployVerticle('groovy:com.example.vertx.WorkerVerticle', new DeploymentOptions() .setConfig(config) .setWorkerPoolName("coffee-making-pool") .setWorkerPoolSize(5) .setInstances(5) // Let's create 5 instances! .setWorker(true)) And now we truly have work happening in parallel, as expected: Log \[vert.x-eventloop-thread-4\] INFO Processing request... \[customer:bernard, coffee:latte, size:large\] \[coffee-making-pool-2\] INFO Processing coffee order 1 for bernard: latte, large, instance:com.example.vertx.WorkerVerticle@21604e3a \[vert.x-eventloop-thread-4\] INFO Processing request... \[customer:denise, coffee:americano, size:small\] \[coffee-making-pool-4\] INFO Processing coffee order 2 for denise: americano, small, instance:com.example.vertx.WorkerVerticle@42940c81 \[vert.x-eventloop-thread-4\] INFO Processing request... \[customer:charline, coffee:cappucino, size:large\] \[coffee-making-pool-0\] INFO Processing coffee order 3 for charline: cappucino, large, instance:com.example.vertx.WorkerVerticle@3c9d2c75 \[coffee-making-pool-0\] INFO Order 3 complete! Calling charline instance:com.example.vertx.WorkerVerticle@3c9d2c75 \[vert.x-eventloop-thread-4\] INFO Processing request... \[customer:herbert, coffee:mocha, size:large\] \[coffee-making-pool-1\] INFO Processing coffee order 4 for herbert: mocha, large, instance:com.example.vertx.WorkerVerticle@ecf6924 \[coffee-making-pool-1\] INFO Order 4 complete! Calling herbert instance:com.example.vertx.WorkerVerticle@ecf6924 \[vert.x-eventloop-thread-4\] INFO Processing request... \[customer:esteban, coffee:cappucino, size:large\] \[coffee-making-pool-3\] INFO Processing coffee order 5 for esteban: cappucino, large, instance:com.example.vertx.WorkerVerticle@258240f4 \[vert.x-eventloop-thread-4\] INFO Processing request... \[customer:esteban, coffee:latte, size:large\] \[coffee-making-pool-2\] INFO Order 1 complete! Calling bernard instance:com.example.vertx.WorkerVerticle@21604e3a \[coffee-making-pool-2\] INFO Processing coffee order 6 for esteban: latte, large, instance:com.example.vertx.WorkerVerticle@21604e3a \[coffee-making-pool-4\] INFO Order 2 complete! Calling denise instance:com.example.vertx.WorkerVerticle@42940c81 The reason for this behavior is that by default worker verticles are not multi-threaded, and are not executed by more than one thread a time. When executing a blocking task, an instance is taken from the instances of workers created at startup, and is executed in a thread taken from the common thread pool -- unless a specific thread pool has been created for these instances, as we did here. Worker verticle instances being executed by a given thread from the pool Another way of getting workers to run in parallel is to make them multi-threaded by calling setMultiThreaded(true) on the worker's Deployment Options. The difference with the single-threaded approach is that the user must make sure that the verticle is thread-safe, and deal with locking and synchronization "manually." How is that reactive? Worker verticles are a good way of implementing work that takes a long time or mostly blocking. If our application starts to grow, we can add new workers, expand the thread pool. But also in true reactive fashion, we can deploy multiple instances of our workers on separate machines to scale horizontally: the event bus will route messages to these workers as needed. Introducing separate verticles to do the work also allows us to decouple the different logical parts of our application: in our example, if anything goes wrong with the processing of the order, we can queue these orders while still taking new ones until the problem is resolved. This is possible because any failure of the worker component doesn't block or prevent the order-taking component. This blog post describes how worker verticles can help building reactive systems, but Vert.x 3 offers a lot more to build applications that can run in a variety of environments and programming languages -- and become the backbone for a much-prized micro service architecture.