A common pattern is where you have a cluster of queue processors that just process jobs as fast as they can, and some other services that need to take the result of these processors and do something with it, maybe storing results in a database.
The Pattern The most robust and scalable way to accomplish this is by combining the standard job queue with the message queue pattern :
Submit Jobs
A service sends jobs to the cluster by opening a job queue and adding jobs to it
Process Jobs
The cluster processes jobs as fast as it can
Send Results
Every time a job gets completed in the cluster, a message is sent to a results message queue with the result data
Store Results
A separate service listens to the results queue and stores the results in a database
Architecture Example Implementation Submitting Service
Worker
Results Service
const Queue = require ( 'bull' ); const jobQueue = new Queue ( 'jobs' ); // Submit jobs await jobQueue . add ({ task: 'process-data' , data: { ... } });
Benefits
Scalability Workers and result processors can scale independently
Decoupling Job processing and result handling are separate concerns
Reliability Results are guaranteed to be delivered and stored
Flexibility Multiple services can consume results from the same queue
This pattern is particularly useful in microservices architectures where you want to maintain loose coupling between services.