What is a message queue? why backend systems need it

When building backend systems, the initial architecture is usually straightforward:

Client -> API -> Business Logic -> Database/Third-party Service -> Response

• A request enters the system, the backend performs all required work, and then returns a response to the client.

• This approach is simple, easy to understand, and often sufficient when the system is still small.

• However, as traffic grows, workloads become heavier, or the system starts communicating with multiple external services, direct synchronous processing begins to introduce several problems.

This is where Message Queues become useful.

1. What Problems Does a Message Queue Solve?

Before looking at how Message Queues work, it is useful to understand what can go wrong when everything is processed directly inside the request lifecycle.

1.1 The API Has to Wait for Work That Is Not Actually Required

Consider an API for creating an order:

POST /orders

After creating the order, the system may also need to:

1. Save the order to the database
2. Send a confirmation email
3. Send a notification
4. Write an audit log
5. Synchronize the order with a CRM

If everything is processed sequentially:

Client
  ↓
API
  ├─ Insert Order       100ms
  ├─ Send Email         700ms
  ├─ Send Notification  300ms
  ├─ Write Audit        100ms
  └─ Call CRM          1500ms
  ↓
Response

The total response time can easily approach three seconds.

From the business perspective, however, the client may only need to know one thing:

Was the order successfully created?

Sending an email, publishing a notification, or synchronizing with a CRM does not necessarily need to finish before the API responds.

But because those tasks are part of the same synchronous request flow, the client has to wait for all of them.

This creates a form of time coupling: the response time becomes dependent on downstream work.

1.2 The Producer Generates Work Faster Than the Consumer Can Process It

Suppose a service processes images.

Its safe processing capacity is:

5 image-processing jobs / second

But suddenly the system receives:

200 requests / second

If every request immediately starts processing:

200 requests
    ↓
200 processing tasks
    ↓
CPU spikes
Memory usage increases
Threads/processes increase
    ↓
Server overload

The problem is not necessarily that each individual task is too expensive.

The problem is that too many tasks are executing at the same time.

The producer is generating workload faster than the consumer can safely process it.

1.3 One Service Failure Can Affect Other Services

Consider a simple flow:

Order Service
     ↓
Email Service

The order is successfully created, but the Email Service is currently unavailable.

If the Order Service calls it directly:

Order Service
     ↓
POST Email Service
     ↓
timeout

The order API may become:

• slower,

• timed out,

• forced to retry inside the request,

• or even fail completely.

But the actual failure only exists in the Email Service.

This is called failure coupling.

A non-critical downstream component can negatively affect a more important upstream workflow.

1.4 Both Services Must Be Online at the Same Time

If Service A communicates synchronously with Service B:

Service A
   ↓
Service B

then Service B must be:

• running,

• reachable over the network,

• healthy,

• able to accept more requests,

• and fast enough to respond before the timeout.

If Service B is restarting or under maintenance, Service A may no longer be able to complete its normal flow.

The two services are coupled to each other's lifecycle.

1.5 The Producer Must Know Who the Consumers Are

Suppose an order creation needs to trigger several systems:

Order Service
 ├─ Email Service
 ├─ Analytics Service
 ├─ Loyalty Service
 └─ Warehouse Service

Later, the system introduces Fraud Detection:

Order Service
 ├─ Email
 ├─ Analytics
 ├─ Loyalty
 ├─ Warehouse
 └─ Fraud Detection

The Order Service gradually becomes responsible for coordinating more and more downstream dependencies.

This increases coupling between services.

Message Queue as a Boundary Between Producer and Consumer

Instead of:

Producer
    ↓
Consumer

we introduce an intermediate layer:

Producer
    ↓
Message Queue
    ↓
Consumer

The producer no longer directly asks the consumer to perform the work.

It only sends a message such as:

"Order 123 was created"

or:

"Send an email to user 456"

The queue stores the message until a consumer is ready to process it.

2. A Simple Message Queue Flow

A basic message-queue system consists of three main parts:

Producer → Queue → Consumer

Producer

The producer creates a message.

For example:

{
  "type": "send_email",
  "userId": 123,
  "template": "order_created"
}

A producer can be:

• an HTTP API,

• a scheduler,

• another service,

• an event handler.

Queue

The queue temporarily stores messages.

For example:

Queue

[Job 1]
[Job 2]
[Job 3]
[Job 4]
[Job 5]

If the consumer is not fast enough, the remaining messages stay in the queue.

Consumer

The consumer reads messages from the queue and performs the actual work:

Queue
  ↓
Consumer
  ↓
sendEmail()

After processing succeeds, the consumer may acknowledge the message:

Consumer
   ↓
ACK
   ↓
Queue removes the message

If processing fails, depending on the system, the message may be:

retried
requeued
sent to a dead-letter queue

The key idea is:

The producer does not need to wait for the consumer to finish processing the work.

3. Fundamental Message Queue Use Cases

3.1 Background Processing and Faster API Response Times

This is one of the most common use cases.

Consider a registration API:

POST /register

After creating a user, the system may need to:

Create user
Send welcome email
Send analytics event
Sync CRM

With direct processing:

Client
  ↓
API
  ├─ Create User
  ├─ Send Email
  ├─ Analytics
  └─ CRM
  ↓
Response

The client waits for the entire flow.

With a queue:

Client
  ↓
API
  ├─ Create User
  └─ Enqueue background jobs
  ↓
Response

Then:

Queue
 ├─ Send Email
 ├─ Analytics
 └─ CRM
      ↓
   Workers

The API only performs the work that is required for the request itself to succeed.

Everything that can be deferred moves to background processing.

Typical Tasks That Fit Well

Examples include:

• Sending emails

• Push notifications

• Generating PDFs
  Resizing images

• Analytics

• Audit processing

• Exporting reports

• Synchronizing CRM systems

• Calling non-critical third-party APIs

However, this does not mean that every operation should be moved to a queue.

If a business transaction requires a step to succeed before returning a successful response, that step still belongs in the main flow.

For example:

• Create booking

• Charge payment

If the business rule says:

A booking is only confirmed after payment succeeds

then simply enqueueing the payment and immediately returning:

booking successful

would be incorrect.

Message Queues are most useful for work that can safely be deferred.

3.2 Concurrency Limiting and Overload Protection

Message Queues are also useful for regulating workload.

Suppose a server can safely process:

10 report-generation jobs at the same time

But the system suddenly receives:

1000 requests

Without workload control:

1000 requests
      ↓
1000 report jobs start immediately
      ↓
CPU 100%
Memory pressure
Database overload
Third-party overloa

With a queue:

1000 jobs
    ↓
Queue
    ↓
Worker Pool

concurrency = 10

Only ten jobs run at a time:

Job 1
Job 2
Job 3
...
Job 10

The remaining jobs stay in the queue:

Queue
──────────────────────
Job 11
Job 12
Job 13
...
Job 1000

When a worker finishes:

Job 4 complete

it takes the next job:

Job 11

As a result, the rate at which work arrives and the rate at which work is executed no longer need to be identical.

The queue becomes a buffer between the two.

Without a queue:

incoming load
     ↓
process immediately

With a queue:

incoming load
     ↓
buffer
     ↓
process according to capacity

This is the foundation of concepts such as:

• concurrency control,

• load leveling,

• backpressure,

• downstream protection.

For example, if a third-party API allows only:

50 requests / second

and your system receives 500 jobs in one second, you do not need to send all 500 requests immediately.

Workers can process them at a controlled rate that respects downstream limits.

3.3 Retry and Fault Isolation

Another major benefit of Message Queues is separating consumer failures from producers.

Consider:

API
 ↓
Queue
 ↓
CRM API

The CRM is temporarily unavailable.

A consumer tries to process a job:

Job
 ↓
CRM
 ↓
FAILED

Instead of failing the original request, the system can retry:

retry 1
 ↓
retry 2
 ↓
retry 3

For example:

Job
 ↓
Worker
 ↓
Third-party API
 ↓
FAIL
 ↓
wait
 ↓
retry

If all retries fail:

Job
 ↓
Dead Letter Queue

or the job may simply be marked as failed for later investigation.

Why This Matters

• Suppose a user successfully creates an order.

• The system also needs to synchronize that order with a CRM.

• The CRM is down for ten minutes.

Without a queue:

Order API
   ↓
CRM
   ↓
timeout

A non-critical system now directly affects the user-facing request.

With a queue:

Order API
   ↓
Create Order
   ↓
Enqueue CRM Sync
   ↓
Response success

If the CRM is unavailable:

Queue
 ↓
Worker
 ↓
CRM FAILED
 ↓
retry later

The order still exists normally.

Only the CRM synchronization is delayed.

This is fault isolation.

A failure in one subsystem does not necessarily propagate into another subsystem.

3.4 Asynchronous Communication Between Services

Message Queues are also commonly used for communication between services without synchronous request-response.

Suppose an order is created:

Order Service

and the service publishes an event:

order.created

The broker receives it:

Order Service
     ↓
 order.created
     ↓
Message Broker

Multiple services may consume that event:

                  order.created
                       ↓
                    Broker
               ┌───────┼────────┐
               ↓       ↓        ↓
            Email   Loyalty  Analytics

The Order Service does not need to know:

where the Email Service is
whether Analytics is currently online
how long Loyalty takes to process the event

It only announces:

An order has been created.

Each consumer independently decides what to do with that event.

This Is Asynchronous Messaging

It should not be thought of as the same thing as UDP.

UDP is closer to:

sender sends
→ receiver may or may not receive it

A message broker usually provides additional mechanisms such as:

persistence
acknowledgment
retry
delivery guarantees
consumer tracking
dead-letter queues

A more accurate description is:

asynchronous communication through a messaging infrastructure.

4. Queue Consumers vs. Similar Background Concepts

A common question when learning Message Queues is:

If a consumer is just a background worker, how is it different from a thread or a cron job?

The important distinction is:

“Background” describes where or how code executes.
“Message Queue” describes how work is delivered from producer to consumer.

These concepts are not direct replacements for one another.

4.1 Background Threads Inside the API Process

For example, in FastAPI or Django:

def send_email():
    ...

def api_handler():
    create_user()

    thread = Thread(target=send_email)
    thread.start()

    return {"success": True}

The architecture looks like:

API Process
├── Request Handler
└── Background Thread

The email no longer blocks the response.

At first glance, this looks similar to a queue.

However, the background thread still belongs to the same lifecycle as the API process.

If that process:

• crashes

• restarts

• is redeployed

• is killed

• runs out of memory

the background task may disappear with it.

Plain threading also does not automatically provide:

• persistent jobs

• retry

• acknowledgment

• dead-letter queues

• distributed workers

• job status

• backpressure

• queue backlog

When Is a Background Thread Enough?

It can be sufficient when the task is:

• small,

• fast,

• non-critical,

• safe to lose,

• does not need retry,

• does not need independent scaling.

Examples:

best-effort logging
local cache updates
non-critical cleanup

4.2 Cron and Schedulers

Cron solves a different problem.

Cron answers:

When should this task be triggered?

For example, in NestJS:

@Cron('0 0 * * *')
async cleanupExpiredData() {
  ...
}

The flow is:

00:00
  ↓
Scheduler
  ↓
cleanupExpiredData()

Cron does not require a queue.

A scheduler can call a function directly:

Scheduler
   ↓
Function

or enqueue work:

Scheduler
   ↓
Queue
   ↓
Worker

Suppose every day at midnight the system needs to generate reports for 100,000 users.

A poor approach would be:

Cron
 ↓
loop through 100000 users
 ↓
generate every report directly

A more scalable approach is:

Cron
 ↓
create jobs
 ↓
Queue
 ↓
Worker Pool

The scheduler is responsible for:

WHEN

The queue is responsible for:

HOW WORK IS DELIVERED

The worker is responsible for:

HOW WORK IS EXECUTED

Celery Is a Common Source of Confusion

Celery is not just a scheduler.

A typical setup looks like:

Django
  ↓
Celery Broker
  ↓
Celery Worker

A Celery Worker is effectively a queue consumer.

Meanwhile:

Celery Beat

is the scheduler.

When both are combined:

Celery Beat
    ↓
Broker
    ↓
Celery Worker

Beat decides when the task should run.

The broker stores and delivers the task.

The worker executes it.

4.3 Queue Worker / Consumer

A queue consumer usually runs independently from the API process.

For example:

API Container
     ↓
Redis / RabbitMQ
     ↓
Worker Container

They can scale independently:

• API x 5

• Queue

• Worker x 20

If API traffic increases:

scale API

If the queue backlog grows:

scale workers

The two groups do not need to scale together.

This is a major architectural difference compared with a background thread inside an API process.

A Simple Mental Model

You can distinguish these concepts using three questions:

• WHERE?

• WHEN?

• HOW?

Background Thread

• WHERE: inside the same API process

• WHEN: triggered by the current application flow

• HOW: function call / shared memory
  

Cron

• WHERE: depends on the implementation

• WHEN: time-based schedule

• HOW: direct function call or enqueue

Queue Consumer

• WHERE: usually an independent worker process or service

• WHEN: when a message becomes available

• HOW: through a queue or broker

They can also work together:

HTTP Request
      │
Cron ─┼─→ Producer
      │
Event ┘
        ↓
      Queue
        ↓
    Worker Pool

5. BullMQ, RabbitMQ, and Kafka

At an introductory level, all three are related to asynchronous processing and messaging.

However, they are optimized for slightly different problems.

BullMQ

BullMQ is built on Redis and is especially useful for job queues and background processing, particularly in Node.js systems.

The basic model is:

API
 ↓
BullMQ / Redis
 ↓
Worker

Typical jobs include:

sendEmail
generateReport
resizeImage
processWebhook
syncCRM

BullMQ is convenient when you need:

background jobs
retry
delayed jobs
concurrency control
job status
rate limiting

A simple way to remember BullMQ is:

There is a job that someone needs to execute.

For example:

GenerateReportJob

RabbitMQ

RabbitMQ is a message broker focused on routing and delivering messages between producers and consumers.

A basic architecture can look like:

Producer
   ↓
Exchange
   ↓
Queue
   ↓
Consumer

RabbitMQ provides powerful routing mechanisms.

Messages can be routed to different queues based on concepts such as:

routing keys
exchange types
bindings

For example:

order.created
      ↓
Exchange
  ┌────┼─────┐
  ↓    ↓     ↓
Email CRM Analytics

A simple way to remember RabbitMQ is:

I need to route and deliver this message to the appropriate consumer.

Kafka

Kafka has a somewhat different mental model from traditional message queues.

Kafka is primarily designed around event streaming and distributed event logs.

A producer writes events to a topic:

Producer
   ↓
Topic

The topic stores an ordered sequence of events:

Topic
──────────────────────────→
Event 1
Event 2
Event 3
Event 4
Event 5

Consumers read events and maintain their own offsets:

Consumer A → offset 5
Consumer B → offset 3

Consumer A reading Event 5 does not prevent Consumer B from reading the same event later.

Kafka is well suited for problems such as:

event streaming
event history
large-scale data pipelines
analytics events
multiple independent consumers
event replay

A simple way to remember Kafka is:

This event happened. Store it in an event log so consumers can read it independently.

Simple Comparison

This is not a strict boundary.

All three have overlapping capabilities.

At the beginning, the important part is not memorizing every feature.

The important part is understanding which problem makes you need one of them.

5. Summary

From the outside, a Message Queue can easily look like:

A way to run code in the background.

That is true, but it is only part of the picture.

The real value of a Message Queue is that it creates a boundary between producer and consumer.

Instead of:

Producer
    ↓
Consumer

the two sides are separated:

Producer
    ↓
Queue
    ↓
Consumer

That boundary reduces several forms of dependency.

Less Time Coupling

The producer does not necessarily have to wait for the consumer to finish.

Producer
 ↓
enqueue
 ↓
continue

Less Speed Coupling

The producer can generate work faster than the consumer for a period of time.

The queue stores the backlog:

Producer → Queue → Consumer
             ↑
           buffer

Less Capacity Coupling

Consumers can process work according to their actual available capacity.

1000 jobs
   ↓
Queue
   ↓
10 workers

Less Lifecycle Coupling

A consumer being temporarily unavailable does not necessarily mean the producer has to fail.

Producer ✓
Queue    ✓
Consumer ✗

The message can remain available until the consumer recovers.

Less Service Coupling

The producer does not necessarily need to know every downstream consumer.

Producer
   ↓
Event
   ↓
Broker
 ┌─┼─┐
 ↓ ↓ ↓
A B C

A more complete way to think about Message Queues is therefore:

A Message Queue is not simply a mechanism for running background jobs. It is an intermediate boundary that reduces the producer's dependency on the consumer's timing, processing speed, availability, and capacity.

Use cases such as:

Background processing
Concurrency limiting
Retry
Fault isolation
Asynchronous communication

all come from the same underlying idea:

Decouple the producer from the consumer

That is the most important concept to understand when learning Message Queues.