Queues, Streams, and Pub/Sub - Which One Do You Need to Know?

In partnership with

Welcome to the Null Pointer Club Newsletter!

If you’ve ever been asked about messaging systems in a system design interview, you know it’s one of those topics that can get tricky fast. Interviewers love to test candidates on Queues, Streams, and Publish-Subscribe (Pub/Sub) models—where they’re used, how they work, and when to pick one over the other.

So Today, We’re here to break it down in a way that sticks and helps you, so you can confidently answer these questions when they come up.

Understanding Messaging Systems

Messaging systems play a crucial role in managing asynchronous communication among various services within a distributed system. They contribute to a seamless data flow, enhance reliability, and support scalability.

You will often come across three common patterns:

1. Message Queues

A queue is a messaging pattern where messages are sent to a broker and processed by consumers in a First-In-First-Out (FIFO) manner. Once a message is processed, it’s removed from the queue.

• Example: A food delivery system where orders (messages) are placed in a queue and assigned to the next available delivery agent.

• Best for: Task processing, job scheduling, and load distribution.

• Popular tools: RabbitMQ, Amazon SQS, Apache ActiveMQ

2. Event Streams

A stream processes and delivers messages in real time, typically maintaining a log of all messages. In contrast to queues, messages remain available even after being consumed, enabling multiple consumers to handle the same message.

• Example: A ride-sharing app where various services (such as location tracking, pricing, and notifications) can simultaneously process the same ride request.

• Best for: Real-time analytics, log processing, event-driven architectures.

• Popular tools: Apache Kafka, AWS Kinesis, Apache Pulsar

3. Publish-Subscribe (Pub/Sub)

In a Pub/Sub model, publishers send messages to topics rather than sending them directly to a queue or stream. Subscribers who are interested in those topics then receive the messages.

• Example: A sports app that sends live score updates to thousands of users who have subscribed to notifications for their favorite teams.

• Best for: Broadcasting updates to multiple consumers, chat applications, and notification systems.

• Popular tools: Google Pub/Sub, Redis Pub/Sub, NATS

Common Interview Questions and How to Answer Them

1. What’s the difference between a Queue, a Stream, and Pub/Sub?

• Queues: Messages are processed once and then deleted. Used for task distribution.

• Streams: Messages are logged and can be processed by multiple consumers. Used for real-time processing.

• Pub/Sub: Messages are broadcast to multiple subscribers, who each receive a copy. Used for event-driven systems.

2. When would you choose a Queue over a Stream?

If you require one-time processing and prefer messages to be deleted after use (similar to job processing), opt for a Queue. Conversely, if you need messages to remain available for multiple consumers (like for analytics processing), choose a Stream.

3. How do you ensure reliability in a messaging system?

• Message persistence: Store messages until they’re acknowledged (e.g., Kafka’s log-based storage).

• Retries and dead-letter queues: Handle failed messages gracefully.

• Load balancing: Use multiple consumers to distribute workload.

• Ordering guarantees: Some systems like Kafka provide message ordering.

4. What are some common challenges in messaging systems?

• Message Duplication: Systems like Kafka and RabbitMQ provide message deduplication strategies.

• Scaling: Queues may need partitioning for high throughput.

• Latency: Streams process messages in real time, but may introduce delays.

Pro Tips

• Understand the trade-offs involved and be prepared to explain when and why you would choose one system over another.

• Provide real-world examples to make your responses more relatable and memorable.

• Familiarize yourself with the tools; even if you haven't directly worked with Kafka or RabbitMQ, understanding their strengths and weaknesses can leave a positive impression on your interviewer.

• Discuss edge cases, such as what occurs if a consumer crashes and how you can ensure message delivery guarantees.

Mastering messaging systems can make or break your system design interview. Keep practicing, and you'll be handling these questions like a pro in no time!

— The Nullpointer Club Team