It works until it doesn't. The email provider is slow, so checkout is slow. Analytics is down, so checkout fails. And every new feature means editing the checkout code again.

That problem is exactly what Apache Kafka is built for. This is the first post in my Learn Kafka series. By the end of it you'll understand the core ideas and have a working producer and consumer running on your machine. Every command and every line of output below comes from a real run.

What Kafka actually is Kafka is a distributed event log. Strip away the jargon and it's three simple ideas:

Something happened, so you write it down as an event: "order ord_101 was placed". Kafka appends that event to a log and keeps it, for days or forever, depending on how you configure it. Any number of services can read that log, each at its own pace, without knowing about each other. The checkout service no longer calls five services. It just announces "an order was placed" and moves on. Whoever cares about orders listens.

A useful way to think about it: a message queue is like a to-do list where finished items get crossed out. Kafka is more like a diary. Entries stay, and anyone can read it from any page.

The six words you need Term What it means Event (message) One record, with a key, a value and a timestamp. Example: key u_1, value {"orderId":"ord_101"} Topic A named stream of events, like orders or payments. Think of it as a table name for events Partition A topic is split into partitions so it can scale. Each partition is an ordered, append-only log Offset The position of an event inside a partition: 0, 1, 2… Consumers remember the offset they've reached Producer Code that writes events to a topic Consumer group One or more consumers that share the work of reading a topic. Each group gets every event The last one is where Kafka gets clever, and we'll see it in action in a minute.

Step 1: Run Kafka locally Since Kafka 4.0, Kafka no longer needs ZooKeeper. It manages itself in what's called KRaft mode, so a local setup is much simpler than older tutorials suggest.

With Docker (easiest):

docker run -d --name kafka -p 9092:9092 apache/kafka:4.3.1 Without Docker: you need Java 17 or newer. Download Kafka from kafka.apache.org/downloads, extract it, then from inside the folder:

KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)" bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties bin/kafka-server-start.sh config/server.properties Either way, you now have a broker listening on localhost:9092.

Step 2: Create a topic We'll create an orders topic with 3 partitions:

Docker

docker exec -it kafka /opt/kafka/bin/kafka-topics.sh
--create --topic orders --partitions 3 --bootstrap-server localhost:9092

Without Docker

bin/kafka-topics.sh --create --topic orders --partitions 3 --bootstrap-server localhost:9092 Created topic orders. Step 3: Set up the Node.js project For Node.js, I'm using Confluent's official client, @confluentinc/kafka-javascript. You'll see a lot of older tutorials use kafkajs, which hasn't been actively maintained for a while. The Confluent client includes a KafkaJS-style API, so the code will look familiar either way.

mkdir kafka-demo && cd kafka-demo npm init -y npm pkg set type=module npm install @confluentinc/kafka-javascript Step 4: Write a producer The producer plays the role of our checkout service. It sends three orders to the orders topic.

// producer.js import { KafkaJS } from "@confluentinc/kafka-javascript";

const kafka = new KafkaJS.Kafka({ kafkaJS: { clientId: "checkout-service", brokers: ["localhost:9092"], logLevel: KafkaJS.logLevel.ERROR, }, });

const producer = kafka.producer(); await producer.connect();

const orders = [ { orderId: "ord_101", userId: "u_1", total: 499 }, { orderId: "ord_102", userId: "u_2", total: 1299 }, { orderId: "ord_103", userId: "u_1", total: 249 }, ];

for (const order of orders) { await producer.send({ topic: "orders", messages: [{ key: order.userId, value: JSON.stringify(order) }], }); console.log(Sent ${order.orderId}); }

await producer.disconnect(); Notice the key: we use userId. That matters, and we'll come back to it.

Step 5: Write a consumer The consumer reads orders and prints them. We pass the consumer group name on the command line so we can run it as different services.

// consumer.js import { KafkaJS } from "@confluentinc/kafka-javascript";

const groupId = process.argv[2] ?? "email-service";

const kafka = new KafkaJS.Kafka({ kafkaJS: { clientId: groupId, brokers: ["localhost:9092"], logLevel: KafkaJS.logLevel.ERROR, }, });

const consumer = kafka.consumer({ kafkaJS: { groupId, fromBeginning: true }, });

await consumer.connect(); await consumer.subscribe({ topics: ["orders"] });

await consumer.run({ eachMessage: async ({ partition, message }) => { const order = JSON.parse(message.value.toString()); console.log( [${groupId}] partition ${partition}, offset ${message.offset}: + order ${order.orderId} for user ${order.userId}, total ₹${order.total} ); }, }); fromBeginning: true means that when a brand-new group starts, it reads the topic from the first event instead of only new ones.

Step 6: Watch consumer groups work Open three terminals and start three consumers. Two belong to email-service, and one is analytics-service:

node consumer.js email-service # terminal 1 node consumer.js email-service # terminal 2 node consumer.js analytics-service # terminal 3 Give them a few seconds to join, then in a fourth terminal run the producer:

node producer.js Sent ord_101 Sent ord_102 Sent ord_103 Here's what each consumer printed on my machine:

terminal 1 (email-service)

[email-service] partition 2, offset 0: order ord_101 for user u_1, total ₹499 [email-service] partition 2, offset 1: order ord_103 for user u_1, total ₹249

terminal 2 (email-service)

[email-service] partition 1, offset 0: order ord_102 for user u_2, total ₹1299

terminal 3 (analytics-service)

[analytics-service] partition 2, offset 0: order ord_101 for user u_1, total ₹499 [analytics-service] partition 2, offset 1: order ord_103 for user u_1, total ₹249 [analytics-service] partition 1, offset 0: order ord_102 for user u_2, total ₹1299 Your partition numbers may differ, but the pattern will be the same. This small output shows the three most important ideas in Kafka.

1. Different groups each get everything analytics-service received all three orders, and so did email-service as a whole. Each group is an independent reader of the log. Adding a new service, say loyalty-service, means starting a consumer with a new group name. The checkout code doesn't change at all.

2. Consumers in the same group share the work The two email-service consumers split the orders between them. Kafka gives each consumer in a group its own partitions, so no order gets emailed twice. If email gets busy, run more consumers in that group, up to one per partition. That's why the partition count matters: it's the ceiling on how many consumers in one group can work in parallel.

3. Same key, same partition, guaranteed order Both orders from user u_1 landed in partition 2, in the order they were sent. Kafka sends every message with the same key to the same partition, and order is only guaranteed within a partition. So if the order of events per user, per account or per device matters to you, use that ID as the key.

Step 7: Replay, Kafka's superpower Stop a consumer and start it again with the same group name. It doesn't print the old orders again, because the group remembers its offset: how far it has read.

Now start one with a new group name:

node consumer.js reporting-service It reads all three orders from the beginning, because the events are still in the log. That's something a traditional queue can't do. You can add a new service months later, or rebuild a broken report, by replaying history.

When you don't need Kafka Kafka is powerful, but it's also one more system to run. You probably don't need it if:

you have a single app and a background job queue would do (BullMQ, a Postgres-backed queue, or your platform's queues); you're handling a few events per minute and no other service needs them; nobody on the team wants to run or pay for a cluster yet. Kafka starts to pay off when several independent services react to the same events, when you need high throughput, or when being able to replay events actually matters to you.

Recap Kafka is an event log: producers append, consumers read, events stay. Topics are split into partitions. Each consumer tracks an offset. Different groups each get every event. Consumers in one group share the partitions. Keys decide the partition, and order is guaranteed within a partition. Since Kafka 4.0 there's no ZooKeeper, so you can be up and running locally in one command. Next in the series In the next post, we'll make this production-shaped: what happens when a consumer crashes halfway through a message, how offset commits work, and how to avoid processing an order twice.

If this helped, follow the Learn Kafka series so you don't miss it, and tell me in the comments what you're planning to build with Kafka.