12 min read

Kafka Architecture Explained: Brokers, Partitions, KRaft

Kafka architecture with a diagram: brokers, topics, partitions, replicas and ISR, the KRaft controller, producers and consumer groups, tested on Kafka 4.4.

kafka kafka-architecture kafka-architecture-diagram kraft distributed-systems system-design
On this page · 12 sections
  1. The Kafka architecture diagram
  2. Brokers and the cluster
  3. Topics, partitions and offsets
  4. Replication: leaders, followers and the ISR
  5. The controller and KRaft
  6. Producers: the write path
  7. Consumers and consumer groups: the read path
  8. What happens when a broker fails
  9. Design decisions that make Kafka fast
  10. Sizing rules of thumb
  11. Frequently asked questions
  12. Keep learning

Kafka’s architecture is small once you see it: a cluster of brokers stores topics, each topic is split into partitions, every partition is copied to a few brokers, and a controller quorum keeps track of who leads what. Producers append to partition leaders; consumers read from them in consumer groups. Everything else (ordering, scaling, failover, delivery guarantees) follows from those few pieces.

This guide walks through each component with a diagram, then runs a real three-node Kafka 4.4 cluster and stops one broker to show what actually happens. Every command output below is copied from that run.

The Kafka architecture diagram

Kafka architecture diagram: two producers write to partition leaders spread over three brokers; each partition of topic orders has one leader and two follower replicas; a KRaft controller quorum holds cluster metadata; a consumer group of two consumers reads the partitions

Read it from left to right:

  1. Producers decide which partition each record goes to (by hashing the key) and send it to that partition’s leader.
  2. Brokers store partitions as append-only logs on disk. Each partition has one leader and some followers that copy it.
  3. The controller quorum (top) stores the cluster’s metadata (which brokers are alive, which replica leads each partition) and tells brokers about changes.
  4. Consumers in a group split the partitions between them and read each one in order, tracking their position with offsets.

Brokers and the cluster

A broker is one Kafka server process. A cluster is a set of brokers that share the same metadata. Brokers do the heavy lifting:

  • accept writes from producers and append them to partition logs on disk,
  • serve reads to consumers and to follower replicas on other brokers,
  • report to the controller and apply the metadata it publishes.

Clients only need one or two broker addresses (bootstrap.servers). They fetch the full cluster metadata from there and then talk directly to whichever broker leads each partition they need. There is no central proxy in the data path, which is a big part of why Kafka scales.

Topics, partitions and offsets

A topic is a named stream of records, like orders or payments. A topic is split into partitions, and each partition is an ordered, append-only log. Each record in a partition gets a sequential offset: 0, 1, 2 and so on.

Two consequences matter in practice:

  • Ordering is per partition, not per topic. Records with the same key land in the same partition, so all events for one key stay in order. There is no global order across partitions.
  • Partitions are the unit of parallelism. One partition is read by at most one consumer in a group, so a topic with 6 partitions can keep at most 6 consumers in one group busy.

Here is that rule on the test cluster: seven records with four different keys, written to a 3-partition topic.

$ printf 'alice:p1\nbob:p2\ncarol:p3\ndave:p4\nalice:p5\nbob:p6\nalice:p7\n' | \
    kafka-console-producer.sh --bootstrap-server localhost:9092 --topic payments \
      --reader-property parse.key=true --reader-property key.separator=:

$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic payments \
    --from-beginning --group billing \
    --formatter-property print.partition=true --formatter-property print.offset=true \
    --formatter-property print.key=true
Partition:0	Offset:0	alice	p1
Partition:0	Offset:1	bob	p2
Partition:0	Offset:2	alice	p5
Partition:0	Offset:3	bob	p6
Partition:0	Offset:4	alice	p7
Partition:1	Offset:0	dave	p4
Partition:2	Offset:0	carol	p3

Every alice record went to partition 0, in the order it was sent; offsets restart at 0 in each partition. Notice too that alice and bob share a partition: hashing spreads keys, but it doesn’t guarantee an even spread with only a few keys. Records without a key are spread by the producer’s sticky partitioner, which fills a batch for one partition before moving to the next.

On disk, each partition is a directory of segment files plus offset and time indexes. Old segments are deleted by retention (time or size) or compacted to keep only the latest value per key. The Kafka storage lesson covers segments, retention and compaction in depth.

Replication: leaders, followers and the ISR

Each partition is stored on replication.factor brokers. One replica is the leader: all writes go to it, and by default all reads too. The others are followers that continuously fetch from the leader to stay up to date.

The in-sync replicas (ISR) are the replicas that are caught up with the leader (a follower that falls behind for longer than replica.lag.time.max.ms, 30 seconds by default, drops out). Only an in-sync replica can be elected leader without losing acknowledged data.

This is the orders topic on the test cluster, created with 3 partitions and replication factor 3:

$ kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic orders
Topic: orders	TopicId: 4B3TSKlDRtCdKi47DG0fwg	PartitionCount: 3	ReplicationFactor: 3	Configs: min.insync.replicas=2
	Topic: orders	Partition: 0	Leader: 2	Replicas: 2,3,1	Isr: 2,3,1	Elr: 	LastKnownElr: 
	Topic: orders	Partition: 1	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2	Elr: 	LastKnownElr: 
	Topic: orders	Partition: 2	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3	Elr: 	LastKnownElr: 

How to read it:

  • Leader: the broker that currently serves partition 0 is broker 2.
  • Replicas: the brokers holding copies. The first one listed is the preferred leader. Kafka spreads leadership so each broker leads about one partition here.
  • Isr: all three replicas are in sync.
  • Elr / LastKnownElr: eligible leader replicas, a newer safety feature (KIP-966). When the ISR shrinks below min.insync.replicas, Kafka remembers which replicas still have all committed data so it can pick a safe leader later. Empty here because nothing has failed.

acks and min.insync.replicas

The producer’s acks setting decides when a write counts as successful:

acksThe leader replies when…Risk
0never; the producer doesn’t waitRecords can vanish silently
1the leader has written itLost if the leader dies before followers copy it
all (-1)every replica in the ISR has itSafe, as long as the ISR is big enough

acks=all on its own isn’t enough: if the ISR shrinks to just the leader, “all in-sync replicas” means one copy. That’s what min.insync.replicas is for. With replication factor 3 and min.insync.replicas=2, an acks=all write needs at least two copies, so the topic keeps accepting writes with one broker down and rejects them (with NotEnoughReplicas) if two are down, rather than pretending they’re durable. Since Kafka 3.0, producers default to acks=all with idempotence on, which also prevents duplicates from retries.

The controller and KRaft

Something has to decide which replica leads each partition, notice when a broker dies, and keep the topic list. That’s the controller.

Older Kafka used ZooKeeper for this. Since Kafka 4.0, ZooKeeper is gone entirely and every cluster runs in KRaft mode: a small group of controller nodes (usually 3, or 5 for very large clusters) runs the Raft consensus protocol over an internal log called __cluster_metadata. One controller is active; the others are hot standbys with a full copy of the metadata. Brokers fetch the metadata log from the controllers and send them heartbeats.

$ kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status
ClusterId:              hzBgkkuDRbyx9OM67laWUA
LeaderId:               2
LeaderEpoch:            1
HighWatermark:          142
MaxFollowerLag:         0
MaxFollowerLagTimeMs:   176
CurrentVoters:          [{"id": 1, "endpoints": ["CONTROLLER://localhost:19091"]}, {"id": 2, "endpoints": ["CONTROLLER://localhost:19092"]}, {"id": 3, "endpoints": ["CONTROLLER://localhost:19093"]}]
CurrentObservers:       []

LeaderId: 2 is the active controller. Raft needs a majority of voters, so 3 controllers tolerate 1 failure and 5 tolerate 2.

Why the switch mattered: metadata now lives in Kafka’s own replicated log, so there’s one system to run and secure, controller failover takes moments instead of reloading all state from ZooKeeper, and clusters can hold far more partitions.

In production, controllers usually run as separate processes (process.roles=controller) from brokers (process.roles=broker). The test cluster used combined mode, where each node is both, which is fine for development:

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:19091,2@localhost:19092,3@localhost:19093
listeners=PLAINTEXT://localhost:9091,CONTROLLER://localhost:19091
advertised.listeners=PLAINTEXT://localhost:9091
controller.listener.names=CONTROLLER
inter.broker.listener.name=PLAINTEXT
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
log.dirs=/tmp/kc/n1
num.partitions=3
default.replication.factor=3
min.insync.replicas=2
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2

Nodes 2 and 3 are the same file with their own node.id, ports and log.dirs. Each node’s storage is formatted once with a shared cluster ID (kafka-storage.sh random-uuid, then kafka-storage.sh format -t <id> -c node1.properties) before the first start.

Producers: the write path

When your code calls send():

  1. The record is serialized and assigned a partition: by key hash if it has a key, by the sticky partitioner if not, or explicitly if you set one.
  2. It’s added to an in-memory batch for that partition. Batches are sent when they’re full (batch.size) or after linger.ms, and can be compressed.
  3. The batch goes to the partition leader, which appends it to its log.
  4. Followers fetch the new records. Once every ISR member has them, the records are committed (below the high watermark) and the leader replies to an acks=all producer.

Consumers can only read committed records, which is why a record acknowledged with acks=all survives a leader failure. The producers lesson covers batching, idempotence and transactions.

Consumers and consumer groups: the read path

Consumers pull: they ask a broker for records from a partition starting at an offset. Kafka doesn’t delete records when they’re read, so many independent applications can read the same topic, each at its own pace.

A consumer group is a set of consumers sharing a group.id. Kafka assigns each partition to exactly one member, so the group processes the topic in parallel while each partition stays in order. If a consumer joins or dies, partitions are reassigned (a rebalance). The group coordinator, a broker, manages membership and stores each group’s committed offsets in the internal __consumer_offsets topic:

$ kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group billing
Consumer group 'billing' has no active members.

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
billing         orders          0          0               0               0               -               -               -
billing         orders          1          0               0               0               -               -               -
billing         orders          2          6               6               0               -               -               -

CURRENT-OFFSET is where the group will resume, LOG-END-OFFSET is the newest record, and LAG is the difference: how far behind the group is. Lag is the single most useful Kafka health metric.

Kafka 4.0 also made the new consumer rebalance protocol (KIP-848) generally available. Clients opt in with group.protocol=consumer, and the broker then computes assignments incrementally, so a rebalance no longer pauses the whole group. The consumer groups lesson and the rebalancing lesson go deeper.

What happens when a broker fails

The best way to understand the architecture is to break it. On the test cluster, broker 2 was both the leader of orders-0 and the active controller. Here’s the topic about 15 seconds after stopping it:

Topic: orders	TopicId: 4B3TSKlDRtCdKi47DG0fwg	PartitionCount: 3	ReplicationFactor: 3	Configs: min.insync.replicas=2
	Topic: orders	Partition: 0	Leader: 3	Replicas: 2,3,1	Isr: 3,1	Elr: 	LastKnownElr: 
	Topic: orders	Partition: 1	Leader: 3	Replicas: 3,1,2	Isr: 3,1	Elr: 	LastKnownElr: 
	Topic: orders	Partition: 2	Leader: 1	Replicas: 1,2,3	Isr: 1,3	Elr: 	LastKnownElr: 
$ kafka-metadata-quorum.sh --bootstrap-server localhost:9091 describe --status | grep Leader
LeaderId:               1
LeaderEpoch:            2

Three things happened automatically:

  1. Controller failover. The two remaining controllers elected node 1 as the new active controller (epoch 1 → 2).
  2. Partition leader election. The new controller noticed broker 2 had stopped heartbeating and made broker 3, the next in-sync replica, leader of orders-0.
  3. ISR shrink. Broker 2 was removed from every ISR.

Writes kept working: an acks=all produce succeeded because two in-sync replicas still met min.insync.replicas=2. Clients found the new leader on their own by refreshing metadata.

After restarting broker 2, it caught up from the leaders and rejoined every ISR within about 20 seconds:

	Topic: orders	Partition: 0	Leader: 3	Replicas: 2,3,1	Isr: 1,2,3	Elr: 	LastKnownElr: 
	Topic: orders	Partition: 1	Leader: 3	Replicas: 3,1,2	Isr: 1,2,3	Elr: 	LastKnownElr: 
	Topic: orders	Partition: 2	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3	Elr: 	LastKnownElr: 

Broker 3 still leads partition 0 for now. Kafka moves leadership back to the preferred replica (broker 2) on its own a few minutes later (auto.leader.rebalance.enable, checked every 5 minutes by default), so load evens out again. On the test cluster, a later --describe showed Partition: 0 Leader: 2 again.

Design decisions that make Kafka fast

  • Sequential disk I/O. Appending to a log and reading it in order is fast even on spinning disks, and it lets the OS page cache do most of the work.
  • Batching and compression on the producer, end to end: brokers store batches as they arrive and send them to consumers without unpacking them.
  • Zero-copy reads. For plaintext connections, brokers can send file data straight from the page cache to the network socket.
  • Pull-based consumers control their own pace, so a slow consumer doesn’t slow down the broker or other consumers.
  • Partitioning spreads load over brokers and consumers with no coordination between partitions.

Sizing rules of thumb

SettingCommon starting pointWhy
Brokers3 or moreReplication factor 3 needs 3 brokers
Controllers3 (5 for very large clusters)Raft majority: survives 1 (or 2) failures
replication.factor3Survives one broker loss with writes still allowed
min.insync.replicas2With acks=all, never acknowledge a single copy
Partitions per topicEnough for peak consumer parallelism, with headroomYou can add partitions later, but key-to-partition mapping changes when you do

Frequently asked questions

What are the main components of Kafka architecture?

Brokers (store and serve data), topics and partitions (how data is organized), replicas with leaders and followers (durability), the KRaft controller quorum (metadata and leader election), producers (write) and consumers in consumer groups (read).

Does Kafka still need ZooKeeper?

No. Kafka 4.0 removed ZooKeeper support; clusters use the built-in KRaft controller quorum. Clusters on older versions must migrate to KRaft on a 3.x release before upgrading to 4.x.

Does Kafka guarantee message order?

Within a partition, yes. Across partitions, no. Give related events the same key (for example a user or order ID) so they go to the same partition.

How many partitions should a topic have?

At least as many as the number of consumers you want working in parallel at peak, plus headroom. More partitions also mean more files, more leader elections on failure and more memory, so don’t over-partition small topics. Kafka partitions explained works through the sizing maths and what adding partitions does to keyed data.

What’s the difference between a broker and a controller?

Brokers hold partition data and serve clients. Controllers hold cluster metadata and make decisions like leader elections. A node can be either, or both in combined mode.

Keep learning

AlgoSimplified logo

AlgoSimplified

Practical engineering courses and technical notes

Share: X LinkedIn

Related articles