10 min read

Kafka Partitions Explained: Keys, Ordering, How Many

How Kafka partitions work: how keys pick a partition, why adding partitions breaks ordering, and how many partitions a topic needs. Tested on Kafka 4.4.

kafka kafka-partitions consumer-groups distributed-systems system-design
On this page · 9 sections
  1. How Kafka picks a partition
  2. The gotcha: not every client uses the same hash
  3. What ordering you actually get
  4. Partitions and consumers
  5. Adding partitions moves your keys
  6. How many partitions does a topic need?
  7. Common mistakes
  8. Frequently asked questions
  9. Keep learning

A Kafka partition is one ordered, append-only log. A topic is just a name for a set of partitions, and almost every practical Kafka question (Is this ordered? Why is one consumer idle? Why did my events arrive out of order after a config change? How many partitions should I create?) is really a question about partitions.

If you need the big picture first (brokers, replication, the controller), read Kafka architecture explained. This post goes deep on partitions only. Everything below was run on Kafka 4.4 with Python confluent-kafka 2.16, and the scripts are in the course code folder.

How Kafka picks a partition

The producer, not the broker, decides which partition a record goes to:

  • Record has a key: the Java client hashes the key bytes with murmur2, clears the sign bit, and takes the result modulo the partition count. Same key and same partition count means the same partition, every time.
  • No key: since Kafka 3.3 the Java client uses the built-in “uniform sticky” behaviour. It fills a batch for one partition, then moves to another, which gives bigger batches than the old one-record-per-partition round robin.
  • You set the partition explicitly in the ProducerRecord: the producer uses it as is.

Because the key rule is plain arithmetic, you can predict it. Here is the Java client’s hash in Python (kafka_partition.py):

def murmur2(data: bytes) -> int:
    m, r = 0x5BD1E995, 24
    length = len(data)
    h = (0x9747B28C ^ length) & 0xFFFFFFFF
    for i in range(0, length - length % 4, 4):
        k = int.from_bytes(data[i:i + 4], "little")
        k = (k * m) & 0xFFFFFFFF
        k ^= k >> r
        k = (k * m) & 0xFFFFFFFF
        h = (h * m) & 0xFFFFFFFF
        h ^= k
    tail, base = length % 4, length - length % 4
    if tail == 3:
        h ^= data[base + 2] << 16
    if tail >= 2:
        h ^= data[base + 1] << 8
    if tail >= 1:
        h ^= data[base]
        h = (h * m) & 0xFFFFFFFF
    h ^= h >> 13
    h = (h * m) & 0xFFFFFFFF
    h ^= h >> 15
    return h

def partition_for(key: str, num_partitions: int) -> int:
    return (murmur2(key.encode()) & 0x7FFFFFFF) % num_partitions

For a 6-partition topic it predicts user-1 → 2, user-2 → 2, user-3 → 5, user-42 → 4, order-1001 → 0. Sending those keys with the Java console producer and reading them back:

$ printf "user-1:a\nuser-2:a\nuser-3:a\nuser-42:a\norder-1001:a\n" | \
    kafka-console-producer.sh --bootstrap-server localhost:9092 --topic pk \
      --reader-property parse.key=true --reader-property key.separator=:
$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic pk --from-beginning \
    --formatter-property print.key=true --formatter-property print.partition=true
Partition:5	user-3	a
Partition:4	user-42	a
Partition:2	user-1	a
Partition:2	user-2	a
Partition:0	order-1001	a

All five match. Note that user-1 and user-2 share partition 2 while partitions 1 and 3 got nothing: with a handful of keys, hashing is lumpy. It evens out with thousands of keys, not five.

The gotcha: not every client uses the same hash

The Java client uses murmur2. librdkafka, which sits under the Python, confluent-kafka-go, .NET, Confluent JavaScript and C/C++ clients, defaults to consistent_random, which hashes keys with CRC32. Same keys, same 6-partition topic, sent from Python (partitioner_compare.py):

key         java librdkafka default murmur2_random
user-1         2                  2              2
user-2         2                  4              2
user-3         5                  0              5
user-42        4                  1              4
order-1001     0                  3              0

Four of the five keys land somewhere else. (Go’s Sarama client is different again: its default hash partitioner uses FNV-1a.) If a Java service and a Python service both write user-3 events to the same topic, those events end up in two partitions, read by two consumers, in no guaranteed order. Nothing errors; you just get occasional out-of-order processing that is miserable to debug.

The fix is one line in every non-Java producer that shares a topic with Java producers:

Producer({"bootstrap.servers": "kafka:9092", "partitioner": "murmur2_random"})

murmur2_random hashes keyed records exactly like the Java client (the third column above), and spreads keyless records randomly. Kafka Streams and ksqlDB also use murmur2, so they agree with Java producers.

What ordering you actually get

Kafka guarantees order within a partition only. A consumer reads partition 3 in offset order; there is no ordering between partition 3 and partition 4. So the real design question is: “which events must be processed in order relative to each other?” Make that thing the key. For a payments service it’s usually the account ID, for an order pipeline the order ID.

Two details keep per-partition order intact on the producer side:

  • Retries. If batch 1 fails and is retried after batch 2 succeeded, order flips. The idempotent producer (on by default since Kafka 3.0 with acks=all) prevents this as long as max.in.flight.requests.per.connection is 5 or less. Don’t turn idempotence off to “go faster”.
  • One logical writer per key. Two services writing the same key race each other. Kafka will store whatever arrives first.

Partitions and consumers

Inside one consumer group, each partition is read by exactly one consumer. One consumer can read many partitions. So the partition count is the ceiling on parallelism for a group. Here are four consumers in one group on a 3-partition topic, described with the CLI:

$ kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group idle-demo --members
GROUP           CONSUMER-ID                                  HOST             CLIENT-ID       #PARTITIONS
idle-demo       rdkafka-2e949297-e951-477c-badc-6390e4c7ffc4 /0:0:0:0:0:0:0:1 rdkafka         1
idle-demo       rdkafka-39fad3a9-71cd-413d-92c6-7b9a1ffc0e33 /0:0:0:0:0:0:0:1 rdkafka         1
idle-demo       rdkafka-ad927ec4-83c1-46d2-bf9c-1ce33913623a /127.0.0.1       rdkafka         0
idle-demo       rdkafka-9a73bc07-cd57-47c3-b71c-7de0882d57a1 /127.0.0.1       rdkafka         1

The fourth consumer has zero partitions. It’s not broken, and it’s not useless either: it’s a hot standby that picks up work in the next rebalance if another member dies. But if you scaled to four pods expecting more throughput, you got none. A different consumer group reading the same topic gets its own full copy of all partitions; groups don’t compete with each other.

When consumers come and go, partitions move between them in a rebalance. How long that pauses processing depends a lot on the assignor; the rebalancing lesson measures eager, cooperative and the new KIP-848 protocol side by side.

Adding partitions moves your keys

You can increase a topic’s partition count at any time (kafka-topics.sh --alter --partitions N). You can never decrease it. And increasing it changes the modulo, so existing keys start going to different partitions. How many? For 10,000 keys (key_moves.py):

6 ->  7 partitions: 86.4% of keys move
6 ->  8 partitions: 74.6% of keys move
6 ->  9 partitions: 67.1% of keys move
6 -> 12 partitions: 50.4% of keys move
6 -> 18 partitions: 67.1% of keys move

Doubling is the gentlest change and still moves half your keys. Here’s one of them moving on the real topic. Before the change user-3 lived on partition 5; after --alter --partitions 8, a new user-3 record goes to partition 3, exactly as the predictor says:

Partition:5	user-3	a
Partition:5	user-3	x
Partition:3	user-3	after

The old user-3 records are still on partition 5, possibly unconsumed if that partition’s consumer is lagging. Partition 3’s consumer can process after before partition 5’s consumer gets to a and x. For a balance update or a state machine (created, paid, shipped), that’s a real bug.

What to do about it:

  • Ordering doesn’t matter for this topic (metrics, click logs, keyless data): just add partitions. Consumers pick them up after the rebalance.
  • Ordering matters, and you can pause producers briefly: stop producers, let consumers drain lag to zero, add partitions, restart producers. No key has records in two places that are both unconsumed.
  • Ordering matters and you can’t pause: create a new topic with the target count, switch producers to it, and let consumers finish the old topic before reading the new one. More work, but nothing reorders.

The cheap option is to choose a partition count you won’t outgrow soon, which is the next section.

How many partitions does a topic need?

Work it out from two numbers you can measure: how fast one partition can be written, and how fast one consumer can process.

Consumer side (usually the limit). If processing a record takes 20 ms (a database write plus an HTTP call), one consumer handles about 50 records/s. To handle 2,000 records/s at peak you need 40 consumers working in parallel, so at least 40 partitions. No broker tuning changes that.

Producer/broker side. Measure one partition on your own hardware:

$ kafka-producer-perf-test.sh --topic perf1 --num-records 500000 --record-size 1024 \
    --throughput -1 --producer-props bootstrap.servers=localhost:9092 acks=all
500000 records sent, 121565.767080 records/sec (118.72 MB/sec), 222.91 ms avg latency, ...

That’s one partition on a single-broker, 8-core test box with replication factor 1 and no network hop, so treat it as an upper bound. A real cluster with replication factor 3 across machines will be lower. Run the same test against your cluster; the point is that a single partition is rarely the write bottleneck for typical event sizes. Slow consumers are.

Then:

partitions >= max(peak_writes / per_partition_write_rate,
                  peak_records / per_consumer_rate)

and add headroom for growth, because adding partitions later moves keys. Some practical choices:

  • Pick a number with many divisors (6, 12, 24, 48). With 12 partitions, 1, 2, 3, 4, 6 or 12 consumers each get an equal share. With 10 partitions and 4 consumers, two consumers do 50% more work than the others.
  • Don’t go to thousands “just in case”. Every partition is a set of files on each replica, a leader to elect when a broker dies, a buffer in every producer (batch.size is per partition) and one more thing to move in each rebalance. Small topics with 200 partitions waste all of that and make batches tiny.
  • Watch for hot partitions. The count doesn’t help if one key dominates. If a single tenant sends 30% of traffic, its partition gets at least 30% no matter how many partitions exist. The options are splitting that key (tenant-7#0 to tenant-7#3, giving up ordering across the split) or giving that tenant its own topic.

Common mistakes

  • Mixed-language producers with default partitioners. Java uses murmur2, librdkafka defaults to CRC32. Set partitioner=murmur2_random on librdkafka clients that share topics with Java.
  • Expecting topic-wide order. Only partitions are ordered. If two events must be ordered, they need the same key.
  • Scaling consumers past the partition count. The extra consumers sit at zero partitions. Check --describe --members before blaming Kafka for slow scaling.
  • Adding partitions to a keyed topic during an incident. It triggers a rebalance and moves most keys while lag is already high, which is exactly when reordering hurts.
  • Using a low-cardinality key. Keying by country or event type puts most traffic on a few partitions. Use the entity whose order matters (user, account, order).
  • Null keys for data that needs order. Keyless records are spread across partitions, so related events scatter.

Frequently asked questions

What is a partition in Kafka?

An ordered, append-only log that stores part of a topic’s records. Each record in it has a sequential offset. Partitions are the unit of ordering, parallelism and replication.

How does Kafka decide which partition a message goes to?

The producer decides. With a key, the Java client uses murmur2(key) modulo the partition count; without a key it uses sticky batching across partitions; or the application can set the partition explicitly. librdkafka-based clients default to a CRC32 hash unless you set partitioner=murmur2_random.

Can you decrease the number of partitions in Kafka?

No. You can only increase it. To go lower, create a new topic with fewer partitions and move producers and consumers to it.

Does adding partitions break ordering?

For keyed topics, yes, temporarily. Most keys map to a new partition after the change, and their older records may still be unconsumed in the old partition. Drain lag before adding partitions, or migrate to a new topic.

How many partitions should a Kafka topic have?

Enough for the consumer parallelism you need at peak (peak records per second divided by what one consumer can process), with headroom, preferably a number with many divisors such as 12 or 24. Measure per-partition throughput on your own cluster rather than copying a number.

What happens if there are more consumers than partitions?

The extra consumers in that group get no partitions and stay idle until a rebalance gives them one, for example when another consumer fails.

Keep learning

AlgoSimplified logo

AlgoSimplified

Practical engineering courses and technical notes

Share: X LinkedIn

Related articles