Read Offset Checkpoint Stored in Kafka

In Kafka 0.8.2+, offset checkpoint is stored in Kafka itself, reducing the dependency on Zookeeper.

Here's how to read the offset checkpoint from the command line:

echo "exclude.internal.topics=false" > myconfig
kafka-console-consumer --topic __consumer_offsets --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" --consumer.config myconfig --zookeeper <zookeeper> [--from-begin]

We can use kafka-console-consumer and read from the topic __consumer_offsets. There are 2 caveats.

First, we need to create a consumer configuration file with the property exclude.internal.topics set to false.

Next, we need to use the proper formatter, OffsetsMessageFormatter, conveniently :) located inside kafka.coordinator.GroupMetadataManager. Here's the code for Kafka 0.9.0 — it's also clearly commented!

Many thanks to Joel Koshy for his presentation (slide 32) and to Rajasekar Elango for the hint.