Jocko
Kafka/distributed commit log service in Go.
Goals of this project:
- Implement Kafka in Go
- Protocol compatible with Kafka so Kafka clients and services work with Jocko
- Make operating simpler
- Distribute a single binary
- Use Serf for discovery, Raft for consensus (and remove the need to run ZooKeeper)
- Smarter configuration settings
- Able to use percentages of disk space for retention policies rather than only bytes and time kept
- Handling size configs when you change the number of partitions or add topics
- Learn a lot and have fun
TODO
- Producing
- Fetching
- Partition consensus and distribution
- Protocol
- Produce
- Fetch
- Metadata
- Create Topics
- Delete Topics
- Consumer group [current task]
- Discovery
- API versioning [more API versions to implement]
- Replication [first draft done - testing heavily now]
Hiatus Writing Book
I’m writing a book for PragProg called Building Distributed Services with Go. You can sign up on this mailing list and get updated when the book’s available. It walks you through building a distributed commit log from scratch. I hope it will help Jocko contributors and people who want to work on distributed services.
Reading
- How Jocko's built-in service discovery and consensus works
- How Jocko's (and Kafka's) storage internals work
Project Layout
├── broker broker subsystem
├── cmd commands
│ └── jocko command to run a Jocko broker and manage topics
├── commitlog low-level commit log implementation
├── examples examples running/using Jocko
│ ├── cluster example booting up a 3-broker Jocko cluster
│ └── sarama example producing/consuming with Sarama
├── protocol golang implementation of Kafka's protocol
├── prometheus wrapper around Prometheus' client lib to handle metrics
├── server API subsystem
└── testutil test utils
└── mock mocks of the various subsystems
Building
Local
-
Clone Jocko
$ go get github.com/travisjeffery/jocko -
Build Jocko
$ cd $GOPATH/src/github.com/travisjeffery/jocko $ make build(If you see an error about
depnot being found, ensure that
$GOPATH/binis in yourPATH)
Docker
docker build -t travisjeffery/jocko:latest .
Contributing
See CONTRIBUTING for details on submitting patches and the contribution workflow.
License
Jocko is under the MIT license, see the LICENSE file for details.
-
GitHub @travisjeffery
-
Twitter @travisjeffery
-
Medium @travisjeffery