storagetapper

StorageTapper is a scalable realtime MySQL change data streaming and transformation service

Github星跟蹤圖

StorageTapper

StorageTapper is a scalable realtime MySQL change data streaming and
transformation service.

Service reads data from MySQL, transforms it into an Avro schema
serialized format, and publishes these events to Kafka. Consumers can then
consume these events directly from Kafka.

Build Status
Go Report Card
codecov

Features

  • Producing events to Kafka
  • Automatic work distribution between instances
  • Avro output format
  • JSON output format
  • HTTP endpoints to control
    • Tables to be ingested
    • Output schema
    • Database address resolver
  • Snapshot is taken from slave to reduce load on master
  • Binlogs streaming from master for better SLA
  • Throttling when taking snapshot

Limitations

  • Tables must have a primary key

Build & Install

Get build dependencies

go get github.com/Masterminds/glide
go get github.com/alecthomas/gometalinter

It's recommended to have local MySQL and Kafka installed, so as many tests
depend on them. Tweak development config to
correspond your setup.
WARNING: Tests run dangerous commands like RESET MASTER and DROP
DATABASE, so for tests don't use any MySQL instances with precious data.

Debian & Ubuntu

cd storagetapper
make deb && dpkg -i ../storagetapper_1.0_amd64.deb

Others

cd storagetapper
make test && make install

Configuration

Storagetapper loads configuration from the following files and location in the
given order:

    /etc/storagetapper/base.yaml
    /etc/storagetapper/production.yaml
    $(HOME)/base.yaml
    $(HOME)/production.yaml
    $(STORAGETAPPER_CONFIG_DIR)/base.yaml
    $(STORAGETAPPER_CONFIG_DIR)/production.yaml

Available options described in Options section

License

This software is licensed under the MIT License.

主要指標

概覽
名稱與所有者uber/storagetapper
主編程語言Go
編程語言Shell (語言數: 3)
平台
許可證MIT License
所有者活动
創建於2017-05-07 00:31:41
推送於2023-03-19 22:39:47
最后一次提交2022-01-11 22:23:26
發布數2
最新版本名稱v1.1.0-beta (發布於 )
第一版名稱v1.0.0-beta (發布於 )
用户参与
星數355
關注者數22
派生數66
提交數250
已啟用問題?
問題數28
打開的問題數18
拉請求數29
打開的拉請求數6
關閉的拉請求數2
项目设置
已啟用Wiki?
已存檔?
是復刻?
已鎖定?
是鏡像?
是私有?