Metrictank
Introduction
Metrictank is a multi-tenant timeseries engine for Graphite and friends.
It provides long term storage, high availability, efficient storage, retrieval and processing for large scale environments.
GrafanaLabs has been running metrictank in production since December 2015.
It currently requires an external datastore like Cassandra or Bigtable, and we highly recommend using Kafka to support clustering, as well
as a clustering manager like Kubernetes. This makes it non-trivial to operate, though GrafanaLabs has an on-premise product
that makes this process much easier.
Features
- 100% open source
- Inspired by the Facebook gorilla paper.
Most notably, the heavily compressed chunks dramatically lower cpu, memory and storage requirements. - Writeback RAM cache, serving most data out of memory.
- Graphite is a first class citizen. As of graphite-1.0.1, metrictank can be used as a graphite CLUSTER_SERVER.
- Can also act as a Graphite server itself, though the functions processing library is only partially implemented, metrictank proxies requests to Graphite if it can't handle the required processing (for those requests it will degrade to just being the backend storage)
- Accurate, flexible rollups by storing min/max/sum/count (which also gives us average).
So we can do consolidation (combined runtime+archived) accurately and correctly,
unlike most other graphite backends like whisper - Flexible tenancy: can be used as single tenant or multi tenant. Selected data can be shared across all tenants.
- Input options: carbon, metrics2.0, kafka (soon: json or msgpack over http)
- Guards against excessive data requests
- Data backfill/import from whisper
Limitations
- No performance/availability isolation between tenants per instance. (only data isolation)
- Minimum computation locality: we move the data from storage to processing code, which is both metrictank and graphite.
- Backlog replaying and queries can be made faster. A Go GC issue may occasionally inflate response times.
- We use metrics2.0 in native input protocol and indexes, but barely do anything with it yet.
- can't overwrite old data. We support reordering the most recent time window but that's it. (unless you restart MT)
Interesting design characteristics (feature or limitation... up to you)
- Upgrades / process restarts requires running multiple instances (potentially only for the duration of the maintenance) and possibly re-assigning the primary role.
Otherwise data loss of current chunks will be incurred. See operations guide - clustering works best with an orchestrator like kubernetes. MT itself does not automate master promotions. See clustering for more.
- Only float64 values. Ints and bools currently stored as floats (works quite well due to the gorilla compression),
No text support. - Only uint32 unix timestamps in second resolution. For higher resolution, consider streaming directly to grafana
- No data locality: doesn't seem needed yet to put related series together.
Docs
installation, configuration and operation.
- Overview
- Quick start using docker
(more docker information) - Installation guides
- Configuration
- Memory server
- Compression tips
- Cassandra
- Kafka
- Inputs
- Metrics
- Operations
- Startup
- Tools
features in-depth
Other
Releases and versioning
-
we aim to keep
masterstable and vet code before merging to master -
We're pre-1.0 but adopt semver for our
0.MAJOR.MINORformat. The rules are simple:- MAJOR version for incompatible API or functionality changes
- MINOR version when you add functionality in a backwards-compatible manner, and
We don't do patch level releases since minor releases are frequent enough.
License
Copyright 2016-2019 Grafana Labs
This software is distributed under the terms of the GNU Affero General Public License.
Some specific packages have a different license:
- schema : apache2
- expr: 2-clause BSD
- mdata/chunk/tsz: 2-clause BSD