Heroic

A scalable time series database based on Bigtable, Cassandra, and Elasticsearch.
Go to https://spotify.github.io/heroic/ for documentation.

This project adheres to the Open Code of Conduct.
By participating, you are expected to honor this code.

Install

Docker

Docker images are available on Docker Hub.

$ docker run -p 8080:8080 -p 9091:9091 spotify/heroic

Heroic will now be reachable at http://localhost:8080/status.

In production it's advised to use a tagged version.

Configuration

For help on how to write a configuration file, see the Configuration Section of the official documentation.

Heroic has been tested with the following services:

• Cassandra (2.1.x, 3.5) when using metric/datastax.
• Cloud Bigtable when using
  metric/bigtable.
• Elasticsearch (7.x) when using
  metadata/elasticsearch or
  suggest/elasticsearch.
• Kafka (0.8.x) when using consumer/kafka.

Developing

Building from source

In order to compile Heroic, you'll need:

• A Java 11 JDK
• Maven 3
• Gradle

The project is built using Gradle:

# full build, runs all tests and builds the shaded jar
./gradlew build

# only compile
./gradlew assemble

# build a single module
./gradlew heroic-metric-bigtable:build

The heroic-dist module can be used to produce a shaded jar that contains all required dependencies:

./gradlew heroic-dist:shadowJar

After building, the entry point of the service is
com.spotify.heroic.HeroicService.
The following is an example of how this can be run:

./gradlew heroic-dist:runShadow <config>

which is the equivalent of doing:

java -jar $PWD/heroic-dist/build/libs/heroic-dist-0.0.1-SNAPSHOT-shaded.jar <config>

Building with Docker

$ docker build -t heroic:latest .

This is a multi-stage build and will first build Heroic via a ./gradlew clean build and then copy the resulting shaded jar into the runtime container.

Running heroic via docker can be done:

$ docker run -d -p 8080:8080 -p 9091:9091 -v /path/to/config.yml:/heroic.yml spotify/heroic:latest

Logging

Logging is captured using SLF4J, and forwarded to
Log4j.

To configure logging, define the -Dlog4j.configurationFile=<path>
parameter. You can use docs/log4j2-file.xml as a base.

Testing

We run tests with Gradle:

# Start pubsub emulator
gcloud beta emulators pubsub start

# run unit tests
./gradlew test

# run integration tests
./gradlew integrationTest

or to run a more comprehensive set of checks:

./gradlew check

This will run:

• unit tests
• integration tests
• Checkstyle
• Coverage Reporting with Jacoco

It is strongly recommended that you run the full test suite before setting up a
pull request, otherwise it will be rejected by Travis.

Remote Integration Tests

Integration tests are configured to run remotely depending on a set of system
properties.

Elasticsearch, Property, Description, ----------, -------------, -D elasticsearch.version=<version>, Use the given client version when building the project, -D it.elasticsearch.remote=true, Run Elasticsearch tests against a remote database, -D it.elasticsearch.seed=<seed>, Use the given seed (default: localhost), -D it.elasticsearch.clusterName=<clusterName>, Use the given cluster name (default: elasticsearch), ##### Datastax, Property, Description, ----------, -------------, -D datastax.version=<version>, Use the given client version when building the project, -D it.datastax.remote=true, Run Datastax tests against a remote database, -D it.datastax.seed=<seed>, Use the given seed (default: localhost), ##### Bigtable, Property, Description, ----------, -------------, -D bigtable.version=<version>, Use the given client version when building the project, -D it.bigtable.remote=true, Run Bigtable tests against a remote database, -D it.bigtable.project=<project>, Use the given project, -D it.bigtable.zone=<zone>, Use the given zone, -D it.bigtable.instance=<instance>, Use the given instance, -D it.bigtable.credentials=<credentials>, Use the given credentials file, The following is an example Elasticsearch remote integration test:

$> mvn -P integration-tests \
    -D elasticsearch.version=5.6.0 \
    -D it.elasticsearch.remote=true \
    clean verify

PubSub

PubSub relies on having the PUBSUB_EMULATOR_HOST environment variable set instead of a system property. Detailed instructions are available in the Google PubSub emulator docs.

Full Cluster Tests

Full cluster tests are defined in heroic-dist/src/test/java.

This way, they have access to all the modules and parts of Heroic.

The JVM RPC module is specifically designed to allow for rapid
execution of integration tests. It allows multiple cores to be defined and
communicate with each other in the same JVM instance.

• See AbstractClusterQueryIT
• JVM-based JvmClusterQueryIT
• gRPC-based GrpcClusterQueryIT

Code Coverage

There's an ongoing project to improve test coverage.
Clicking the above graph will bring you to codecov.io, where you can find areas to focus on.

Bypassing Validation

To bypass automatic formatting and checkstyle validation you can use the
following stanza:

// @formatter:off
final List<String> list = ImmutableList.of(
   "Welcome to...",
   "... The Wild West"
);
// @formatter:on

To bypass a FindBugs error, you should use the @SupressFBWarnings annotation.

@SupressFBWarnings(value="FINDBUGS_ERROR_CODE", justification="I Know Better Than FindBugs")
public class IKnowBetterThanFindbugs() {
    // ...
}

Module Orientation

The Heroic project is split into a couple of modules.

The most critical one is heroic-component. It contains
interfaces, value objects, and the basic set of dependencies necessary to glue
different components together.

Submodules include metric, suggest,
metadata, and aggregation. The first three
contain various implementations of the given backend type, while the latter
provides aggregation methods.

heroic-core contains the
com.spotify.heroic.HeroicCore
class which is the central building block for setting up a Heroic instance.

heroic-elasticsearch-utils is a collection of
utilities for interacting with Elasticsearch. This is separate since we have
more than one backend that needs to talk with elasticsearch.

Finally there is heroic-dist, a small project that depends on all module. Here is where everything is bound together into a distribution
— a shaded jar. It also provides the entry-point for services, namely
com.spotify.heroic.HeroicService
or through an interactive shell com.spotify.heroic.HeroicShell.
The shell can either be run standalone or connected to an existing Heroic instance for administration.

Contributing

Guidelines for contributing can be found here.