OpenCensus Specs

This is a high level design of the OpenCensus library, some of the API examples may be written
(linked) in C++, Java or Go but every language should translate/modify them based on language
specific patterns/idioms. Our goal is that all libraries have a consistent "look and feel".

This repository uses terminology (MUST, SHOULD, etc) from RFC 2119.

Overview

Today, distributed tracing systems and stats collection tend to use unique protocols and
specifications for propagating context and sending diagnostic data to backend processing systems.
This is true amongst all the large vendors, and we aim to provide a reliable implementation in
service of frameworks and agents. We do that by standardizing APIs and data models.

OpenCensus provides a tested set of application performance management (APM) libraries, such as
Metrics and Tracing, under a friendly OSS license. We acknowledge the polyglot nature of modern
applications, and provide implementations in all main programming languages including C/C++,
Java, Go, Ruby, PHP, Python, C#, Node.js, Objective-C and Erlang.

Ecosystem Design

Layers

Service exporters

Each backend service SHOULD implement this API to export data to their services.

OpenCensus library

This is what the following section of this document is defining and explaining.

Manually instrumented frameworks

We are going to instrument some of the most popular frameworks for each language using the
OpenCensus library to allow users to get traces/stats when they use these frameworks.

Tools for automatic instrumentation

Some of the languages may support libraries for automatic instrumentation. For example, Java
applications can use byte-code manipulation (monkey patching) to provide an agent that
automatically instruments an application. Note: not all the languages support this.

Application

This is customer's application/binary.

Library Design

Namespace and Package

• For details about the library package names structure see Namespace and Package.

Components

This section focuses on the important components that each OpenCensus library must have to
support all required functionalities.

Here is a layering structure of the proposed OpenCensus library:

Context

Some of the features for distributed tracing and tagging need a way to propagate a specific
context (trace, tags) in-process (possibly between threads) and between function calls.

The key elements of the context support are:

• Every implementation MUST offer an explicit or implicit generic Context propagation mechanism
  that allows different sub-contexts to be propagated.
• Languages that already have this support, like Go (context.Context) or C#
  (Activity), MUST use the language supported generic context instead of building their
  own.
• For an explicit generic context implementation you can look at the Java
  io.grpc.Context.

Trace

Trace component is designed to support distributed tracing (see the Dapper paper).
OpenCensus allows functionality beyond data collection and export. For example, it allows
tracking of the active spans and keeping local samples for interesting requests.

The key elements of the API can be broken down as:

• A Span represents a single operation within a trace. Spans can be nested to form a trace tree.
• Libraries must allow users to record tracing events for a span (attributes, annotations, links,
  etc.).
• Spans are carried in the Context. Libraries MUST provide a way of getting, manipulating,
  and replacing the Span in the current context.
• Libraries SHOULD provide a means of dynamically controlling the trace global configuration at runtime
  (e.g. trace sampling rate/probability).
• Libraries SHOULD keep track of active spans and in memory samples based on latency/errors and
  offer ways to access the data.
• Because context must also be propagated across processes, library MUST offer the functionality
  that allows any transport (e.g RPC, HTTP, etc.) systems to encode/decode the “trace context” for
  placement on the wire.

Links

• Details about Trace package can be found here.
• Data model is defined at the Trace Data Model.

Tags

Tags are values propagated through the Context subsystem inside a process and among processes by
any transport (e.g RPC, HTTP, etc.). For example tags are used by the Stats component to break
down measurements by arbitrary metadata set in the current process or propagated from a remote
caller.

The key elements of the Tags component are:

• A tag: this is a key-value pair, where the keys and values are strings. The API allows for
  creating, modifying and querying objects representing a tag value.
• A set of tags (with unique keys) carried in the Context. Libraries MUST provide a means
  of manipulating the tags in the context, including adding new tags, replacing tag values, deleting
  tags, and querying the current value for a given tag key.
• Because tags must also be propagated across processes, library MUST offer the functionality that
  allows RPC systems to encode/decode the set of tags for placement on the wire.

Links

• Details about Tags package can be found here.

Stats

The Stats component is designed to record measurements, dynamically break them down by
application-defined tags, and aggregate those measurements in user-defined ways. It is designed
to offer multiple types of aggregation (e.g. distributions) and be efficient (all measurement
processing is done as a background activity); aggregating data enables reducing the overhead of
uploading data, while also allowing applications direct access to stats.

The key elements the API MUST provide are:

• Defining what is to be measured (the types of data collected, and their meaning), and how data
  will be aggregated (e.g. into a distribution, cumulative aggregation vs. deltas, etc.). Libraries
  must offer ways for customer to define Metrics that make sense for their application, and support
  a canonical set for RPC/HTTP systems.
• Recording data - APIs for recording measured values. The recorded data is then broken down by tags
  carried in the context (e.g. a tag can have a value that describes the current RPC service/method
  name; when RPC latency is recorded, this can be made in a generic call, without having to specify
  the exact method), and aggregated as needed (e.g. a histogram of all latency values).
• Accessing the aggregated data. This can be filtered by data type, resource name, etc. This
  allows applications to easily get access to their own data in-process.

Links

• Details about Stats package can be found here.
• Data model is defined at the Stats Data Model.

Supported propagation formats

• Library MUST support the W3C Distributed Tracing format for Trace and Tags components.
• Binary encoding is defined at the BinaryEncoding document.