= NGINX-to-Prometheus log file exporter
:tip-caption: :bulb:
:note-caption: :information_source:
:important-caption: :heavy_exclamation_mark:
:caution-caption: :fire:
:warning-caption: :warning:
:toc:
:toc-placement!:
:toc-title:

image:https://travis-ci.org/martin-helmich/prometheus-nginxlog-exporter.svg?branch=master[link="https://travis-ci.org/martin-helmich/prometheus-nginxlog-exporter",Build status]
image:https://quay.io/repository/martinhelmich/prometheus-nginxlog-exporter/status[link="https://quay.io/repository/martinhelmich/prometheus-nginxlog-exporter",Docker Repository on Quay]
image:https://goreportcard.com/badge/github.com/martin-helmich/prometheus-nginxlog-exporter[link="https://goreportcard.com/report/github.com/martin-helmich/prometheus-nginxlog-exporter", Go Report Card]

Helper tool that continuously reads an NGINX log file and exports metrics to https://prometheus.io/[Prometheus].

[discrete]
== Contents

toc::[]

== Usage

You can either use a simple configuration, using command-line flags, or create
a configuration file with a more advanced configuration.

Use the command-line:

./prometheus-nginxlog-exporter \
  -format="<FORMAT>" \
  -listen-port=4040 \
  -namespace=nginx \
  [PATHS-TO-LOGFILES...]

Use the configuration file:

./prometheus-nginxlog-exporter -config-file /path/to/config.hcl

== Collected metrics

This exporter collects the following metrics. This collector can listen on
multiple log files at once and publish metrics in different namespaces. Each
metric uses the labels method (containing the HTTP request method) and
status (containing the HTTP status code).

[IMPORTANT]

Keep in mind that some of these metrics will require certain values to be present
in your access log format (for example, the http_upstream_time_seconds metric
will require your access to contain the variable $upstream_response_time.

Metrics are exported at the /metrics path.

These metrics are exported:, ===, <namespace>_http_response_count_total, The total amount of processed HTTP requests/responses., <namespace>_http_response_size_bytes, The total amount of transferred content in bytes., <namespace>_http_upstream_time_seconds, A summary vector of the upstream response times in seconds. Logging these needs to be specifically enabled in NGINX using the $upstream_response_time variable in the log format., <namespace>_http_upstream_time_seconds_hist, Same as <namespace>_http_upstream_time_seconds, but as a histogram vector. Also requires the $upstream_response_time variable in the log format., <namespace>_http_response_time_seconds, A summary vector of the total response times in seconds. Logging these needs to be specifically enabled in NGINX using the $request_time variable in the log format., <namespace>_http_response_time_seconds_hist, Same as <namespace>_http_response_time_seconds, but as a histogram vector. Also requires the $request_time variable in the log format., ===

Additional labels can be configured in the configuration file (see below).

== Configuration file

You can specify a configuration file to read at startup. The configuration file
is expected to be either in https://github.com/hashicorp/hcl[HCL] or YAML format. Here's an example file:

[source,hcl]

listen {
port = 4040
address = "10.1.2.3"
}

consul {
enable = true
address = "localhost:8500"
datacenter = "dc1"
scheme = "http"
token = ""
service {
id = "nginx-exporter"
name = "nginx-exporter"
tags = ["foo", "bar"]
}
}

namespace "app1" {
format = "$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for""
source {
files = [
"/var/log/nginx/app1/access.log"
]
}
labels {
app = "application-one"
environment = "production"
foo = "bar"
}

histogram_buckets = [.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]
}

namespace "app2" {
format = "$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" $upstream_response_time"
source {
files = [
"/var/log/nginx/app2/access.log"
]
}
}

The same file as YAML file:

[source,yaml]

listen:
port: 4040
address: "10.1.2.3"

consul:
enable: true
address: "localhost:8500"
datacenter: dc1
scheme: http
token: ""
service:
id: "nginx-exporter"
name: "nginx-exporter"
tags: ["foo", "bar"]

namespaces:

• name: app1
  format: "$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for""
  source:
  files:
  - /var/log/nginx/app1/access.log
  labels:
  app: "application-one"
  environment: "production"
  foo: "bar"
  histogram_buckets: [.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]
• name: app2
  format: "$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" $upstream_response_time"
  source:
  files:
  - /var/log/nginx/app2/access.log

Advanced features

Log sources

Currently, the exporter supports reading log data from

1. files
2. syslog

All log sources can be configured on a per-namespace basis using the source property.

Reading from files

When reading from log files, all that is needed is a files property:

namespace "test" {
  source {
    files = ["/var/log/nginx/access.log"]
    // ...
  }
}

Reading from syslog

The exporter can also open and listen on a Syslog port and read logs from there. Configuration works as follows:

[source,hcl]

namespace "test" {
source {
syslog {
listen_address = "udp://127.0.0.1:8514" <1>
format = "rfc3164" <2>
tags = ["nginx"]
}

// ...

}
}

<1> The listen_address might be either a TCP or UDP address. UNIX sockets are not supported (yet -- pull requests are welcome)
<2> The format may be one of rfc3164, rfc5424, rfc6587 or auto. If omitted, it will default to auto.

Have a look at http://nginx.org/en/docs/syslog.html[the respective section of the NGINX documentation] on how to set up NGINX to log into syslog.

Experimental features

The exporter contains features that are currently experimental and may change without prior notice.
To use these features, either set the -enable-experimental flag or add a enable_experimental option
to your configuration file.

Dynamic re-labeling

Re-labeling lets you add arbitrary fields from the parsed log line as labels to your metrics.
To add a dynamic label, add a relabel statement to your configuration file:

[source,hcl]

namespace "app-1" {
// ...

relabel "host" {
from = "server_name"
whitelist = [ <1>
"host-a.com",
"host-b.de"
]
}
}

<1> The whitelist property is optional; if set, only the supplied values will be added as label.
All other values will be subsumed under the "other" label value. See #16 for a more detailed
discussion around the reasoning.

Dynamic relabeling also allows you to aggregate your metrics by request path (which replaces
the experimental feature originally introduced in #23):

[source,hcl]

namespace "app1" {
// ...

relabel "request_uri" {
from = "request"
split = 2

match "^/users/[0-9]+" {
  replacement = "/users/:id"
}

match "^/profile" {
  replacement = "/profile"
}

}
}

The YAML configuration for relabelings works similar to the HCL configuration:

[source,yaml]

namespaces:

• name: app1
  relabel_configs:
  • target_label: request_uri
    from: request
    split: 2
    matches:
    • regexp: "^/users/[0-9]+"
      replacement: "/users/:id"

If your regular expression contains groups, you can also use the matched values of those in the replacement value:

[source,hcl]

relabel "request_uri" {
from = "request"
split = 2

match "^/(users, profiles)/[0-9]+" {
replacement = "/$1/:id"
}
}

Running the collector

Systemd

You can find an example unit file for this service https://github.com/martin-helmich/prometheus-nginxlog-exporter/blob/master/systemd/prometheus-nginxlog-exporter.service[in this repository]. Simply copy the unit file to /etc/systemd/system:

$ wget -O /etc/systemd/system/prometheus-nginxlog-exporter.service https://raw.githubusercontent.com/martin-helmich/prometheus-nginxlog-exporter/master/systemd/prometheus-nginxlog-exporter.service
$ systemctl enable prometheus-nginxlog-exporter
$ systemctl start prometheus-nginxlog-exporter

The shipped unit file expects the binary to be located in /usr/local/bin/prometheus-nginxlog-exporter and the configuration file in /etc/prometheus-nginxlog-exporter.hcl. Adjust to your own needs.

Docker

You can also run this exporter from the Docker image quay.io/martinhelmich/prometheus-nginxlog-exporter:

$ docker run --name nginx-exporter -v logs:/mnt/nginxlogs -p 4040:4040 quay.io/martinhelmich/prometheus-nginxlog-exporter mnt/nginxlogs/access.log

Command-line flags and arguments can simply be appended to the docker run command, for example to use a
configuration file:

$ docker run --name nginx-exporter -p 4040:4040 -v logs:/mnt/nginxlogs -v /path/to/config.hcl:/etc/prometheus-nginxlog-exporter.hcl quay.io/martinhelmich/prometheus-nginxlog-exporter -config-file /etc/prometheus-nginxlog-exporter.hcl

== Frequently Asked Questions

I have started the exporter, but it is not exporting any application-specific metrics!

This may have several issues:

1. Make sure that the access log files that your exporter is listening on are present. The exporter will exit with an error code if a file is present but cannot be opened (for example, due to bad permissions), but will wait for a file if it does not yet exist.
2. Make sure that the exporter can parse the lines from your access log files. Pay attention to the <namespace>_parse_errors_total metric, which will indicate how many log lines could not be parsed.

The exporter exports the <namespace>_http_response_count_total metric, but not [other metric that is mentioned in the README]!

Most metrics require certain values to be present in the access log files that are not present in the NGINX default configuration. Especially, make sure that the access log contains the http://nginx.org/en/docs/http/ngx_http_upstream_module.html#var_upstream_response_time[`$upstream_response_time`], http://nginx.org/en/docs/http/ngx_http_log_module.html#var_request_time[`$request_time`] and/or http://nginx.org/en/docs/http/ngx_http_core_module.html#variables[`$body_bytes_sent`] variables. These need to be enabled in the NGINX configuration (more precisely, the log_format setting) and then added to the format specified for the exporter.

How can I configure NGINX to export these variables?

Have a look at NGINX's https://www.nginx.com/resources/admin-guide/logging-and-monitoring/[Logging and Monitoring] guide. It contains some good examples that contain the $request_time and $upstream_response_time:

log_format upstream_time '$remote_addr - $remote_user [$time_local] '
                         '"$request" $status $body_bytes_sent '
                         '"$http_referer" "$http_user_agent"'
                         'rt=$request_time uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time"';

Credits

• https://github.com/hpcloud/tail[tail], MIT license
• https://github.com/satyrius/gonx[gonx], MIT license
• https://github.com/prometheus/client_golang[Prometheus Go client library], Apache License
• https://github.com/hashicorp/hcl[HashiCorp configuration language], Mozilla Public License