gonx 
gonx is Nginx access log reader library for Go. In fact you can use it for any format.
Usage
The library provides Reader type and two constructors for it.
Common constructor NewReader gets opened file (any io.Reader in fact) and log format of type string as argumets.
Format is in form os nginx log_format string.
reader := gonx.NewReader(file, format)
NewNginxReader provides more magic. It gets log file io.Reader, nginx config file io.Reader
and log_format name string as a third. The actual format for Parser will be extracted from
given nginx config.
reader := gonx.NewNginxReader(file, nginxConfig, format_name)
Reader implements io.Reader. Here is example usage
for {
rec, err := reader.Read()
if err == io.EOF {
break
}
// Process the record... e.g.
}
See more examples in example/*.go sources.
Performance
NOTE All benchmarks was made on my old 11" MacBook Air 2011, so you should get the better results for your brand new hardware ;-)
I have a few benchmarks for parsing string log record into Entry using gonx.Parser
BenchmarkParseSimpleLogRecord 100000 19457 ns/op
BenchmarkParseLogRecord 20000 84425 ns/op
And here is some real wold stats. I got ~300Mb log file with ~700K records and process with simple scripts.
- Reading whole file line by line with
bufio.Scannerwithout any other processing takes a one second. - Read in the same manner plus parsing with
gonx.Parsertakes about 80 seconds - But for reading this file with
gonx.Readerwhich parses records using separate goroutines it takes about 45 seconds (but I want to make it faster)
Format
As I said above this library is primary for nginx access log parsing, but it can be configured to parse any
other format. NewReader accepts format argument, it will be transformed to regular expression and used
for log line by line parsing. Format is nginx-like, here is example
`$remote_addr [$time_local] "$request"`
It should contain variables in form $name. The regular expression will be created using this string
format representation
`^(?P<remote_addr>[^ ]+) \[(?P<time_local>[^]]+)\] "(?P<request>[^"]+)"$`
Reader.Read returns a record of type Entry (which is customized map[string][string]). For this example
the returned record map will contain remote_addr, time_local and request keys filled with parsed values.
Stability
This library API and internal representation can be changed at any moment, but I guarantee that backward
capability will be supported for the following public interfaces.
func NewReader(logFile io.Reader, format string) *Readerfunc NewNginxReader(logFile io.Reader, nginxConf io.Reader, formatName string) (reader *Reader, err error)func (r *Reader) Read() (record Entry, err error)
Changelog
All major changes will be noticed in release notes.
Roadmap
I have no roadmap for this project at the moment for a few reasons. At the first, it is a simple library and I want to keep it. At the second, there is no feature requests, and for me, this library do its job. A few things may happen: the default binary, to use this not only as a library, and performance improvements if they will be needed.
Contributing
Fork the repo, create a feature branch then send me pull request. Feel free to create new issues or contact me using email.