flumedb
A modular database made for moving logs with streams.
architecture
Flume is a modular database comprised of an Append Only Log
and Streaming Views on that log.
This makes a star shaped pipeline - or rather, there is a pipeline
from the log to each view, but the log is part of every pipeline.
The Log is the main storage element and provides durability.
Views stream that data and build up their own model. They
can point back to the main storage (normalized) or materialize it
(denormalized). The views can use a structure optimized for
the queries they provide, instead of durability, because the
can rebuild from the main log.
In my previous modular database design, level-sublevel
indexes had to be written atomically with the data. This was
okay for map style indexes, but made simple aggregations such
as a count of all records difficult. Worse, rebuilding the indexes
needed a batch migration between two versions of the database.
This made developing interesting applications on level-sublevel quite difficult (such as secure-scuttlebutt)
In flume, each view remembers a version number, and if the version
number changes, it just rebuilds the view. This means view code can
be easily updated, or new views added. It just rebuilds the view on
startup. (though, this may take a few minutes on larger data)
The trick is that each view exposes a observable, flumeview.since,
this represents it's current state - the sequence number the view is up to.
An observable is like an event meets a value. it's an changing value that you can observe.
Events, promises, or streams are similar, but not quite the right choice here.
Note, views are async. The main log may callback before the view is
fully up to date, but if a read is made to an as yet unsynced view,
it just waits for the view building to complete. This may make
the call take longer (applications should show a progress bar
for reindexing) but most of the application code can just assume
that indexes are always up to date, because if you get a callback
from an append to the log, a subsequent view read will return
data consistent with that.
This gives us the freedom to have async views,
which gives us the freedom to have many different sorts of views!
example
Take one flumelog-* module and zero or more flumeview-* modules,
and glue them together with flumedb.
var MemLog = require('flumelog-memory') // just store the log in memory
var Reduce = require('flumeview-reduce') //just a reduce function.
var Flume = require('flumedb')
var db = Flume(MemLog())
//the api of flumeview-reduce will be mounted at db.sum...
.use('sum', Reduce(1, function (acc, item) {
return (acc