Description
Package slowpoke is a simple key/value store written using Go's standard library only. Keys are stored in memory (with persistence), values stored on disk.
Description on russian: https://habr.com/post/354224/
Motivation
Replace Bolt with a simpler and more efficient engine.
Slowpoke (from version 2.0) based on pudge
How it works
Keys are stored in memory with persistence to disk. Values stored on disk only.
Slowpoke is parallel
Server
GRPC Server example: okdb
Complex examples
zen platform for authors and their subscribers with a minimalistic design and user-friendly interface.
golang-gin-realworld-example-app
This codebase was created to demonstrate a fully fledged fullstack application built with Golang/Gin/Slowpoke including CRUD operations, authentication, routing, pagination, and more.
Basic example
package main
import (
"fmt"
"github.com/recoilme/slowpoke"
)
func main() {
// create database
file := "test/example.db"
// close all opened database
defer slowpoke.CloseAll()
// init key/val
key := []byte("foo")
val := []byte("bar")
//store
slowpoke.Set(file, key, val)
// get
res, _ := slowpoke.Get(file, key)
//result
fmt.Println(string(res))
}
Lazy example
func TestGob(t *testing.T) {
file := "test/gob.db"
DeleteFile(file)
defer CloseAll()
type Post struct {
Id int
Content string
Category string
}
for i := 0; i < 20; i++ {
post := &Post{Id: i, Content: "Content:" + strconv.Itoa(i)}
err := SetGob(file, post.Id, post)
ch(err, t)
}
for i := 0; i < 20; i++ {
var post = new(Post)
err := GetGob(file, i, post)
ch(err, t)
fmt.Println("i:", i, "Post:", post)
}
}
Advanced example
type Post struct {
Id int
Content string
Category string
}
func main() {
posts := "test/posts"
tags := "test/tags"
var pairs [][]byte
for i := 0; i < 40; i++ {
id := make([]byte, 4)
binary.BigEndian.PutUint32(id, uint32(i))
post := &Post{Id: i, Content: "Content:" + strconv.Itoa(i), Category: "Category:" + strconv.Itoa(i/10)}
b, _ := json.Marshal(post)
pairs = append(pairs, id)
pairs = append(pairs, b)
tag := fmt.Sprintf("%s:%08d", strconv.Itoa(i/10), i)
//store only tags keys
slowpoke.Set(tags, []byte(tag), nil)
}
//store posts fast
slowpoke.Sets(posts, pairs)
//get last 2 post key with offset 2
limit := uint32(2)
offset := uint32(2)
order := false //desc
keys, _ := slowpoke.Keys(posts, nil, limit, offset, order)
fmt.Println(keys) //[[0 0 0 37] [0 0 0 36]]
//get key/ values
res := slowpoke.Gets(posts, keys)
for k, v := range res {
if k%2 == 0 {
fmt.Print(binary.BigEndian.Uint32(v))
} else {
var p Post
json.Unmarshal(v, &p)
fmt.Println(p)
}
}
//37{37 Content:37 Category:3}
//36{36 Content:36 Category:3}
//free from memory
slowpoke.Close(posts)
slowpoke.Close(tags)
//open Db and read tags by prefix 2:* in ascending order
tagsKeys, _ := slowpoke.Keys(tags, []byte("2:*"), 0, 0, true)
for _, v := range tagsKeys {
fmt.Print(string(v) + ", ")
}
//2:00000020, 2:00000021, 2:00000022, 2:00000023, 2:00000024, 2:00000025, 2:00000026, 2:00000027, 2:00000028, 2:00000029,
}
Api
All methods are thread-safe.
- Set/Sets/SetGob
Store val and key. If the file does not exist it will be created.
- Get/Gets/GetGob
Return the value for the given key or nil and an error. Get will open the database if necessary.
- Keys
Return keys in ascending/descending order.
With limit and offset.
If from is not nil, return keys lexicographically greater than the from value.
If from ends with asterix *, return keys with the prefix equal to from without the asterix.
Status
Used in production (master branch)
Benchmark
Some tests, MacBook Pro (Retina, 13-inch, Early 2015)
Test 1
Number of keys: 1000000
Minimum key size: 16, maximum key size: 64
Minimum value size: 128, maximum value size: 512
Concurrency: 2
| pogreb | goleveldb | bolt | badgerdb | pudge | slowpoke | pudge(mem) | |
|---|---|---|---|---|---|---|---|
| 1M (Put+Get), seconds | 187 | 38 | 126 | 34 | 23 | 23 | 2 |
| 1M Put, ops/sec | 5336 | 34743 | 8054 | 33539 | 47298 | 46789 | 439581 |
| 1M Get, ops/sec | 1782423 | 98406 | 499871 | 220597 | 499172 | 445783 | 1652069 |
| FileSize,Mb | 568 | 357 | 552 | 487 | 358 | 358 | 358 |