Hayabusa
Hayabusa: A Simple and Fast Full-Text Search Engine for Massive System Log Data
Concept
- Pure python implement
- Parallel SQLite processing engine
- SQLite3 FTS(Full Text Search)
- Core-scale architecture
Architecture
-
Design of the directory structure
- By specifying a search range of time in ”the directory path + yyyy + mm + dd + hh + min.db”, the search program can select the search time systematically.
/targetdir/yyyy/mm/dd/hh/min.db -
StoreEngine
- sample code
import os.path import sqlite3 db_file = ’test.db’ log_file = ’1m.log’ if not os.path.exists(db_file): conn = sqlite3.connect(db_file) conn.execute("CREATE VIRTUAL TABLE SYSLOG USING FTS3(LOGS)"); conn.close() conn = sqlite3.connect(db_file) with open(log_file) as fh: lines = [[line] for line in fh] conn.executemany(’INSERT INTO SYSLOG VALUES ( ? )’, lines) conn.commit() -
SearchEngine
- sample command
$ python search_engine.py -h usage: search_engine.py [-h] [--time TIME] [--match MATCH] [-c] [-s] [-v] optional arguments: -h, --help show this help message and exit --time TIME time explain regexp(YYYY/MM/DD/HH/MIN). eg: 2017/04/27/10/* --match MATCH matching keyword. eg: noc or 'noc Login' -e exact match -c count -s sum -v verbose $ python search_engine.py --time 2017/05/11/13/* --match 'keyword' -c -
Architecture image
Search condition
-
case-insensitive
- no distinguish uppercase or lowercase
-
Exact match
-e --match '192.168.0.1' -
AND
--match 'Hello World' -
OR
--match 'Hello OR World' -
NOT
--match 'Hello World -Wide' -
PHRASE
--match '"Hello World"' --match '\"192.168.0.1\"' <- IP address case(same as -e flag) --match '\"192.168.0.1\" src sent' <- PHRASE + AND search -
asterisk(*)
--match 'H* World' -
HAT
--match '^Hello World'
Development environment
- CentOS 7.3
- Python 3.5.1(use anaconda packages)
- SQLite3(version 3.9.2)
Dependency softwares
- Python 3
- SQLite3
- GNU Parallel
Benchmark
Compare with Apache Spark
-
Hayabusa and Spark time comparison
-
Comarison of distributes Spark environment and the stand-alone Hayabusa
