HSE

HSE:异构内存存储引擎。「HSE: Heterogeneous-memory storage engine」

Github stars Tracking Chart

HSE:异构内存存储引擎

HSE 是一个快速可嵌入的键值存储,专为 SSD 和持久性内存设计。HSE 通过协调跨 DRAM 和多类固态存储的数据放置,优化了性能和耐久性。

HSE 是支持数据库、软件定义存储(SDS)、高性能计算(HPC)、物联网(IoT)和机器学习(ML)的理想选择。

主要特点

  • 丰富的键值运算符集
  • 带有快照隔离的完整事务,跨越多个独立的键值集合
  • 用于在快照视图上迭代的游标
  • 用于优化单一数据存储中的混合用例工作负载的数据模型
  • 键和值的压缩
  • 灵活的耐久性控制
  • 可配置的数据编排方案
  • 可嵌入任何应用程序的本地C语言库

优势:

  • 可扩展到 TB 级的数据和每个存储的数千亿个密钥
  • 有效地处理成千上万的并发操作
  • 相对于许多工作负载的普通替代方案,极大地提高了吞吐量、延迟、写放大和读放大的能力
  • 可选择结合多种类型的固态存储,以优化性能和耐用性

入门

HSE 项目文档 包含了所有你需要开始使用 HSE 的信息。

构建和安装

按照下面的步骤来构建和安装 HSE。

依赖性

你可能需要安装额外的软件包来为你的特定 Linux 发行版和环境构建或运行 HSE。参见 INSTALL.md,了解几种常见的 Linux 发行版所需的软件包的例子。

Poetry

我们使用 Poetry 来管理 Python 的依赖性。按照 文档 中的要求安装 Poetry。

构建 HSE

克隆 hse 仓库并签出最新的发布标签。这个标签必须是针对 HSE 2.0 或更高版本的。

比如说:

git clone https://github.com/hse-project/hse.git
cd hse
git checkout <release tag>

用 Poetry 安装 Python 依赖项

poetry install

在 Poetry shell 中使用 Meson 和 Ninja 进行构建和安装:

poetry shell
meson setup build
meson compile -C build
meson install -C build

默认的安装目录是 /opt/hse。这可以通过配置 -Dprefix=$prefix 或 -prefix=$prefix 来覆盖。在构建和安装 HSE 后,你可以退出 Poetry shell。

exit

其他参考资料

关于运行测试套件和对 HSE 的贡献的信息位于 CONTRIBUTING.md 文件中。

我们将 HSE 与几个常见的应用程序集成,以展示其能力:

Overview

Name With Ownerhse-project/hse
Primary LanguageC
Program languageC (Language Count: 6)
PlatformLinux
License:
Release Count44
Last Release Namer3.1.0-rc0 (Posted on 2023-02-22 10:33:27)
First Release Namer1.7.0.rc1 (Posted on )
Created At2020-04-17 12:09:48
Pushed At2023-09-04 19:53:49
Last Commit At2023-08-17 16:38:43
Stargazers Count670
Watchers Count35
Fork Count65
Commits Count2.2k
Has Issues Enabled
Issues Count30
Issue Open Count4
Pull Requests Count517
Pull Requests Open Count4
Pull Requests Close Count43
Has Wiki Enabled
Is Archived
Is Fork
Is Locked
Is Mirror
Is Private

HSE: Heterogeneous-Memory Storage Engine

HSE is a fast embeddable key-value store designed for SSDs and
persistent memory.
HSE optimizes performance and endurance by orchestrating data
placement across DRAM and multiple classes of solid-state storage.

HSE is ideal for powering Databases, Software-Defined Storage (SDS),
High-Performance Computing (HPC), Internet of Things (IoT),
and Machine Learning (ML).

Key Features:

  • Rich set of key-value operators
  • Full transactions with snapshot-isolation spanning multiple independent
    key-value collections
  • Cursors for iterating over snapshot views
  • Data model for optimizing mixed use-case workloads in a single data store
  • Key and value compression
  • Flexible durability controls
  • Configurable data orchestration schemes
  • Native C library that can be embedded in any application

Benefits:

  • Scales to terabytes of data and hundreds of billions of keys per store
  • Efficiently handles thousands of concurrent operations
  • Dramatically improves throughput, latency, write-amplification,
    and read-amplification versus common alternatives for many workloads
  • Optionally combines multiple classes of solid-state storage to
    optimize performance and endurance

Getting Started

The HSE project documentation
contains all the information you need to get started with HSE.

Building and Installing

Follow the steps below to build and install HSE.

Dependencies

You may need to install additional packages to build or run HSE for your
particular Linux distribution and environment.
See INSTALL.md
for examples of the packages required for several common Linux distributions.

Poetry

We use Poetry to manage Python dependencies.
Install Poetry as per the docs.

Building HSE

Clone the hse repo and checkout
the latest release tag. This tag must be for HSE version 2.0 or higher.

For example

git clone https://github.com/hse-project/hse.git
cd hse
git checkout <release tag>

Install Python dependencies using Poetry.

poetry install

Build and install using Meson and Ninja from within the Poetry shell.

poetry shell
meson setup build
meson compile -C build
meson install -C build

The default install directory is /opt/hse. This can be overridden by
configuring the build with either -Dprefix=$prefix or --prefix=$prefix.
After building and installing HSE you can exit the Poetry shell.

exit

Additional References

Information on running test suites and contributing to HSE is located in the
CONTRIBUTING.md file.

We integrated HSE with several common applications to demonstrate its
capabilities.

To the top