Momy

MongoDB 到 MySQL 的复制。(MongoDB to MySQL replication)

Github星跟蹤圖

Momy

Momy 是一个简单的 cli 工具,用于将 MongoDB 实时复制到 MySQL。

  • 对 NoSQL 数据库中的数据启用 SQL 查询
  • 使能够被 Excel 访问

安装

通过 npm 安装

$ npm install -g momy

或者使用 docker

$ docker run -it --rm -v $(pwd):/workdir cognitom/momy

您可能想要创建一个别名,例如

$ echo 'alias momy="docker run -it --rm -v $(pwd):/workdir cognitom/momy"' >> ~/.bashrc

请在下面查看有关 Docker 配置的更多详细信息。

预备

MongoDB

Momy 在 MongoDB 中使用Replica Set(副本集)功能。 但是您实际上不必在 MongoDB 之间进行复制。 请按照以下步骤操作。

启动一个没有数据的新mongo实例:

$ mongod --replSet "rs0" --oplogSize 100

打开另一个终端,然后转到MongoDB Shell:

$ mongo
....
> rs.initiate()

rs.initiate() 命令准备复制所需的集合。

MySQL

启动 MySQL 实例,并创建要使用的新数据库。同步时将创建或更新表。您还将看到 mongo_to_mysql。这是存储同步信息所必需的。(请勿删除)

配置

创建一个新的 momyfile.json 文件,如下所示:

{
  "src": "mongodb://localhost:27017/dbname",
  "dist": "mysql://root:password@localhost:3306/dbname",
  "prefix": "t_",
  "case": "camel",
  "collections": {
    "collection1": {
      "_id": "number",
      "createdAt": "DATETIME",
      "field1": "number",
      "field2": "string",
      "field3": "boolean",
      "field4.subfield": "string"
    },
    "collection2": {
      "_id": "string",
      "createdAt": "DATETIME",
      "field1": "number",
      "field2": "string",
      "field3": "boolean",
      "field4": "TEXT"
    }
  }
}

  • src:MongoDB 服务器的 URL
  • dist:MySQL 服务器的 URL
  • prefix:表名的可选前缀。 在上面的示例中,表的名称为 t_collection1。
  • fieldCase:可选。 snake 或 camel。 请参阅以下部分。
  • 排除:可选。 字符或要排除的字符范围:"\uFFFD"
  • 包含:可选。 字符或包含以下字符的范围:"\u0000-\u007F"
  • 集合:将集合和字段设置为同步

_id 字段是每个集合的必填字段,应为字符串或数字。

栏位名称和类型

"<field_name>": "<field_tipe>"

或者,field_name 可以点连接:

"<field_name>.<sub_name>": "<field_tipe>"

例如,如果您有 { a: { b: { c: 'hey!' } } } 则 "a.b.c": "string"

当前支持以下本机类型:

  • BIGINT
  • TINYINT
  • VARCHAR
  • DATE
  • DATETIME
  • TIME
  • TEXT

还有一些别名:

  • number => BIGINT
  • boolean => TINYINT
  • string => VARCHAR

字段名称规范化:

某些系统(例如 Microsoft Access)不允许点连接的字段名称,因此 address.street 会导致错误。 在这种情况下,请使用fieldCase:

  • snake: address.street --> address_street
  • camel: address.street --> addressStreet

注意:如果您设置 fieldCase 值,则 _id 字段的名称也将更改为 id,不带_。

用法

在第一次运行时,我们需要从MongoDB导入所有数据:

$ momy --config momyfile.json --import

然后启动守护程序以流式传输数据:

$ momy --config momyfile.json/p>
<p><field_name><field_tipe>或</field_tipe></field_name>

或者

$ forever momy --config momyfile.json

Docker的用法

首先,为您的容器创建一个网络:

$ docker network create my-net

然后,启动数据库服务器:

$ docker run \
    --name my-mongod \
    --detach --rm \
    --network my-net \
    --mount type=volume,source=my-mongo-store,target=/data/db \
    mongo --replSet "rs0"
$ docker run \
    --name my-mysqld \
    --detach --rm \
    --network my-net \
    --mount type=volume,source=my-mysql-store,target=/var/lib/mysql \
    --env MYSQL_ALLOW_EMPTY_PASSWORD=yes \
    mysql

如果这是第一次运行上述容器,则需要初始化它们:

$ docker exec my-mongod mongo --eval 'rs.initiate()'
$ docker exec my-mysqld mysql -e 'CREATE DATABASE momy;'

像这样创建 momyfile.json:

{
  "src": "mongodb://my-mongod:27017/momy",
  "dist": "mysql://root@my-mysqld:3306/momy",
  "collections": {...}
}

注意:您必须更改用户名,密码,端口等。 以适合您的环境。

好的,让我们使用 --import 选项运行 momy:

$ docker run \
    --interactive --tty --rm \
    --network my-net \
    --mount type=bind,source=$(pwd),target=/workdir \
    cognitom/momy --import

一切顺利? 然后,停止容器(Ctrl + C)。 现在,您可以将其作为守护程序运行:

$ docker run \
    --detach --rm \
    --restart unless-stopped \
    --network my-net \
    --mount type=bind,source=$(pwd),target=/workdir \
    cognitom/momy

对于贡献者

请参阅 dev 目录。

许可

MIT

该库最初由 @doubaokun 作为 MongoDB-to-MySQL 创建,并由 @cognitom 重写。

(The first version translated by vz on 2020.07.18)

主要指標

概覽
名稱與所有者cognitom/momy
主編程語言JavaScript
編程語言JavaScript (語言數: 3)
平台Docker, Linux, Mac, Windows
許可證MIT License
所有者活动
創建於2015-10-20 04:07:44
推送於2024-05-08 06:15:43
最后一次提交2024-05-08 15:15:35
發布數10
最新版本名稱v0.8.1 (發布於 )
第一版名稱v0.3.0 (發布於 2015-11-20 21:09:38)
用户参与
星數98
關注者數8
派生數36
提交數134
已啟用問題?
問題數22
打開的問題數9
拉請求數5
打開的拉請求數3
關閉的拉請求數5
项目设置
已啟用Wiki?
已存檔?
是復刻?
已鎖定?
是鏡像?
是私有?

Build Status
NPM Status
Codecov Status

Momy

Momy is a simple cli tool for replicating MongoDB to MySQL in realtime.

  • Enable SQL query on data in NoSQL database
  • Enable to be accessed by Excel / Access

Momy

Installation

Install via npm:

$ npm install -g momy

Or use docker:

$ docker run -it --rm -v $(pwd):/workdir cognitom/momy

You might want to create an alias, for example

$ echo 'alias momy="docker run -it --rm -v $(pwd):/workdir cognitom/momy"' >> ~/.bashrc

See more detail about Docker configurations below.

Preparation

MongoDB

Momy uses Replica Set feature in MongoDB. But you don't have to replicate between MongoDB actually. Just follow the steps below.

Start a new mongo instance with no data:

$ mongod --replSet "rs0" --oplogSize 100

Open another terminal, and go to MongoDB Shell:

$ mongo
....
> rs.initiate()

rs.initiate() command prepare the collections that is needed for replication.

MySQL

Launch MySQL instance, and create the new database to use. The tables will be created or updated when syncing. You'll see mongo_to_mysql, too. This is needed to store the information for syncing. (don't remove it)

Configuration

Create a new momyfile.json file like this:

{
  "src": "mongodb://localhost:27017/dbname",
  "dist": "mysql://root:password@localhost:3306/dbname",
  "prefix": "t_",
  "case": "camel",
  "collections": {
    "collection1": {
      "_id": "number",
      "createdAt": "DATETIME",
      "field1": "number",
      "field2": "string",
      "field3": "boolean",
      "field4.subfield": "string"
    },
    "collection2": {
      "_id": "string",
      "createdAt": "DATETIME",
      "field1": "number",
      "field2": "string",
      "field3": "boolean",
      "field4": "TEXT"
    }
  }
}
  • src: the URL of the MongoDB server
  • dist: the URL of the MySQL server
  • prefix: optional prefix for table name. The name of the table would be t_collection1 in the example above.
  • fieldCase: optional. snake or camel. See the section below.
  • exclusions: optional. Chars or a range of chars to exclude: "\uFFFD"
  • inclusions: optional. Chars or a range of chars to include: "\u0000-\u007F"
  • collections: set the collections and fields to sync

_id field is required for each collection and should be string or number.

Field names and types

"<field_name>": "<field_tipe>"

or, field_name could be dot-concatenated:

"<field_name>.<sub_name>": "<field_tipe>"

For example, if you have { a: { b: { c: 'hey!' } } } then "a.b.c": "string"

Currently these native types are supported:

  • BIGINT
  • TINYINT
  • VARCHAR
  • DATE
  • DATETIME
  • TIME
  • TEXT

There're also some aliases:

  • number => BIGINT
  • boolean => TINYINT
  • string => VARCHAR

Field name normalization: fieldCase

Some system like Microsoft Access don't allow dot-concatenated field names, so address.street will cause an error. For such a case, use fieldCase:

  • snake: address.street --> address_street
  • camel: address.street --> addressStreet

Note: if you set fieldCase value, the name of _id field will change into id without _, too.

Usage

At the first run, we need to import all the data from MongoDB:

$ momy --config momyfile.json --import

Then start the daemon to streaming data:

$ momy --config momyfile.json

or

$ forever momy --config momyfile.json

Usage with Docker

First thing first, create a network for your containers:

$ docker network create my-net

Then, launch database servers:

$ docker run \
    --name my-mongod \
    --detach --rm \
    --network my-net \
    --mount type=volume,source=my-mongo-store,target=/data/db \
    mongo --replSet "rs0"
$ docker run \
    --name my-mysqld \
    --detach --rm \
    --network my-net \
    --mount type=volume,source=my-mysql-store,target=/var/lib/mysql \
    --env MYSQL_ALLOW_EMPTY_PASSWORD=yes \
    mysql

If this is the first time to run the containers above, you need to initialize them:

$ docker exec my-mongod mongo --eval 'rs.initiate()'
$ docker exec my-mysqld mysql -e 'CREATE DATABASE momy;'

Create momyfile.json like this:

{
  "src": "mongodb://my-mongod:27017/momy",
  "dist": "mysql://root@my-mysqld:3306/momy",
  "collections": {...}
}

Note: you must change username, password, port, ...etc. to fit your environment.

OK, let's run momy with --import option:

$ docker run \
    --interactive --tty --rm \
    --network my-net \
    --mount type=bind,source=$(pwd),target=/workdir \
    cognitom/momy --import

Everything goes well? Then, stop the container (Ctrl + C). Now you can run it as a daemon:

$ docker run \
    --detach --rm \
    --restart unless-stopped \
    --network my-net \
    --mount type=bind,source=$(pwd),target=/workdir \
    cognitom/momy

For contributors

See dev directory.

License

MIT

This library was originally made by @doubaokun as MongoDB-to-MySQL and rewritten by @cognitom.