HBase in Docker

在Docker中运行HBase。(HBase running in Docker.)

Github stars Tracking Chart

在 Docker 中运行 HBase

此配置构建一个docker容器,以便在容器内的文件上运行HBase(带有嵌入式Zookeeper)。

请注意

此处的方法需要编辑本地服务器的 /etc/hosts 文件以添加容器主机名的条目。这是因为 HBase 使用主机名将连接数据传回容器(从它的内部Zookeeper)。

希望这可以通过 Docker 的新网络得到改善,但尚未得到修复。

构建映像

  $ docker build -t dajobe/hbase

拉映像

如果您想拉出已经构建的映像,请使用此

  $ docker pull dajobe/hbase

更多详情参见:https://hub.docker.com/r/dajobe/hbase/

运行HBase

手动运行HBase:

$ mkdir data
id=$(docker run --name=hbase-docker -h hbase-docker -d -v $PWD/data:/data dajobe/hbase)

运行它并通过编辑在本地调整主机系统 /etc/hosts 将DNS主机名'hbase-docker' 别名为 容器,请使用以下命令:

  $ ./start-hbase.sh

这将要求您输入您的 sudo 密码来编辑主机机器的 /etc/hosts 文件。

如果要在同一主机上运行多个 hbase docker,则可以使用 '-h' / '--hostname' 参数为它们提供不同的主机名。 你可能不得不给他们不同的端口。未经测试。

如果要自定义使用的主机名,请设置 docker 命令行上的 HBASE_DOCKER_HOSTNAME envariable

查找Hbase状态

如果 docker 容器 DNS 名称为 'hbase-docker',则为主状态

  http://hbase-docker:16010/master-status

区域服务器状态页面是从上一页链接的。

Thrift UI

  http://hbase-docker:9095/thrift.jsp

REST 服务器 UI

  http://hbase-docker:8085/rest.jsp

(嵌入式)Zookeeper 状态

  http://hbase-docker:16010/zk.jsp

请参阅 HBase 日志

如果您想查看最新的日志实时使用:

  $ docker attach $id

然后 ^C 分离。

要查看自 HBase 服务器启动以来的所有日志,请使用:

  $ docker logs $id

和 ^C 再次分离。

要查看各个日志文件而不使用 docker ,请查看如上所述调用数据卷目录,例如 $PWD/data/logs。

测试 HBase 正在通过 Thrift 上的 python 工作

在这里,我将连接到名为 “hbase-docker” 的 docker 容器(例如由 start-hbase.sh 脚本创建)。 端口 9090 是 Thrift API 端口,因为 [Happybase] [1] [2] 使用 Thrift与HBase通信。

$ python
Python 2.7.15 (default, Jan 12 2019, 21:07:57)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import happybase
>>> connection = happybase.Connection('hbase-docker', 9090)
>>> connection.create_table('table-name', { 'family': dict() } )
>>> connection.tables()
['table-name']
>>> table = connection.table('table-name')
>>> table.put('row-key', {'family:qual1': 'value1', 'family:qual2': 'value2'})
>>> for k, data in table.scan():
...   print k, data
...
row-key {'family:qual1': 'value1', 'family:qual2': 'value2'}
>>>

(对于happybase的简单安装: sudo pip install happybase 虽然我使用 pip install --user happybase 来获取它仅供我使用)

测试 HBase 正在使用 Java

$ docker run --rm -it --link $id:hbase-docker dajobe/hbase hbase shell
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/2.0/book.html#shell
Version 2.1.2, r1dfc418f77801fbfb59a125756891b9100c1fc6d, Sun Dec 30 21:45:09 PST 2018
Took 0.0472 seconds
hbase(main):001:0> status
1 active master, 0 backup masters, 1 servers, 0 dead, 2.0000 average load
Took 0.7255 seconds
hbase(main):002:0> list
TABLE
table-name
1 row(s)
Took 0.0509 seconds
=> ["table-name"]
hbase(main):003:0>

显示上面的 happybase 示例中的 table-name 表。

或者,如果主机上有Hbase发行版,则可以使用 bin/hbase shell,如果已将 hbase 配置设置为连接到主机 hbase-docker zookeeper 端口 2181,则通过配置属性hbase.zookeeper.quorum 获取服务器。

本地代理 HBase UI

如果你在远程机器上运行docker,在本地浏览器中看到这些服务器私有URL是很方便的,所以这里有一个〜/.ssh/config 片段来做

Host my-docker-server
Hostname 1.2.3.4
    LocalForward 127.0.0.1:16010 127.0.0.1:16010
    LocalForward 127.0.0.1:9095 127.0.0.1:9095
    LocalForward 127.0.0.1:8085 127.0.0.1:8085

当您使用 ssh my-docker-server 时,ssh 连接到 docker 服务器并将端口16010/16030 上的本地计算机上的请求转发到连接到 hbase 容器的远程端口。

最重要的是,您可以使用这些网址查看正在发生的事情:

要查看容器中发生了什么,由于本地计算机和容器都使用 localhost(也称为127.0.0.1),即使链接也能正常工作!

备注

[1] http://happybase.readthedocs.org/en/latest/

[2] https://github.com/wbolster/happybase

Overview

Name With Ownerdajobe/hbase-docker
Primary LanguageShell
Program languageShell (Language Count: 4)
PlatformDocker, Linux, Mac, Windows
License:
Release Count0
Created At2013-08-23 00:32:28
Pushed At2022-09-27 01:35:07
Last Commit At2020-09-22 20:19:16
Stargazers Count328
Watchers Count18
Fork Count188
Commits Count57
Has Issues Enabled
Issues Count26
Issue Open Count19
Pull Requests Count6
Pull Requests Open Count2
Pull Requests Close Count1
Has Wiki Enabled
Is Archived
Is Fork
Is Locked
Is Mirror
Is Private

HBase in Docker

This configuration builds a docker container to run HBase (with
embedded Zookeeper) running on the files inside the container.

NOTE

The approach here requires editing the local server's /etc/hosts
file to add an entry for the container hostname. This is because
HBase uses hostnames to pass connection data back out of the
container (from it's internal Zookeeper).

Hopefully this can be improved with Docker's newer networking
but this hasn't been fixed yet.

Build Image

$ docker build -t dajobe/hbase .

Pull image

If you want to pull the image already built then use this

$ docker pull dajobe/hbase

More details at https://hub.docker.com/r/dajobe/hbase/

Run HBase

To run HBase by hand:

$ mkdir data
$ id=$(docker run --name=hbase-docker -h hbase-docker -d -v $PWD/data:/data dajobe/hbase)

To run it and adjust the host system's locally by editing
/etc/hosts to alias the DNS hostname 'hbase-docker' to the
container, use this:

$ ./start-hbase.sh

This will require you to enter your sudo password to edit the host
machine's /etc/hosts file

If you want to run multiple hbase dockers on the same host, you can
give them different hostnames with the '-h' / '--hostname' argument.
You may have to give them different ports though. Not tested.

If you want to customize the hostname used, set the
HBASE_DOCKER_HOSTNAME envariable on the docker command line

Find Hbase status

Master status if docker container DNS name is 'hbase-docker'

http://hbase-docker:16010/master-status

The region servers status pages are linked from the above page.

Thrift UI

http://hbase-docker:9095/thrift.jsp

REST server UI

http://hbase-docker:8085/rest.jsp

(Embedded) Zookeeper status

http://hbase-docker:16010/zk.jsp

See HBase Logs

If you want to see the latest logs live use:

$ docker attach $id

Then ^C to detach.

To see all the logs since the HBase server started, use:

$ docker logs $id

and ^C to detach again.

To see the individual log files without using docker, look into
the data volume dir eg $PWD/data/logs if invoked as above.

Test HBase is working via python over Thrift

Here I am connecting to a docker container with the name 'hbase-docker'
(such as created by the start-hbase.sh script). The port 9090 is the
Thrift API port because [Happybase][1] [2] uses Thrift to talk to HBase.

$ python
Python 2.7.15 (default, Jan 12 2019, 21:07:57)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import happybase
>>> connection = happybase.Connection('hbase-docker', 9090)
>>> connection.create_table('table-name', { 'family': dict() } )
>>> connection.tables()
['table-name']
>>> table = connection.table('table-name')
>>> table.put('row-key', {'family:qual1': 'value1', 'family:qual2': 'value2'})
>>> for k, data in table.scan():
...   print k, data
...
row-key {'family:qual1': 'value1', 'family:qual2': 'value2'}
>>>

(Simple install for happybase: sudo pip install happybase although I
use pip install --user happybase to get it just for me)

Test HBase is working from Java

$ docker run --rm -it --link $id:hbase-docker dajobe/hbase hbase shell
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
For Reference, please visit: http://hbase.apache.org/2.0/book.html#shell
Version 2.1.2, r1dfc418f77801fbfb59a125756891b9100c1fc6d, Sun Dec 30 21:45:09 PST 2018
Took 0.0472 seconds
hbase(main):001:0> status
1 active master, 0 backup masters, 1 servers, 0 dead, 2.0000 average load
Took 0.7255 seconds
hbase(main):002:0> list
TABLE
table-name
1 row(s)
Took 0.0509 seconds
=> ["table-name"]
hbase(main):003:0>

Showing the table-name table made in the happybase example above.

Alternatively if you have the Hbase distribution available on the
host you can use bin/hbase shell if the hbase configuration has
been set up to connect to host hbase-docker zookeeper port 2181 to
get the servers via configuration property hbase.zookeeper.quorum

Proxy HBase UIs locally

If you are running docker on a remote machine, it is handy to see
these server-private urls in a local browser so here is a
~/.ssh/config fragment to do that

Host my-docker-server
Hostname 1.2.3.4
    LocalForward 127.0.0.1:16010 127.0.0.1:16010
    LocalForward 127.0.0.1:9095 127.0.0.1:9095
    LocalForward 127.0.0.1:8085 127.0.0.1:8085

When you ssh my-docker-server ssh connects to the docker server and
forwards request on your local machine on ports 16010 / 16030 to the
remote ports that are attached to the hbase container.

The bottom line, you can use these URLs to see what's going on:

  • http://localhost:16010/master-status for the Master Server
  • http://localhost:9095/thrift.jsp for the thrift UI
  • http://localhost:8085/rest.jsp for the REST server UI
  • http://localhost:16010/zk.jsp for the embedded Zookeeper

to see what's going on in the container and since both your local
machine and the container are using localhost (aka 127.0.0.1), even
the links work!

Notes

[1] http://happybase.readthedocs.org/en/latest/

[2] https://github.com/wbolster/happybase

To the top