Genie

分布式大数据业务流程服务。(Distributed Big Data Orchestration Service.)

Github stars Tracking Chart

Genie是由Netflix开发的联合工作协调引擎。 Genie提供RESTful API来运行各种大数据工作,如Hadoop,Pig,Hive,Spark,Presto,Sqoop等。 它还提供用于管理许多分布式处理集群的元数据的API以及在其上运行的命令和应用程序。\r\n

Genie为计算资源的用户提供可扩展的联合作业和资源管理。

>从最终用户的角度来看,Genie提取了各种(潜在的瞬态)计算资源(如YARN,Presto,Mesos群集等)的物理细节。然后,它提供API来提交和监视这些集群上的作业,而无需用户自己安装任何客户端或了解集群和命令的详细信息。

管理员将使用配置API来注册集群以及与Genie一起运行的命令/应用程序。 Genie节点可以预先安装所有客户端,如果正确配置,Genie将在运行时下载并安装它们。用户可以查找可用的集群和命令,并提交要处理的作业。一旦提交作业,用户可以查询Genie的工作状态和输出。

该模式的一大优点在于它为客户端资源提供了可扩展性。这解决了一个非常常见的问题,其中单个机器被配置为将作业提交到大型集群并且机器过载的入口点。 Genie允许使用一组可以增加和减少数量的机器来处理不断增加的负载,提供了非常可扩展的解决方案。

Overview

Name With OwnerNetflix/genie
Primary LanguageJava
Program languageJava (Language Count: 10)
Platform
License:Apache License 2.0
Release Count238
Last Release Namev4.3.12 (Posted on )
First Release Name0.20 (Posted on 2013-06-21 19:59:50)
Created At2013-06-20 20:35:56
Pushed At2024-04-08 23:06:01
Last Commit At2024-04-08 15:16:00
Stargazers Count1.7k
Watchers Count519
Fork Count363
Commits Count3.2k
Has Issues Enabled
Issues Count188
Issue Open Count7
Pull Requests Count915
Pull Requests Open Count5
Pull Requests Close Count90
Has Wiki Enabled
Is Archived
Is Fork
Is Locked
Is Mirror
Is Private

Genie

Download
License
Issues
NetflixOSS Lifecycle

Introduction

Genie is a federated job orchestration engine developed by Netflix. Genie provides REST-ful APIs to run a variety of big
data jobs like Hadoop, Pig, Hive, Spark, Presto, Sqoop and more. It also provides APIs for managing the metadata of many
distributed processing clusters and the commands and applications which run on them.

Documentation

See the official website to find documentation about Genie and specific
documentation for various releases.

Demo

Genie has demo instructions available for all 3.x.x releases. Please see the release you're interested in demoing
on the releases page. Click on the release and then demo docs.

Builds

Genie builds are run on Travis CI here., Branch, Build, Coverage (coveralls.io), Coverage (codecov.io), :------:, :-------------------------------------------------------------------------------------------------------------:, :------------------------------------------------------------------------------------------------------------------------------------------------------:, :----------------------------------------------------------------------------------------------------------------------------------:, master, Build Status, Coverage Status, codecov, 3.3.x, Build Status, Coverage Status, codecov, 3.2.x, Build Status, Coverage Status, codecov, 3.1.x, Build Status, Coverage Status, codecov, 3.0.x, Build Status, Coverage Status, codecov, ## Docker

Successful builds will also generate a docker image which is published to Docker Hub.

App Image

App Image

This is the image for the sample Spring Boot all in one jar. You can use docker pull netflixoss/genie-app:{version}
to test the one you want.

You can run via docker run -t --rm -p 8080:8080 netflixoss/genie-app:{version}

Python Client

The Genie Python client has been moved into its own repo.

To the top