Apache InLong

Apache InLong - 海量数据的一站式整合框架。「Apache InLong - a one-stop integration framework for massive data.」

Github星跟踪图

A one-stop, full-scenario integration framework for massive data

GitHub Actions
CodeCov
Maven Central
GitHub release
License
Twitter
Slack

What is Apache InLong?

Stargazers Over Time Contributors Over Time
Stargazers over time Contributor Over Time

Apache InLong is a one-stop, full-scenario integration framework for massive data that supports Data Ingestion, Data Synchronization and Data Subscription, and it provides automatic, secure and reliable data transmission capabilities. InLong also supports both batch and stream data processing at the same time, which offers great power to build data analysis, modeling and other real-time applications based on streaming data.

InLong (应龙) is a divine beast in Chinese mythology who guides the river into the sea, and it is regarded as a metaphor of the InLong system for reporting data streams.

InLong was originally built at Tencent, which has served online businesses for more than 8 years, to support massive data (data scale of more than 80 trillion pieces of data per day) reporting services in big data scenarios. The entire platform has integrated 5 modules: Ingestion, Convergence, Caching, Sorting, and Management, so that the business only needs to provide data sources, data service quality, data landing clusters and data landing formats, that is, the data can be continuously pushed from the source to the target cluster, which greatly meets the data reporting service requirements in the business big data scenario.

For getting more information, please visit our project documentation at https://inlong.apache.org/.
inlong-structure-en.png

Features

Apache InLong offers a variety of features:

  • Ease of Use: a SaaS-based service platform. Users can easily and quickly report, transfer, and distribute data by publishing and subscribing to data based on topics.
  • Stability & Reliability: derived from the actual online production environment. It delivers high-performance processing capabilities for 10 trillion-level data streams and highly reliable services for 100 billion-level data streams.
  • Comprehensive Features: supports various types of data access methods and can be integrated with different types of Message Queue (MQ). It also provides real-time data extract, transform, and load (ETL) and sorting capabilities based on rules. InLong also allows users to plug features to extend system capabilities.
  • Service Integration: provides unified system monitoring and alert services. It provides fine-grained metrics to facilitate data visualization. Users can view the running status of queues and topic-based data statistics in a unified data metric platform. Users can also configure the alert service based on their business requirements so that users can be alerted when errors occur.
  • Scalability: adopts a pluggable architecture that allows you to plug modules into the system based on specific protocols. Users can replace components and add features based on their business requirements.

When should I use InLong?

InLong aims to provide a one-stop, full-scenario integration framework for massive data, users can easily build stream-based data applications. It supports Data Ingestion, Data Synchronization and Data Subscription at the same time, and is suitable for environments that need to quickly build a data reporting platform, as well as an ultra-large-scale data reporting environment that InLong is very suitable for, and an environment that needs to automatically sort and land the reported data.

You can use InLong in the following ways:

Supported Data Nodes (Updating)

Type Name Version
Extract Node Auto Push None
File None
Kafka 2.x
MongoDB >= 3.6
MQTT >= 3.1
MySQL 5.6, 5.7, 8.0.x
Oracle 11,12,19
PostgreSQL 9.6, 10, 11, 12
Pulsar 2.8.x
Redis 2.6.x
SQLServer 2012, 2014, 2016, 2017, 2019
Load Node Auto Consumption None
ClickHouse 20.7+
Elasticsearch 6.x, 7.x
Greenplum 4.x, 5.x, 6.x
HBase 2.2.x
HDFS 2.x, 3.x
Hive 1.x, 2.x, 3.x
Iceberg 0.12.x
Hudi 0.12.x
Kafka 2.x
MySQL 5.6, 5.7, 8.0.x
Oracle 11, 12, 19
PostgreSQL 9.6, 10, 11, 12
SQLServer 2012, 2014, 2016, 2017, 2019
TDSQL-PostgreSQL 10.17
Doris >= 0.13
StarRocks >= 2.0
Kudu >= 1.12.0
Redis >= 3.0

Build InLong

More detailed instructions can be found at Quick Start section in the documentation.

Requirements:

CodeStyle:

mvn spotless:apply

Compile and install:

mvn clean install -DskipTests

(Optional) Compile using docker image:

docker pull maven:3.6-openjdk-8
docker run -v `pwd`:/inlong  -w /inlong maven:3.6-openjdk-8 mvn clean install -DskipTests

after compile successfully, you could find distribution file at inlong-distribution/target.

Deploy InLong

Develop InLong

Contribute to InLong

Contact Us

Documentation

License

© Contributors Licensed under an Apache-2.0 license.

主要指标

概览
名称与所有者apache/inlong
主编程语言Java
编程语言Shell (语言数: 16)
平台Bare Metal, Docker, Kubernetes, Linux
许可证Apache License 2.0
所有者活动
创建于2020-01-03 07:13:14
推送于2025-04-17 06:56:08
最后一次提交2025-04-17 14:56:08
发布数19
最新版本名称inlong-sdk/dataproxy-sdk-twins/dataproxy-sdk-golang/v1.0.0 (发布于 )
第一版名称0.12.0-incubating-RC0 (发布于 2021-12-22 16:16:23)
用户参与
星数1.4k
关注者数67
派生数532
提交数5.1k
已启用问题?
问题数6234
打开的问题数14
拉请求数5141
打开的拉请求数7
关闭的拉请求数399
项目设置
已启用Wiki?
已存档?
是复刻?
已锁定?
是镜像?
是私有?