DevStats

📈CNCF 创建的开发人员贡献分析和图表工具。「📈CNCF-created tool for analyzing and graphing developer contributions」

Github星跟蹤圖

Build Status
CII Best Practices

GitHub archives and git Grafana visualization dashboards

Authors: Łukasz Gryglicki lgryglicki@cncf.io, Justyna Gryglicka jgryglicka@cncf.io.

This is a toolset to visualize GitHub archives using Grafana dashboards.

GHA2DB stands for GitHub Archives to DashBoards.

More information about Kubernetes dashboards here.

Kubernetes and Helm

Please see example Helm chart for an example Helm deployment.

Please see Helm chart for a full Helm deployment.

Please see LF Helm chart for the LF Helm deployment (it is a data deployment, has no Grafana, and uses ElasticSearch in addition to Postgres to store data).

Please see GraphQL Helm chart for GraphQL foundation DevStats deployment.

Please see Kubernetes dashboard if you want to enable a local dashboard to explore the cluster state.

Please see bare metal example to see an example of bare metal deployment.

The rest of this document describes the current bare metal deployment on metal.equinix.com used by CNCF projects.

Presentations

  • Presentations are available here.
  • Direct link.
  • Another direct link.

Talks

Architecture

DevStats is deployed using Helm on Kubernetes running on bare metal servers provided by Equinix.

DevStats is written in Go, it uses GitHub archives, GitHub API and git as its main data sources.

Under the hood, DevStats uses the following CNCF projects:

  • Helm (for deployment).
  • containerd (as a Kubernetes container runtime, CRI).
  • cert-manager (for HTTPS/SSL certificates).
  • OpenEBS (for local storage volumes support).
  • MetalLB (as a load balancer for bare metal servers).
  • CoreDNS (Kubernetes internal DNS).

And other projects, including:

  • Equinix (bare metal servers provider).
  • Ubuntu (containers base operating system).
  • kubeadm (for installing Kubernetes).
  • NFS (for shared write network volumes support).
  • NGINX (for ingress).
  • Calico (as networking for Kubernetes, CNI).
  • Golang (DevStats is written in Go).
  • PostgreSQL (DevStats database is Postgres).
  • patroni (HA deployment of PostgreSQL database, tweaked for DevStats).
  • GitHub archives (main data source).
  • GitHub API (data source).
  • git (data source).
  • Grafana (UI).
  • Let's Encrypt (provides HTTPS/SSL certificates).
  • Travis CI (continuous integration & testing).

Please check this for a detailed architecture description.

Deploying on your own project(s)

See the simple DevStats example repository for single project deployment (Homebrew), follow instructions to deploy for your own project.

Goal

We want to create a toolset for visualizing various metrics for the Kubernetes community (and also for all CNCF projects).

Everything is open source so that it can be used by other CNCF and non-CNCF open source projects.

The only requirement is that project must be hosted on a public GitHub repository/repositories.

Data hiding

If you want to hide your data (replace with anon-#) please follow the instructions here.

Forking and installing locally

This toolset uses only Open Source tools: GitHub archives, GitHub API, git, Postgres databases, and multiple Grafana instances.
It is written in Go and can be forked and installed by anyone.

Contributions and PRs are welcome.
If you see a bug or want to add a new metric please create an issue and/or PR.

To work on this project locally please fork the original repository, and:

Please see Development for local development guide.

For more detailed description of all environment variables, tools, switches, etc, please see Usage.

Metrics

We want to support all kinds of metrics, including historical ones.
Please see requested metrics to see what kind of metrics are needed.
Many of them cannot be computed based on the data sources currently used.

Repository groups

There are some groups of repositories that are grouped together as a repository groups.
They are defined in scripts/kubernetes/repo_groups.sql.

To setup default repository groups:

  • PG_PASS=pwd ./kubernetes/setup_repo_groups.sh.

This is a part of kubernetes/psql.sh script and kubernetes psql dump already has groups configured.

In an All CNCF project repository groups are mapped to individual CNCF projects scripts/all/repo_groups.sql:

Company Affiliations

We also want to have per company statistics. To implement such metrics we need a mapping of developers and their employers.

There is a project that attempts to create such mapping cncf/gitdm.

DevStats has an import tool that fetches company affiliations from cncf/gitdm and allows to create per company metrics/statistics. It also uses companies.yaml file to map company acquisitions (any data generated by a company acquired by another company is assigned to the latter using a mapping from companies.yaml).

If you see errors in the company affiliations, please open a pull request on cncf/gitdm and the updates will be reflected on https://k8s.devstats.cncf.io a couple of days after the PR has been accepted. Note that gitdm supports mapping based on dates, to account for developers moving between companies.

New affiliations are imported into DevStats about 1-2 times/month.

Architecture

For architecture details please see architecture file.

Detailed usage is here

Adding new metrics

Please see metrics to see how to add new metrics.

Adding new projects

To add a new project on a bare metal deployment follow adding new project instructions.

See cncf/devstats-helm:ADDING_NEW_PROJECTS.md for information about how to add more projects on Kubernetes/Helm deployment.

Grafana dashboards

Please see dashboards to see a list of already defined Grafana dashboards.

Exporting data

Please see exporting.

Detailed Usage instructions

Servers

The servers to run devstats are generously provided by Equinix bare metal hosting as part of CNCF's Community Infrastructure Lab.

One line run all projects

  • Use GHA2DB_PROJECTS_OVERRIDE="+cncf" PG_PASS=pwd devstats.
  • Or add this command using crontab -e to run every hour HH:08.

Checking projects activity

  • Use: PG_PASS=... PG_DB=allprj ./devel/activity.sh '1 month,,' > all.txt.
  • Example results here - all CNCF project activity during January 2018, excluding bots.

主要指標

概覽
名稱與所有者cncf/devstats.archive
主編程語言Shell
編程語言Shell (語言數: 8)
平台
許可證Apache License 2.0
所有者活动
創建於2017-07-29 21:11:44
推送於2023-03-29 13:08:59
最后一次提交2023-03-29 13:08:50
發布數8
最新版本名稱v0.7.0 (發布於 )
第一版名稱v0.1.0 (發布於 )
用户参与
星數443
關注者數27
派生數147
提交數7.6k
已啟用問題?
問題數346
打開的問題數1
拉請求數36
打開的拉請求數0
關閉的拉請求數12
项目设置
已啟用Wiki?
已存檔?
是復刻?
已鎖定?
是鏡像?
是私有?