Chaos Mesh

Note:

This readme and related documentation are Work in Progress.

Chaos Mesh is a cloud-native Chaos Engineering platform that orchestrates chaos on Kubernetes environments. At the current stage, it has the following components:

• Chaos Operator: the core component for chaos orchestration. Fully open sourced.
• Chaos Dashboard: a visualized panel that shows the impacts of chaos experiments on the online services of the system; under development; curently only supports chaos experiments on TiDB(https://github.com/pingcap/tidb).

See the following demo video for a quick view of Chaos Mesh:

Chaos Operator

Chaos Operator injects chaos into the applications and Kubernetes infrastructure in a manageable way, which provides easy, custom definitions for chaos experiments and automatic orchestration. There are three components at play:

Controller-manager: used to schedule and manage the lifecycle of CRD objects

Chaos-daemon: runs as daemonset with privileged system permissions over network, Cgroup, etc. for a specifc node

Sidecar: a special type of container that is dynamically injected into the target Pod by the webhook-server, which can be used for hijacking I/O of the application container.

Chaos Operator uses Custom Resource Definition (CRD) to define chaos objects. The current implementation supports three types of CRD objects for fault injection, namely PodChaos, NetworkChaos, and IOChaos, which correspond to the following major actions (experiments):

• pod-kill: The selected pod is killed (ReplicaSet or something similar may be needed to ensure the pod will be restarted).
• pod-failure: The selected pod will be unavailable in a specified period of time.
• netem chaos: Network chaos such as delay, duplication, etc.
• network-partition: Simulate network partition.
• IO chaos: Simulate file system faults such as I/O delay, read/write errors, etc.

Prerequisites

Before deploying Chaos Mesh, make sure the following items have been installed. If you would like to have a try on your machine, you can refer to get-started-on-your-local-machine section.

• Kubernetes >= v1.12
• RBAC enabled (optional)
• Helm version >= v2.8.2

Deploy Chaos Mesh

Get the Helm files

git clone https://github.com/pingcap/chaos-mesh.git
cd chaos-mesh/

Create custom resource type

To use Chaos Mesh, you must first create the related custom resource type.

kubectl apply -f manifests/
kubectl get crd podchaos.pingcap.com

Install Chaos Mesh

• Install Chaos Mesh with Chaos Operator only in docker environment

# create namespace chaos-testing
kubectl create ns chaos-testing
# helm 2.X
helm install helm/chaos-mesh --name=chaos-mesh --namespace=chaos-testing
# helm 3.X
helm install chaos-mesh helm/chaos-mesh --namespace=chaos-testing
# check Chaos Mesh pods installed
kubectl get pods --namespace chaos-testing -l app.kubernetes.io/instance=chaos-mesh

• Install Chaos Mesh with Chaos Operator only in containerd environment (Kind)

# create namespace chaos-testing
kubectl create ns chaos-testing
# helm 2.X
helm install helm/chaos-mesh --name=chaos-mesh --namespace=chaos-testing --set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock
# helm 3.X
helm install chaos-mesh helm/chaos-mesh --namespace=chaos-testing --set chaosDaemon.runtime=containerd --set chaosDaemon.socketPath=/run/containerd/containerd.sock
# check Chaos Mesh pods installed
kubectl get pods --namespace chaos-testing -l app.kubernetes.io/instance=chaos-mesh

• Install Chaos Mesh with Chaos Operator and Chaos Dashboard

# helm 2.X
helm install helm/chaos-mesh --name=chaos-mesh --namespace=chaos-testing --set dashboard.create=true
# helm 3.X
helm install chaos-mesh helm/chaos-mesh --namespace=chaos-testing --set dashboard.create=true

Get started on your local machine

Warning:

This deployment is for testing only. DO NOT USE in production!

You can try Chaos Mesh on your local K8s environment deployed using kind or minikube.

Deploy your local K8s environment

Deploy with kind

1. Clone the code

   git clone --depth=1 https://github.com/pingcap/chaos-mesh && \
   cd chaos-mesh
   

2. Run the script and create a local Kubernetes cluster. Make sure you have installed kind.

   hack/kind-cluster-build.sh
   

3. To connect the local Kubernetes cluster, set the default configuration file path of kubectl to kube-config.

   export KUBECONFIG="$(kind get kubeconfig-path)"
   

4. Verify whether the Kubernetes cluster is on and running

   kubectl cluster-info
   

5. Install chaos-mesh on kind kubernetes cluster as suggested in Install Chaos Mesh.

Deploy with minikube

1. Start a minikube kubernetes cluster. Make sure you have installed minikube.

   minikube start --kubernetes-version v1.15.0 --cpus 4 --memory "8192mb" # we recommend that you allocate enough RAM (more than 8192 MiB) to the VM
   

2. Install helm

   curl https://raw.githubusercontent.com/helm/helm/master/scripts/get, bash
   helm init
   

3. Check whether helm tiller pod is running

   kubectl -n kube-system get pods -l app=helm
   

4. Install chaos-mesh on minikube kubernetes cluster as suggested in Install Chaos Mesh.

Note:

There are some known restrictions for Chaos Operator deployed on minikube clusters:

• netem chaos is only supported for minikube clusters >= version 1.6.

  In minikube, the default virtual machine driver's image doesn't contain the sch_netem kernel module in smaller versions. You can use none driver (if your host is Linux with the sch_netem kernel module loaded) to try these chaos actions on minikube or build an image with sch_netem by yourself.

Deploy target cluster

After Chaos Mesh is deployed, we can deploy the target cluster to be tested, or where we want to inject faults. For illustration purposes, we use TiDB as our sample cluster.

You can follow the instructions in the following two documents to deploy a TiDB cluster:

• Deploy using kind
• Deploy using minikube

Define chaos experiment config file

The chaos experiement configuration is defined in a .yaml file. The following sample file (pod-kill-example.yaml) defines a chaos experiment to kill one tikv pod randomly every 60 seconds:

apiVersion: pingcap.com/v1alpha1
kind: PodChaos
metadata:
  name: pod-failure-example
  namespace: chaos-testing
spec:
  action: pod-failure # the specific chaos action to inject; supported actions: pod-kill/pod-failure
  mode: one # the mode to run chaos action; supported modes are one/all/fixed/fixed-percent/random-max-percent
  duration: "60s" # duration for the injected chaos experiment
  selector: # pods where to inject chaos actions
    namespaces:
      - tidb-cluster-demo  # the namespace of the system under test (SUT) you've deployed
    labelSelectors:
      "app.kubernetes.io/component": "tikv" # the label of the pod for chaos injection
  scheduler: # scheduler rules for the running time of the chaos experiments about pods.
    cron: "@every 5m"

Create a chaos experiment

kubectl apply -f pod-failure-example.yaml
kubectl get podchaos --namespace=chaos-testing

You can see the QPS performance (by running a benchmark against the cluster affected by the chaos experiment from TiDB Grafana dashboard:

Update a chaos experiment

vim pod-failure-example.yaml # modify pod-failure-example.yaml to what you want
kubectl apply -f pod-failure-example.yaml

Delete a chaos experiment

kubectl delete -f pod-failure-example.yaml

Watch your chaos experiments in Dashboard

Chaos Dashboard is currently only available for TiDB clusters. Stay tuned for more supports or join us in making it happen.

Note:

If Chaos Dashboard was not installed in your earlier deployment, you need to install it by upgrading Chaos Mesh:

helm upgrade chaos-mesh helm/chaos-mesh --namespace=chaos-testing --set dashboard.create=true

A typical way to access it is to use kubectl port-forward

kubectl port-forward -n chaos-testing svc/chaos-dashboard 8080:80

Then you can access http://localhost:8080 in browser.

FAQs

Refer to FAQs.

Community

Please reach out for bugs, feature requests, and other issues via:

• Following us on Twitter at @chaos_mesh.
• The #sig-chaos-mesh channel in the TiDB Community slack workspace.
• Filing a issue or opening a PR against this repo.

Roadmap

See ROADMAP

License

Chaos Mesh is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.