Titus Control Plane
Overview
Titus is the Netflix Container Management Platform that manages containers and provides integrations to the infrastructure
ecosystem. This repository contains the control plane components which are responsible for accepting job requests and
scheduling containers on agents.
Documentation & Getting Started
Building and Testing
For reference, check some of the scripts commonly used by committers in the scripts/dev/ folder.
Building
./gradlew build
Run Tests
./gradlew test
Run All Tests (including integration)
./gradlew testAll
Extensions
There are several extensions in the titus-ext folder for integrations with various systems. In order to use
these extensions, a wrapper project that reconfigures the guice bindings is needed. A tutorial project for binding
the different implementations is coming soon.
Local testing with docker-compose
docker-compose together with docker-engine
can be used to stand up a local cluster with all components necessary to run titus containers. Each component
(titus-master, titus-gateway, mesos-master, zookeeper, titus-agent) will run as a separate docker container, and Titus
containers will be launched as nested containers (docker-in-docker) inside the agent container.
The last versions known to work:
- docker-engine
18.06.0-ce - docker-compose
1.22.0
To build and launch all components:
docker-compose build
docker-compose up -d
Ports 7001 (http) and 7104 (grpc) from titus-gateway will be exposed locally:
curl localhost:7001/api/v2/status
curl localhost:7001/api/v2/leader
Logs can be inspected with docker-compose logs. For agent logs, journald can be
used in a shell inside the agent container:
docker-compose exec agent bash
(agent) $ journalctl
# Note: some warnings and errors are to be expected on those logs
# Not all systemd units will work when not on an EC2 VM
# list nested docker containers launched by Titus
(agent) $ docker ps
By default, all titus-agent containers will join a cluster named unknown-instanceGroup.
Before any tasks can be scheduled, that cluster needs to be activated. Note that
this is necessary every time the Titus master is restarted, since instanceGroup
state is not being persisted:
curl localhost:7001/api/v3/agent/instanceGroups/unknown-instanceGroup/lifecycle \
-X PUT -H "Content-type: application/json" -d \
'{"instanceGroupId": "unknown-instanceGroup", "lifecycleState": "Active"}'
If you get an error saying that unknown-instanceGroup does not exist, wait 10s and try again. Agents take some time to
get registered.
After that, jobs can be submitted normally. AWS integrations will not work in this mode:
curl localhost:7001/api/v3/jobs \
-X POST -H "Content-type: application/json" -d \
'{
"applicationName": "localtest",
"owner": {"teamEmail": "me@me.com"},
"container": {
"image": {"name": "alpine", "tag": "latest"},
"entryPoint": ["/bin/sleep", "1h"],
"securityProfile": {"iamRole": "test-role", "securityGroups": ["sg-test"]}
},
"batch": {
"size": 1,
"runtimeLimitSec": "3600",
"retryPolicy":{"delayed": {"delayMs": "1000", "retries": 3}}
}
}'
The status of the new job, and its task (container) can be queried with:
curl localhost:7001/api/v3/jobs
curl localhost:7001/api/v3/tasks
# to delete a task or a job:
curl localhost:7001/api/v3/jobs/:id -X DELETE
curl localhost:7001/api/v3/tasks/:id -X DELETE
Components can also be built and deployed individually. E.g.:
# build and deploy an agent
docker-compose build agent
docker-compose up -d agent
# build and deploy titus-master
docker-compose build master
docker-compose up -d master
To add more agents to the cluster:
docker-compose up -d --scale agent=2
Note that it can take ~10s for a new titus-agent to be detected and registered with the default configuration.
After you are done:
# tear everything down
docker-compose down
# or only stop titus-master
docker-compose stop master
LICENSE
Copyright (c) 2018 Netflix, Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.