llbuild

A low-level build system.

llbuild is a set of libraries for building build systems. Unlike most build
system projects which focus on the syntax for describing the build, llbuild is
designed around a reusable, flexible, and scalable general purpose build
engine capable of solving many "build system"-like problems. The project also
includes additional libraries on top of that engine which provide support for
constructing bespoke build systems (like swift build) or for building from
Ninja manifests.

llbuild currently includes:

• A flexible core engine capable of discovering new work on the fly.

• Scalability for dependency graphs reaching millions of nodes.

• Support for building Ninja manifests (e.g., for building LLVM, Clang, and
  Swift).

• An llbuild-native build description format designed for extensibility.

• Library-based design intended to support embedding and reuse.

Usage

The project currently produces three top-level products; llbuild, swift-build-tool,
and libllbuild / llbuild.framework.

llbuild Command Line Tool

The llbuild tool provides a command line interface to various feature of the
llbuild libraries. It has several subtools available, see llbuild --help for
more information. The most important subtool is the Ninja build support:

Ninja Build Support

You can use llbuild to build Ninja-based projects using:

$ llbuild ninja build

This tool supports a subset of the command line arguments supported by Ninja
itself, to allow it to be used as a compatible replacement, even by tools like
CMake that depend on particular Ninja command line flags during their
configuration process.

As a convenience, if you invoke llbuild via a symlink called ninja then it
will automatically use this subtool. This supports installing llbuild as ninja
into your PATH and then using it as an alternative to Ninja for building
arbitrary projects (like LLVM, Clang, and Swift). This is also how we self-host
llbuild (via the CMake Ninja generator).

The llbuild ninja subtool also provides additional commands which are
primarily only useful for developers interested in working on the Ninja
support. These commands allow testing the lexer, parser, and manifest loading
components independently and are used as part of the test suite.

If you want to rep your use of llbuild in public, you can proudly leverage our
conveniently provided sticker (PSD version adjacent). ?

Build Trace Files

Inspired by Buck, llbuild ninja supports a --profile PATH option to generate
a
Chromium trace for
visualizing where time is spent during a build. For example, the following graph
is for a build of llbuild itself:

swift-build-tool Command Line Tool

The swift-build-tool product is the command line interface to the build system
used by the Swift Package Manager. It is
built as part of the Swift project build and incorporated
into the Swift language snapshots.

This tool is built on top of the BuildSystem library.

libllbuild Library

The libllbuild library exposes a C API for the llbuild libraries, which can be
used directly by third-parties or to build additional language bindings. See
bindings for example Swift and Python bindings that use this
library.

This API is what is used, for example, in Xcode as the basis for the new build
system introduced in Xcode 9.

Motivation

The design of llbuild is a continuation of the LLVM philosophy of applying
library-based design to traditional developer tools.
Clang has followed this approach to deliver a high
performance compiler and assembler while also enabling new tools like
clang-format or the libclang interfaces for code completion and
indexing. However, the rigid command line interface between traditional build
systems and the compiler still limits the optimizations and features which can
be implemented in Clang.

llbuild is designed to allow construction of more feature rich build
environments which integrate external tools -- like the compiler -- using APIs
instead of command line interfaces. By allowing the build system and tools to
communicate directly and to be co-designed, we believe we can unlock additional
optimization opportunities and create more robust, easy-to-use build systems.

For more information, see
A New Architecture for Building Software
from the 2016 LLVM Developer's Conference.

Philosophy

In the abstract, build systems are used to perform a task while also being:

• Incremental: Outputs should be efficiently rebuilt given a small change to the
  inputs, by leveraging the ability to save partial outputs from a prior build.

• Consistent: Equivalent inputs should always produce the same result as building from clean.

• Persistent: Results should be stored so that builds can be interrupted and
  resumed after failure without needing to redo the full computation.

• Parallel & Efficient: It must be possible to perform independent elements of
  the computation in parallel, in order to compute the result as efficiently as
  possible.

When viewed in this light, it is clear that the core technology of a build
system is applicable to any complex, long-running computation in which it is
common for the user to only modify a small portion of the input before wanting
the recompute the result. For example, a movie editor application will commonly
need to rerender small portions of the overall movie in response to interactive
edits in order to support preview of the final result. However, such
applications frequently do not take full advantage of the ability to store and
partially recompute the results because of the complexity of correctly managing
the dependencies between parts of the computation.

Part of the goal in designing llbuild around a general purpose build engine is
to allow its use in contexts which are not traditionally thought of as requiring
a "build system".

Documentation

Technical documentation is available at
llbuild.readthedocs.io.

Bug Reports

Bug reports should be filed in the Swift OSS Jira in
the llbuild component.

Open Projects

llbuild is a work in progress. Some of the more significant open projects which
we hope to tackle are:

• Support for using file signatures instead of timestamps for change detection.

• Support richer data types for communication between tasks.

  Tasks currently only compute a single scalar value as their result. We
  would like to support richer data types for tasks results, for example tasks
  should be able to compute sets of results, and have the engine automatically
  communicate the addition or removal of individual items in the set to
  downstream consumers.

• Support a more sophisticated database implementation.

  The current implementation uses a SQLite3 database for storing build
  results. This was a pragmatic choice for bring up, but it can be a performance
  bottleneck for some applications, and we do not need the flexibility of a full
  SQL database. We would like to evaluate the tradeoffs of designing a custom
  solution for llbuild.

• Support transparent distributed builds.

  We would like llbuild to have facilities for transparently distributing a
  build across an array of worker machines.

• Support automatic auditing of build consistency.

  Few build systems diagnose problems effectively. Frequently, undeclared inputs
  or misbehaving tools can cause inconsistent build results. We would like
  llbuild to automatically diagnose these problems, for example by periodically or
  speculatively rebuilding items which are not expected to have changed and
  comparing the results.

• Performance tuning of core engine queues.

  The core build engine does its work using a number of queues of work items,
  and locking for the subset which support concurrent manipulation. We would
  like to investigate moving the shared queues to using a lock-free data
  structure and to micro-optimize the queues in general, in order to support
  very fine-grained task subdivisions without negatively impacting performance.

FAQ

Q. Why does llbuild include some parts of LLVM?

A. As a low-level, embeddable component, we want llbuild itself to have a
simple build process without any significant build time dependencies. However,
we also wanted to take advantage of some of the data structures and support
facilities that have been developed for LLVM. For now, our solution is to
incorporate some parts of LLVM's Support libraries into the repository, with the
hope that over time LLVM will either factor out those libraries in a way that
makes it easier to reuse them, or that we will develop our own exclusive set of
support data structures and utilities and drop use of the LLVM ones.

Q. Why does llbuild include Ninja support?

A. llbuild includes a Ninja compatibility layer which allows building projects
which use Ninja manifests using the llbuild core engine. We developed this
support as a proof of concept for the core engine, and as a way to bootstrap
ourselves (we develop llbuild using the CMake Ninja generator and llbuild to
build itself). This support is also valuable for allowing direct benchmarking
comparisons of llbuild.

Our implementation of Ninja support also includes a separate library for
programmatically loading Ninja manifests, which may prove useful to other
projects wishing to use or manipulate Ninja files.

We intend to continue to maintain the Ninja support to keep compatibility with
the main project.

Acknowledgements

llbuild is heavily influenced by modern build systems like
Shake, Buck, and
Ninja. We would particularly like to thank Neil
Mitchell for his work describing the Shake algorithm which provided the
inspiration for the mechanism llbuild uses to allow additional work to be
discovered on the fly.

For similar projects, see Adapton
and Salsa.

License

Copyright (c) 2014 - 2018 Apple Inc. and the Swift project authors.
Licensed under Apache License v2.0 with Runtime Library Exception.

See http://swift.org/LICENSE.txt for license information.

See http://swift.org/CONTRIBUTORS.txt for Swift project authors.