Commit a54dfc73 authored by Adam Simpkins's avatar Adam Simpkins

Include fbcode_builder sources in the folly github repository

Summary:
Include the fbcode_builder sources in the folly repository under
build/fbcode_builder/.  These build scripts will make it easier. to set
up continuous integration for the opensource github repository.

Note that for folly this is potentially slightly confusing, since this
will now add a top-level build directory in addition to the existing
folly/build subdirectory.  However, it seems easiest for now to follow
the default fbcode_builder conventions.

This does not yet include configuration files for the folly build--I
will add those in a subsequent diff.

Test Plan:
Confirmed the CI build work with the upcoming configuration files.

Reviewed By: yfeldblum

Differential Revision: D6700052

fbshipit-source-id: ba026835e866bda5eed0776ec8bbf1ac51014034
parent e8696645
# Facebook projects that use `fbcode_builder` for continuous integration
# share this Travis configuration to run builds via Docker.
sudo: required
env:
global:
- travis_cache_dir=$HOME/travis_ccache
# Travis times out after 50 minutes. Very generously leave 10 minutes
# for setup (e.g. cache download, compression, and upload), so we never
# fail to cache the progress we made.
- docker_build_timeout=40m
cache:
# Our build caches can be 200-300MB, so increase the timeout to 7 minutes
# to make sure we never fail to cache the progress we made.
timeout: 420
directories:
- $HOME/travis_ccache # see docker_build_with_ccache.sh
# Ugh, `services:` must be in the matrix, or we get `docker: command not found`
# https://github.com/travis-ci/travis-ci/issues/5142
matrix:
include:
- env: ['os_image=ubuntu:14.04', gcc_version=4.9]
services: [docker]
# Unlike 14.04, this Debian Stable has 4.9 as the system compiler, so
# there is no risk of 4.8/4.9 ABI incompatibilities.
- env: ['os_image=debian:8.6', gcc_version=4.9]
services: [docker]
- env: ['os_image=ubuntu:16.04', gcc_version=5]
services: [docker]
script:
# Travis seems to get confused when `matrix:` is used with `language:`
- sudo apt-get install python2.7
# We don't want to write the script inline because of Travis kludginess --
# it looks like it escapes " and \ in scripts when using `matrix:`.
- ./build/fbcode_builder/travis_docker_build.sh
notifications:
webhooks: https://code.facebook.com/travis/webhook/
# Facebook-internal CI builds don't have write permission outside of the
# source tree, so we install all projects into this directory.
/facebook_ci
## Debugging Docker builds
To debug a a build failure, start up a shell inside the just-failed image as
follows:
```
docker ps -a | head # Grab the container ID
docker commit CONTAINER_ID # Grab the SHA string
docker run -it SHA_STRING /bin/bash
# Debug as usual, e.g. `./run-cmake.sh Debug`, `make`, `apt-get install gdb`
```
## A note on Docker security
While the Dockerfile generated above is quite simple, you must be aware that
using Docker to run arbitrary code can present significant security risks:
- Code signature validation is off by default (as of 2016), exposing you to
man-in-the-middle malicious code injection.
- You implicitly trust the world -- a Dockerfile cannot annotate that
you trust the image `debian:8.6` because you trust a particular
certificate -- rather, you trust the name, and that it will never be
hijacked.
- Sandboxing in the Linux kernel is not perfect, and the builds run code as
root. Any compromised code can likely escalate to the host system.
Specifically, you must be very careful only to add trusted OS images to the
build flow.
Consider setting this variable before running any Docker container -- this
will validate a signature on the base image before running code from it:
```
export DOCKER_CONTENT_TRUST=1
```
Note that unless you go through the extra steps of notarizing the resulting
images, you will have to disable trust to enter intermediate images, e.g.
```
DOCKER_CONTENT_TRUST= docker run -it YOUR_IMAGE_ID /bin/bash
```
# Easy builds for Facebook projects
This is a Python 2.6+ library designed to simplify continuous-integration
(and other builds) of Facebook projects.
For external Travis builds, the entry point is `travis_docker_build.sh`.
## Using Docker to reproduce a CI build
If you are debugging or enhancing a CI build, you will want to do so from
host or virtual machine that can run a reasonably modern version of Docker:
``` sh
./make_docker_context.py --help # See available options for OS & compiler
# Tiny wrapper that starts a Travis-like build with compile caching:
os_image=ubuntu:14.04 \
gcc_version=4.9 \
make_parallelism=2 \
travis_cache_dir=~/travis_ccache \
./travis_docker_build.sh &> build_at_$(date +'%Y%m%d_%H%M%S').log
```
**IMPORTANT**: Read `fbcode_builder/README.docker` before diving in!
Setting `travis_cache_dir` turns on [ccache](https://ccache.samba.org/),
saving a fresh copy of `ccache.tgz` after every build. This will invalidate
Docker's layer cache, foring it to rebuild starting right after OS package
setup, but the builds will be fast because all the compiles will be cached.
To iterate without invalidating the Docker layer cache, just `cd
/tmp/docker-context-*` and interact with the `Dockerfile` normally. Note
that the `docker-context-*` dirs preserve a copy of `ccache.tgz` as they
first used it.
# What to read next
The *.py files are fairly well-documented. You might want to peruse them
in this order:
- shell_quoting.py
- fbcode_builder.py
- docker_builder.py
- make_docker_context.py
As far as runs on Travis go, the control flow is:
- .travis.yml calls
- travis_docker_build.sh calls
- docker_build_with_ccache.sh
This library also has an (unpublished) component targeting Facebook's
internal continuous-integration platform using the same build-step DSL.
# Contributing
Please follow the ambient style (or PEP-8), and keep the code Python 2.6
compatible -- since `fbcode_builder`'s only dependency is Docker, we want to
allow building projects on even fairly ancient base systems.
#!/bin/bash -uex
set -o pipefail # Be sure to `|| :` commands that are allowed to fail.
#
# Future: port this to Python if you are making significant changes.
#
# Parse command-line arguments
build_timeout="" # Default to no time-out
print_usage() {
echo "Usage: $0 [--build-timeout TIMEOUT_VAL] SAVE-CCACHE-TO-DIR"
echo "SAVE-CCACHE-TO-DIR is required. An empty string discards the ccache."
}
while [[ $# -gt 0 ]]; do
case "$1" in
--build-timeout)
shift
build_timeout="$1"
if [[ "$build_timeout" != "" ]] ; then
timeout "$build_timeout" true # fail early on invalid timeouts
fi
;;
-h|--help)
print_usage
exit
;;
*)
break
;;
esac
shift
done
# There is one required argument, but an empty string is allowed.
if [[ "$#" != 1 ]] ; then
print_usage
exit 1
fi
save_ccache_to_dir="$1"
if [[ "$save_ccache_to_dir" != "" ]] ; then
mkdir -p "$save_ccache_to_dir" # fail early if there's nowhere to save
else
echo "WARNING: Will not save /ccache from inside the Docker container"
fi
rand_guid() {
echo "$(date +%s)_${RANDOM}_${RANDOM}_${RANDOM}_${RANDOM}"
}
id=fbcode_builder_image_id=$(rand_guid)
logfile=$(mktemp)
echo "
Running build with timeout '$build_timeout', label $id, and log in $logfile
"
if [[ "$build_timeout" != "" ]] ; then
# Kill the container after $build_timeout. Using `/bin/timeout` would cause
# Docker to destroy the most recent container and lose its cache.
(
sleep "$build_timeout"
echo "Build timed out after $build_timeout" 1>&2
while true; do
maybe_container=$(
egrep '^( ---> Running in [0-9a-f]+|FBCODE_BUILDER_EXIT)$' "$logfile" |
tail -n 1 | awk '{print $NF}'
)
if [[ "$maybe_container" == "FBCODE_BUILDER_EXIT" ]] ; then
echo "Time-out successfully terminated build" 1>&2
break
fi
echo "Time-out: trying to kill $maybe_container" 1>&2
# This kill fail if we get unlucky, try again soon.
docker kill "$maybe_container" || sleep 5
done
) &
fi
build_exit_code=0
# `docker build` is allowed to fail, and `pipefail` means we must check the
# failure explicitly.
if ! docker build --label="$id" . 2>&1 | tee "$logfile" ; then
build_exit_code="${PIPESTATUS[0]}"
# NB: We are going to deliberately forge ahead even if `tee` failed.
# If it did, we have a problem with tempfile creation, and all is sad.
echo "Build failed with code $build_exit_code, trying to save ccache" 1>&2
fi
# Stop trying to kill the container.
echo $'\nFBCODE_BUILDER_EXIT' >> "$logfile"
if [[ "$save_ccache_to_dir" == "" ]] ; then
echo "Not inspecting Docker build, since saving the ccache wasn't requested."
exit "$build_exit_code"
fi
img=$(docker images --filter "label=$id" -a -q)
if [[ "$img" == "" ]] ; then
docker images -a
echo "In the above list, failed to find most recent image with $id" 1>&2
# Usually, the above `docker kill` will leave us with an up-to-the-second
# container, from which we can extract the cache. However, if that fails
# for any reason, this loop will instead grab the latest available image.
#
# It's possible for this log search to get confused due to the output of
# the build command itself, but since our builds aren't **trying** to
# break cache, we probably won't randomly hit an ID from another build.
img=$(
egrep '^ ---> (Running in [0-9a-f]+|[0-9a-f]+)$' "$logfile" | tac |
sed 's/Running in /container_/;s/ ---> //;' | (
while read -r x ; do
# Both docker commands below print an image ID to stdout on
# success, so we just need to know when to stop.
if [[ "$x" =~ container_.* ]] ; then
if docker commit "${x#container_}" ; then
break
fi
elif docker inspect --type image -f '{{.Id}}' "$x" ; then
break
fi
done
)
)
if [[ "$img" == "" ]] ; then
echo "Failed to find valid container or image ID in log $logfile" 1>&2
exit 1
fi
elif [[ "$(echo "$img" | wc -l)" != 1 ]] ; then
# Shouldn't really happen, but be explicit if it does.
echo "Multiple images with label $id, taking the latest of:"
echo "$img"
img=$(echo "$img" | head -n 1)
fi
container_name="fbcode_builder_container_$(rand_guid)"
echo "Starting $container_name from latest image of the build with $id --"
echo "$img"
# ccache collection must be done outside of the Docker build steps because
# we need to be able to kill it on timeout.
#
# This step grows the max cache size to slightly exceed than the working set
# of a successful build. This simple design persists the max size in the
# cache directory itself (the env var CCACHE_MAXSIZE does not even work with
# older ccaches like the one on 14.04).
#
# Future: copy this script into the Docker image via Dockerfile.
(
# By default, fbcode_builder creates an unsigned image, so the `docker
# run` below would fail if DOCKER_CONTENT_TRUST were set. So we unset it
# just for this one run.
export DOCKER_CONTENT_TRUST=
# CAUTION: The inner bash runs without -uex, so code accordingly.
docker run --user root --name "$container_name" "$img" /bin/bash -c '
build_exit_code='"$build_exit_code"'
# Might be useful if debugging whether max cache size is too small?
grep " Cleaning up cache directory " /tmp/ccache.log
export CCACHE_DIR=/ccache
ccache -s
echo "Total bytes in /ccache:";
total_bytes=$(du -sb /ccache | awk "{print \$1}")
echo "$total_bytes"
echo "Used bytes in /ccache:";
used_bytes=$(
du -sb $(find /ccache -type f -newermt @$(
cat /FBCODE_BUILDER_CCACHE_START_TIME
)) | awk "{t += \$1} END {print t}"
)
echo "$used_bytes"
# Goal: set the max cache to 750MB over 125% of the usage of a
# successful build. If this is too small, it takes too long to get a
# cache fully warmed up. Plus, ccache cleans 100-200MB before reaching
# the max cache size, so a large margin is essential to prevent misses.
desired_mb=$(( 750 + used_bytes / 800000 )) # 125% in decimal MB: 1e6/1.25
if [[ "$build_exit_code" != "0" ]] ; then
# For a bad build, disallow shrinking the max cache size. Instead of
# the max cache size, we use on-disk size, which ccache keeps at least
# 150MB under the actual max size, hence the 400MB safety margin.
cur_max_mb=$(( 400 + total_bytes / 1000000 )) # ccache uses decimal MB
if [[ "$desired_mb" -le "$cur_max_mb" ]] ; then
desired_mb=""
fi
fi
if [[ "$desired_mb" != "" ]] ; then
echo "Updating cache size to $desired_mb MB"
ccache -M "${desired_mb}M"
ccache -s
fi
# Subshell because `time` the binary may not be installed.
if (time tar czf /ccache.tgz /ccache) ; then
ls -l /ccache.tgz
else
# This `else` ensures we never overwrite the current cache with
# partial data in case of error, even if somebody adds code below.
rm /ccache.tgz
exit 1
fi
'
)
echo "Updating $save_ccache_to_dir/ccache.tgz"
# This will not delete the existing cache if `docker run` didn't make one
docker cp "$container_name:/ccache.tgz" "$save_ccache_to_dir/"
# Future: it'd be nice if Travis allowed us to retry if the build timed out,
# since we'll make more progress thanks to the cache. As-is, we have to
# wait for the next commit to land.
echo "Build exited with code $build_exit_code"
exit "$build_exit_code"
#!/usr/bin/env python
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
'''
Extends FBCodeBuilder to produce Docker context directories.
In order to get the largest iteration-time savings from Docker's build
caching, you will want to:
- Use fine-grained steps as appropriate (e.g. separate make & make install),
- Start your action sequence with the lowest-risk steps, and with the steps
that change the least often, and
- Put the steps that you are debugging towards the very end.
'''
import logging
import os
import shutil
import tempfile
from fbcode_builder import FBCodeBuilder
from shell_quoting import (
raw_shell, shell_comment, shell_join, ShellQuoted
)
from utils import recursively_flatten_list, run_command
class DockerFBCodeBuilder(FBCodeBuilder):
def _user(self):
return self.option('user', 'root')
def _change_user(self):
return ShellQuoted('USER {u}').format(u=self._user())
def setup(self):
# Please add RPM-based OSes here as appropriate.
#
# To allow exercising non-root installs -- we change users after the
# system packages are installed. TODO: For users not defined in the
# image, we should probably `useradd`.
return self.step('Setup', [
# Docker's FROM does not understand shell quoting.
ShellQuoted('FROM {}'.format(self.option('os_image'))),
# /bin/sh syntax is a pain
ShellQuoted('SHELL ["/bin/bash", "-c"]'),
] + self.install_debian_deps() + [self._change_user()])
def step(self, name, actions):
assert '\n' not in name, 'Name {0} would span > 1 line'.format(name)
b = ShellQuoted('')
return [ShellQuoted('### {0} ###'.format(name)), b] + actions + [b]
def run(self, shell_cmd):
return ShellQuoted('RUN {cmd}').format(cmd=shell_cmd)
def workdir(self, dir):
return [
# As late as Docker 1.12.5, this results in `build` being owned
# by root:root -- the explicit `mkdir` works around the bug:
# USER nobody
# WORKDIR build
ShellQuoted('USER root'),
ShellQuoted('RUN mkdir -p {d} && chown {u} {d}').format(
d=dir, u=self._user()
),
self._change_user(),
ShellQuoted('WORKDIR {dir}').format(dir=dir),
]
def comment(self, comment):
# This should not be a command since we don't want comment changes
# to invalidate the Docker build cache.
return shell_comment(comment)
def copy_local_repo(self, repo_dir, dest_name):
fd, archive_path = tempfile.mkstemp(
prefix='local_repo_{0}_'.format(dest_name),
suffix='.tgz',
dir=os.path.abspath(self.option('docker_context_dir')),
)
os.close(fd)
run_command('tar', 'czf', archive_path, '.', cwd=repo_dir)
return [
ShellQuoted('ADD {archive} {dest_name}').format(
archive=os.path.basename(archive_path), dest_name=dest_name
),
# Docker permissions make very little sense... see also workdir()
ShellQuoted('USER root'),
ShellQuoted('RUN chown -R {u} {d}').format(
d=dest_name, u=self._user()
),
self._change_user(),
]
def _render_impl(self, steps):
return raw_shell(shell_join('\n', recursively_flatten_list(steps)))
def debian_ccache_setup_steps(self):
source_ccache_tgz = self.option('ccache_tgz', '')
if not source_ccache_tgz:
logging.info('Docker ccache not enabled')
return []
dest_ccache_tgz = os.path.join(
self.option('docker_context_dir'), 'ccache.tgz'
)
try:
try:
os.link(source_ccache_tgz, dest_ccache_tgz)
except OSError:
logging.exception(
'Hard-linking {s} to {d} failed, falling back to copy'
.format(s=source_ccache_tgz, d=dest_ccache_tgz)
)
shutil.copyfile(source_ccache_tgz, dest_ccache_tgz)
except Exception:
logging.exception(
'Failed to copy or link {s} to {d}, aborting'
.format(s=source_ccache_tgz, d=dest_ccache_tgz)
)
raise
return [
# Separate layer so that in development we avoid re-downloads.
self.run(ShellQuoted('apt-get install -yq ccache')),
ShellQuoted('ADD ccache.tgz /'),
ShellQuoted(
# Set CCACHE_DIR before the `ccache` invocations below.
'ENV CCACHE_DIR=/ccache '
# No clang support for now, so it's easiest to hardcode gcc.
'CC="ccache gcc" CXX="ccache g++" '
# Always log for ease of debugging. For real FB projects,
# this log is several megabytes, so dumping it to stdout
# would likely exceed the Travis log limit of 4MB.
#
# On a local machine, `docker cp` will get you the data. To
# get the data out from Travis, I would compress and dump
# uuencoded bytes to the log -- for Bistro this was about
# 600kb or 8000 lines:
#
# apt-get install sharutils
# bzip2 -9 < /tmp/ccache.log | uuencode -m ccache.log.bz2
'CCACHE_LOGFILE=/tmp/ccache.log'
),
self.run(ShellQuoted(
# Future: Skipping this part made this Docker step instant,
# saving ~1min of build time. It's unclear if it is the
# chown or the du, but probably the chown -- since a large
# part of the cost is incurred at image save time.
#
# ccache.tgz may be empty, or may have the wrong
# permissions.
'mkdir -p /ccache && time chown -R nobody /ccache && '
'time du -sh /ccache && '
# Reset stats so `docker_build_with_ccache.sh` can print
# useful values at the end of the run.
'echo === Prev run stats === && ccache -s && ccache -z && '
# Record the current time to let travis_build.sh figure out
# the number of bytes in the cache that are actually used --
# this is crucial for tuning the maximum cache size.
'date +%s > /FBCODE_BUILDER_CCACHE_START_TIME && '
# The build running as `nobody` should be able to write here
'chown nobody /tmp/ccache.log'
)),
]
This diff is collapsed.
#!/usr/bin/env python
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
'Demo config, so that `make_docker_context.py --help` works in this directory.'
config = {
'fbcode_builder_spec': lambda _builder: {
'depends_on': [],
'steps': [],
},
'github_project': 'demo/project',
}
#!/usr/bin/env python
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
'''
Reads `fbcode_builder_config.py` from the current directory, and prepares a
Docker context directory to build this project. Prints to stdout the path
to the context directory.
Try `.../make_docker_context.py --help` from a project's `build/` directory.
By default, the Docker context directory will be in /tmp. It will always
contain a Dockerfile, and might also contain copies of your local repos, and
other data needed for the build container.
'''
import os
import tempfile
import textwrap
from docker_builder import DockerFBCodeBuilder
from parse_args import parse_args_to_fbcode_builder_opts
def make_docker_context(
get_steps_fn, github_project, opts=None, default_context_dir=None
):
'''
Returns a path to the Docker context directory. See parse_args.py.
Helper for making a command-line utility that writes your project's
Dockerfile and associated data into a (temporary) directory. Your main
program might look something like this:
print(make_docker_context(
lambda builder: [builder.step(...), ...],
'facebook/your_project',
))
'''
if opts is None:
opts = {}
valid_versions = (
('ubuntu:14.04', '4.9'), ('ubuntu:16.04', '5'), ('debian:8.6', '4.9')
)
def add_args(parser):
parser.add_argument(
'--docker-context-dir', metavar='DIR',
default=default_context_dir,
help='Write the Dockerfile and its context into this directory. '
'If empty, make a temporary directory. Default: %(default)s.',
)
parser.add_argument(
'--user', metavar='NAME', default=opts.get('user', 'nobody'),
help='Build and install as this user. Default: %(default)s.',
)
parser.add_argument(
'--prefix', metavar='DIR',
default=opts.get('prefix', '/home/install'),
help='Install all libraries in this prefix. Default: %(default)s.',
)
parser.add_argument(
'--projects-dir', metavar='DIR',
default=opts.get('projects_dir', '/home'),
help='Place project code directories here. Default: %(default)s.',
)
parser.add_argument(
'--os-image', metavar='IMG', choices=zip(*valid_versions)[0],
default=opts.get('os_image', 'debian:8.6'),
help='Docker OS image -- be sure to use only ones you trust (See '
'README.docker). Choices: %(choices)s. Default: %(default)s.',
)
parser.add_argument(
'--gcc-version', metavar='VER',
choices=set(zip(*valid_versions)[1]),
default=opts.get('gcc_version', '4.9'),
help='Choices: %(choices)s. Default: %(default)s.',
)
parser.add_argument(
'--make-parallelism', metavar='NUM', type=int,
default=opts.get('make_parallelism', 1),
help='Use `make -j` on multi-CPU systems with lots of RAM. '
'Default: %(default)s.',
)
parser.add_argument(
'--local-repo-dir', metavar='DIR',
help='If set, build {0} from a local directory instead of Github.'
.format(github_project),
)
parser.add_argument(
'--ccache-tgz', metavar='PATH',
help='If set, enable ccache for the build. To initialize the '
'cache, first try to hardlink, then to copy --cache-tgz '
'as ccache.tgz into the --docker-context-dir.'
)
opts = parse_args_to_fbcode_builder_opts(
add_args,
# These have add_argument() calls, others are set via --option.
(
'docker_context_dir',
'user',
'prefix',
'projects_dir',
'os_image',
'gcc_version',
'make_parallelism',
'local_repo_dir',
'ccache_tgz',
),
opts,
help=textwrap.dedent('''
Reads `fbcode_builder_config.py` from the current directory, and
prepares a Docker context directory to build {github_project} and
its dependencies. Prints to stdout the path to the context
directory.
Pass --option {github_project}:git_hash SHA1 to build something
other than the master branch from Github.
Or, pass --option {github_project}:local_repo_dir LOCAL_PATH to
build from a local repo instead of cloning from Github.
Usage:
(cd $(./make_docker_context.py) && docker build . 2>&1 | tee log)
'''.format(github_project=github_project)),
)
# This allows travis_docker_build.sh not to know the main Github project.
local_repo_dir = opts.pop('local_repo_dir', None)
if local_repo_dir is not None:
opts['{0}:local_repo_dir'.format(github_project)] = local_repo_dir
if (opts.get('os_image'), opts.get('gcc_version')) not in valid_versions:
raise Exception(
'Due to 4/5 ABI changes (std::string), we can only use {0}'.format(
' / '.join('GCC {1} on {0}'.format(*p) for p in valid_versions)
)
)
if opts.get('docker_context_dir') is None:
opts['docker_context_dir'] = tempfile.mkdtemp(prefix='docker-context-')
elif not os.path.exists(opts.get('docker_context_dir')):
os.makedirs(opts.get('docker_context_dir'))
builder = DockerFBCodeBuilder(**opts)
context_dir = builder.option('docker_context_dir') # Mark option "in-use"
# The renderer may also populate some files into the context_dir.
dockerfile = builder.render(get_steps_fn(builder))
with os.fdopen(os.open(
os.path.join(context_dir, 'Dockerfile'),
os.O_RDWR | os.O_CREAT | os.O_EXCL, # Do not overwrite existing files
0o644,
), 'w') as f:
f.write(dockerfile)
return context_dir
if __name__ == '__main__':
from utils import read_fbcode_builder_config, build_fbcode_builder_config
# Load a spec from the current directory
config = read_fbcode_builder_config('fbcode_builder_config.py')
print(make_docker_context(
build_fbcode_builder_config(config),
config['github_project'],
))
#!/usr/bin/env python
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
'Argument parsing logic shared by all fbcode_builder CLI tools.'
import argparse
import logging
from shell_quoting import raw_shell, ShellQuoted
def parse_args_to_fbcode_builder_opts(add_args_fn, top_level_opts, opts, help):
'''
Provides some standard arguments: --debug, --option, --shell-quoted-option
Then, calls `add_args_fn(parser)` to add application-specific arguments.
`opts` are first used as defaults for the various command-line
arguments. Then, the parsed arguments are mapped back into `opts`,
which then become the values for `FBCodeBuilder.option()`, to be used
both by the builder and by `get_steps_fn()`.
`help` is printed in response to the `--help` argument.
'''
top_level_opts = set(top_level_opts)
parser = argparse.ArgumentParser(
description=help,
formatter_class=argparse.RawDescriptionHelpFormatter
)
add_args_fn(parser)
parser.add_argument(
'--option', nargs=2, metavar=('KEY', 'VALUE'), action='append',
default=[
(k, v) for k, v in opts.items()
if k not in top_level_opts and not isinstance(v, ShellQuoted)
],
help='Set project-specific options. These are assumed to be raw '
'strings, to be shell-escaped as needed. Default: %(default)s.',
)
parser.add_argument(
'--shell-quoted-option', nargs=2, metavar=('KEY', 'VALUE'),
action='append',
default=[
(k, raw_shell(v)) for k, v in opts.items()
if k not in top_level_opts and isinstance(v, ShellQuoted)
],
help='Set project-specific options. These are assumed to be shell-'
'quoted, and may be used in commands as-is. Default: %(default)s.',
)
parser.add_argument('--debug', action='store_true', help='Log more')
args = parser.parse_args()
logging.basicConfig(
level=logging.DEBUG if args.debug else logging.INFO,
format='%(levelname)s: %(message)s'
)
# Map command-line args back into opts.
logging.debug('opts before command-line arguments: {0}'.format(opts))
new_opts = {}
for key in top_level_opts:
val = getattr(args, key)
# Allow clients to unset a default by passing a value of None in opts
if val is not None:
new_opts[key] = val
for key, val in args.option:
new_opts[key] = val
for key, val in args.shell_quoted_option:
new_opts[key] = ShellQuoted(val)
logging.debug('opts after command-line arguments: {0}'.format(new_opts))
return new_opts
#!/usr/bin/env python
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
'''
Almost every FBCodeBuilder string is ultimately passed to a shell. Escaping
too little or too much tends to be the most common error. The utilities in
this file give a systematic way of avoiding such bugs:
- When you write literal strings destined for the shell, use `ShellQuoted`.
- When these literal strings are parameterized, use `ShellQuoted.format`.
- Any parameters that are raw strings get `shell_quote`d automatically,
while any ShellQuoted parameters will be left intact.
- Use `path_join` to join path components.
- Use `shell_join` to join already-quoted command arguments or shell lines.
'''
import os
from collections import namedtuple
class ShellQuoted(namedtuple('ShellQuoted', ('do_not_use_raw_str',))):
'''
Wrap a string with this to make it transparent to shell_quote(). It
will almost always suffice to use ShellQuoted.format(), path_join(),
or shell_join().
If you really must, use raw_shell() to access the raw string.
'''
def __new__(cls, s):
'No need to nest ShellQuoted.'
return super(ShellQuoted, cls).__new__(
cls, s.do_not_use_raw_str if isinstance(s, ShellQuoted) else s
)
def __str__(self):
raise RuntimeError(
'One does not simply convert {0} to a string -- use path_join() '
'or ShellQuoted.format() instead'.format(repr(self))
)
def __repr__(self):
return '{0}({1})'.format(
self.__class__.__name__, repr(self.do_not_use_raw_str)
)
def format(self, **kwargs):
'''
Use instead of str.format() when the arguments are either
`ShellQuoted()` or raw strings needing to be `shell_quote()`d.
Positional args are deliberately not supported since they are more
error-prone.
'''
return ShellQuoted(self.do_not_use_raw_str.format(**dict(
(k, shell_quote(v).do_not_use_raw_str) for k, v in kwargs.items()
)))
def shell_quote(s):
'Quotes a string if it is not already quoted'
return s if isinstance(s, ShellQuoted) \
else ShellQuoted("'" + str(s).replace("'", "'\\''") + "'")
def raw_shell(s):
'Not a member of ShellQuoted so we get a useful error for raw strings'
if isinstance(s, ShellQuoted):
return s.do_not_use_raw_str
raise RuntimeError('{0} should have been ShellQuoted'.format(s))
def shell_join(delim, it):
'Joins an iterable of ShellQuoted with a delimiter between each two'
return ShellQuoted(delim.join(raw_shell(s) for s in it))
def path_join(*args):
'Joins ShellQuoted and raw pieces of paths to make a shell-quoted path'
return ShellQuoted(os.path.join(*[
raw_shell(shell_quote(s)) for s in args
]))
def shell_comment(c):
'Do not shell-escape raw strings in comments, but do handle line breaks.'
return ShellQuoted('# {c}').format(c=ShellQuoted(
(raw_shell(c) if isinstance(c, ShellQuoted) else c)
.replace('\n', '\n# ')
))
#!/usr/bin/env python
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
import specs.folly as folly
import specs.wangle as wangle
import specs.zstd as zstd
from shell_quoting import ShellQuoted
def fbcode_builder_spec(builder):
# This API should change rarely, so build the latest tag instead of master.
builder.add_option(
'no1msd/mstch:git_hash',
ShellQuoted('$(git describe --abbrev=0 --tags)')
)
builder.add_option(
'rsocket/rsocket-cpp/yarpl/build:cmake_defines', {'BUILD_TESTS': 'OFF'}
)
return {
'depends_on': [folly, wangle, zstd],
'steps': [
# This isn't a separete spec, since only fbthrift uses mstch.
builder.github_project_workdir('no1msd/mstch', 'build'),
builder.cmake_install('no1msd/mstch'),
builder.github_project_workdir(
'rsocket/rsocket-cpp', 'yarpl/build'
),
builder.step('configuration for yarpl', [
builder.cmake_configure('rsocket/rsocket-cpp/yarpl/build'),
]),
builder.cmake_install('rsocket/rsocket-cpp/yarpl'),
builder.fb_github_cmake_install('fbthrift/thrift'),
],
}
#!/usr/bin/env python
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
import specs.fbthrift as fbthrift
import specs.folly as folly
import specs.gmock as gmock
from shell_quoting import ShellQuoted
def fbcode_builder_spec(builder):
builder.add_option('jedisct1/libsodium:git_hash', 'stable')
return {
'depends_on': [folly, fbthrift, gmock],
'steps': [
builder.github_project_workdir('jedisct1/libsodium', '.'),
builder.step('Build and install jedisct1/libsodium', [
builder.run(ShellQuoted('./autogen.sh')),
builder.configure(),
builder.make_and_install(),
]),
builder.github_project_workdir('zeromq/libzmq', '.'),
builder.step('Build and install zeromq/libzmq', [
builder.run(ShellQuoted('./autogen.sh')),
builder.configure(),
builder.make_and_install(),
]),
builder.fb_github_project_workdir('fbzmq/fbzmq/build', 'facebook'),
builder.step('Build and install fbzmq/fbzmq/build', [
builder.cmake_configure('fbzmq/fbzmq/build'),
# we need the pythonpath to find the thrift compiler
builder.run(ShellQuoted(
'PYTHONPATH="$PYTHONPATH:"{p}/lib/python2.7/site-packages '
'make -j {n}'
).format(p=builder.option('prefix'), n=builder.option('make_parallelism'))),
builder.run(ShellQuoted('make install')),
]),
],
}
#!/usr/bin/env python
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
def fbcode_builder_spec(builder):
return {
'steps': [
builder.fb_github_cmake_install('folly/folly'),
],
}
#!/usr/bin/env python
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
def fbcode_builder_spec(builder):
builder.add_option(
'google/googletest:cmake_defines',
{'BUILD_GTEST': 'ON'}
)
return {
'steps': [
builder.github_project_workdir('google/googletest', 'build'),
builder.cmake_install('google/googletest'),
],
}
#!/usr/bin/env python
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
import specs.folly as folly
import specs.wangle as wangle
def fbcode_builder_spec(builder):
return {
'depends_on': [folly, wangle],
'steps': [
builder.fb_github_autoconf_install('proxygen/proxygen'),
],
}
#!/usr/bin/env python
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
import specs.folly as folly
def fbcode_builder_spec(builder):
# Projects that simply depend on Wangle need not spend time on tests.
builder.add_option('wangle/wangle/build:cmake_defines', {'BUILD_TESTS': 'OFF'})
return {
'depends_on': [folly],
'steps': [
builder.fb_github_cmake_install('wangle/wangle/build'),
],
}
#!/usr/bin/env python
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
from shell_quoting import ShellQuoted
def fbcode_builder_spec(builder):
# This API should change rarely, so build the latest tag instead of master.
builder.add_option(
'facebook/zstd:git_hash',
ShellQuoted('$(git describe --abbrev=0 --tags)')
)
return {
'steps': [
builder.github_project_workdir('facebook/zstd', '.'),
builder.step('Build and install zstd', [
builder.make_and_install(make_vars={
'PREFIX': builder.option('prefix'),
})
]),
],
}
#!/bin/bash -uex
# .travis.yml in the top-level dir explains why this is a separate script.
# Read the docs: ./make_docker_context.py --help
os_image=${os_image?Must be set by Travis}
gcc_version=${gcc_version?Must be set by Travis}
make_parallelism=${make_parallelism:-4}
# ccache is off unless requested
travis_cache_dir=${travis_cache_dir:-}
# The docker build never times out, unless specified
docker_build_timeout=${docker_build_timeout:-}
cur_dir="$(readlink -f "$(dirname "$0")")"
if [[ "$travis_cache_dir" == "" ]]; then
echo "ccache disabled, enable by setting env. var. travis_cache_dir"
ccache_tgz=""
elif [[ -e "$travis_cache_dir/ccache.tgz" ]]; then
ccache_tgz="$travis_cache_dir/ccache.tgz"
else
echo "$travis_cache_dir/ccache.tgz does not exist, starting with empty cache"
ccache_tgz=$(mktemp)
tar -T /dev/null -czf "$ccache_tgz"
fi
docker_context_dir=$(
cd "$cur_dir/.." # Let the script find our fbcode_builder_config.py
"$cur_dir/make_docker_context.py" \
--os-image "$os_image" \
--gcc-version "$gcc_version" \
--make-parallelism "$make_parallelism" \
--local-repo-dir "$cur_dir/../.." \
--ccache-tgz "$ccache_tgz"
)
cd "${docker_context_dir?Failed to make Docker context directory}"
# Make it safe to iterate on the .sh in the tree while the script runs.
cp "$cur_dir/docker_build_with_ccache.sh" .
exec ./docker_build_with_ccache.sh \
--build-timeout "$docker_build_timeout" \
"$travis_cache_dir"
#!/usr/bin/env python
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals
'Miscellaneous utility functions.'
import itertools
import logging
import os
import subprocess
import sys
from contextlib import contextmanager
def recursively_flatten_list(l):
return itertools.chain.from_iterable(
(recursively_flatten_list(i) if type(i) is list else (i,))
for i in l
)
def run_command(*cmd, **kwargs):
'The stdout of most fbcode_builder utilities is meant to be parsed.'
logging.debug('Running: {0} with {1}'.format(cmd, kwargs))
kwargs['stdout'] = sys.stderr
subprocess.check_call(cmd, **kwargs)
@contextmanager
def make_temp_dir(d):
os.mkdir(d)
try:
yield d
finally:
if os.path.exists(d):
os.rmdir(d)
@contextmanager
def push_dir(d):
old_dir = os.getcwd()
os.chdir(d)
try:
yield d
finally:
os.chdir(old_dir)
def read_fbcode_builder_config(filename):
# Allow one spec to read another
scope = {'read_fbcode_builder_config': read_fbcode_builder_config}
with open(filename) as config_file:
exec(config_file.read(), scope)
return scope['config']
def steps_for_spec(builder, spec, processed_modules=None):
'''
Sets `builder` configuration, and returns all the builder steps
necessary to build `spec` and its dependencies.
Traverses the dependencies in depth-first order, honoring the sequencing
in each 'depends_on' list.
'''
if processed_modules is None:
processed_modules = set()
steps = []
for module in spec.get('depends_on', []):
if module not in processed_modules:
processed_modules.add(module)
steps.extend(steps_for_spec(
builder,
module.fbcode_builder_spec(builder),
processed_modules
))
steps.extend(spec.get('steps', []))
return steps
def build_fbcode_builder_config(config):
return lambda builder: builder.build(
steps_for_spec(builder, config['fbcode_builder_spec'](builder))
)
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment