Make targets
The Makefile in the repository root is the single entry point for building images, running the checks for every component, and starting the stack. Run make help to print the annotated targets.
Run all commands from the repository root. Pass variables on the command line, for example make generation FLAVOR=gpu.
Variables
| Variable | Default | Description |
|---|---|---|
FLAVOR | cpu | Which generation to build: cpu or gpu. Every artifact path and image name includes the flavor, so the two flavors do not overwrite each other. See flavors and catalogues. |
TAG | UTC timestamp (YYYYMMDD-HHMMSS) | Tag added to the worker image, alongside latest. |
ANALYZE_EXCLUDE | google | Space-separated packages make analyze skips, with everything under them. |
ROOTFS_REQUIREMENTS | pipeline/rootfs-requirements.txt (empty) | File of distribution names, relative to the repository root, that the root filesystem installs on top of ROOTFS_BASE_IMAGE. make rootfs-requirements writes one from a plan. |
ROOTFS_REQUIREMENTS_STRICT | false | When true, the image build fails if a listed distribution does not install. Otherwise the failures are listed in /etc/readie-rootfs-skipped.txt in the root filesystem. |
ROOTFS_PYTHONPATH | /usr/local/lib/python3.12/dist-packages | Where the root filesystem's Python keeps packages. The python:3.12 images use /usr/local/lib/python3.12/site-packages. |
ROOTFS_APT_PACKAGES | empty | Debian or Ubuntu packages, comma separated, that the root filesystem installs before the distributions. Use it for the system libraries that wheels load and the base image lacks. |
ROOTFS_EXTRA_INDEX_URL | empty | Extra package indexes for pins that the default one lacks, comma separated when there are several. With any set, the install looks at every index for the best match, so each listed index can supply any package name. Set it empty for a base whose pins are all on PyPI. |
PLAN_DIR | pipeline/plan/<flavor> | Where make rootfs-requirements writes the plan and the list. |
REUSE_PLAN_DIR | empty | A plan directory from an earlier make rootfs-requirements, relative to the repository root. When set, make capture mounts it into the container and reuses that plan instead of planning again. When empty, capture plans inside the container. |
READIE_PLANNER | greedy | Checkpoint planner: greedy or fixed. |
READIE_MAX_CHECKPOINTS | 15 | Most checkpoints a plan may produce. |
READIE_SIZE_BUDGET_MB | 2048.0 | Total size budget for checkpoints, in MB. |
READIE_ALPHA | set in the Makefile | Size-versus-time weight. See cost model. |
The four READIE_* planner values are set with := in the Makefile. A value passed on the command line takes precedence, for example make capture READIE_ALPHA=<seconds-per-mb>. A value that is only exported in the shell does not.
These values can differ from the pipeline's own defaults (for example 8 checkpoints), because make capture always passes its own values to the pipeline. See Configuration reference.
The following names are derived from FLAVOR:
| Name | Value |
|---|---|
ARTIFACTS_DIR | pipeline/out/<flavor> |
CATALOGUE_DIR | catalogues |
PIPELINE_IMAGE | readie-pipeline-<flavor>:latest |
ANALYZER_IMAGE | readie-pipeline-analyzer-<flavor>:latest |
WORKER_BASE | readie-worker-base-<flavor>:latest |
WORKER_IMAGE | readie-worker-<flavor> |
ROOTFS_BASE_IMAGE | gcr.io/kaggle-images/python for cpu, gcr.io/kaggle-gpu-images/python for gpu |
Everyday targets
| Target | What it does |
|---|---|
all | Generate protos, then lint, type-check and test everything. |
install | Sync every Python virtual environment from its lockfile and download Go modules. |
lint | Lint every component, and lint and format-check the protos. |
type | Type-check the four Python components. The Go worker has no type target. |
test | Test every component. The worker tests run with the race detector. Most tests use fakes and run anywhere. |
help | List the annotated targets. |
The command make lint type test is the full check that CI repeats.
Per-component targets
A pattern rule forwards <component>-<verb> to that component's own Makefile. The components are router, pkg, executor, pipeline and worker.
make worker-test # same as: make -C worker test
make router-lint
make pkg-type
make pipeline-test
make pipeline-corpus FLAVOR=gpu ARGS="--total 500" # append 500 new items to the flavor's corpus
pipeline-corpus forwards to the pipeline's own corpus target. It needs the corpus extra and the credentials listed in Configuration reference. make generation does not build a corpus.
See Repo map for the contents of each component.
Proto targets
Generated code is committed. After editing a .proto file, run make protos and commit the result. See Changing protos.
| Target | What it does |
|---|---|
protos | Regenerate all gRPC stubs (Python and Go). |
protos-python | Regenerate the router and client Python stubs. |
protos-go | Regenerate the worker's Go stubs. |
protos-lint | Lint and format-check the protos with buf. |
protos-fmt | Format the protos in place. |
protos-breaking | Check the protos for breaking changes against main. |
clean-protos | Remove the generated Python stubs. |
Checkpoint build targets
These targets build the images that hold checkpoints and require Docker. Targets with yes or amd64 only in the gVisor column work only on an amd64 Linux host with real gVisor, and do not work on Apple Silicon or in CI. For the procedure, see Build checkpoints.
| Target | gVisor | What it does |
|---|---|---|
rootfs-requirements | no | Plan on the host with plan --no-spec and write the distributions the plan needs to $(PLAN_DIR)/rootfs-requirements.txt. Needs uv, and no Docker or gVisor. See Build checkpoints. |
pipeline-image | no | Build the offline pipeline image. |
analyzer-image | amd64 only | Build the analyzer image (the pipeline on the base image itself). |
analyze | amd64 only | Measure every package the flavor's base image installs. Writes pipeline/data/metadata/<flavor>.json, including the version of each distribution and the name of the base image. Runs with no network. |
capture | yes | Plan and capture checkpoints into pipeline/out/<flavor>. Runs a privileged container. Fails if no manifest is produced, then copies the catalogue to catalogues/<flavor>.json. |
worker-base | no | Build the base image the worker runs on: gVisor and the root filesystem. It does not contain the checkpoints and does not need a capture first. |
worker-image | no | Build the worker image on top of the base. This is a Go build and takes seconds. Builds the base first if it is missing. |
generation | yes | Run capture, worker-base and worker-image in order. docker compose then mounts pipeline/out/<flavor>/checkpoints and manifest.json into the worker. |
clean-artifacts | no | Delete captured checkpoints and the manifest for the flavor. |
To redeploy a worker code change without capturing again, run make worker-image alone, then recreate the container.
Run targets
These targets wrap Docker Compose. See Run the stack locally and Deploy with TLS.
| Target | What it does |
|---|---|
run-local | Start NGINX, the router and a worker with docker-compose.yml plus docker-compose.local.yml (plaintext on port 50051). |
run-prod | Same, with docker-compose.prod.yml (TLS on port 443). |
shutdown | Stop the containers with docker compose down --remove-orphans. |
Documentation targets
The documentation site has its own targets in the root Makefile:
| Target | What it does |
|---|---|
docs-install | Install the documentation site's npm dependencies (npm ci). Needs Node 20 or newer. |
docs-tools | Install protoc-gen-doc, the generator behind the protobuf reference page. |
docs-protos | Regenerate the protobuf reference page (architecture/protos) from protos/*.proto. Needs protoc and protoc-gen-doc on your PATH. |
docs-build | Build the static site. Fails on any broken link or anchor. |
docs-start | Run the site with live reload. |
docs-serve | Build the site, then serve the result locally. |