Running AOSP Module Builds on Mac with Apple Container + Rosetta

macOS 26 shipped Apple Container. I wanted to see if it could run AOSP module builds on a MacBook to reduce dependency on CI. Short answer: yes, but there are a few things to handle.

Background

My AOSP development workflow has always had a pain point: the MacBook can only edit code — builds have to go to remote CI. Every change means “push → wait for CI build → check result.” A typo costs 30 minutes to discover.

macOS can’t build AOSP natively — the build system assumes Linux, prebuilt toolchains are x86_64 binaries, and it requires case-sensitive filesystem semantics. So I push code and wait.

After Apple Container came out, I wanted to try running module builds locally — at least to validate once before pushing to CI.

Environment

• MacBook Pro M3 Max, 128 GB RAM
• macOS 26, Apple Container v1.0.0
• AOSP Android 14

Apple Container is Apple’s Linux container runtime built on Virtualization.framework. The features relevant here:

• ARM64 Linux containers run natively on Apple Silicon
• --rosetta flag transparently executes x86_64 binaries
• virtiofs mounts host directories with near-native I/O
• Supports persistent ext4 volumes
• Sub-second container startup

The idea: mount the AOSP source tree via virtiofs, run ARM64 tools natively, and let Rosetta handle x86_64 prebuilts (Go, clang, aapt2, etc.).

Dockerfile

Ubuntu 24.04 ARM64 with AOSP build dependencies, plus x86_64 runtime libraries for Rosetta:

FROM ubuntu:24.04

# ARM64 build dependencies
RUN apt-get update && apt-get install -y \
    git-core gnupg flex bison build-essential zip curl \
    zlib1g-dev libxml2-utils xsltproc unzip fontconfig \
    libncurses-dev python3 openjdk-21-jdk \
    rsync bc cpio lz4

# x86_64 libs for Rosetta (AOSP prebuilts are x86_64)
RUN dpkg --add-architecture amd64 \
    && echo "deb [arch=amd64] http://archive.ubuntu.com/ubuntu noble main" \
       > /etc/apt/sources.list.d/amd64.list \
    && apt-get update \
    && apt-get install -y libc6:amd64 libstdc++6:amd64 zlib1g:amd64

Launch:

container run --rm --rosetta --cap-add ALL \
    -v /path/to/aosp:/aosp \
    -v build-cache:/tmp/aosp-out \
    -e OUT_DIR=/tmp/aosp-out \
    aosp-builder:v1

At this point the environment is set up, but running lunch directly will fail. Three things need to be handled.

Three Compatibility Issues to Handle

1. macOS Metadata Directories Leaking Through virtiofs

macOS HFS+/APFS volumes have several system metadata directories: .Spotlight-V100, .DocumentRevisions-V100, .TemporaryItems, .fseventsd.

When mounted into Linux via virtiofs, these directories show up in readdir() results, but lstat() on them returns ENOENT — they appear in listings but can’t be stat’d.

Soong’s module-finder scans the entire source tree looking for Android.bp files. When it hits these directories, it fatals:

finder.go: error: lstat .Spotlight-V100: no such file or directory

The fix is to add these to ExcludeDirs in build/soong/ui/build/finder.go:

// Before
ExcludeDirs: []string{".git", ".repo"},
// After
ExcludeDirs: []string{".git", ".repo",
    ".Spotlight-V100", ".DocumentRevisions-V100",
    ".TemporaryItems", ".fseventsd"},

I tried several other approaches — creating dummy directories at those paths (virtiofs mount overrides them), overlayfs on top of virtiofs (stale NFS handle errors), symlinks to /dev/null (breaks other path resolution). Only the ExcludeDirs change worked cleanly.

2. Rosetta Needs x86_64 Runtime Libraries

--rosetta translates x86_64 instructions, but doesn’t bring libc. First time running an AOSP prebuilt tool:

$ prebuilts/go/linux-x86/bin/go version
rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2

You need libc6:amd64, libstdc++6:amd64, and zlib1g:amd64 from Ubuntu’s x86_64 repositories.

One gotcha here: Ubuntu ARM64’s ports.ubuntu.com doesn’t carry amd64 packages. You need to add a separate source pointing to archive.ubuntu.com and restrict the default sources to Architectures: arm64 to avoid conflicts. The Dockerfile above already handles this.

3. Artifact Path Strict Checking

AOSP’s artifact path checker validates that files don’t violate partition boundaries. Under certain vendor configurations it flags files like vendor/lib/libhidltransport.so and hard-fails.

On CI this is usually handled by BUILD_BROKEN_ARTIFACT_PATH_REQUIREMENT := true in the board config, but in the container this flag may not propagate correctly.

Pragmatic fix for local builds: in build/make/core/artifact_path_requirements.mk, change maybe-print-list-and-error to maybe-print-list-and-warning. CI stays unaffected.

virtiofs COW: Patches That Leave No Trace

All three patches are applied automatically in the container entrypoint. The important part is virtiofs copy-on-write — all modifications only exist inside the container. The host source tree is never touched. When the container stops, the patches are gone.

This means I can freely sed build system files inside the container without worrying about dirtying the source tree.

#!/bin/bash
# entrypoint.sh — applied on every container start

# Exclude macOS metadata directories
sed -i 's/ExcludeDirs:.*\[\]string{".git", ".repo"}/.../' \
    build/soong/ui/build/finder.go

# Disable duplicate product definition
mv device/.../AndroidProducts.mk{,.container-disabled}

# Downgrade artifact path check
sed -i 's/maybe-print-list-and-error/maybe-print-list-and-warning/' \
    build/make/core/artifact_path_requirements.mk

source build/envsetup.sh
exec bash

Persistent Build Cache

AOSP builds produce a lot of intermediate artifacts (Soong ninja files, kati makefiles, compiled intermediates). Without a cache, every run is a cold build.

Apple Container supports persistent ext4 volumes:

container volume create -s 50G build-cache
container run -v build-cache:/tmp/aosp-out ...

The volume persists across container restarts.

Measured Results

Cache used about 8.6 GB (4.2 GB Soong cache).

The warm cache still regenerates 1024 glob shards (Soong re-scans the source tree every time), but skips ninja file generation.

Is a Full Build Feasible?

Technically yes — all compilation tools run through Rosetta. But every compiler invocation has translation overhead. A module build with ~2500 steps takes 19-29 minutes. A full build has 50,000+ steps.

Estimated 4-8 hours vs. CI’s 30-60 minutes. Not practical. Full builds stay on CI.

Use Cases

Local builds aren’t fast — 19 minutes is still 19 minutes. But what’s saved is the round-trip of “push → wait → find error → fix → push again.” That’s where the real time waste is.

Notes After Working Through This

1. virtiofs leaks host filesystem quirks. macOS metadata directories, extended attributes, and similar things all pass through to Linux. Build systems that do filesystem scanning will hit these.

2. Rosetta translates instructions, not libraries. You need to provide x86_64 libc, libstdc++, and zlib yourself.

3. virtiofs COW makes container patches cost-free. This is a useful property for local dev environments that need to differ from CI — changes revert when the container exits.

4. Build cache is the line between “usable” and “annoying.” The 37% difference means that without a cache, nobody would want to use this workflow.

5. Module builds are the sweet spot. Full builds are technically possible but pointless. Let each tool do what it’s good at.

Setup Steps

1. macOS 26+ on Apple Silicon
2. Install Apple Container v1.0.0+
3. container system start (one-time — downloads the VM kernel)
4. Build the container image (Ubuntu 24.04 + ARM64 deps + amd64 Rosetta libs)
5. Create a persistent volume for build cache
6. Write an entrypoint that handles the three compatibility issues
7. -v /path/to/aosp:/aosp to mount the AOSP tree
8. lunch <target> && mmm <module>

The specific patches will vary by AOSP version and device configuration, but the three categories (filesystem metadata, Rosetta runtime libs, build system strict checks) should be universal.

I packaged the Containerfile, entrypoint, and scripts into a repo: wangchauyan/aosp-container. Clone, set AOSP_ROOT in config.env, and ./run.sh.

Tested on MacBook Pro M3 Max (128 GB RAM), macOS 26, Apple Container v1.0.0, AOSP Android 14.