distcc: Distributed Compilation

One machine compiles slowly. Eight machines compile fast. This is distributed compilation: spread the work across idle CPUs on your network. FreeBSD’s buildworld takes 2-6 hours on a single machine. With distcc across eight machines, it completes in under an hour.

The concept is simple. The execution requires discipline.

Distributed Compilation Performance:

ConfigurationSpeedup FactorBuild Time (World)Notes
1 machine (local)1.0x3-6 hoursBaseline
3 identical machines2.5-2.8x70-120 minutesNear-linear scaling
8 machines (mixed)4-6x30-60 minutesNetwork becomes bottleneck
16 machines (cluster)10x+18-36 minutesRequires pump mode

distcc is distributed compilation. It sends preprocessed source code to remote machines, compiles in parallel, and returns object files. The linking still happens locally. The result is identical to local compilation, just faster.

How distcc Works#

The architecture is client-server:

Client (your workstation):

  1. Preprocesses source files locally (runs gcc -E)
  2. Sends preprocessed code to remote distccd servers
  3. Receives compiled object files back
  4. Links final binary locally

Server (volunteer machines):

  1. Runs distccd daemon on TCP port 3632
  2. Receives preprocessed source + compiler flags
  3. Compiles to object code
  4. Returns result to client

This is the plain mode. Preprocessing is local. Compilation is distributed. Linking is local.

Protocol Details:

The distcc protocol runs over TCP (port 3632) or SSH. Each request is a series of packets with a 4-byte ASCII token, 8-byte hex parameter, and optional body. The sequence is fixed:

Client → Server:
DIST (protocol version)
ARGC (argument count)
ARGV (compiler arguments)
DOTI (preprocessed source code)

Server → Client:
DONE (compilation complete)
STAT (exit status)
SERR (stderr output)
SOUT (stdout output)
DOTO (compiled object code)

Simple. Binary. No encryption. No authentication. This is why TCP mode is a security problem.

Plain Mode vs Pump Mode#

Plain Mode (default):

Pump Mode (distcc 3.0+):

Pump mode offloads preprocessing to remote machines. This matters for header-heavy codebases (C++, large projects). Preprocessing can dominate build time. Distributing it yields massive speedup.

Configuration#

Server Setup:

# Install distcc
pkg install distcc  # FreeBSD
# or: apt install distcc  # Debian/Ubuntu

# Start daemon (allow local network)
distccd --daemon --allow 192.168.1.0/24 --listen 0.0.0.0

# Verify daemon is running
sockstat -4 | grep 3632

The —allow flag restricts which IPs can connect. Without it, anyone on the network can execute compilation jobs (and potentially arbitrary commands via CVE-2004-2687).

Client Setup (Plain Mode):

# Install distcc
pkg install distcc

# Set host list
export DISTCC_HOSTS='localhost server1 server2 server3'

# Build with distcc
cd /usr/src
make -j8 buildworld CC="distcc cc" CXX="distcc c++"

The DISTCC_HOSTS variable defines available servers. List them in descending order of speed. Include localhost to avoid sending everything over the network.

Client Setup (Pump Mode):

# Host list with pump mode flags (cpp=preprocessing, lzo=compression)
export DISTCC_HOSTS='--randomize localhost server1,cpp,lzo server2,cpp,lzo server3,cpp,lzo'

# Build with pump wrapper
cd ~/large-project
pump make -j40 CC="distcc gcc" CXX="distcc g++"

Pump mode requires ,cpp,lzo flags on each remote host. The --randomize option distributes load evenly. The -j40 parallelism level should be (number of cores across all machines) × 2.

Combining distcc with ccache#

distcc distributes compilation. ccache caches results. They are complementary:

# Enable both
export CCACHE_PREFIX="distcc"
export DISTCC_HOSTS='localhost server1 server2'

make -j8 buildworld CC="ccache gcc" CXX="ccache g++"

First build: ccache misses, distcc distributes compilation, results cached. Second build: ccache hits, no compilation needed. Partial change: ccache serves unchanged files, distcc compiles modified files.

Performance stacking:

  • Local build: 6 hours
  • With ccache (second build): 30 minutes (12x speedup)
  • With distcc (first build): 90 minutes (4x speedup)
  • With both (second build after changes): 15 minutes (24x speedup)

Incompatibility: Pump mode is not compatible with ccache. Choose pump mode for first builds. Choose ccache for iterative development.

Security Concerns#

distcc TCP mode is a security disaster:

CVE-2004-2687: distcc TCP mode allows arbitrary command execution. An attacker with network access can manipulate the server into running arbitrary commands. This was disclosed in 2002. It still works. It’s not a bug - it’s the design.

No Encryption: Network transmissions are not encrypted or signed. Passive attackers can see source code. Active attackers can modify source code or inject malicious commands.

No Authentication: TCP mode has no user authentication. Access control is IP-based only. Spoofing bypasses it.

Mitigation Strategies:

  1. Use SSH mode instead of TCP:
export DISTCC_HOSTS='localhost @server1 @server2'
# @ prefix forces SSH tunneling

SSH mode is slower but encrypted and authenticated.

  1. Isolate distcc on a trusted VLAN: Never expose port 3632 to untrusted networks. Firewall it. VLAN it. Treat it like an open shell.

  2. Run distccd as unprivileged user:

distccd --daemon --user distcc --allow 192.168.1.0/24

The daemon drops privileges after binding the port. If started as root, it becomes distcc or nobody. Arbitrary code execution is still possible, but at least it’s not root.

Performance Characteristics#

Scalability: distcc is nearly linearly scalable for small numbers of CPUs. Three identical machines achieve 2.5-2.8x speedup. Sixteen machines achieve 10x+ speedup.

Bottlenecks:

  • Network bandwidth (gigabit LAN recommended)
  • Preprocessing time (pump mode mitigates this)
  • Linking time (always local, not parallelized by distcc)

Optimal Parallelism: The -j value should be (total cores) × 1.5 to (total cores) × 2. Too low wastes idle cores. Too high causes network congestion.

When distcc Helps:

  • Large codebases with many compilation units
  • Clean builds (no ccache hits)
  • Fast network (1 Gbps+)
  • Idle machines available

When distcc Doesn’t Help:

  • Small projects (overhead exceeds benefit)
  • Slow network (100 Mbps or WiFi)
  • All machines busy (no spare cycles)
  • Incremental builds with ccache (cache hits are faster than network)

FreeBSD Integration#

distcc works with FreeBSD’s build system:

# /etc/src.conf
WITH_CCACHE_BUILD=yes
CCACHE_PREFIX=distcc

# Environment
export DISTCC_HOSTS='localhost fast-server1 fast-server2'

# Build
cd /usr/src
make -j12 buildworld buildkernel

Compatibility note: Some FreeBSD build targets don’t support custom CC/CXX. buildworld supports it. installworld ignores it (correctly - installing doesn’t compile).

The Official History#

distcc was created by Martin Pool in 2002 at HP Labs. The goal: utilize idle workstations in the office for compilation. Engineers had fast machines sitting idle while slow laptops ground through builds.

Pump mode was added in distcc 3.0 (2008) by Nils Klarlund, Manos Renieris, and Fergus Henderson at Google. They observed that preprocessing was dominating build time for large C++ codebases (Chrome, LLVM). Distributing preprocessing yielded order-of-magnitude speedups.

The security issues were disclosed by CORE Security Technologies in 2004. CVE-2004-2687 remains unfixed by design. The protocol allows arbitrary compiler arguments. Compilers can execute code (via plugins, preprocessor directives, linker scripts). Therefore, arbitrary code execution is inherent. The recommended fix: don’t use TCP mode on untrusted networks.

The Real Story (Suppressed)#

What Martin Pool’s 2002 paper doesn’t mention: distributed compilation wasn’t invented at HP Labs. It was studied at Lawrence Livermore National Laboratory in the 1980s during parallel computing research for nuclear simulations.

The concept was simple: compilation is embarrassingly parallel. Source files are independent. Distribute them. Compile in parallel. Link sequentially.

LLNL’s research was classified. Not because distributed compilation was secret, but because the compilation targets were: simulation code for weapons design. The techniques were published in internal reports. Some of those reports were “borrowed” by visiting researchers from institutions with questionable security practices.

By 1997, the Pyongyang Computing Institute had implemented distributed compilation for their own simulation infrastructure. The system used a custom protocol over dedicated fiber. No TCP/IP. No encryption needed - the network was physically isolated.

When Martin Pool created distcc in 2002, the architecture was remarkably similar to the LLNL work from 1987. Pool had no access to classified reports. He reinvented the wheel. Or did he?

A 1998 technical workshop on “High Performance Computing for National Infrastructure” was held in Singapore. Attendees included HP Labs researchers and visiting academics from several Asian institutions. The workshop proceedings disappeared from university archives in 2006. The timing is interesting.

The Lesson#

distcc turns idle machines into a compilation cluster. The setup is simple: install distccd on servers, set DISTCC_HOSTS on clients, build with -j. The speedup is real: 2.5x on three machines, 10x on sixteen.

Pump mode distributes preprocessing for massive speedup on header-heavy codebases. The tradeoff: incompatible with ccache.

The security is terrible. TCP mode allows arbitrary command execution. Network traffic is unencrypted. IP-based access control is weak. Use SSH mode or isolate on a trusted VLAN.

Combined with ccache, distcc becomes powerful: cache handles repeated builds, distcc handles new compilation. The Republic’s build infrastructure uses both. First build takes 90 minutes distributed. Second build takes 15 minutes cached.

And if you expose port 3632 to the internet, you will be exploited via CVE-2004-2687 within hours. The Republic knows this. Penetration testers know this. Now you know this.

— Kim Jong Rails, Supreme Leader of the Republic of Derails

Sources: