The Architecture of Necessity: Building Self-Hosted GitHub Runners in the Shadow of Limits

A technical archaeology of constraint, emergence, and the systems we build when the universe says no.

The failure arrived without ceremony, as failures often do in our interconnected digital ecosystems. One moment, the continuous integration pipelines of Red Duck Labs hummed with their familiar rhythm- code commits triggering builds, tests cascading through their predetermined sequences, deployments flowing like clockwork through their carefully orchestrated stages. The next moment: silence. Not the dramatic silence of catastrophic system failure, but the more insidious quiet of artificial scarcity.

Image

GitHub's hosted runners had reached their limits. Not technical limits- those we understand, respect, even admire for their honest brutality. These were different: the soft boundaries of commercial constraint, the invisible walls that emerge when your computational appetite exceeds the parameters of someone else's business model. Every build queued indefinitely. Every deployment frozen in amber. The entire development lifecycle of Red Duck Labs suspended in a state that was neither failure nor success, but something far more maddening: bureaucratic limbo.

In that moment of enforced stillness, watching months of carefully crafted automation rendered inert by GitHub's account limitations, the architecture of necessity began to crystallize. Like the ancient civilizations that built their greatest monuments not in times of abundance, but when pressed against the hard edges of survival- we would build our own infrastructure. Not because it was elegant or cost-effective, but because the alternative was either stagnation or a truly massive bill to upgrade to GitHub Enterprise.

The Emergence of Self-Sufficiency

The decision to build self-hosted GitHub runners emerged from this crucible of constraint with the inexorable logic of evolution. When external dependencies fail; successful systems develop internal capabilities. When shared resources become unreliable; resilient architectures become self-contained. The path forward was clear, even if the implementation would prove to be less than simple.

What began as a straightforward requirement- "we need our own runners"- quickly revealed itself to be a complex systems problem with layers of interdependence. Each component we thought we understood spawned new requirements, new constraints, new opportunities for elegant solutions or spectacular failure.

The first challenge emerged in the seemingly simple task of containerization. GitHub's official runner images, designed for their hosted environment, carried assumptions about their operational context that proved incompatible with our needs. We required not just a runner, but a complete development ecosystem: Python 3.13 for our machine learning pipelines, Node.js 22 for our frontend builds, Terraform for infrastructure management, kubectl for Kubernetes orchestration, and a constellation of security tools- kubeconform, kubesec, Trivy- that formed the defensive perimeter of our development process.

# The foundation: GitHub's runner, but transformed
FROM ghcr.io/actions/actions-runner:latest

# Layer by layer, we build our computational environment
RUN add-apt-repository -y ppa:deadsnakes/ppa && \
    apt-get update && \
    apt-get install -y python3.13 python3.13-dev python3.13-venv

As we performed the dependency archaeology, tracing the connections between packages, the Dockerfile grew organically; accumulating layers of functionality like sedimentary rock, each stratum telling the story of a problem encountered and solved.

Image

The Docker-in-Docker Problem

A larger challenge emerged from a fundamental requirement: our runners needed to build Docker images. This created what systems theorists might recognize as a recursive containment problem- containers that must create containers, a computational ouroboros that required careful architectural consideration.

Docker-in-Docker (DinD) represents one of those solutions that feels simultaneously obvious and impossible. The runner container, itself an isolated computational environment, must spawn a Docker daemon capable of building and managing other containers. It's a nested virtualization problem that pushes against the boundaries, nay, the very fabric of what containerization was designed to do.

# The template that enables recursive containerization
containerMode:
  type: "dind"

template:
  spec:
    containers:
    - name: runner
      image: registry.digitalocean.com/redducklabs/github-runner:latest
      command: ["/home/runner/run.sh"]
      resources:
        limits:
          cpu: "2"
          memory: "4Gi"

The solution required not just technical implementation, but a philosophical acceptance of computational recursion. Our runners would exist in a state of controlled schizophrenia- simultaneously the environment that builds containers and the container being built within. The Docker daemon running inside each runner pod became a kind of computational engine, capable of spawning entire application ecosystems from nothing more than text files and imagination.

The Scaling Paradox

As the system took shape, a new class of problems emerged around the concept of scale. How many runners should exist at any given moment? Too few, and builds queue behind each other, creating the same bottlenecks we sought to escape. Too many, and computational resources idle expensively, consuming cluster capacity without purpose.

The solution lay in embracing the ephemeral nature of computational demand. GitHub's Actions Runner Controller (ARC) provided the foundation for dynamic scaling, but the real intelligence lay in understanding the rhythms of development work. Morning commits in Pacific time zones, afternoon deployments in European markets, the predictable surge of activity as deadlines approach- all of these patterns could be encoded into scaling policies.

# The delicate balance of computational availability
minRunners: 2    # Always ready, never waiting
maxRunners: 4    # Burst capacity for peak demand

But the numbers tell only part of the story. The real architecture of scaling lies in the feedback loops between demand and supply, the way the system learns to anticipate need before it becomes urgent. Each runner that spins up represents a bet on future computational demand. Each one that scales down represents a judgment about the ebb and flow of development activity.

The Management Ecosystem

Perhaps the most sophisticated aspect of the entire system emerged not from the runners themselves, but from the management infrastructure that surrounds them. Our runners required a comprehensive suite of monitoring, scaling, and emergency procedures.

The GitHub Actions workflows that manage the runners represent a kind of meta-automation- automation that manages automation. The scale-runners.yml workflow can dynamically adjust capacity based on demand. The runner-status.yml workflow provides continuous health monitoring. The emergency-stop.yml workflow offers a controlled shutdown procedure for crisis situations.

# The emergency stop: controlled destruction when control is lost
- name: Scale to zero
  run: |
    echo "🛑 INITIATING EMERGENCY STOP..."
    helm upgrade "${RELEASE_NAME}" \
      --set minRunners=0 \
      --set maxRunners=0 \
      --wait --timeout 120s

Each workflow represents a different aspect of operational intelligence. They embody the accumulated wisdom of failure modes encountered and resolved, edge cases discovered through painful experience, and the gradual evolution of a system that must be both robust enough to handle routine operations and flexible enough to adapt to unexpected circumstances.

The Security Substrate

Beneath all of this computational choreography runs a security substrate that would be invisible to casual observation but represents a critical aspect of the system. Self-hosted runners, by their very nature, represent a potential attack vector- computational resources that execute arbitrary code from external sources.

The security model layers multiple defensive strategies. Container isolation provides the first boundary, ensuring that runner processes cannot escape their computational sandbox. Resource limits prevent any single job from consuming excessive CPU or memory. Network policies could restrict communication between runners and sensitive internal systems. Image scanning with Trivy identifies vulnerabilities before they can be exploited.

# Security through constraint: minimal privileges, maximum isolation
USER runner
WORKDIR /home/runner

# The runner process: powerful but contained
CMD ["/home/runner/run.sh"]

But the most sophisticated security measure is architectural: the ephemeral nature of the runners themselves. Each job executes in a fresh computational environment that is destroyed immediately upon completion. No state persists between executions. No accumulated cruft can build up over time. Each runner pod is born clean, lives purposefully, and dies completely- a computational lifecycle that eliminates entire classes of security vulnerabilities through the simple expedient of not existing long enough for them to matter.

The Economics of Computational Independence

The financial architecture of the system reveals itself through careful analysis of resource utilization patterns. GitHub's hosted runners operate on a consumption model- you pay for what you use, but you're constrained by what they're willing to provide. Self-hosted runners invert this relationship: higher upfront infrastructure costs in exchange for unlimited computational capacity and complete control over the execution environment.

Image

The break-even analysis depends on usage patterns, but for organizations with consistent CI/CD demands, the mathematics favor independence. More importantly, the value of eliminating external dependencies; the ability to deploy critical fixes without waiting for external resource availability- often exceeds the pure financial calculation.

Lessons from the Architecture of Constraint

The Red Duck Labs runner system is a practical response to a resource constraint. It reflects a preference for self-sufficiency: systems that can operate on their own reduce the risk introduced by external dependencies.

The system isn’t perfect. It needs maintenance, monitoring, and occasional intervention. It uses cluster resources that could serve other workloads and adds complexity compared to hosted runners. In return, it gives us predictable availability and a controlled environment without the usage limits or dependencies of a third-party service.

In the end, the constraints led us to a working solution and a clearer operating model: keep critical infrastructure close, and make sure the team understands and controls it.

Constraints are a given. Within them, we built runners that meet our needs on our schedule. When external limits slow us down, we run our own, one container at a time.

The complete Red Duck Labs runners implementation is available at github.com/redducklabs/redducklabs-runners, where the architecture of necessity continues to evolve in response to the ever-changing demands of computational reality.