The Comptime Architecture: Building a Metacircular Compiler

· 12 min read

Most compilers are monoliths written in C++ with architecture frozen decades ago. Koru’s compiler compiles itself - and the architecture emerged through conversation, not specs. Let’s talk about how we built a metacircular compiler through narrative development.

The Promise: Code That Compiles Itself

Here’s what happens when you run koruc my_app.kz:

Pass 1 (Frontend): Static compilation

my_app.kz → AST → backend.zig

Pass 2 (Backend): Dynamic orchestration

backend.zig → Zig compiles it → ./backend runs → output_emitted.zig

Pass 3 (Final): Runtime executable

output_emitted.zig → Zig compilation → my_app

The backend isn’t just generated code - it’s a compiler as an executable. When you run ./backend, your [comptime] events execute as a regular program. They transform the AST and emit output_emitted.zig. Compilation is just a program running!

The Two-Phase Architecture

Phase 1: Frontend - Static Pipeline

The frontend (koruc) is a traditional compiler pipeline, but with a twist - it’s designed to be replaced by the backend it generates.

Source → Parse → Type Check → Transform → Emit → backend.zig

Current reality: Fixed, deterministic pipeline

  • Parses Koru source → AST
  • Validates types and branch coverage
  • Runs optimization passes (dead code elimination, fusion)
  • Emits serialized AST + compiler orchestration code

Key insight: The frontend is bootstrapping infrastructure. It generates a compiler (backend.zig) that will eventually replace it. The frontend’s job is to become obsolete.

Phase 2: Backend - Dynamic Orchestration

Here’s where it gets wild. backend.zig isn’t just compiled code - it’s a running compiler:

// This is backend.zig - generated by the frontend
pub const PROGRAM_AST = SourceFile{
    // Your entire program, serialized
};

pub fn main() void {
    // This runs when you execute ./backend (Zig runtime!)
    const result = compiler.coordinate.default.handler(.{
        .ast = &PROGRAM_AST,
        .allocator = allocator,
    });

    // Koru [comptime] events execute HERE
    // Program transformations happen HERE
    // The output (output_emitted.zig) is WRITTEN by this program
}

The magic: The backend compiles to an executable that runs as a program. When you execute ./backend, its main() function runs - this is regular Zig runtime, but it’s Koru’s compilation phase:

  • Events with [comptime] annotation execute when backend runs (Zig runtime!)
  • They can read the AST, transform it, emit new code
  • The result (output_emitted.zig) is generated by a running program

Key insight: Koru comptime = Zig runtime (for the backend). Compilation isn’t magic - it’s just a program running!

Wait, So No Zig Comptime?

You could use Zig’s comptime features inside your [comptime] procs if you wanted:

~proc optimize {
    // This proc runs when ./backend executes (Zig runtime)
    const result = comptime analyze(ast);  // Zig comptime WITHIN the proc
    // But the proc itself runs at Zig runtime!
}

But Koru’s compilation model is simpler: Compilation is just running an executable. The backend is a program. You run it. It writes output_emitted.zig. Done.

This means you can debug compilation with a regular debugger, profile it with regular tools, and understand it as regular code. No special “comptime” environment - just a program that generates code.

Aspiration: Fully dynamic, user-controlled pipeline

  • Users write compiler passes as regular Koru events
  • ~compiler.coordinate events orchestrate the compilation
  • Optimization passes are just events that transform ProgramAST
  • The pipeline is data, not hardcoded logic

How We Got Here: No Specs, Just Conversations

This architecture wasn’t designed in a 50-page document. It emerged through dialogue. Let me show you the narrative development in action.

Discovery #1: “Why is main.zig doing emission?”

Recently, we hit a regression - module scoping tests were failing. The compiler was generating code like:

// Generated at TOP LEVEL - wrong scope!
const test_lib_user_create_handler = struct {
    pub fn handler(...) Output {
        const user = User{ ... };  // ❌ Error: User not in scope!
    }
};

// User type defined INSIDE module struct
pub const koru_test_lib = struct {
    pub const user = struct {
        pub const User = struct { ... };  // User is HERE
    };
};

Investigating the code, we found main.zig (frontend) was generating 150+ lines of module handler emission logic - duplicating work that visitor_emitter.zig already did correctly!

The conversation that fixed it:

“Hold on. I DON’T UNDERSTAND WHY main.zig IS DOING EMISSION. Wasn’t the WHOLE POINT that BOTH the frontend AND the backend USED THE SAME EMISSION LIBRARY?”

That question revealed an architectural violation:

  • ❌ main.zig generating handlers (wrong - frontend should be minimal)
  • ✅ visitor_emitter.zig generates ALL user code (correct - library reuse)

The fix: Delete 192 lines from main.zig. Trust the library.

Result: 5 tests went from ❌ → ✅

Discovery #2: “Profiling should work at compile time too”

The profiling feature lets you instrument your entire program with one line:

~[profile]import "$std/profiler"

But tests were failing:

backend_output_emitted.zig:43:21: error: struct 'main_module' has no member named 'profiler_event'

The profiler’s universal tap (~* -> *) was trying to call profiler_event during backend compilation, but the event wasn’t there!

The realization:

~event profiler { source: []const u8, ... }  // No annotation
| done {}

With no annotation, events default to runtime-only. But the profiler needs to work during BOTH compilation (instrumenting) AND runtime (collecting data).

The conversation:

“Don’t we need to be able to mark events ~[comptime|runtime] for them to be emitted as available BOTH during compile AND runtime?”

That’s the design! But the filtering logic only checked module-level annotations, not item-level.

The fixes:

  1. Check annotations at BOTH item and module level
  2. Don’t filter @import statements by phase (always available)
  3. Mark profiler events with [comptime|runtime]

Result: 4 more tests went from ❌ → ✅

The Score: 102 → 111/112 baseline

Started with 10 tests failing. Fixed through conversation. Now within 1 test of perfect parity with main branch.

No specs. No requirements docs. Just narrative development.

The Architecture Emerges

Let’s formalize what we discovered:

Frontend Responsibilities

  1. Parse source → AST
  2. Validate types → Catch errors early
  3. Serialize AST → Embed in backend.zig
  4. Generate bootstrap → compiler.* events for orchestration
  5. Emit nothing else → Trust the backend

Key files:

  • main.zig - Entry point, minimal
  • parser.zig - Koru → AST
  • visitor_emitter.zig - THE emission library (used by both frontend and backend!)

Backend Responsibilities

  1. Deserialize AST → Reconstruct program structure
  2. Execute comptime events → Run as a regular program (Zig runtime!)
  3. Orchestrate compilation → Call compiler.coordinate
  4. Emit final code → Generate output_emitted.zig
  5. Support profiling → Emit comptime AND runtime handlers

Key files:

  • backend.zig - Generated per-program (contains serialized AST)
  • backend_output_emitted.zig - Generated comptime handlers
  • compiler_bootstrap.kz - Compiler events (coordinate, emit, etc.)
  • visitor_emitter.zig - SAME emission library frontend uses!

The Metacircular Vision

Right now, the frontend is written in Zig. But the backend can already run Koru code at compile time. The endgame:

Frontend in Koru:

// Instead of Zig code in main.zig:
~compiler.parse(source: Source)
| parsed { ast: ProgramAST } |> compiler.validate(ast: ast)
  | valid v |> compiler.optimize(ast: v.ast)
    | optimized o |> compiler.emit(ast: o.ast)
      | emitted e |> save(e.code, "backend.zig")

Backend in Koru:

// Already working today!
~compiler.coordinate(ast: ProgramAST)
| coordinated c |> compiler.emit(ast: c.ast)
  | emitted e |> save(e.code, "output_emitted.zig")

When both phases are Koru, the compiler compiles itself completely. True metacircularity.

Library Reuse: The Key Insight

The breakthrough was realizing: THE SAME CODE SHOULD EMIT IN BOTH PHASES.

Before:

  • main.zig had emission logic (150+ lines)
  • visitor_emitter.zig had emission logic (different code!)
  • Duplication → bugs

After:

  • visitor_emitter.zig is THE emission library
  • Frontend uses it: visitor_emitter.emit(ast, .runtime_only)
  • Backend uses it: visitor_emitter.emit(ast, .comptime_only)
  • Backend uses it again: visitor_emitter.emit(ast, .runtime_only) for final output

One library. Three uses. Zero duplication.

Phase Annotations: Ambient Infrastructure

The annotation system controls what code appears in which phase:

// Runtime only (default)
~event fetch { url: []const u8 }
| success { data: []const u8 }

// Comptime only
~[comptime] event optimize { ast: ProgramAST }
| optimized { ast: ProgramAST }

// BOTH phases (profiler, debugging, instrumentation)
~[comptime|runtime] event profile { source: []const u8, ... }
| done {}

The magic: Imports are ALWAYS available (both phases):

const std = @import("std");  // Available everywhere!

This enables code like:

~[comptime|runtime] proc profiler {
    std.debug.print("Profile: {s}
", .{source});  // std works in both phases!
}

What This Enables

Full-Program Profiling

~[profile]import "$std/profiler"

// Automatically instruments EVERYTHING
// Profiler events run at compile time (setup)
// AND runtime (data collection)

Compile-Time Optimization

~[comptime] event inline_all { ast: ProgramAST }
| inlined { ast: ProgramAST }

// Runs DURING backend compilation
// Transforms program before runtime code generation

Meta-Programming That Just Works

~compiler.coordinate(ast: my_program)
| coordinated c |> custom_pass(c.ast)
  | optimized o |> compiler.emit(o.ast)

The Philosophy

Traditional compilers are built bottom-up:

  1. Write architecture docs
  2. Implement according to spec
  3. Debug when reality diverges from spec
  4. Update spec (maybe)

Koru’s compiler was built narrative-first:

  1. Write code that tells a story
  2. Tests fail → story has plot holes
  3. Ask “why is this confusing?”
  4. Rewrite the story to be clearer
  5. Tests pass → story makes sense

The architecture emerged. We didn’t plan for visitor_emitter to be shared between frontend and backend - we discovered it when duplication caused bugs. We didn’t spec out [comptime|runtime] annotations - we needed them when profiling didn’t work.

The code IS the documentation. The tests ARE the verification. The narrative IS the design.

Current Status

Test Results: 111/112 baseline (99.1% parity with main branch)

What Works:

  • ✅ Two-phase compilation (frontend → backend → output)
  • ✅ Metacircular backend (Koru code runs during compilation)
  • ✅ Library reuse (visitor_emitter used by both phases)
  • ✅ Phase annotations ([comptime], [runtime], [comptime|runtime])
  • ✅ Profiling at both compile-time and runtime
  • ✅ Module scoping (handlers emitted in correct scope)
  • ✅ Import availability (std accessible everywhere)

What’s Coming:

  • 🎯 Fully dynamic backend pipeline (user-defined compiler passes)
  • 🎯 Frontend in Koru (self-hosting compiler)
  • 🎯 Plugin system (compiler extensions as imports)
  • 🎯 Live compilation (incremental updates during development)

The Future Is Narrative

We built a metacircular compiler through conversation. No architecture documents. No upfront specs. Just questions:

  • “Why is main.zig doing emission?”
  • “Should profiling work at compile time?”
  • “What if events could transform programs?”

Each question led to discovery. Each discovery improved the architecture. Each improvement was validated by tests.

This is narrative development. The compiler’s story became clearer through iteration. And now it compiles itself.

The future of compilers isn’t better specs. It’s better storytelling.

Want to discuss the architecture? Found a plot hole in the narrative? Open an issue or PR on GitHub.