Budgeted Interpreter: Gas for Your API

· 7 min read

Budgeted Interpreter: Gas for Your API

What if your API had gas limits like Ethereum?

Not arbitrary rate limits. Not “429 Too Many Requests” after the fact. But semantic metering that tracks what operations cost and stops execution before you exceed your budget.

And when you run out of gas? Your resources clean themselves up.

The Problem with Traditional Rate Limiting

// Traditional: Count requests
if (user.requestCount > 100) {
  return 429; // Too bad, so sad
}

// But what about...
// - A request that opens 50 database connections?
// - A request that reads 10GB from disk?
// - A request that spawns 1000 subprocesses?

One request is not like another. A “get user profile” is not the same as “export entire database.”

And when you do hit the limit mid-execution, what happens to the resources you’ve already acquired? Traditional rate limiting doesn’t even ask that question.

Event Costs in Koru

In Koru, events have costs. You declare them in scope registration:

~std.runtime:register(scope: "api") {
    get_user(1)           // Cheap: just a lookup
    list_users(10)        // More expensive: pagination
    export_data(100)      // Very expensive: bulk operation
    open_file(10)         // Acquires a resource
    close_file(1)         // Releases it
}

These aren’t arbitrary numbers. They’re relative costs that let you express “this operation costs 100x more than that one.”

Running with a Budget

When you invoke the runtime, you provide a budget:

~std.runtime:run(source: user_code, scope: "api", budget: 1000)
| result r   |> respond_200(body: r.value)
| exhausted e |> respond_429(
    message: "Budget exhausted",
    used: e.used,
    last_event: e.last_event,
    handles: e.handles
)

If the user tries to run code that exceeds their budget, they get exhausted with details:

  • How much budget was consumed
  • Which event pushed them over the limit
  • How many handles are still undischarged after auto-discharge

No ambiguity. No “you made too many requests.” Just “you tried to export_data but only had 50 tokens left and it costs 100.”

Obligations from Phantom Types

Here’s where it gets interesting. Koru has phantom types for compile-time resource tracking:

~pub event open { path: []const u8 }
| opened { file: []const u8[opened!] }    // Creates "opened" obligation (handle ID)

~pub event close { file: []const u8[!opened] }  // Discharges obligation
| closed {}

The [opened!] annotation means “this output creates an obligation.” The [!opened] means “this input discharges one.”

The scope registration automatically extracts obligations from event signatures:

~std.runtime:register(scope: "api") {
    open(10)    // Cost: 10. Creates: "opened". Extracted from signature!
    read(5)     // Cost: 5.  No obligations.
    close(1)    // Cost: 1.  Discharges: "opened". Extracted!
}

No duplication. The type system already knows which events acquire resources and which release them.

The Handle Pool

At runtime, the interpreter tracks all undischarged obligations in a handle pool:

[Request starts with budget: 100]
~open(path: "data.txt")     → Handle h1: "opened", cost: 10 (remaining: 90)
~open(path: "index.txt")    → Handle h2: "opened", cost: 10 (remaining: 80)
~read(file: h1)             → cost: 5 (remaining: 75)
~close(file: h2)            → h2 discharged, cost: 1 (remaining: 74)
[Request ends normally]
→ h1 still undischarged - auto-discharge triggered!
→ ~close(file: h1) called automatically

The interpreter knows which event discharges each obligation because it extracted that from the phantom types. When execution ends - whether normally or due to budget exhaustion - it walks the undischarged handles and calls the appropriate cleanup events.

Auto-Discharge on Exhaustion

This is the killer feature. What happens when budget runs out mid-execution?

// User code
~open(path: "a.txt")
| opened f1 |>
    open(path: "b.txt")
    | opened f2 |>
        open(path: "c.txt")   // Budget exhausted here!
        | opened f3 |>
            // Never reached
            close(file: f3) |>
            close(file: f2) |>
            close(file: f1)

Traditional systems would leak f1 and f2. The user code never got to close them.

With auto-discharge:

[Budget exhausted at open("c.txt")]
[AUTO-DISCHARGE] Invoked 'close' for handle 'f2' [opened]
[AUTO-DISCHARGE] Invoked 'close' for handle 'f1' [opened]
| exhausted { used: 100, last_event: "open", handles: 0 }

The interpreter automatically calls close for each undischarged handle. Resources don’t leak, even when execution is terminated early.

Scope Isolation

Handles are tagged with their owning scope. A handle created in scope “api” can’t be discharged by code running in scope “admin”:

// Scope "api" - limited capabilities
~std.runtime:register(scope: "api") {
    open(10)
    read(5)
    close(1)
}

// Scope "admin" - full access
~std.runtime:register(scope: "admin") {
    scope(api)              // Include api events
    delete_file(50)         // Plus dangerous ones
    format_disk(10000)
}

If code in “api” scope opens a file, and you later run code in “admin” scope, the admin code can’t close the api’s file handle. Different namespaces.

Persistent Sessions

For multi-turn interactions (REPL, chat, LLM tool calling), you need handles to persist across requests. The interpreter accepts an external handle pool (auto-discharge is disabled for external pools):

// Your session management
var session_pool = HandlePool.init(allocator);

// First request: open a file
interpreter.run(source1, scope, budget, &session_pool);
// session_pool now contains handle for the opened file

// Second request: read from it
interpreter.run(source2, scope, budget, &session_pool);
// Same pool, same handle available

// Session ends: bridge decides when to discharge remaining handles
for (session_pool.getUndischarged()) |handle| {
    // ... cleanup
}

The interpreter provides the mechanism. Your application provides the policy - session timeouts, budget refills, user tiers.

Why This Matters

For LLM Tool Calling: Claude calls your API. Each tool invocation has a cost. If Claude goes wild opening files, budget exhaustion triggers auto-cleanup. No leaked resources.

For Multi-Tenant SaaS: Different customers get different budgets. The metering is in the interpreter, not sprinkled throughout your handlers. And resource cleanup is automatic.

For Sandboxed Execution: Run untrusted code with hard limits. No infinite loops draining resources. No “oops, they opened 10,000 files” - even if they try, they’ll hit budget limits and everything gets cleaned up.

The Full Picture

  1. Compile time: Phantom types declare which events create/discharge obligations
  2. Scope registration: Costs and obligations extracted from event signatures
  3. Runtime: Budget tracking, handle pools, scope isolation
  4. Cleanup: Auto-discharge on normal exit and budget exhaustion

It’s gas for your API. But unlike Ethereum, when you run out of gas, you don’t leave the blockchain in a half-mutated state.

Your resources clean themselves up.