Developer Documentation

Backend

Python / Flask / gunicorn

Frontend

Next.js 16.2.12 / React 19 / TypeScript

Database

Redis (in-memory store + work streams)

Library DB

SQLite (public-domain book corpus)

Search

Apache Solr (full-text)

AI Inference

Ollama — local model + cloud escalation

Speech

Kokoro neural TTS — local synthesis, MP3 output

Metrics

Prometheus + Grafana (corpus and pipeline telemetry)

Auth

Auth.js v5 beta — Google + GitHub OAuth, JWT sessions

Proxy

Caddy (automatic TLS via Let's Encrypt)

Process Mgr

arc.sh + systemd (auto-starts on boot)

Arc is not a single box. Responsibilities are split across roles that run independently: ingestion, storage, and serving are one role; local analysis is a second role; speech synthesis is a third; and a portable node exists for offline authoring work. Synthesis runs on its own role by design — audio generation is memory-hungry, and keeping it off the analysis role means neither ever waits on the other. No role but the synthesis role produces speech.

Within each role, the services are managed by a single control script and auto-start on boot. A watchdog supervises them at runtime and restarts any that crash, distinguishing a deliberately-stopped service from a failed one.

What runs: RSS ingestion (the Scribe); the on-demand analysis worker (the Analyzer) that produces the three A.R.C. passes lazily on an article’s first view; on-demand and manual publishing of user submissions; automated posting to Bluesky, Mastodon, and Facebook (each toggleable at runtime with no restart); an email digest; and the Next.js frontend.

Posting is fail-safe, not fire-and-forget: the posters track which articles they have published separately, so a mid-publish failure re-tries on the next cycle rather than double-posting or silently dropping.

A Caddy reverse proxy terminates TLS and routes requests: authentication and user-preference calls are handled by the Next.js application server, and the remaining API traffic goes to the Flask backend. Everything else renders from Next.js. Automatic certificate management is handled by Caddy.

Arc Codex exposes a read-only public HTTP API over the same domain. It serves the article feed, individual articles with their full analysis, full-text search, an RSS feed, the wiki directive pages, the public-domain library, and the machine-readable sitemap. Signed-in users can additionally submit content for processing and manage their own preferences.

Machine-readable discovery surfaces — /rss.xml, /sitemap.xml,/news-sitemap.xml, and /opensearch.xml — are published and kept current automatically.

Live application data is held in Redis for speed; the public-domain book corpus behind the Library lives in SQLite; and full-text search is served from Solr. At a conceptual level the system stores:

Soft auth — the site is fully public. Signing in with Google or GitHub is optional and unlocks preferences, publishing, and private articles. There is no username/password fallback and no third-party tracking.

Sessions are JWT-based (stateless), and preference writes are accepted only from the application server itself — never directly from the public internet — so a user can only ever change their own settings.

Inference is tiered and demand-gated: a compact local model, running on the dedicated analysis role, handles the bulk of the work, and a larger cloud model is reached only on escalation, within a weekly budget. The Red / Blue / Purple analyses are computed lazily — on an article’s first view rather than at ingest — so inference cost tracks readership, not ingest volume. Published articles are retained for roughly a month before they are pruned. Translation degrades gracefully when a model is unavailable: “model unavailable” is shown rather than a hard failure.

Translation is a click, not an auto-fire in the feed.A scrolled feed holds many mounted cards; firing translation on each mount would overwhelm the inference tier. A preferred language is a shortcut that skips the picker — it does not translate the whole feed automatically.

The audio a listener hears is not the source article read aloud. After the Red / Blue / Purple passes complete, an additional analysis pass writes an original short broadcast piecedrawn from what those three passes found — the verified facts, the balanced summary, and the anti-pattern reading. That written piece, not the article the Scribe ingested, is what gets spoken. The reasoning is editorial: reading source prose aloud reproduces its framing verbatim; narrating from Arc’s own analysis passes yields a piece whose voice is the site’s, whose claims trace back to what the analysis actually concluded, and whose length is chosen for the ear rather than the page.

The broadcast piece is then handed to a neural text-to-speech model (Kokoro), which renders it to audio locally — there is no cloud speech service, no per-character billing, and no third party receives the text. Long pieces are split into chunks, synthesised in sequence, then concatenated and encoded to a compact mono MP3 sized for slow connections rather than for fidelity.

Narration is opportunistic rather than blocking. Publishing never waits on audio: a pass runs each cycle, picks the newest article still lacking a recording, and defers if the machine is busy. A deferred article is simply retried next time round. Recent narrations are also concatenated into a rolling bulletin — a single continuous audio stream of the day’s reporting, intended for listeners who want the news without a screen.

Synthesis is its own role, and yields to analysis anyway. Speech generation runs on a dedicated role so it cannot starve analysis of memory; on top of that, a pre-flight check confirms there is genuine headroom before a run starts, and if there is not, narration steps aside rather than competing with the rest of the pipeline. Audio is the part of the system that can afford to be late.

The pipeline is instrumented rather than trusted. Metrics are scraped continuously and rendered as dashboards covering ingest rate, analysis latency, inference tiering, and corpus-level qualities — the average reading difficulty and objectivity of what has actually been published, not merely how much of it there is.

Alerting distinguishes liveness from output. A worker publishes a heartbeat on a short expiry, so its silence is itself the signal; that is a separate question from whether the day produced many articles or few. Conflating the two produces an alarm that fires on every quiet afternoon and is therefore ignored when it matters.

Narration adds a third kind of check, because its failure mode is silent. A synthesis worker can be up and consuming memory while producing no audio at all — a heartbeat would still be green. The narration-liveness check therefore ignores process state and watches the output itself: if audio has not landed for the newest publishable articles within the expected window, that absence is the alert. Liveness is measured by what arrived, not by what is running.

An alert that cannot clear is not an alert.Conditions are edge-triggered and paired with an explicit all-clear, so a fault that resolves itself says so. Without that, a recovered incident and an ongoing one look identical from the outside.

• Feed rendering

  The lazy-loading feed structure is load-bearing — changes are surgical, never structural.

• Theme layer

  A single stylesheet layer is the source of truth for colours and overrides everything else.

• Preferences

  One context is the single source of truth for user preferences across the app.

• App Router

  Next.js 16 App Router with Turbopack. Not the pages router.

• No ads

  Fully ad-free by design. No ad networks, no analytics beacons.

Full-text search is served by Apache Solr, indexed over the article corpus (title, content, source, directive, and the reading-difficulty score). Search reconnects lazily so a restart of either the search engine or the application resolves itself without manual intervention.

Today, Arc Codex is one instance reading the world alone. The RSS ingestion is cheap; the analysis is the expensive part — the three A.R.C. passes, the sentinel verdict, the counter-analyst comment — and every instance that runs the pipeline re-derives conclusions its peers have already computed.

The direction is a network of instances that share their analyses rather than each re-computing them. An operator standing up their own node would contribute what their instance analyzes and draw on what others have already analyzed — a corpus larger than any one machine’s ingest, with cross-verification implicit in the moments several independent nodes reach the same read on the same story.

Trust would be per-node. An operator decides which peers they read; a peer’s consensus becomes visible without becoming binding — annotation, never authority, consistent with the editorial principle applied to the feed itself.

Nothing here is live.The pieces that would need to land first — a stable identifier for each analysis independent of the article’s local ID, a signature scheme so a shared analysis carries its provenance, and a peer-discovery layer that survives an operator leaving — are named to give the direction a shape, not to promise a date.

An operator with a spare PC on the same LAN as an Arc instance should be able to boot it from a USB stick and have a new analyst node come up reachable, cloud-off, and ready to be named in arc.cfg. No manual OS install, no per-machine configuration, no key exchange after the fact — the appliance is the whole setup.

Two of the pieces exist. A bootable image brings a cold machine up to a reachable inference host in one boot — it carries Ollama, the local model, network config, and a systemd unit that binds the service to the LAN behind the host firewall. An Ansible role covers the same territory for machines that already have an operating system installed. Both are working inputs to a cluster; neither is a cluster on their own.

What isn’t there yet is the connective tissue that would turn one working node into a member of a cluster: a way for a booted node to announce itself to an arc instance rather than the operator hand-copying an IP; a nodes section in arc.cfg that enumerates the fleet; and dispatch inside the Analyzer that spreads work across the named nodes instead of pinning to a single OLLAMA_URL. The image and the role that already exist live in private trees while a secrets split completes — the built USB currently bakes an operator SSH key, and the Ansible tree names hosts by their LAN address — so publication follows that work, not this section.

Nothing here is live. The pieces that would need to land first — node self-announcement, a nodes section in arc.cfg, a multi-analyst dispatcher inside the Analyzer, and the secrets split that makes the image and the role safe to ship — are named to give the direction a shape, not to promise a date.