Arc — a private AI platform: one OpenAI-compatible API over cloud models and home GPUs

Solution Architect / AI Engineer

Since 2026

Arc — a private AI platform: one OpenAI-compatible API over cloud models and home GPUs — cover image
Java 21Spring Boot 4Python 3.12asyncioMongoDBQdrantOpenAI-compatible APIServer-Sent EventsWebSocket tunnelMCPOllamallama.cppOpenAI / Gemini / Anthropic / DeepSeek APIsaiohttphttpxPydanticSearXNGPlaywrighttrafilaturafaster-whisperAES-256-GCMnginxsystemdPowerShellMavenJUnitpytest

Key Achievements

  • Consolidated a fragmented AI infrastructure into a single platform serving nine production applications, achieving 290/290 successful requests in the migration acceptance replay.
  • Reduced end-to-end latency of a recorded AI turn from 157 seconds to 5.5 seconds through unified execution budgeting and improved request orchestration.
  • Built an OpenAI-compatible API integrating four cloud AI providers and self-hosted GPU inference, with per-key access policies, provider routing, rate limits, and cost accounting.
  • Verified strict provider isolation across 1,000 randomized failure scenarios, with zero requests routed to unauthorized providers.
  • Designed an outbound-only multiplexed WebSocket tunnel connecting private GPU nodes to a public gateway without exposing home network addresses or opening inbound ports.
  • Implemented resumable streaming that delivered a complete 919-word response despite connection interruptions every 3 seconds over a 57-second test.
  • Developed fault-tolerant task execution with node failover, safe replay rules, and protection against duplicated side effects or retraction of already streamed content.
  • Achieved 205 short responses per minute across two home GPU nodes at eight concurrent requests, with 3.7-second p95 latency and zero rejected requests in the benchmark.
  • Designed an evidence-driven research engine combining deterministic extraction and validation with small-batch LLM evaluation, reducing token consumption in a 30-candidate research scenario.
  • Built a 60-task coding benchmark with isolated execution and hidden tests to evaluate local models, supported by 2,768 Java tests, 3,129 Python tests, and replay of 475 recorded production tickets.

Details

Arc is a private AI infrastructure platform that unifies cloud AI providers and self-hosted GPU inference behind a single OpenAI-compatible API. Applications can use standard OpenAI SDKs while Arc handles provider selection, access policies, workload scheduling, fault tolerance, execution budgets, and cost tracking.

Built to replace a fragmented multi-service AI stack, Arc has been running in production since October 2026, serving nine applications through one unified interface. The migration was validated by replaying recorded application traffic, with all 290 acceptance requests passing.

The architecture is built around four core components:

  • Edge — API compatibility, authentication, provider routing, access policies, and cost accounting.

  • Hub — workload scheduling, execution budgets, tool orchestration, and request lifecycle management.

  • Nodes — GPU-powered inference and processing workers that connect securely without inbound ports.

  • Research Engine — evidence-driven research combining deterministic data processing with targeted LLM evaluation.

The platform is designed to handle both interactive requests and long-running AI workloads, while preserving control over privacy, costs, reliability, and infrastructure resources.

Edge: one entrance for everything

Every request enters through the Edge, a Java 21 / Spring Boot service. It authenticates the key, applies the key's policy and picks a provider only from the allowed intersection: enabled globally, allowed for this key, capable of this request, not cooling down.

  • Standard API. Responses, Chat Completions, Conversations, Embeddings, Files, Vector Stores, audio transcription, image generation and Models, in the OpenAI wire format. Answers stream over SSE and a stream can be resumed from any event; long work runs in the background with polling or signed webhooks.

  • Routing and fallback. The model name decides the path. A provider's model id goes to OpenAI, Gemini, Anthropic or DeepSeek; arc, arc-research and arc-whisper go to the home nodes; auto lets the router choose. A failing provider is replaced by the next one until the first token has been streamed, and never after.

  • Key policy. Per key: allowed providers and models, spend per day and per month in USD, requests per minute, capabilities (tools, images, audio, research, background work) and a price ceiling per million tokens.

  • Provider key pools. Several keys per provider, each with its own rate limits. A key that hits a limit cools down, and cooldowns survive a restart.

  • Prices and cost. Price tables are synced daily from the providers' pricing pages and pass guard rails before they are accepted. Every response carries usage.cost, computed once against the price snapshot of that moment, with cached and reasoning tokens counted correctly. An unknown price is reported as unknown, never as zero.

Hub: the queue and the turn

A request for a home model becomes a ticket in the Hub, which shares a process with the Edge. The Hub owns the queue, the registry of nodes and the budget of every turn.

  • Queue. Strict priorities (a person in a chat goes before batch work, research goes last), no preemption, and fairness inside a priority: the key with the fewest running tickets goes first, so one client cannot starve another.

  • Turn controller. A turn is one object with one budget in four dimensions: seconds, tokens, USD and tool calls. It runs in four stages (before the model, answer or plan, the tool loop, the checks after), and any stage can end the turn with what it has. At 60% of the budget the model is told to finish; a reserve is kept, so a turn always ends with an answer and an honest note about what was not done.

  • Checks. A JSON schema with one retry. Facts such as URLs, versions and quantities must appear in the evidence the turn gathered. An answer that claims an action ("booked") without a successful tool call is sent back. A tool with side effects runs only after the user has confirmed the proposal.

  • Trace. One trace id runs from the first byte to the model call. Every model call and every tool call is a record with its duration, tokens and cost, so a slow turn is explained stage by stage.

  • Tools. Application tools are called by the Hub over MCP, so an application's secrets never reach a home machine. Web tools run on the node.

Nodes: home GPUs that come and go

A node is one Python process on a home machine with a GPU. At start it inspects the hardware and its local model server (Ollama) and announces a manifest: models, context sizes, executors and how many tasks it carries. Nothing is listed by hand. The executors cover chat, completion by schema, research, transcription, video, vision and embeddings.

The Hub sends chat to the strongest GPU and spills to the next node when the expected wait passes 30 seconds; research starts on the smallest GPU and spills upward. A node may vanish in the middle of a task: the ticket is replayed on another node, but only if no reader could have seen content, never after a tool with side effects has run, and at most three times. A workstation lends its GPU on its owner's terms: a tray agent pauses the node, lets the running turns finish, frees the GPU, and can bring the node back after the machine has been idle.

The tunnel: no inbound ports

Home machines accept no connections. Each node dials out to the VPS over a purpose-built multiplexed WebSocket tunnel, extended for Arc with per-node tokens and routing by node. Tasks ride down the lanes the node opened, and tokens stream back the same way. If a lane drops in the middle of an answer, the node keeps working and the Hub re-attaches on another lane from the last event it saw, without losing a token.

Egress: the web sees the VPS

The web requests of the tools (search, page fetches, a headless browser) are sent by the node as frames through the same tunnel and performed by the gateway on the VPS. A site sees the address of the VPS; the home address never appears. The gateway refuses private and loopback destinations and every port except 80 and 443. Paid search and fetch APIs are called only by the Hub, so their keys never leave the VPS, and their cost is charged to the budget of the turn. robots.txt is binding, and CAPTCHAs and paywalls are not bypassed.

Research engine

A research turn follows one rule: the code enumerates, the model judges. A mid-size local model judges a screenful of text well and enumerates a long page badly, so everything that is enumeration, deduplication, normalisation or verification is deterministic code. The model writes the contract of the answer and then judges candidates in small batches. An item is confirmed by a primary source, by two independent sources or by a deterministic witness; unconfirmed items stay in the answer, marked as such. Decisions run at temperature 0 and pages are cached for the day, so the same question gives the same answer tomorrow.

Packs: a product is configuration

A themed assistant for a business is a pack: a prompt, a tool policy, knowledge requirements, a budget and a set of knobs (depth, research, egress, language). A tenant is one customer of a pack, with its own knowledge base and tool server. The node code is the same for every pack: a new assistant is a pack plus data, with no new code.

Storage and secrets

MongoDB holds tickets, events, conversations, packs and the call log; Qdrant holds knowledge and evidence as 1024-dimension, int8-quantized vectors. A node never connects to either. Provider keys are encrypted field by field with AES-256-GCM and bound to their place in the database, so a copied ciphertext does not decrypt anywhere else; API and node tokens are stored as bcrypt hashes.

Coder bench

A separate tool answers a practical question: which coding tasks can be handed to a local model. Sixty original tasks in Python and Java, some cut from this project's own history, are judged only by hidden tests run in a sandbox without network. The model gets one repair round, in which it sees nothing but the failures of the public tests. Every task is validated before use: the reference solution must pass, and the untouched workspace and every plausible wrong solution must fail.

Gallery

Need reliable AI infrastructure combining cloud models and private GPU inference?

I designed and developed Arc end to end, from its OpenAI-compatible API and policy-based routing to private GPU orchestration, fault-tolerant streaming, research workflows, and automated testing. The platform replaced a fragmented AI stack and now serves nine applications through a unified interface. If your product needs the same skills, write to me — a few lines about the task are enough — or look through my CV first.

Get in touch