Building a Bloomberg-grade observatory in twelve weeks: the architecture pattern

Why the same skeleton keeps winning #

The deluair platform family now spans thirteen working observatories. Argus tracks macro indicators across one hundred and ninety economies. Aegis monitors sanctions, export controls, and counterparty risk. Athena watches higher education finance. Ceres covers agricultural commodities. Promethean follows the AI capex cycle. Salus tracks public health surveillance. Sisyphus catalogues productivity research. Hercules indexes infrastructure pipelines. Five additional platforms in fintech, climate, and labor sit at varying stages of internal beta. Despite the breadth of subject matter, the underlying engineering looks almost identical from one to the next.

That convergence was not planned, it was discovered. The first three observatories were built bespoke, with different storage layers, different web frameworks, and different orchestration tools. By the fourth, the team noticed that ninety percent of the work was identical plumbing, and only the domain logic actually varied. Standardizing the skeleton freed the analysts to spend their time on what clients pay for, namely the questions the data is supposed to answer, rather than on yet another bespoke ETL pipeline.

The four-layer pattern #

Every deluair observatory is built from four layers, stacked in the same order. The bottom layer is a set of asynchronous Python collectors, each one wrapping a single upstream source: a central bank API, a government bulletin, a regulator filing portal, a commercial vendor feed, or a parsed PDF release. Collectors share a small library that handles retries with exponential backoff, polite rate limiting, on-disk response caching, and structured logging. A failed collector never blocks its siblings, and the cache layer means a re-run after a network blip costs almost nothing.

The second layer is a SQLite warehouse. SQLite is unfashionable for a data platform, but it is also the right answer for the size of these problems. A typical observatory holds between two and forty gigabytes of normalized time series, documents, and metadata. SQLite handles that comfortably on a single node, with FTS5 providing full text search across documents and headlines, and aiosqlite giving the FastAPI layer non-blocking access. There is no Postgres to operate, no Elasticsearch cluster to babysit, and the entire warehouse can be backed up by copying a file.

FastAPI plus Jinja, not a single page app #

The third layer is a FastAPI service that exposes both an HTML dashboard and a JSON API from the same router tree. Pages are rendered with Jinja templates and progressively enhanced with HTMX for partial updates and D3.js for charts. There is no React build step, no bundler configuration, and no separate frontend deployment. A junior analyst can ship a new dashboard panel in an afternoon by writing a query, a template fragment, and a route, with no JavaScript toolchain to navigate.

The same FastAPI app exposes a JSON API on identical paths, suffixed with a content negotiation header or an explicit prefix. Clients who want to pull data into their own systems get clean, documented endpoints without a parallel codebase. OpenAPI documentation comes free from FastAPI, which means the API is self describing for both human integrators and AI agents that consume it as a tool.

The brain layer and how it earns its keep #

The fourth layer is what clients usually mean when they say they want an AI tool. A Claude-backed analytical brain sits behind a small set of tool definitions that mirror the JSON API. Given a question, the brain plans a sequence of warehouse queries, runs them through the same FastAPI endpoints the dashboard uses, and returns a narrative answer with citations back to the underlying rows. Prompt caching keeps the per-question cost in the low cents, even with large schemas in context.

The brain is also the engine behind scheduled briefings. Each morning, a small orchestrator asks Claude to summarize what changed in the warehouse overnight, flag anomalies against rolling baselines, and draft a client-ready memo. Analysts then edit rather than author, which is where the productivity gain actually lives. The Delphi forecasting layer and the Aegis risk layer both anchor on this pattern: Delphi turns warehouse history into probabilistic forecasts, and Aegis turns warehouse documents into counterparty risk narratives.

Twelve weeks, four phases #

The build runs in four overlapping phases. Weeks one and two are specification: a senior analyst and an engineer interview the client, inventory the upstream sources, and write a one-page schema plus a list of the questions the platform must answer. Weeks three through six are collectors: one engineer pairs with the analyst to write and harden the asyncio collectors, with daily smoke runs against the live sources. Weeks seven through nine are analytics, where the warehouse schema firms up, derived tables and indicators land, and the brain layer is wired to the API. Weeks ten through twelve are dashboards and API hardening, with iterative client review every Friday.

Staffing is small on purpose. A typical engagement runs with one engineer, one domain analyst, and a fractional designer, with a partner reviewing weekly. The narrow team is possible because the skeleton is reused, the frontend is server rendered, and the warehouse is a file. Most of the calendar risk lives in the collectors, where upstream sources misbehave in surprising ways, which is why phase two gets four weeks rather than three.

Three engagements, same skeleton #

A North American agribusiness commissioned a Ceres derivative to monitor fertilizer feedstock pricing, ocean freight, and trade policy in a single view. The engagement shipped in eleven weeks with eighty-seven collectors, a fourteen gigabyte warehouse, and a daily brief that the procurement desk now reads before the morning call. The brain answers ad hoc questions about basis risk and lane economics, with citations back to source documents.

A sovereign development bank in Southeast Asia asked for an Aegis variant tuned to sanctions exposure across its loan book. The team delivered in thirteen weeks, with a delay caused by a late upstream change at a sanctions list provider. The platform now screens new counterparties in under a minute and produces audit-grade memos that the compliance committee accepts without rework.

A workforce intelligence firm needed a labor market observatory across thirty-eight metropolitan areas, blending job posting data, wage surveys, and unemployment insurance microdata. Twelve weeks, one engineer, one labor economist, and the standard skeleton. The client now resells query access to its own customers, which is the cleanest validation of the pattern that the team has yet seen.

What the pattern is not #

The skeleton is not the right answer for every problem. Workloads that need true multi-writer concurrency, terabyte-scale warehouses, or sub-second analytical queries across billions of rows belong on a different stack. The deluair pattern is tuned for the meaty middle of the market: domain observatories where the data fits on a laptop, the question volume is human scale, and the value lives in narrative synthesis rather than raw throughput. That covers the majority of economic intelligence work that clients actually buy.

The other thing the pattern is not is a product. Each observatory is a bespoke deployment for a single client or a small consortium, with the skeleton accelerating delivery rather than commoditizing it. The defensible asset is the library of collectors, the schema patterns, and the brain prompts accumulated across thirteen builds, not a SaaS that anyone could subscribe to. That is a deliberate strategic choice, and it is what makes a twelve week calendar realistic in a domain where greenfield builds usually take a year.

Sources #

• FastAPI documentation
• SQLite FTS5 reference
• aiosqlite asynchronous SQLite
• D3.js visualization library
• Anthropic API documentation