How we use AI: the architecture behind our own agent fleet

Why this page exists

The uncomfortable truth about AI consulting in 2026 is that the barrier to sounding credible has collapsed. Anyone can write a services page about agents. Very few of the firms writing them have an agent that has survived contact with a production system, a real schedule, and a bill.

So instead of asking you to take our word for it, here is the actual architecture: the tool contracts, the security boundaries, the scheduling math, the failure modes we hit and what we changed. If you are evaluating us, this is the document to read. If you are building something similar, take what is useful.

1. Ask Civic AI: an LLM assistant with a hard tool contract

Ask Civic AI is the natural language query feature on the executive dashboard of Civic Suite, our municipal operations platform. Staff type a plain English question about permits, work orders, utility revenue, or code cases and get an answer grounded in rows from the database, along with the underlying data rendered as a table.

It is deployed and switched on. The platform exposes its feature flags at GET /api/config, and that endpoint reports aiEnabled: true today, so you can confirm the feature is live without asking us. The tenant behind it is our reference city, a seeded demo dataset rather than a real municipality’s records, and we would rather say so than let you assume otherwise. What that proves is the engineering: the tool contract, the scoping layer, the audit trail, and the cost controls below are the production code paths, exercised by real model calls, and they are what would carry a real tenant.

The request path

Staff question
      |
      v
POST /api/ops/ai/query
      |
      v
Model tool-use loop  (max 8 tool rounds)
      |
      v
validateParams + department scoping        <-- the security boundary
      |
      v
Read-only tool functions -> Postgres (Prisma)
      |
      v
{ answer, data[], usage }  -> rendered + written to AuditLog

The one rule that makes it safe

There is no freeform text to SQL anywhere in this system. The model never writes a query and never sees the schema. It sees a fixed allow list of tool definitions and the JSON results those tools return. Every database call underneath is a parameterized function we wrote and reviewed.

This matters more than it sounds. The staff member’s question is untrusted input, the same as any form field on the public internet. Prompt instructions are not a security control, because a sufficiently clever question can talk its way around them. So the real boundary is placed where it belongs: parameter validation and department scoping run in ordinary code, after the model has spoken and before anything touches Postgres. A department head scoped to permitting cannot pull code enforcement case records no matter what they type, because the scoping layer does not consult the model about it.

To be precise about what that scoping does and does not cover: the six entity tools are department scoped, so row level access to another department’s records is refused. The four aggregate tools are available to all staff by design, because they return the same executive rollups the standard dashboard already shows that user. That is a deliberate decision rather than a gap, and it is the kind of detail worth pinning down before a system goes anywhere near a real tenant.

The tool allow list

Ten tools. All read only. Row limits are validated and hard capped at 100. Adding an eleventh is a six step checklist: write the read only runner, add allow list validation, register the schema, map the department scope, document it, then unit test the validation and scoping plus an integration test against a mocked client. The documentation step is in the checklist because a tool nobody documented is a tool nobody can audit, and we know that one bites: preparing this page we found our own architecture note still describing an earlier six tool version of this allow list. The code was right, the document had drifted. The table above was rebuilt by reading the registry and the scoping map, not the doc.

Operating limits, all configurable

Every query writes an audit log entry with action ai.query, recording the question asked (truncated to its first 200 characters), which tools ran, and the token usage. The API key is never logged. For a government buyer that audit trail is not a nice to have, it is the thing that makes the feature adoptable at all.

2. The autonomous publishing pipeline

Every weekday, a pipeline drafts an article on municipal reporting and ERP topics, generates original artwork for it, publishes it to this site through the live admin API, verifies the rendered page, and then promotes it to LinkedIn, Facebook, and X. On a normal day no human touches any step. The article you can read on our blog today arrived that way.

It runs on GitHub Actions rather than on our cluster, deliberately. That gives it internet access and repo scoped secrets, and it means the pipeline has no dependency on any laptop, browser session, or local sandbox being awake. It is infrastructure, not a script someone runs.

The chain

cron '0 1 * * 2-6' UTC   =  Mon-Fri 21:00 ET, the evening BEFORE each post
      |
      v
publish.js
      |-- draft the article                    (OpenAI text model)
      |-- generate a dramatic background       (OpenAI image model)
      |-- composite the hook thumbnail         (resvg + a vendored display face)
      |-- POST through the live admin API
      |-- fetch the rendered page and verify it
      |
      v  (workflow_run, on success only)
post-to-social.js
      |-- resolve SOCIAL_ANCHOR -> an absolute instant (10:00 America/New_York)
      |-- hand that instant to Postiz
      |
      v
LinkedIn / Facebook / X

Why it generates the night before

This is the part worth stealing. The obvious design is to generate in the morning and post immediately. We built that first, and it kept missing the mid morning window that the whole schedule was designed to hit.

The reason turned out to be measurable. GitHub’s cron is a queue, not a clock. Across ten scheduled runs we logged it firing between 50 minutes and 2 hours 31 minutes late, with a median lag of 1 hour 25 minutes. Anything chained directly off that cron lands at an unpredictable hour. By contrast, we decoded the real publish times of 17 scheduled social posts from their platform activity IDs and measured the scheduler we hand off to at a median of 11.7 seconds late, worst case 16 minutes. That is two orders of magnitude tighter.

So the fix was to stop chaining and start anchoring. Generation moved to 21:00 ET the evening before, and the social step now resolves an absolute publish instant, daylight saving aware, and hands that to the scheduler that is actually accurate. Generating the night before leaves about 13 hours of slack, which makes the upstream lag irrelevant rather than merely survivable. Even a run that fires 2 hours 31 minutes late still queues the post for the same slot. The ratio between those two schedulers is roughly 435 to one.

One caveat we will not paper over: the cron itself is expressed in UTC and is not daylight saving aware, so that 21:00 ET generation time becomes 20:00 ET once the clocks change in November. Only the social anchor resolves its timezone properly at runtime. The slack absorbs the drift, which is exactly why the design tolerates it, but it is a known rough edge rather than an accident.

The side effect is a genuine SEO win we did not plan: each article is live from the evening before, so search engines crawl the page well ahead of the social traffic. The Friday run rolls its post over the weekend to Monday morning, because our readers are government staff who do not open LinkedIn on a Saturday.

What the run record actually says

Six weeks of production is six weeks, not a track record we are going to inflate into years. And we are publishing the gaps alongside the successes, because a run record with no gaps in it is usually a run record nobody checked. Seven weekdays have no article: a cluster in early July, and one on 5 August when the scheduling change described above moved the cron and the first weekday after the change did not fire. The one outright failure, on 15 July, was recovered by a person triggering the workflow again half an hour later. There is no automatic retry, so on that day a human did touch it.

What the record does demonstrate is that the pipeline runs unattended on a normal day, survives provider failures without shipping something broken, and has been through a model migration and a scheduling redesign while staying in service. That is a real claim. Uninterrupted perfection would not be.

The failure modes we actually hit

We list these because a vendor who has never hit a failure mode has never run anything. The interesting question is not whether an agent pipeline breaks. It is whether anyone finds out when it does.

3. Thirteen production websites on one pipeline

We operate thirteen public production websites, plus a set of authenticated internal applications, on a single k3s cluster. Every one of them is built, tested, shipped, and monitored by the same automated path. The sites are live and you can load any of them right now:

Alongside them run the authenticated applications: the Civic Suite platform and its dashboard, two lead generation engines, an internal command center, and a hardware controller. Same cluster, same deployment contract.

The deployment contract

push to main
      |
      v
build + push container images to GHCR, tagged with the commit SHA and :latest
      |
      v
kubectl set image deploy/{backend,frontend}  ->  rollout status
      |
      v
post-deploy smoke check:  /  /health  /rss.xml  /blog

rollback = set image back to the previous SHA

Tagging by commit SHA rather than only by :latest is what makes the rollback a single command instead of an incident. That is not an AI decision, it is ordinary engineering discipline, and it is exactly the discipline that agent written code needs wrapped around it.

What has to pass before anything ships

On this site, one aggregate check gates the branch, and it depends on all of the following. Nothing merges until it is green, and branches must be up to date with main before they can merge at all.

• Backend and frontend lint, plus a backend import smoke test.
• Backend tests at 90 percent coverage across statements, branches, functions, and lines.
• Frontend tests at 100 percent lines, functions, and statements, and 95 percent branches, with a single documented per-file exception.
• Automation tests on the publishing pipeline itself, currently gated at 70 lines and 72 branches and ratcheting upward.
• npm audit at high severity, blocking, on both backend and frontend.
• A full production build including the prerender pass and a blocking SEO verifier.
• Docker build verification for both images.
• Secret scanning, static analysis, and container scanning on a separate security workflow.

The SEO verifier is the one we are most attached to, because it catches the failure that is invisible until it has cost you three months of rankings. It walks every URL in the sitemap and fails the build if the prerendered HTML is missing a self referencing canonical, does not match the title and description in the route registry, lacks its heading inside the main element, or is missing the structured data that route is supposed to emit. Adding a page is therefore one commit that touches the route registry, the sitemap configuration, and the structured data map together. The build refuses to let those drift apart.

Scheduled jobs handle the rest: a nightly logical backup of the shared Postgres cluster, a nightly demo data reseed, a lead crawl every two hours with an hourly health check, a weekly corroboration pass, and weekly search engine sitemap submission. TLS certificates renew themselves. Storage is replicated. None of that is glamorous and all of it is why the fleet stays up.

4. What we do not let agents do

This is the section most AI pages leave out, which is precisely why it is the most useful one. Every limit below is enforced in code or in repository configuration, not by policy statement.

There is a pattern here. In every one of these systems, the model supplies language and judgement, and ordinary testable code supplies the connections and the rules about what may happen. Agents are given leverage over drafting and diagnosis, which is where they are genuinely superhuman, and given no authority over consequences. That division is the whole design.

Check it yourself

Claims on a consulting website are worth what you can verify. Here is what you can confirm without talking to us:

• Load /api/config on the Civic Suite demo and read the aiEnabled flag.
• Open the blog and check the publication date on the newest article, then come back tomorrow morning.
• Load any of the thirteen domains listed above.
• View source on this page. The prerendered HTML, canonical tag, and structured data were produced by the same verified build pipeline described in section three.

What this means for your project

We are not asking you to buy a pipeline that publishes blog posts. We are showing you that the same pattern, a tightly scoped tool contract, validation as the real security boundary, human approval on anything consequential, and a hard audit trail, is one we have already carried into production and kept running.

If you want that pattern applied to your workflow, see AI agent development for how the engagement works, or the AI consulting overview for the wider practice. For cities and counties specifically, start with AI consulting for local government or the fixed scope AI readiness assessment.

Frequently asked questions