Building an AI Analyst: The Tool Registry Pattern

The wrong way first

The obvious approach to "let an AI answer questions about my data" is to hand the model SQL access. It is also the approach that ends badly: no scoping, no cost control, and a model that will cheerfully write a query scanning every row you own.

The better pattern is a tool registry — a fixed set of callable functions, each with a typed schema, each enforcing its own authorization.

What a tool looks like

{
  name: 'get_funnel',
  description:
    'Conversion funnel with per-step drop-off for a time range. Use for ' +
    '"conversion rate", "where are users dropping off", funnel questions.',
  inputSchema: {
    type: 'object',
    properties: { dateRange: { type: 'string', description: '…' } },
    additionalProperties: false
  },
  async run(args, ctx) { /* … */ }
}

Three things matter here.

The description is a prompt. The model reads it to decide whether this tool answers the user's question. Listing the phrasings people actually use — "where are users dropping off" — matters more than describing the return type.

`additionalProperties: false` stops the model inventing arguments. Without it, a hallucinated groupBy silently does nothing and the answer is wrong in a way nobody notices.

`ctx` carries authorization, not the arguments. The caller sets ctx.siteId to a site the user can actually access. Tools never widen that scope, and the model cannot pass a different site id because it is not an input.

Result envelopes

Returning raw rows wastes the opportunity. An envelope carries the answer in several forms at once:

{
  summary: '412 visitors, 1,180 pageviews, 38% bounce (30d).',
  data: { /* the numbers */ },
  render: { type: 'kpi' },
  download: null,
  deepLink: { label: 'Open Dashboard', to: '/?dateRange=30d' }
}

The chat panel renders a stat card and a deep link. A terminal MCP client prints the summary. The model reasons over data. None of them need a separate code path.

The guardrail nobody thinks about until it bites

A 90-day query on a busy site returns a lot of rows. Sent to a model, that is slow and expensive, and the reply is often worse — models reason poorly over thousands of rows of noise.

Every envelope is capped before it reaches the model:

const MAX_ROWS = 100;
const MAX_DATA_CHARS = 20_000;

Arrays are truncated, and a note is appended to the summary so the model knows it is seeing a sample: "Showing the first 100 of 4,210 rows." Without that note, the model will confidently describe the sample as the whole dataset.

Non-array objects get a size backstop too, since a single wide object can be as large as a thousand rows.

Caching is shared with the dashboard

Tool calls hit the same coalescing cache the REST routes use, with the same keys. A question the dashboard already answered is a cache hit, and asking the same thing twice in one conversation never re-runs the query.

That detail matters more than it sounds: conversations are repetitive. "Show me top pages" followed by "now break that down by country" often re-reads the same underlying range.

Site scoping, concretely

Most tools require ctx.siteId. A few — listing the sites a user can see — operate on ctx.userId instead and are flagged siteless. The executor enforces which is which:

if (tool.siteless) {
  if (!ctx.userId) throw new Error('ctx.userId is required');
} else if (!ctx.siteId) {
  throw new Error('ctx.siteId is required');
}

It is a small check, but it is the difference between a tool layer and a security incident.

What to write first

Start with the five questions people actually ask: how is traffic, what are my top pages, where does traffic come from, how are conversions doing, and what is happening right now. Those five cover most of what a dashboard gets opened for.