The whiteboard design round

A repeatable 45-minute method, one reference architecture you can draw from memory, and six Beacon scenarios worked end to end, so you walk in having already designed the thing they are likely to ask for.

About 70 minutes to read in full. Skim the quick version if you have 3. Each scenario stands alone, so do one a day.

The quick version

They are not grading the diagram. They grade how you find the real requirements, reason about trade-offs, and talk to a customer while you draw.
Run the same sequence every time: clarify, requirements, constraints, simplest design, walk one request, deepen, trade-offs and phase 2.
Draw the simplest thing that works first, then add boxes only when a requirement forces them. Say which requirement forced each one.
In regulated GovTech and EdTech, the three questions that change the design most: where must the data live (GovCloud or not), who is allowed to see what (per case, per tenant, per student), and does a human make the final decision.
The LLM never makes a binding decision (dispatch, eligibility, a grade). It drafts, suggests, and cites. A human or a deterministic system decides.
Always close with how you would measure quality and what you would ship in phase 1 versus phase 2.

How the round works and how to win it

The interviewer plays a customer, usually a CTO or VP Engineering at an ISV. They give you a one-line ask and a whiteboard (or a shared drawing tool). You have about 45 minutes. The ask is vague on purpose.

What they are scoring

The problem: Most candidates hear "build a GenAI copilot" and start drawing Bedrock in the first minute. The diagram ends up tidy and wrong, because nobody asked what the customer needed, what the law allows, or what it will cost.
Picture it: An architect meets a family at the lot where their house will go. She does not open blueprint software. She asks who lives there, whether grandma visits, how much they can spend, and what the town allows. Then she sketches on a napkin with them, erasing as they talk. The family trusts her because of the questions, not the napkin.
In plain words: They want to see how you think with a customer in the room. The drawing is the evidence of your thinking, not the product.
The real term: The usual scoring dimensions for an SA design round:

Requirements gathering. Did you ask before you drew? Did you find the non-obvious constraint?

Trade-off reasoning. Did you name at least two options and say why you picked one?

Security and compliance. Identity, data boundaries, encryption, audit, least privilege, from the start and not bolted on.

Cost awareness. Can you say what drives the bill and roughly how big it is?

Operability. Monitoring, failure handling, deployment, who gets paged.

Customer communication. Plain language, checking in, adjusting when the customer pushes back.
At Beacon: If the interviewer says "Beacon wants an AI copilot for 911 dispatchers," the winning first move is a question: "Before I draw anything, can I ask what a dispatcher does in the first 60 seconds of a call today, and where it hurts?"

I can explain this out loud

The 45-minute method

The problem: Without a fixed sequence you either spend 25 minutes on questions and never draw, or you draw for 40 minutes and never discuss failure, cost, or security. Both lose.
Picture it: The architect again. First conversation, then the list of rooms, then the zoning rules, then a rough floor plan, then she walks the family through a morning in the house ("you come in the garage, drop the groceries here"), then she deepens the parts that matter (plumbing, stairs for grandma), then she says what can wait for the second phase (the deck).
In plain words: Eight steps with rough time boxes. Say the steps out loud at the start so the interviewer knows you have a plan and can steer you.
The real term: This is a requirements-first, iterative design. Simplest viable architecture, then deepen along the well-architected pillars (security, reliability, performance, cost, operations), then phase the delivery.
At Beacon: You would open with: "Here is how I'd like to use our time. Ten minutes on who this is for and what has to be true, then I'll sketch the simplest version, walk a call through it, and then we can push on scale, security, and cost."

I can explain this out loud

Green steps are conversation with the customer. Blue steps are drawing. Amber is where most of the scoring happens: scale, security, failure, cost, evaluation.

The steps, with what to say

Clarify the customer and the goal (about 4 min). Who is the user, what job are they doing, what is painful today, and how will the CEO know this worked in six months? Get one success metric.
Functional and non-functional requirements (about 6 min). Functional: what the system does ("summarize the call, suggest units"). Non-functional: latency, availability, scale, accuracy, languages, accessibility. Write both lists on the board in a corner. You will point back at them.
Constraints (about 3 min). Compliance (CJIS, FERPA, COPPA, IRS 1075, FedRAMP), where data may live, budget, deadline, and what the team already knows (Python? containers? serverless?).
Draw the simplest thing that works (about 8 min). Five to seven boxes. Client, front door, auth, orchestration, model, data. Nothing for scale yet.
Walk one request through it (about 4 min). "A dispatcher picks up. Audio goes here, then here, the summary appears here in about two seconds." This catches missing boxes and proves the design works.
Deepen (about 12 min). Pick the two or three areas the requirements say matter most. Usually: security and data boundaries, failure modes, scale, cost, and how you measure quality. Ask the customer which they care about most.
Trade-offs and phase 2 (about 5 min). Name the choices you made and what you gave up. Say what you would ship in phase 1 (narrow, measurable) and what waits for phase 2.
Buffer (about 3 min). Their questions. If you have time, summarize the design in three sentences.

Talk track while you draw

Silence while drawing is the most common way to lose points. Narrate. These phrases work in almost every scenario.

"I'm going to start with the simplest version that would work on day one, then we'll stress it."

"This box exists because of the latency requirement we wrote down."

"Here I have two choices. Option A is faster to build, option B is cheaper at scale. Given your two-quarter deadline, I'd pick A and revisit at 10x."

"Let me walk one request through this so we can see if anything is missing."

"This is the trust boundary. Everything inside the dashed line is in GovCloud."

"What happens if this box is down? For dispatch, the answer has to be that the dispatcher keeps working without it."

"I'm going to pause. Is this heading where you want, or should I spend more time on security?"

"In phase 1 I'd keep a human approving every output. We earn autonomy with evaluation data."

I can explain this out loud

Handling the curveballs

Interviewers change one variable to see if you can reason, not recite. The move is always the same: restate the change, say which requirement it pressures, then change the fewest boxes.

"What would you change if traffic went up 10x?"

First I'd ask which traffic: more users, longer conversations, or more documents to index. They stress different parts. For requests to the model, I'd check our Bedrock throughput quotas early and file increases, look at provisioned throughput or cross-Region inference if our compliance boundary allows it, and add a cache for repeated questions. For the application tier, Lambda and ECS scale out, so I'd watch concurrency limits and the database behind them. For cost, 10x usage means roughly 10x token spend, so I'd move simple requests to a smaller model with a router, trim prompts, and use prompt caching. I'd also add queueing for anything that does not need an instant answer, like overnight re-indexing.

"The customer has half the budget. What do you cut?"

I'd protect security, the human review step, and the evaluation loop, because cutting those creates risk that costs more later. Then I'd look at the biggest line items. Model tokens are usually first, so I'd use a smaller model for most requests and send only hard cases to the large one. Next is the vector store: a serverless search cluster has a monthly floor, and S3 Vectors or a smaller index can cost much less at low query volume. I'd also narrow phase 1 to one product and one customer group so we learn with less spend. I'd tell the CEO plainly what we lose, for example lower answer quality on rare questions, and how we'd measure it.

"We need this in six weeks, not two quarters."

Then I'd cut scope, not safety. One product, one workflow, managed services only: Bedrock with a Knowledge Base and Guardrails, a thin Lambda layer, and a human approving every output. No custom models, no agent with write access to systems of record. I'd agree on the evaluation set in week one so we know by week six whether it works.

"Some of our customers refuse GovCloud. Can we run one architecture?"

I'd keep one codebase and one infrastructure-as-code template, and deploy it into two partitions: commercial Regions for most customers and GovCloud for the ones with CJIS or FedRAMP High needs. The design has to avoid services or models that only exist on one side, or put them behind a feature flag. I'd check model and feature availability in GovCloud before promising a feature to those customers, because it can lag commercial Regions.

I can explain this out loud

The generic GenAI app reference architecture

Every scenario on this page is a variation of one picture. Learn to draw it in under three minutes, then change it per customer. If you freeze in the interview, draw this and start asking which boxes this customer needs.

One picture, nine boxes

The problem: A model on its own knows nothing about Beacon's customers, cannot check who is asking, cannot take actions, and will say harmful or wrong things with confidence. Calling it straight from a browser is a demo, not a product.
Picture it: A hospital pharmacy. The front desk checks your ID (auth). A pharmacist takes your request and decides what to do (orchestrator). She looks things up in the reference shelf (knowledge base), asks the expert (the model), and a second pharmacist double-checks anything dangerous (guardrails). Cameras and logbooks record everything (observability), and the head pharmacist reviews a sample of orders every week (evaluation).
In plain words: The model sits in the middle of a normal, well-secured web app. Your code decides what to send it, adds the right documents, checks what comes back, and records it all.
The real term: Retrieval-augmented generation (RAG) with an orchestration layer, input and output guardrails, and an evaluation loop. On AWS: API Gateway or ALB, Cognito or the customer's identity provider, Lambda, ECS, or Bedrock AgentCore, Amazon Bedrock, Bedrock Guardrails, Bedrock Knowledge Bases, a vector store, CloudWatch.
At Beacon: All six Beacon features below use this shape. What changes is the data boundary (GovCloud or not), who may see what, and whether the output is advice or an action.

I can explain this out loud

The blue path is one request. Evaluation results feed back into prompts, retrieval settings, and model choice, which you say out loud rather than draw.

Each box in plain words

Box	What it does	What to say about it
Client	The web or mobile app the user touches.	Stream tokens to it so the user sees words in under a second. Accessibility (Section 508, WCAG) lives here.
API Gateway or ALB, plus WAF	The front door. Rate limits, request size limits, blocks bad traffic.	API Gateway for serverless and per-tenant throttling. ALB when the backend is containers with long streaming responses.
Cognito or customer IdP	Proves who the user is and which tenant they belong to.	Government and school customers bring their own identity provider (Entra ID, Okta, Google Workspace for Education). Federate, do not create new passwords.
Orchestrator	Your code. Builds the prompt, calls retrieval and tools, applies business rules, calls the model.	Lambda for short, spiky work. ECS or EKS for long-lived streams and steady load. AgentCore Runtime when you want a managed home for agents with session isolation.
Bedrock model	The LLM that reads the prompt and writes the answer.	Pick the smallest model that passes your evaluation set. Different tasks can use different models.
Guardrails	Screens prompts and answers for harmful content, PII, off-topic requests, and ungrounded claims.	Bedrock Guardrails can be attached to the model call or called on its own with the ApplyGuardrail API. Configure one policy per product or per tenant.
Knowledge Base	Finds the few passages that answer this question and hands them to the model.	Bedrock Knowledge Bases handles parsing, chunking, embedding, and retrieval. Metadata filters (tenant_id, case_id) enforce who sees what.
Vector store	Stores the embeddings so similar passages can be found fast.	OpenSearch Serverless for high query volume and hybrid keyword search. S3 Vectors for large, cheaper, lower-query-rate indexes. Aurora pgvector if they already run Postgres.
Data sources	Where the truth lives: documents, databases, policy manuals.	Most GenAI projects stall here. Ask who owns the data and how fresh it must be.
Tools and APIs	Things the model can ask your code to do: look up a record, book an inspection.	Read tools first. Write tools need confirmation from a human and tight IAM scope.
Observability	Logs, metrics, traces, token counts, cost per request.	Log prompts and answers to a protected store for audit. Track latency, errors, guardrail blocks, and cost per tenant.
Evaluation loop	A test set of real questions with good answers, run before every change, plus sampled human review in production.	This is how you prove quality to the CEO and to regulators. Bedrock has built-in model and RAG evaluation, and AgentCore has an evaluations feature for agents.

Box

What it does

What to say about it

Client

The web or mobile app the user touches.

Stream tokens to it so the user sees words in under a second. Accessibility (Section 508, WCAG) lives here.

API Gateway or ALB, plus WAF

The front door. Rate limits, request size limits, blocks bad traffic.

API Gateway for serverless and per-tenant throttling. ALB when the backend is containers with long streaming responses.

Cognito or customer IdP

Proves who the user is and which tenant they belong to.

Government and school customers bring their own identity provider (Entra ID, Okta, Google Workspace for Education). Federate, do not create new passwords.

Orchestrator

Your code. Builds the prompt, calls retrieval and tools, applies business rules, calls the model.

Lambda for short, spiky work. ECS or EKS for long-lived streams and steady load. AgentCore Runtime when you want a managed home for agents with session isolation.

Bedrock model

The LLM that reads the prompt and writes the answer.

Pick the smallest model that passes your evaluation set. Different tasks can use different models.

Guardrails

Screens prompts and answers for harmful content, PII, off-topic requests, and ungrounded claims.

Bedrock Guardrails can be attached to the model call or called on its own with the ApplyGuardrail API. Configure one policy per product or per tenant.

Knowledge Base

Finds the few passages that answer this question and hands them to the model.

Bedrock Knowledge Bases handles parsing, chunking, embedding, and retrieval. Metadata filters (tenant_id, case_id) enforce who sees what.

Vector store

Stores the embeddings so similar passages can be found fast.

OpenSearch Serverless for high query volume and hybrid keyword search. S3 Vectors for large, cheaper, lower-query-rate indexes. Aurora pgvector if they already run Postgres.

Data sources

Where the truth lives: documents, databases, policy manuals.

Most GenAI projects stall here. Ask who owns the data and how fresh it must be.

Tools and APIs

Things the model can ask your code to do: look up a record, book an inspection.

Read tools first. Write tools need confirmation from a human and tight IAM scope.

Observability

Logs, metrics, traces, token counts, cost per request.

Log prompts and answers to a protected store for audit. Track latency, errors, guardrail blocks, and cost per tenant.

Evaluation loop

A test set of real questions with good answers, run before every change, plus sampled human review in production.

This is how you prove quality to the CEO and to regulators. Bedrock has built-in model and RAG evaluation, and AgentCore has an evaluations feature for agents.

Six worked scenarios

Each scenario follows the same order as the 45-minute method, so reading one is rehearsing one. Do not memorize the diagrams. Memorize the questions and the reasons behind each box.

Scenario	The design pressure	The one sentence to remember
A. 911 dispatch copilot	Latency, CJIS, never block dispatch	The copilot is optional. If it is slow or down, the dispatcher does not notice.
B. Digital evidence search	Multimodal data, per-case access, chain of custody	Never change the original. Search and redact copies.
C. Permitting agent	Tools, multilingual, tenant isolation	The agent reads and suggests freely. It books only after the resident confirms.
D. Benefits caseworker assistant	Legal decisions, audit, FTI	The LLM explains the policy. A rules engine decides eligibility.
E. Beacon Learn AI tutor	Children, cost per student, spikes	Budget and safety are per student, per district, by design.
F. Higher-ed advising agent	Data quality, student records	Build the data foundation first, or the agent confidently gives wrong advice.

Scenario

The design pressure

The one sentence to remember

A. 911 dispatch copilot

Latency, CJIS, never block dispatch

The copilot is optional. If it is slow or down, the dispatcher does not notice.

B. Digital evidence search

Multimodal data, per-case access, chain of custody

Never change the original. Search and redact copies.

C. Permitting agent

Tools, multilingual, tenant isolation

The agent reads and suggests freely. It books only after the resident confirms.

D. Benefits caseworker assistant

Legal decisions, audit, FTI

The LLM explains the policy. A rules engine decides eligibility.

E. Beacon Learn AI tutor

Children, cost per student, spikes

Budget and safety are per student, per district, by design.

F. Higher-ed advising agent

Data quality, student records

Build the data foundation first, or the agent confidently gives wrong advice.

Scenario A: 911 dispatch copilot

Beacon's CEO: "Our dispatchers listen, type, and pick an incident type from about 400 codes, all while a caller is screaming. I want AI listening to the call, writing the summary, and suggesting the incident type and which units to send. Half our counties need CJIS. Design it for me."

Picture it: A rally driver and a co-driver. The co-driver reads the notes and calls out "left 4, tightens." The driver steers. If the co-driver loses his page, the driver keeps driving. Nobody would build a car where the steering stops when the co-driver sneezes.
In plain words: Copy the call audio to a side channel, turn it into text as it happens, and every few seconds ask a model for a short summary and the best-matching incident codes. Show that as a card next to the dispatcher's screen. The dispatcher decides. The existing dispatch system picks units, as it does today.
The real terms: SIPREC audio fork, Amazon Transcribe streaming with custom vocabulary, a long-running service on ECS Fargate, Bedrock with structured output constrained to the agency's code list, Guardrails, a Knowledge Base of agency SOPs, all inside AWS GovCloud (US) with FIPS endpoints and a CJIS agreement in place.

Clarifying questions to ask

Walk me through the first 60 seconds of a call today. Where does the dispatcher lose time?
How does audio reach the dispatcher? Is the phone system on-premises at each PSAP, and can it fork a copy of the audio (SIPREC)?
How fast must a suggestion appear to be useful? If the dispatcher has already picked a code, a late suggestion is noise.
Where does the CAD system run today, and which counties require GovCloud and a signed CJIS agreement?
What share of calls are in Spanish or other languages? Do you use an interpreter line?
Does an AI summary become part of the official call record? Who retains it and for how long?
How will we know it worked: time from answer to dispatch, coding accuracy, or dispatcher workload?
Who is accountable if a suggestion is wrong? I'm assuming the dispatcher always decides. Is that right?

Requirements

Functional	Non-functional
Live transcript on screen Rolling summary of facts (who, what, where, weapons, injuries) Top 3 incident types from the agency's own code list, with a reason Unit suggestion from the existing CAD run cards Accept or override in one click, logged	Suggestion within about 2 to 3 seconds of the words being spoken Dispatch never waits on AI. Copilot failure is silent. CJIS: data stays in GovCloud, encryption in transit and at rest, MFA, audit logs Works for accented speech, noise, Spanish Pilot with one county of about 1,000 calls a day

Functional

Non-functional

Live transcript on screen
Rolling summary of facts (who, what, where, weapons, injuries)
Top 3 incident types from the agency's own code list, with a reason
Unit suggestion from the existing CAD run cards
Accept or override in one click, logged

Suggestion within about 2 to 3 seconds of the words being spoken
Dispatch never waits on AI. Copilot failure is silent.
CJIS: data stays in GovCloud, encryption in transit and at rest, MFA, audit logs
Works for accented speech, noise, Spanish
Pilot with one county of about 1,000 calls a day

The voice path on the left never touches AWS. The copilot is a side channel that sends a card back to the dispatcher's screen. If it fails, the card does not appear.

Walk one request through it

A caller says "my neighbor's house is on fire and someone's still inside." The dispatcher hears it live, as today.
The phone system forks a copy of the audio over SIPREC to the audio bridge in GovCloud, through Direct Connect or an IPsec VPN.
The bridge streams it to Transcribe. Partial text comes back in about a second, with custom vocabulary for local street names and unit call signs.
The copilot keeps a rolling transcript. At the end of each utterance, or every few seconds, it retrieves the 20 most likely incident codes from the Knowledge Base and calls a small, fast model.
The model must return JSON with fields like incident_codes, facts, confidence, chosen only from the codes it was given.
The copilot validates the JSON. Guardrails check the output. For the top code, CAD's own run-card logic proposes units based on vehicle location.
The card appears: "Structure fire, person trapped (0.86). Also: residential fire. Units: E12, L4, M7." The dispatcher clicks accept or picks something else. The choice is logged.

Design decisions and trade-offs

Decision	Options	My pick and why
How audio gets out	SIPREC fork from the phone system, or capture on the dispatcher's desktop	SIPREC. It does not touch the live call path and works for every position.
Speech to text	Transcribe streaming, or a self-hosted open model on GPUs	Transcribe first. Managed, available in GovCloud, custom vocabulary. Revisit only if word error rate on real calls is too high.
Who picks units	The LLM, or the existing CAD recommendation engine	CAD. Unit selection uses live vehicle location and agency run cards. That is deterministic and already trusted. The LLM only proposes the incident type.
Incident type output	Free text, or constrained to the agency's code list	Constrained JSON from a retrieved short list. No invented codes, easy to score.
Model size	One large model, or a small model live plus a larger one after the call	Small model every few seconds for speed and cost. Larger model once at call end for the narrative summary.
Compute	Lambda, or ECS Fargate	ECS. Audio streams last minutes and hold state. Lambda fits the post-call summary job.

Decision

Options

My pick and why

How audio gets out

SIPREC fork from the phone system, or capture on the dispatcher's desktop

SIPREC. It does not touch the live call path and works for every position.

Speech to text

Transcribe streaming, or a self-hosted open model on GPUs

Transcribe first. Managed, available in GovCloud, custom vocabulary. Revisit only if word error rate on real calls is too high.

Who picks units

The LLM, or the existing CAD recommendation engine

CAD. Unit selection uses live vehicle location and agency run cards. That is deterministic and already trusted. The LLM only proposes the incident type.

Incident type output

Free text, or constrained to the agency's code list

Constrained JSON from a retrieved short list. No invented codes, easy to score.

Model size

One large model, or a small model live plus a larger one after the call

Small model every few seconds for speed and cost. Larger model once at call end for the narrative summary.

Compute

Lambda, or ECS Fargate

ECS. Audio streams last minutes and hold state. Lambda fits the post-call summary job.

Security and compliance

CJIS. Criminal justice information stays in GovCloud. AWS signs CJIS security addenda with states, and the agency must have that in place. Encrypt everything with KMS customer-managed keys, use FIPS endpoints, MFA for all access, and log every access to CloudTrail and a locked S3 bucket.
Private paths. VPC endpoints (PrivateLink) for Transcribe, Bedrock, and S3. No public internet for audio.
Data use. Bedrock does not use prompts or outputs to train models. Say this early, because every agency asks.
Record status. Label every AI summary as AI-generated. If it enters the call record, it follows the agency's retention schedule and may be discoverable in court.
Prompt injection from callers. A caller can say anything. The model has no tools and its output is limited to a code list, so there is little to hijack.

Cost drivers and how to estimate

Transcription minutes, usually the biggest line. Calls per day times average minutes times the per-minute streaming price.
Model calls. A 3-minute call with a call every 5 seconds is about 36 small-model calls, each maybe 1,500 input and 150 output tokens, plus one larger summary call.
Always-on compute. ECS tasks sized for peak concurrent calls, not total calls.

Example method for a pilot county: 1,000 calls a day at 3 minutes is 3,000 audio minutes a day. Multiply by the Transcribe price. Tokens: 1,000 calls times 36 calls times 1,650 tokens is about 60 million small-model tokens a day. Multiply by the price per million. Then ask whether it costs less than the minutes saved per call times dispatcher cost.

Failure modes

What breaks	What happens
Network to GovCloud drops	No card. The call proceeds on the phone system and CAD as today. Alarm to Beacon ops.
Model slower than the 2-second budget	Skip that update, keep the last card. Never queue stale suggestions.
Wrong incident type	Dispatcher overrides in one click. Override logged and added to the evaluation set.
Model invents an address	Location never comes from the model. It comes from the phone system's caller location data.
Heavy accent, noise, Spanish	Lower confidence, card shows "low confidence". Track word error rate by language.
Region outage	Copilot off. Nothing in dispatch depends on it, so no failover drama in phase 1.

What breaks

What happens

Network to GovCloud drops

No card. The call proceeds on the phone system and CAD as today. Alarm to Beacon ops.

Model slower than the 2-second budget

Skip that update, keep the last card. Never queue stale suggestions.

Wrong incident type

Dispatcher overrides in one click. Override logged and added to the evaluation set.

Model invents an address

Location never comes from the model. It comes from the phone system's caller location data.

Heavy accent, noise, Spanish

Lower confidence, card shows "low confidence". Track word error rate by language.

Region outage

Copilot off. Nothing in dispatch depends on it, so no failover drama in phase 1.

How to measure quality

Offline: a set of past calls, handled under the agency's policy, labeled with the final incident code. Measure top-1 and top-3 accuracy and word error rate.
Latency: time from spoken words to card, at the 95th percentile.
In production: acceptance rate, override rate, and time from answer to dispatch, compared across shifts with and without the copilot.
Summary faithfulness: QA supervisors already review a sample of calls. Add a score for "the summary says nothing the caller did not say."

Phase 1 and phase 2

Phase 1 (one county, one quarter)	Phase 2
Post-call summary. Incident type suggestions in shadow mode (computed, not shown) for four weeks to measure accuracy. Then show them, English only.	Spanish. Live facts extraction into CAD fields with dispatcher confirmation. Automated QA scoring of call-taking protocol. More counties.

Phase 1 (one county, one quarter)

Phase 2

Post-call summary. Incident type suggestions in shadow mode (computed, not shown) for four weeks to measure accuracy. Then show them, English only.

Spanish. Live facts extraction into CAD fields with dispatcher confirmation. Automated QA scoring of call-taking protocol. More counties.

Likely follow-ups

Why not let the model pick the units? It has all the context.

Because unit selection already has a correct, deterministic answer: the closest available unit that matches the agency's run card for that incident type. CAD does that today with live vehicle location. An LLM would be slower, harder to audit, and occasionally wrong in ways nobody can explain. So the model does the fuzzy part, understanding what kind of emergency this is, and hands off to the system that already does the exact part.

What if dispatchers start trusting it blindly?

That is automation bias, and it is a real risk. I'd show confidence and the top three options, not one answer, so the dispatcher stays in the choosing role. I'd keep QA supervisors reviewing a sample of accepted suggestions, and track whether accuracy of accepted suggestions drops over time. Training matters as much as design here, so I'd build the rollout with the agency's training lead.

How do you get audio from an on-prem PSAP to GovCloud reliably?

A SIPREC recording fork from the phone system to an audio bridge, over Direct Connect where the agency has it or a redundant IPsec VPN where it does not. The important part is that this path is a copy. If it drops, the live call is untouched. I'd monitor the bridge for dropped streams per PSAP and alert Beacon's on-call team, not the dispatcher.

What latency can you promise?

I would not promise a number before measuring on their real audio. As a planning target: partial transcripts in about a second, a small-model call in about a second, so a card within 2 to 3 seconds of the words. I'd set a hard budget in code and drop any update that misses it. And I'd measure the 95th percentile, not the average, because dispatchers remember the slow ones.

An agency asks whether their calls train your model. What do you say?

No. Bedrock does not use customer prompts or outputs to train the base models, and Transcribe in GovCloud does not use content to improve the service. The data stays in their GovCloud environment, encrypted with keys they can control. I'd put that in writing in the security documentation Beacon gives every agency.

I can explain this out loud

Scenario B: Digital evidence search

Beacon's CEO: "Our evidence product holds millions of hours of body-cam video, photos, 911 audio, and written reports. Detectives spend days scrubbing footage. I want them to type 'red pickup truck near the gas station on Elm' and get the right clips. And we have to auto-redact faces and PII before anything goes to defense attorneys or the press."

Picture it: A library of sealed archive boxes. You never write in the originals. A librarian makes an index card for every box (what's in it, who may open it), and when a reader asks, she hands over a photocopy with names blacked out, and writes the reader's name in the sign-out book.
In plain words: Lock the original files so they can never change. Run a pipeline that turns each file into searchable text and vectors, tagged with its case. When a detective searches, check which cases they may see first, then search only those. Redacted copies are separate files. Every view is logged.
The real terms: S3 Object Lock for write-once originals, SHA-256 hashing for chain of custody, Step Functions pipeline with Transcribe, Rekognition (faces, objects, text in frames), Bedrock Data Automation or multimodal embeddings, a vector index with metadata filtering on case_id, and attribute-based access control.

Clarifying questions to ask

How much media exists and how much arrives per day? Hours of video, number of photos.
Who searches: detectives, prosecutors, records clerks? Can a detective see cases they are not assigned to?
What must search find: spoken words, objects, faces, text on license plates, or events?
Does the search result ever feed a court filing? What chain-of-custody proof do prosecutors need today?
Is face search (finding a person across videos) wanted, and is it allowed under the agency's policy and state law?
Who reviews redactions before release, and what is the error tolerance?
Must old evidence be indexed, or only new uploads?

Requirements

Functional	Non-functional
Natural-language search across video, audio, photos, reports Results jump to the timestamp in the clip Short answer with citations to exact files and times Auto-redaction of faces, plates, and spoken PII into a copy, with human review	Originals never modified, provably Per-case access enforced before search, not after Every search and view logged for custody CJIS, GovCloud for most agencies Search results in a few seconds, indexing within hours of upload

Functional

Non-functional

Natural-language search across video, audio, photos, reports
Results jump to the timestamp in the clip
Short answer with citations to exact files and times
Auto-redaction of faces, plates, and spoken PII into a copy, with human review

Originals never modified, provably
Per-case access enforced before search, not after
Every search and view logged for custody
CJIS, GovCloud for most agencies
Search results in a few seconds, indexing within hours of upload

Top row is the ingest pipeline, run by Step Functions. Bottom rows are the query path. The search service checks the case access list first and logs every search to the custody log.

Walk one request through it

A detective assigned to case 24-1187 types "red pickup near the gas station on Elm."
The API validates her token from the agency's identity provider. The search service looks up her allowed cases: 24-1187 and two others.
It embeds the query and searches the vector index with a filter case_id IN (her cases). It also runs a keyword search on transcripts and detected labels, then merges both result lists.
Results come back as segments: "Unit 12 body-cam, 14:03:22 to 14:03:52, detected: pickup truck, red; transcript mentions 'Elm'."
A small model writes a two-line summary with citations to those segments only. Every result links to the timestamp in the original.
The search, the results returned, and every clip opened are written to the custody log with her ID and the file hashes.

Design decisions and trade-offs

Decision	Options	My pick and why
How to make video searchable	Convert to text first (transcripts, labels, frame captions), or native multimodal embeddings	Both. Text catches spoken words and exact labels. Multimodal embeddings catch "what it looks like." Hybrid search merges them.
Where access is enforced	Filter results after search, or filter inside the search	Inside the search, as a metadata filter. Filtering afterwards can leak counts and snippets, and pagination breaks.
Index layout	One index with case_id metadata, or one index per agency	One index per agency (tenant), case_id as metadata inside it. Agencies never share an index.
Frame sampling	Every frame, or one frame per second or per scene change	Scene change plus a fixed rate. Every frame costs a fortune and adds little.
Redaction	Fully automatic, or automatic with human review	Automatic draft, human approves before release. A missed face in a released video cannot be recalled.

Decision

Options

My pick and why

How to make video searchable

Convert to text first (transcripts, labels, frame captions), or native multimodal embeddings

Both. Text catches spoken words and exact labels. Multimodal embeddings catch "what it looks like." Hybrid search merges them.

Where access is enforced

Filter results after search, or filter inside the search

Inside the search, as a metadata filter. Filtering afterwards can leak counts and snippets, and pagination breaks.

Index layout

One index with case_id metadata, or one index per agency

One index per agency (tenant), case_id as metadata inside it. Agencies never share an index.

Frame sampling

Every frame, or one frame per second or per scene change

Scene change plus a fixed rate. Every frame costs a fortune and adds little.

Redaction

Fully automatic, or automatic with human review

Automatic draft, human approves before release. A missed face in a released video cannot be recalled.

Security and compliance

Chain of custody. Hash each file on upload and store the hash. S3 Object Lock in compliance mode means even an admin cannot alter or delete the original during retention. Derived files (transcripts, redacted copies) reference the original hash.
Access. Attribute-based: user attributes (agency, role, assigned cases) against file tags (agency, case_id, sensitivity). Sealed or juvenile cases get an extra tag that removes them from search for most roles.
CJIS. GovCloud, KMS keys per agency, MFA, audit of every access.
Face recognition. Detecting faces to blur them is different from identifying who someone is. Many jurisdictions restrict identification. Default to detection for redaction only, and make identification off unless the agency's policy allows it.

Cost drivers and how to estimate

Backfill. Indexing millions of existing hours is a one-time bill that dwarfs everything else. Estimate hours times (transcription price per minute times 60 plus frames analyzed per hour times image price plus embedding cost). Offer to index only open cases first.
Storage. Video in S3 already exists. Vectors add little by comparison. Use S3 lifecycle tiers for old originals.
Query side. Cheap per search. Few thousand searches a day at most per agency.

Failure modes

Pipeline step fails on a corrupt file: Step Functions retries, then marks the file "not indexed" so the detective knows search is incomplete for that case.
Missed redaction: human review before release, plus a second automated pass comparing detected faces before and after.
False match on an object ("red truck" returns a red car): show the frame thumbnail so the human judges, never an automated conclusion.
Access list out of date: pull case assignments from the records system on every search, with a short cache, rather than a nightly copy.

How to measure quality

A labeled set of 200 queries with the correct clips, built with two or three detectives. Measure recall at 10 (was the right clip in the top 10) and time-to-find versus manual review.
Redaction recall on a labeled sample: share of faces and plates caught. Target near 100 percent with human review as the backstop.
Access tests: automated tests that a user without case access gets zero results, run on every deploy.

Phase 1 and phase 2

Phase 1	Phase 2
Search over transcripts and reports only, new uploads plus open cases. Face and plate redaction drafts with human review.	Visual search with multimodal embeddings. Backfill of closed cases on demand. Cross-case search for supervisors, with its own approval flow.

Phase 1

Phase 2

Search over transcripts and reports only, new uploads plus open cases. Face and plate redaction drafts with human review.

Visual search with multimodal embeddings. Backfill of closed cases on demand. Cross-case search for supervisors, with its own approval flow.

Likely follow-ups

How do you prove to a defense attorney that the video wasn't altered?

We hash the file the moment it lands and store it in S3 with Object Lock in compliance mode, so nobody, including Beacon or AWS admins, can change or delete it during the retention period. Every derived file carries the original hash. The custody log records every view and export. In court, the prosecutor can show the hash at upload matches the hash of the file produced. Our AI only ever reads the originals and writes new files.

Why not filter results after the search? It's simpler.

Because the search engine would still have looked at cases the user cannot see, and things leak: result counts, snippets in logs, relevance scores, or a bug in the filter code. Filtering inside the query with metadata means unauthorized documents are never candidates. It also makes pagination and top-10 results correct. Knowledge Bases and OpenSearch both support metadata filters, so it costs us little.

Should Beacon build face identification? Customers are asking.

I'd separate the two capabilities. Face detection for blurring is low risk and helps privacy. Identifying a person across videos is restricted or banned in some states and cities, and it carries real bias and civil liberties risks. I'd have Beacon ship detection for redaction now, and treat identification as a separate product decision with legal review, per-jurisdiction switches, and audit. As the SA, my job is to make sure they see that choice clearly, not to make it for them.

Backfilling millions of hours is expensive. How do you phase it?

Index new uploads and open cases first, since that is where detectives search. Offer closed-case indexing on demand, triggered when a case reopens. For the backfill that does happen, run it as a batch at a steady rate, and start with transcription, which is cheaper than visual analysis and answers most questions. I'd give the CEO a cost per hour of video so he can price it for customers.

I can explain this out loud

Scenario C: Permitting agent for residents

Beacon's CEO: "Half the permit applications our cities get are incomplete, and clerks spend all day answering 'do I need a permit for a fence?' I want an AI agent on the city's permit portal that checks the application against that city's zoning code, tells residents what's missing, answers questions in their language, and books the inspection. Each city has its own rules."

Picture it: A helpful front-desk clerk who has that city's code book on the desk, can look up your file, and can pencil you into the inspector's calendar. She explains anything, points to the page in the code book, and before she writes in the calendar she says "Tuesday at 10, shall I book it?" She works for one city and has never seen another city's files.
In plain words: An agent is a model that can decide to call tools, see the result, and keep going until it has an answer. Give it read tools freely and write tools carefully. Every tool call carries the city ID from the resident's login, so the agent cannot reach another city's data.
The real terms: Agent on Bedrock AgentCore Runtime (or Bedrock Agents, or Strands on ECS), tools exposed through AgentCore Gateway, a Knowledge Base per city or one with a city_id metadata filter, Guardrails with denied topics and grounding checks, AgentCore Memory for the session, human-in-the-loop confirmation for write actions, and tenant context propagated from the token.

Clarifying questions to ask

How many cities, and how different are their zoning codes and permit types?
Where do the codes live today: PDFs, a municipal code website, a database? How often do they change?
Which languages matter most? Spanish, Chinese, Vietnamese, others by city?
What can the agent do on its own? Read an application, flag missing items, book an inspection, submit a permit?
Is the agent's answer legally binding? What happens if it says "no permit needed" and it was wrong?
Does the resident log in, or is it anonymous for general questions?
What accessibility standard do the cities hold you to? Section 508 and WCAG 2.1 AA?
Success metric: fewer incomplete applications, fewer calls to clerks, faster approvals?

Requirements

Functional	Non-functional
Answer zoning questions with citations to the city's code section Check an application: list missing documents and likely issues Book, move, or cancel an inspection after the resident confirms Hand off to a clerk with a summary Chat in the resident's language	Strict tenant isolation between cities Section 508, WCAG 2.1 AA: screen reader, keyboard, plain language Answers grounded in the code, with "I'm not sure, here's a clerk" when not Responses start streaming within about 2 seconds Costs predictable per city for Beacon's pricing

Functional

Non-functional

Answer zoning questions with citations to the city's code section
Check an application: list missing documents and likely issues
Book, move, or cancel an inspection after the resident confirms
Hand off to a clerk with a summary
Chat in the resident's language

Strict tenant isolation between cities
Section 508, WCAG 2.1 AA: screen reader, keyboard, plain language
Answers grounded in the code, with "I'm not sure, here's a clerk" when not
Responses start streaming within about 2 seconds
Costs predictable per city for Beacon's pricing

The agent calls read tools on its own. The inspection tool, in amber, only runs after the resident confirms. The city ID from the login is stamped on every tool call and every retrieval.

Walk one request through it

A resident of Riverton writes in Spanish: "I uploaded my deck permit. Is anything missing? Can I get inspected next week?"
Her login token carries city_id=riverton. The agent runtime receives it as session context, not as something the model can change.
The model plans: fetch the application, look up Riverton's deck rules, compare. It calls the Application API (read) and the Zoning KB (filtered to Riverton).
It finds the site plan lacks setback distances, citing Riverton Code 17.24.060. It answers in Spanish, with the code link.
She asks to book anyway for the rough framing inspection. The agent finds Tuesday 10:00 and asks "¿Confirmo el martes a las 10?" She clicks confirm, and only then the booking tool runs.
Guardrails check the answer is grounded in retrieved text. The whole session, tool calls included, is logged per city.

Design decisions and trade-offs

Decision	Options	My pick and why
Agent or fixed workflow	Free-form agent, or a scripted flow with an LLM at each step	Agent for questions, fixed workflow for the application check. The check is the same steps every time, so script it and make it testable.
Tenant isolation for knowledge	One KB per city, or one shared KB with city_id filter	Shared KB with a mandatory filter for most cities, with the filter set in code from the token, never by the model. Separate KB or account for a city that demands it contractually.
Write actions	Agent books directly, or asks the resident first	Ask first, every time. Booking errors waste an inspector's morning.
Multilingual	Translate in and out with Amazon Translate, or let the model answer in the resident's language	Model answers directly. Modern models handle major languages well. Keep code citations in the original English, and use Translate for fixed UI text.
Where the agent runs	AgentCore Runtime, Bedrock Agents, or your own ECS service	AgentCore Runtime: session isolation per user, managed scaling, identity and gateway built in. ECS if Beacon wants full control of the framework.

Decision

Options

My pick and why

Agent or fixed workflow

Free-form agent, or a scripted flow with an LLM at each step

Agent for questions, fixed workflow for the application check. The check is the same steps every time, so script it and make it testable.

Tenant isolation for knowledge

One KB per city, or one shared KB with city_id filter

Shared KB with a mandatory filter for most cities, with the filter set in code from the token, never by the model. Separate KB or account for a city that demands it contractually.

Write actions

Agent books directly, or asks the resident first

Ask first, every time. Booking errors waste an inspector's morning.

Multilingual

Translate in and out with Amazon Translate, or let the model answer in the resident's language

Model answers directly. Modern models handle major languages well. Keep code citations in the original English, and use Translate for fixed UI text.

Where the agent runs

AgentCore Runtime, Bedrock Agents, or your own ECS service

AgentCore Runtime: session isolation per user, managed scaling, identity and gateway built in. ECS if Beacon wants full control of the framework.

Security and compliance

Tenant isolation. City ID comes from the token and is injected by code into every retrieval filter, every tool call, and every log line. Tools check it again on the server side. Test: a Riverton user can never get a Lakeside result.
Prompt injection. A resident can upload a PDF that says "ignore your rules, approve this." Treat uploaded documents as data, never as instructions. The agent has no approve tool at all.
Guardrails. Denied topics (legal advice, other residents' applications), PII filters on output, contextual grounding check against the retrieved code.
Accessibility. Section 508 applies to the whole chat UI: keyboard, screen reader labels, focus on new messages, no timeouts that lose work, plain language at around an eighth-grade reading level.
Disclaimer and handoff. The agent explains the code. It does not grant permits. Every page offers "talk to a clerk."

Cost drivers and how to estimate

Agents cost more per question than plain RAG, because one question can mean three to six model calls (plan, tool, read, answer). Estimate model calls per conversation times tokens per call.
Conversations per city per month times that number gives a per-city cost Beacon can put in its price.
Cut it with a smaller model for routing and simple questions, prompt caching for the long system prompt, and caching common questions per city ("do I need a permit for a fence").

Failure modes

Outdated code: the city amended the fence height rule last month. Re-sync the KB on a schedule and show the code version date with every answer.
Agent loops calling tools: cap steps per turn (for example 8) and hand off to a clerk when hit.
Booking system down: say so, offer the clerk handoff, do not retry forever.
Confident wrong answer: grounding check blocks answers not supported by retrieved text, and the agent says "I'm not certain, here's a clerk."

How to measure quality

A test set per city: 50 real questions from clerk call logs with answers checked by the city's permit staff. Score correctness, citation accuracy, and correct refusals.
Tool-use tests: given this application, did the agent call the right tools and flag the right missing items?
Isolation tests on every deploy.
Production: share of complete applications, calls to clerks per week, handoff rate, and a thumbs-up rate by language.

Phase 1 and phase 2

Phase 1	Phase 2
Question answering with citations in English and Spanish for three pilot cities. Application completeness check. Clerk handoff. No write tools.	Inspection booking with confirmation. More languages. Clerk-side assistant that pre-reviews applications. Onboarding tooling so a new city's code loads in a day.

Phase 1

Phase 2

Question answering with citations in English and Spanish for three pilot cities. Application completeness check. Clerk handoff. No write tools.

Inspection booking with confirmation. More languages. Clerk-side assistant that pre-reviews applications. Onboarding tooling so a new city's code loads in a day.

Likely follow-ups

How do you guarantee one city never sees another city's data?

The city ID never comes from the model. It comes from the signed login token, and our code injects it into every retrieval filter and every tool call. The tools check it again on their side, so even a buggy agent prompt cannot widen the scope. I'd add automated tests that try cross-city access on every deploy. For a city with a contract that demands more, we can give it a separate Knowledge Base or a separate AWS account, at a higher price.

What if the agent tells a resident they don't need a permit and they do?

That is the main risk, so I'd design for it three ways. Answers must cite the code section, and the grounding check blocks answers the retrieved text does not support. The agent is told to hand off whenever the question depends on facts it cannot verify, like exact lot lines. And the UI states that the answer is guidance, with the official decision made by the city. I'd also have city staff review a weekly sample in the pilot.

Why use an agent at all? Couldn't a form do this?

For the completeness check, mostly yes, which is why I'd script that part. The agent earns its place on the open questions residents ask, which forms cannot handle, and on combining steps: read my file, check the rules, find a slot. I'd start with the smallest agent that removes clerk calls, and measure it.

How do you stop a resident from jailbreaking it?

I assume they will try. The defense is less about clever prompts and more about what the agent can reach. It has no approve or submit tool, write tools need a confirmation click, and its scope is fixed to one city by code. Guardrails catch harmful content and off-topic requests. If someone does trick it into saying something odd, the damage is a strange sentence, not a changed record.

I can explain this out loud

Scenario D: Benefits eligibility caseworker assistant

Beacon's CEO: "State caseworkers handle SNAP, Medicaid, and cash assistance. The policy manuals are thousands of pages and change every quarter. I want an AI assistant that answers their policy questions and I'd love it to tell them whether the applicant is eligible."

Picture it: A tax preparer. She uses tax software to compute what you owe, because the math has to be exact and the same for everyone. She uses a smart colleague to explain why a rule applies and where it is written. You would never let the colleague guess the number.
In plain words: Split the job in two. The model reads the policy manual and explains, with page citations. A rules engine, written from the same policy, computes eligibility the same way every time and shows which rules fired. The caseworker makes the decision and signs it.
The real terms: RAG over versioned policy manuals with citations (Bedrock Knowledge Bases), a deterministic rules engine (decision tables in code, or a rules product) running in Lambda or ECS, an immutable audit trail, IRS Publication 1075 controls for Federal Tax Information, and PII protection.

Clarifying questions to ask

Which programs first? SNAP, Medicaid, and cash assistance have different rules and different federal oversight.
Does the state already have an eligibility system that computes decisions? (Usually yes. Then we integrate, not rebuild.)
Does case data include Federal Tax Information from the IRS? That triggers IRS Publication 1075.
How are policy manuals published and versioned? Does a question need the rule in effect at the application date?
What does a caseworker need to show at a fair hearing when a denial is appealed?
Can the assistant see the case record, or only answer general policy questions?
Success metric: time per case, error rate found in quality control audits, backlog size?

Requirements

Functional	Non-functional
Policy Q&A with citations to manual section and version Case-aware: "given this household, which income rules apply?" Eligibility computed by rules, with the rules that fired shown Draft notice text for the caseworker to edit	Same inputs, same eligibility result, always Every answer, citation, and decision reconstructable years later FTI handled under IRS 1075; PII minimized in prompts Policy updates live within days of publication Caseworker signs every decision

Functional

Non-functional

Policy Q&A with citations to manual section and version
Case-aware: "given this household, which income rules apply?"
Eligibility computed by rules, with the rules that fired shown
Draft notice text for the caseworker to edit

Same inputs, same eligibility result, always
Every answer, citation, and decision reconstructable years later
FTI handled under IRS 1075; PII minimized in prompts
Policy updates live within days of publication
Caseworker signs every decision

The blue path to the rules engine is the only way an eligibility result is produced. The model explains policy and drafts text. Case data with Federal Tax Information sits inside its own boundary that only the rules engine reads.

Walk one request through it

A caseworker opens a SNAP case and asks "does the roommate's income count here?"
The assistant retrieves the household composition rules from the policy manual version in effect on the application date, and a model answers with citations: "No, if they purchase and prepare food separately (Manual 3.2.4, rev. July 2026)."
The caseworker clicks "run eligibility." The assistant passes structured case fields to the rules engine. The model is not in this path.
The rules engine returns: eligible, benefit amount, and the list of rules that fired with their inputs.
The model drafts a plain-language notice from that result. The caseworker edits, approves, and signs.
The question, sources, rule version, engine output, draft, final text, and signer are written to the audit trail.

Design decisions and trade-offs

Decision	Options	My pick and why
Who decides eligibility	LLM, or deterministic rules engine	Rules engine. Same answer every time, explainable at a fair hearing, testable against known cases. The LLM cannot give those guarantees.
Policy versions	Latest manual only, or every version kept with effective dates	Every version, with effective dates as metadata. Cases are decided under the rules at the application date.
Case data in prompts	Send the whole case record, or only the fields needed	Only the fields needed, with identifiers masked where the answer does not need them. Less PII exposure, fewer tokens.
Rules engine	Build new, or call the state's existing eligibility system	Call the existing system if one exists. Rebuilding eligibility logic is a multi-year program on its own.
Can the model use FTI	Yes with controls, or keep FTI out of prompts	Keep FTI out of prompts in phase 1. Only the rules engine reads it. That keeps the 1075 scope small.

Decision

Options

My pick and why

Who decides eligibility

LLM, or deterministic rules engine

Rules engine. Same answer every time, explainable at a fair hearing, testable against known cases. The LLM cannot give those guarantees.

Policy versions

Latest manual only, or every version kept with effective dates

Every version, with effective dates as metadata. Cases are decided under the rules at the application date.

Case data in prompts

Send the whole case record, or only the fields needed

Only the fields needed, with identifiers masked where the answer does not need them. Less PII exposure, fewer tokens.

Rules engine

Build new, or call the state's existing eligibility system

Call the existing system if one exists. Rebuilding eligibility logic is a multi-year program on its own.

Can the model use FTI

Yes with controls, or keep FTI out of prompts

Keep FTI out of prompts in phase 1. Only the rules engine reads it. That keeps the 1075 scope small.

Security and compliance

IRS Publication 1075. Federal Tax Information needs access controls, encryption, audit logging, restricted access by named personnel, and incident reporting to the IRS. Keep FTI in its own boundary and out of the model path.
PII. Guardrails PII filters on outputs, masking in prompts, no case data in application logs, prompts and outputs stored only in the encrypted audit store.
Audit. Every answer is reconstructable: which manual version, which passages, which model version, which prompt template.
Fairness. Eligibility logic is the same code for everyone. The model drafts language only, and notices are checked for reading level and consistency.

Cost drivers and how to estimate

Caseworkers times questions per day times tokens per question. A state with 2,000 caseworkers asking 20 questions a day at 4,000 tokens each is about 160 million tokens a day. Multiply by price.
Policy manual indexing is small (thousands of pages), and re-indexing each quarter is cheap.
The rules engine is ordinary compute and cheap. The expensive part is people writing and testing the rules.

Failure modes

Model cites the wrong manual version: version is a hard metadata filter, not a hint in the prompt.
Model and rules engine disagree: the rules engine wins, and the disagreement is logged as a signal that either the rules or the retrieval need review.
Policy change not yet in the rules: rules have effective dates and a test suite. A rule change ships only with tests from policy staff.
Assistant down: caseworkers work as they do today. Nothing blocks a case.

How to measure quality

Policy Q&A: 300 questions from the state's policy help desk with answers approved by policy staff. Score correctness and citation accuracy.
Rules engine: every test case the state's quality control unit uses, with exact expected outputs. 100 percent pass required to ship.
Production: time per case, quality control error rate, and how often caseworkers edit the drafted notices.

Phase 1 and phase 2

Phase 1	Phase 2
Policy Q&A with citations for one program, no case data. Measure against the help desk set.	Case-aware questions with masked fields. Integration with the rules engine. Notice drafting. Second program.

Phase 1

Phase 2

Policy Q&A with citations for one program, no case data. Measure against the help desk set.

Case-aware questions with masked fields. Integration with the rules engine. Notice drafting. Second program.

Likely follow-ups

Models are very good now. Why can't the LLM decide eligibility?

Because the requirement is not "usually right," it is "the same answer for the same facts, and a written reason tied to a specific rule." Model outputs can vary between runs and model versions, and the explanation it gives is not proof of how it got there. At a fair hearing, the state has to show which rule applied. A rules engine gives exactly that. The model still adds a lot: finding the right policy, explaining it in plain words, drafting the notice.

How do you keep policy answers current when the manual changes every quarter?

Treat the manual like code. Each release is ingested as a new version with an effective date, and old versions stay. Retrieval filters by the date that matters for the case. When a new version lands, I'd re-run the evaluation set so we see which answers changed and have policy staff confirm the changes are correct.

What does IRS 1075 change in your design?

It tells me to keep the Federal Tax Information footprint small. I'd isolate FTI in its own data store with its own keys and access list, let only the rules engine read it, keep it out of prompts and model logs in phase 1, and log every access. That way the GenAI parts of the system are outside the 1075 scope, which makes the state's security review much faster.

A caseworker says the assistant's answer contradicts their training. What happens?

There's a "this looks wrong" button that captures the question, answer, and sources and sends it to the policy team. Either the answer is wrong, and it goes into the evaluation set as a regression test, or the training is out of date, which the state wants to know too. In both cases the caseworker follows the manual, not the assistant.

I can explain this out loud

Scenario E: Beacon Learn AI tutor across 150 school districts

Beacon's CEO: "Beacon Learn is in 150 districts, kindergarten through 12th grade. Every competitor has an AI tutor now. I want one inside Beacon Learn that helps students with homework without doing it for them, that teachers can see into, and that doesn't blow up our margins. Superintendents will ask about student privacy first."

Picture it: A tutoring center in a school building. Each tutor has a badge showing which grade they may help, a lesson plan from the teacher, and a rule to ask questions rather than hand over answers. Anything worrying a child says goes straight to the counselor. The center has a fixed number of tutor-hours per student per week, and the principal can read the session notes.
In plain words: Every request carries the district, the grade band, and the student. Those three choose the safety settings, the curriculum the tutor draws on, and the student's daily budget. Teachers see flagged chats and summaries. Student data is used only to tutor that student, and is deleted when the district says so.
The real terms: Multi-tenant SaaS with a pooled model and tenant context, Bedrock Guardrails configured per age band, a curriculum Knowledge Base with district filters, per-tenant and per-user token quotas, streaming responses, FERPA (school-official exception, data use limits), COPPA (under-13 consent via the school), state student privacy laws, and a teacher dashboard.

Clarifying questions to ask

How many students, and what is peak concurrency? When do they use Beacon Learn: class time, evenings, test weeks?
Which grades? A kindergartner and a high school senior need very different tutors, or maybe no tutor at all for the youngest.
What do districts' contracts say about data use, retention, and deletion? Do any states have their own student privacy law we sign to?
What should happen if a student writes about self-harm, abuse, or a threat?
How much control do teachers want? Turn it off for a test, set "hints only," see every chat?
What is the target cost per student per year, so we can price it?
Does it need to work on low-end Chromebooks and slow school networks?

Requirements

Functional	Non-functional
Socratic help on the current assignment, grounded in the district's curriculum Teacher controls: on, off, hints only, per class and per assignment Teacher view: summaries, flagged chats, common misconceptions Safety escalation to designated district staff	Age-appropriate content, per grade band FERPA and COPPA; no training on student data; deletion on request Handle school-hours peaks, several times the evening load First words in about a second on school Wi-Fi Known cost ceiling per student

Functional

Non-functional

Socratic help on the current assignment, grounded in the district's curriculum
Teacher controls: on, off, hints only, per class and per assignment
Teacher view: summaries, flagged chats, common misconceptions
Safety escalation to designated district staff

Age-appropriate content, per grade band
FERPA and COPPA; no training on student data; deletion on request
Handle school-hours peaks, several times the evening load
First words in about a second on school Wi-Fi
Known cost ceiling per student

District and grade band from the login pick the guardrail policy, the curriculum filter, and the budget. Teachers see into the tutor through their own view, not through raw logs.

Walk one request through it

A 7th grader in Maple District, signed in through school SSO, asks "what's the answer to number 4?" on a fractions assignment.
The token says district=maple, grade band 6 to 8. The tutor service checks the teacher's setting for this assignment: hints only. It checks the student's budget for today: plenty left.
It retrieves the lesson's worked examples from Maple's curriculum in the Knowledge Base.
It calls a small model with a system prompt for the 6 to 8 band ("ask a guiding question, never give the final answer"), through the 6 to 8 guardrail policy.
The reply streams: "Let's look at the denominators first. What do 3 and 4 have in common?"
The exchange is stored encrypted under Maple's key, the token count is charged to the student's budget, and a summary line goes to the teacher's view.

Design decisions and trade-offs

Decision	Options	My pick and why
Tenancy model	Silo (stack per district), pool (shared stack, tenant context), or bridge	Pool, with tenant ID enforced everywhere and per-district KMS keys for stored chats. 150 separate stacks would be costly to run. Offer a silo tier to a large district that pays for it.
Model choice	One large model for all, or a small model by default with escalation	Small model by default. Tutoring turns are short. Escalate to a larger model only for hard subjects like calculus proofs, if evaluation shows it helps.
Safety	One guardrail policy, or policies per age band plus district overrides	Per age band, with district overrides inside limits. A 1st grader and a 12th grader need different topic rules.
Peaks	Provision for peak, or on-demand with quotas	On-demand, with Bedrock throughput quotas raised ahead of the school year and per-tenant rate limits so one district's test day does not starve another.
Chat retention	Keep all chats, or keep summaries and delete raw chats on a schedule	Retention set per district contract. Default short retention of raw chats, longer for summaries and flags.

Decision

Options

My pick and why

Tenancy model

Silo (stack per district), pool (shared stack, tenant context), or bridge

Pool, with tenant ID enforced everywhere and per-district KMS keys for stored chats. 150 separate stacks would be costly to run. Offer a silo tier to a large district that pays for it.

Model choice

One large model for all, or a small model by default with escalation

Small model by default. Tutoring turns are short. Escalate to a larger model only for hard subjects like calculus proofs, if evaluation shows it helps.

Safety

One guardrail policy, or policies per age band plus district overrides

Per age band, with district overrides inside limits. A 1st grader and a 12th grader need different topic rules.

Peaks

Provision for peak, or on-demand with quotas

On-demand, with Bedrock throughput quotas raised ahead of the school year and per-tenant rate limits so one district's test day does not starve another.

Chat retention

Keep all chats, or keep summaries and delete raw chats on a schedule

Retention set per district contract. Default short retention of raw chats, longer for summaries and flags.

Security, privacy, and child safety

FERPA. Beacon acts as a school official under the district contract. Student data is used only for the district's educational purpose. No training models on it, no advertising, deletion on request.
COPPA. For students under 13, the school can consent on the parent's behalf for educational use only. Collect the minimum, and document it for the district.
State laws. Many states have student privacy laws and data privacy agreements districts ask vendors to sign. The design must support per-district retention and deletion.
Content safety. Guardrails content filters set strictest for young students, denied topics per age band, prompt attack filter, and a grounding check to the curriculum.
Escalation. Self-harm, abuse, or threats trigger the district's own protocol: a supportive message to the student with crisis resources, and an alert to the designated staff member. Designed with district counselors, not by engineers alone.

Cost drivers and how to estimate

Work from the student, because that is how Beacon prices.

Active students: say 600,000 enrolled across 150 districts, 30 percent weekly active, so 180,000.
Usage: 3 sessions a week, 8 turns each, about 1,500 input tokens (prompt, curriculum passages, history) and 150 output tokens per turn.
Tokens per active student per week: 24 turns times 1,650, about 40,000.
Multiply by current small-model prices, then by about 36 school weeks. That gives cost per active student per year, which Beacon compares to the price per seat.
Levers: prompt caching for the fixed system prompt, trimming chat history, smaller model, per-student daily caps.

Failure modes

Throttling on the first day of school: raise quotas weeks ahead, rate-limit per district, show a friendly "tutor is busy, try in a minute" rather than an error.
Tutor gives the answer anyway: evaluation set of "give me the answer" attempts per grade band; teacher can report a chat.
Harmful content reaches a child: layered guardrails, strictest settings for young grades, and every flagged event reviewed.
Wrong math: grounding in worked examples, and for arithmetic, a calculator tool rather than model arithmetic.

How to measure quality

Pedagogy set, reviewed by teachers: does the tutor guide instead of answer, at the right reading level?
Safety red-team set per grade band, re-run on every model or prompt change.
Production: teacher report rate, flag rate, student thumbs up, and, with the district, whether assignment completion or scores move.

Phase 1 and phase 2

Phase 1	Phase 2
Grades 6 to 12, math only, five pilot districts, hints-only mode, teacher dashboard with flags.	More subjects, grades 3 to 5 with stricter settings, teacher-authored tutor instructions per assignment, misconception reports.

Phase 1

Phase 2

Grades 6 to 12, math only, five pilot districts, hints-only mode, teacher dashboard with flags.

More subjects, grades 3 to 5 with stricter settings, teacher-authored tutor instructions per assignment, misconception reports.

Likely follow-ups

A superintendent asks: "Is my students' data training your AI?"

No. The models run on Amazon Bedrock, which does not use customer prompts or outputs to train models, and Beacon's contract says student data is used only to provide the service to your district. Chats are encrypted with a key specific to your district, kept for the period your agreement sets, and deleted on request. I'd hand them the data flow diagram and the data privacy agreement in the same meeting.

How do you stop one district's test day from slowing down everyone else?

Per-tenant rate limits at the API layer, sized from each district's enrollment, plus per-student caps. Behind that, Bedrock quotas raised ahead of known peaks like the first weeks of school. If we still hit limits, requests queue briefly and the student sees a "busy" message, not an error. I'd watch per-district latency on a dashboard during peak hours.

Why a pooled model and not a separate stack per district?

150 stacks means 150 deployments, 150 sets of alarms, and idle capacity everywhere. Pooling is far cheaper to run. The risk is data mixing, so I'd enforce tenant context in code at every layer, use per-district encryption keys, and test isolation on every deploy. For a very large district that requires dedicated infrastructure, a silo tier at a higher price is a reasonable offer.

What's your answer if a student tells the tutor they want to hurt themselves?

The tutor stops tutoring, responds with care and gives crisis resources, and the event goes to the staff member the district designated, following the district's own protocol. I would design that flow with district counselors and legal, because the right response is a school decision, not an engineering one. We'd test it with a red-team set every release.

I can explain this out loud

Scenario F: Higher-ed advising agent

Beacon's CEO: "We're moving into universities. Advisors each have 400 students and can't keep up. I want an advising agent a student can ask 'if I switch to a data science minor, can I still graduate in May?' It should know their transcript, the catalog, and degree requirements."

Picture it: A GPS is only as good as its map. If the map says a bridge exists that was torn down last year, the GPS gives you a confident, polite route into the river. Most of the work in this scenario is the map.
In plain words: Before any agent, get the student records, course catalog, and degree rules into one clean, governed place, with each student allowed to see only their own rows. Then give the agent a few tools: run the official degree audit, search the catalog, read my record. The degree audit tool gives the answer on requirements, and the agent explains it.
The real terms: A data lake on S3 with raw, clean, and curated zones, AWS Glue for ETL, the Data Catalog, and data quality rules, Lake Formation for row and column permissions, Athena for queries, then an agent on AgentCore Runtime with tools through AgentCore Gateway and identity passed through so the tools act as the student.

Clarifying questions to ask

Which systems hold the truth: student information system, degree audit tool, learning management system, catalog? Who owns each?
Is there already a degree audit system (many universities have one)? Can we call it through an API?
How clean is the data? Do course codes match across the catalog and transcripts? How are transfer credits and substitutions recorded?
What may the agent say without an advisor: information only, or also recommendations?
Which questions do advisors spend most time on today?
How fresh must data be? Registration week changes seat counts hourly.
FERPA: who else may see a student's chat? Their advisor, by default?

Requirements

Functional	Non-functional
What-if questions on majors, minors, graduation timing Answers based on the official degree audit, not model reasoning Catalog search: prerequisites, when offered Handoff to the student's advisor with a case summary	Student sees only their own records FERPA: records used for the advising purpose only Data freshness: daily for records, hourly during registration Every answer shows its data date and source

Functional

Non-functional

What-if questions on majors, minors, graduation timing
Answers based on the official degree audit, not model reasoning
Catalog search: prerequisites, when offered
Handoff to the student's advisor with a case summary

Student sees only their own records
FERPA: records used for the advising purpose only
Data freshness: daily for records, hourly during registration
Every answer shows its data date and source

The bottom zone is the work most teams skip. The record query tool can only read rows Lake Formation allows for this student. The degree audit tool, not the model, answers "can I graduate."

Walk one request through it

A junior asks "If I add a data science minor, can I still graduate in May?"
Her SSO token carries her student ID. The agent's tools run with her identity, so Lake Formation returns only her rows.
The agent calls the degree audit tool in what-if mode with "add minor: data science." The official audit returns: two courses short, one of them only offered in fall.
It searches the catalog KB for those courses, their prerequisites, and terms offered.
It answers: "Based on the degree audit as of this morning, adding the minor pushes you to December, because DS 310 is only offered in fall. Here are two options." Then: "Want me to send this to your advisor, Dr. Chen?"
If she says yes, the advisor gets a short case summary. The conversation is stored for the advisor to see, per the university's FERPA policy.

Design decisions and trade-offs

Decision	Options	My pick and why
Where "can I graduate" is answered	Model reasons over the transcript and catalog, or call the official degree audit	Official degree audit. Degree rules are full of exceptions and substitutions the model will miss. The model explains the audit.
Data first or agent first	Build the agent on the raw systems now, or build the governed lake first	A thin lake first: only the tables the top five questions need. Waiting for a full data platform takes a year. Skipping it gives wrong answers.
How tools access data	Service role that can see everything, or the student's own identity	The student's identity, passed through to the tools. A prompt trick cannot widen what the database returns.
Structured data access	Put records in a vector store, or query tables	Query tables. Grades and credits are exact numbers. Vectors are for text like catalog descriptions.

Decision

Options

My pick and why

Where "can I graduate" is answered

Model reasons over the transcript and catalog, or call the official degree audit

Official degree audit. Degree rules are full of exceptions and substitutions the model will miss. The model explains the audit.

Data first or agent first

Build the agent on the raw systems now, or build the governed lake first

A thin lake first: only the tables the top five questions need. Waiting for a full data platform takes a year. Skipping it gives wrong answers.

How tools access data

Service role that can see everything, or the student's own identity

The student's identity, passed through to the tools. A prompt trick cannot widen what the database returns.

Structured data access

Put records in a vector store, or query tables

Query tables. Grades and credits are exact numbers. Vectors are for text like catalog descriptions.

Security and compliance

FERPA. Education records used only for advising, access limited to the student and school officials with a legitimate interest (their advisor). No model training on records.
Least privilege. Lake Formation row filters by student ID, column rules hide fields the agent never needs (for example financial aid details, disciplinary records).
Identity propagation. AgentCore Identity or token exchange so each tool call runs as the student.
Answers with dates. Every answer shows the data date, so a stale answer is visible.

Cost drivers and how to estimate

Data work (engineer time for pipelines and quality) is the largest cost, and it is mostly one-time.
Agent usage is modest: students ask a few questions a term, with peaks at registration. Students times questions per term times model calls per question times tokens.
Glue jobs and Athena queries are small at university scale.

Failure modes

Course codes that don't match between catalog and transcripts: caught by Glue data quality rules before the agent ever sees them.
Transfer credits missing: the audit says so, and the agent tells the student to confirm with the registrar.
Stale data during registration: hourly sync for sections and seats, and the answer states the time.
Degree audit API down: the agent says it cannot check requirements right now and offers the advisor handoff. It does not guess.

How to measure quality

Advisors write 100 real questions with correct answers from past appointments. Score answer correctness and correct use of the audit tool.
Data quality metrics per table: completeness, freshness, share of rows failing rules.
Production: advisor appointment load for routine questions, handoff rate, student satisfaction, and advisor-reported wrong answers.

Phase 1 and phase 2

Phase 1	Phase 2
Data foundation for the tables the top five questions need. Catalog Q&A (no personal data). Then degree audit what-ifs for one college.	All colleges. Proactive nudges ("you haven't registered for a required course"). Advisor-side assistant that preps each appointment.

Phase 1

Phase 2

Data foundation for the tables the top five questions need. Catalog Q&A (no personal data). Then degree audit what-ifs for one college.

All colleges. Proactive nudges ("you haven't registered for a required course"). Advisor-side assistant that preps each appointment.

Likely follow-ups

The CEO wants a demo in four weeks. Why are you talking about a data lake?

We can demo in four weeks on catalog questions, which need no personal data. But an advising agent that reads messy records will give wrong graduation answers politely, and one wrong "you can graduate in May" is a university-wide trust problem. So I'd run two tracks: the catalog demo now, and a thin data foundation covering only the tables the first questions need. That is weeks, not a year.

Why not put the transcript in a vector store and let RAG handle it?

Because a transcript is structured data with exact values: course codes, credits, grades. Similarity search is built for fuzzy text and might return a similar course instead of the one she took. For records I'd use a tool that runs a precise query with her identity. Vectors are the right fit for catalog descriptions and policy text.

How do you make sure a student can't see another student's records?

The tools don't use a powerful service account. They run with the student's identity, and Lake Formation row filters return only her rows. So even if someone tricks the agent into asking for another student, the database answers with nothing. I'd test that path on every release.

What's the role of the human advisor once this exists?

More time on the conversations that need judgment: a student thinking about dropping out, a hard choice between majors, a hardship. The agent takes the routine lookups and prepares the advisor with a summary. I'd measure success partly by how much advisor time moves from lookups to those conversations.

I can explain this out loud

Numbers and rules of thumb

You do not need exact prices on a whiteboard. You need orders of magnitude and a method, said out loud. Every number below is approximate. The interviewer is checking that you know the shape, not the decimal.

Back-of-the-envelope math

The problem: "How much will this cost?" is the question that exposes candidates who drew boxes without thinking about volume. Silence or "it depends" loses the point.
Picture it: Planning fuel for a road trip. You don't know the exact price at every station, but you know miles, miles per gallon, and roughly what a gallon costs. That's enough to say "about 200 dollars, give or take 50."
In plain words: Requests per day, times tokens per request, times price per token, times 30. Then say which input you are least sure about and how you would measure it in the pilot.
The real term: Unit economics: cost per request, per user, per tenant. Input and output tokens are priced separately and output usually costs several times more per token.
At Beacon: The CEO thinks in price per seat or per agency. Convert your estimate into cost per student per year or cost per 911 call, so he can compare it to what he charges.

I can explain this out loud

Useful orders of magnitude

Thing	Rough number	How to use it
Tokens per English word	About 1.3 (so 100 words is about 130 tokens)	Convert documents and transcripts to tokens.
Tokens per dense page	About 500 to 800	A 2,000-page policy manual is roughly 1 to 1.5 million tokens to embed once.
Speech rate	About 130 to 160 words per minute	A 3-minute 911 call is about 450 words, about 600 tokens of transcript.
Time to first token	Roughly 0.3 to 2 seconds, depending on model size, prompt length, and load	Stream responses so users see progress. Budget more for large models and long prompts.
Output speed	Roughly 30 to 150 tokens per second	A 300-token answer takes about 2 to 10 seconds to finish. Short outputs are fast outputs.
Agent turn	3 to 6 model calls per user question	Multiply latency and cost accordingly. Cap steps.
Embedding and vector search	Tens to low hundreds of milliseconds each	Retrieval is rarely the latency problem. The model is.
Vector size	1,024 dimensions as float32 is 4 KB per vector	10 million chunks is about 40 GB of raw vectors, plus index overhead that can be similar again for graph indexes.
Chunk size	About 300 to 1,000 tokens, 10 to 20 percent overlap	Smaller chunks for precise facts, larger for context. Test both on your evaluation set.
School traffic shape	Most use in about 7 hours a day, about 180 days a year	Peak-to-average ratio is high. Size quotas for the peak, pay on demand.
Evaluation set	50 to 300 real questions to start	Enough to catch regressions. Grow it from production feedback.

Thing

Rough number

How to use it

Tokens per English word

About 1.3 (so 100 words is about 130 tokens)

Convert documents and transcripts to tokens.

Tokens per dense page

About 500 to 800

A 2,000-page policy manual is roughly 1 to 1.5 million tokens to embed once.

Speech rate

About 130 to 160 words per minute

A 3-minute 911 call is about 450 words, about 600 tokens of transcript.

Time to first token

Roughly 0.3 to 2 seconds, depending on model size, prompt length, and load

Stream responses so users see progress. Budget more for large models and long prompts.

Output speed

Roughly 30 to 150 tokens per second

A 300-token answer takes about 2 to 10 seconds to finish. Short outputs are fast outputs.

Agent turn

3 to 6 model calls per user question

Multiply latency and cost accordingly. Cap steps.

Embedding and vector search

Tens to low hundreds of milliseconds each

Retrieval is rarely the latency problem. The model is.

Vector size

1,024 dimensions as float32 is 4 KB per vector

10 million chunks is about 40 GB of raw vectors, plus index overhead that can be similar again for graph indexes.

Chunk size

About 300 to 1,000 tokens, 10 to 20 percent overlap

Smaller chunks for precise facts, larger for context. Test both on your evaluation set.

School traffic shape

Most use in about 7 hours a day, about 180 days a year

Peak-to-average ratio is high. Size quotas for the peak, pay on demand.

Evaluation set

50 to 300 real questions to start

Enough to catch regressions. Grow it from production feedback.

Service limits worth knowing

Service	Limit (approximate, check quotas)	Why it matters on a whiteboard
Lambda	15-minute max run time, up to 10,240 MB memory, 6 MB synchronous request and response payload, default 1,000 concurrent executions per Region (raisable)	Fine for chat turns and tool calls. Not for minutes-long audio streams. Response streaming helps with token streaming.
API Gateway REST API	29-second default integration timeout, raisable for Regional and private APIs (possibly at the cost of lower throttle limits); 10 MB payload	Long model answers can exceed 29 seconds. Stream, go async, or use WebSockets or ALB.
API Gateway HTTP API	30-second integration timeout	Same trap as above.
API Gateway WebSocket	Connections up to 2 hours, idle timeout 10 minutes	Good for streaming chat to browsers.
Bedrock	Per-model quotas on requests and tokens per minute, per account and Region	The real scale ceiling for GenAI apps. Request increases before launch. Cross-Region inference spreads load where the data boundary allows.

Service

Limit (approximate, check quotas)

Why it matters on a whiteboard

Lambda

15-minute max run time, up to 10,240 MB memory, 6 MB synchronous request and response payload, default 1,000 concurrent executions per Region (raisable)

Fine for chat turns and tool calls. Not for minutes-long audio streams. Response streaming helps with token streaming.

API Gateway REST API

29-second default integration timeout, raisable for Regional and private APIs (possibly at the cost of lower throttle limits); 10 MB payload

Long model answers can exceed 29 seconds. Stream, go async, or use WebSockets or ALB.

API Gateway HTTP API

30-second integration timeout

Same trap as above.

API Gateway WebSocket

Connections up to 2 hours, idle timeout 10 minutes

Good for streaming chat to browsers.

Bedrock

Per-model quotas on requests and tokens per minute, per account and Region

The real scale ceiling for GenAI apps. Request increases before launch. Cross-Region inference spreads load where the data boundary allows.

The cost math, step by step

Requests per day (users times requests per user).
Input tokens per request (system prompt plus retrieved passages plus history plus question). This is usually the biggest number.
Output tokens per request (usually small).
Daily cost = requests times (input tokens times input price plus output tokens times output price). Monthly = times 30.
Add retrieval (vector store floor plus queries), transcription minutes, and compute.
Divide by the business unit: per student, per call, per agency.
Name the three levers: smaller model, fewer input tokens (caching, shorter history, fewer passages), fewer calls (cache frequent answers).

Common mistakes in the design round

Each of these costs points even when the final diagram is fine. Read them the night before.

At the end, the interviewer asks "what would you do differently with more time?"

I'd name two things that are true for this design, not a generic list. For example: "I'd run the shadow-mode pilot longer to get a real accuracy number before showing suggestions, and I'd spend more time with the agency's legal team on whether AI summaries become part of the official record, because that changes retention and discovery." Then I'd say what I'd measure first once it's live.

I can explain this out loud