Rapid prototyping, POCs and pilots with measurable success criteria

Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.

Concept: POCs, pilots and technical evaluations · also covers Rapid prototyping, KPIs and measurable outcomes · Step 4 of 34 in AI forward deployed engineer · version 1

Practice this

On this page

Prototypes, proofs of concept and pilots are how an FDE reduces uncertainty quickly and gives the customer evidence for a go or no-go decision. The work is only useful if the question being answered, the time allowed and the pass criteria are fixed before building starts. This lesson covers the distinction between the stages, how to write measurable criteria, how to bound the effort, and how to build and evaluate fast.

Read more

How it is done in practice

The team writes a one-page POC or pilot charter: the question, the appetite in weeks, the dataset, the criteria with thresholds, the decision owner and the next step on pass. The FDE builds a thin Streamlit or notebook front end over a pipeline function within days and demos on real customer cases. Every run is logged, a customer expert labels a sample each week, and failures are added to the evaluation set. A weekly readout shows the metrics against the thresholds and the baseline. At the end date the team presents results against the charter and the customer decides, instead of extending by default.

Technologies

StreamlitJupyterGradioPythonpytestClaude APILangfuseLangSmithBraintrustVercel

Diagram

flowchart LR
  A[Charter: question, appetite, criteria] --> B[Prototype on real cases]
  B --> C[Log traces]
  C --> D[Expert review and labels]
  D --> E[Eval set grows]
  E --> F[Run evals on each change]
  F --> B
  F --> G{Criteria met by end date}
  G -->|yes| H[Pilot or production rollout]
  G -->|no| I[Stop or reshape]
flowchart LR
  A[Charter: question, appetite, criteria] --> B[Prototype on real cases]
  B --> C[Log traces]
  C --> D[Expert review and labels]
  D --> E[Eval set grows]
  E --> F[Run evals on each change]
  F --> B
  F --> G{Criteria met by end date}
  G -->|yes| H[Pilot or production rollout]
  G -->|no| I[Stop or reshape]

Code

python
import streamlit as st
from pipeline import extract_claim  # the part you keep

st.title("Claim intake POC")
doc = st.file_uploader("Upload a claim document", type=["pdf", "txt"])

if doc and st.button("Extract"):
    with st.spinner("Running pipeline"):
        result = extract_claim(doc.read())
    st.subheader("Extracted fields")
    st.json(result.fields)
    st.subheader("Evidence")
    for cite in result.citations:
        st.markdown(f"- page {cite.page}: {cite.quote}")
    ok = st.radio("Is this correct?", ["yes", "no"], index=None)
    if ok:
        result.save_label(ok)  # feeds the eval set

Subtopics

Prototype, POC and pilot: what each one proves

Three different questions

A prototype answers whether an idea is worth pursuing: it is built in hours or days, often on sample data, to make the concept concrete for stakeholders. A proof of concept answers whether it is technically feasible on this customer's data and systems, and it targets the riskiest assumption, such as extraction accuracy or an integration path. A pilot answers whether it delivers value in real use: a limited group of real users runs real cases over a few weeks, with production-grade access control, logging and support, even if scale and polish are limited.

Avoiding the stalled POC

A POC without an agreed exit drifts into an open-ended free trial. Before starting, write down the decision it informs, who makes that decision, the date, the criteria, and what happens on pass: typically a paid pilot or production rollout with a named budget owner. Also agree what the customer must supply, such as data samples, a subject matter expert for labeling and access to a test environment. If these cannot be agreed, that is itself a signal about the engagement.

Writing measurable success criteria

Specific, measurable, achievable, relevant

Anthropic's guidance on defining success asks for criteria that are specific about the task, measurable with a number or a well-defined qualitative scale, achievable given current model capability, and relevant to the application's purpose. A weak criterion is that the assistant should answer well. A strong one states the metric, the dataset and the threshold, for example an F1 score of at least a stated value on a held-out set of labeled customer cases. Most use cases need several dimensions at once: task fidelity, consistency, tone, privacy preservation, latency and price.

Tying criteria to the baseline

Criteria carry weight with a sponsor when they are expressed against the current process. Use the baseline from discovery: handling time per case, error or rework rate, backlog age, cost per case. A pilot criterion then reads as a comparison, such as median handling time reduced by a stated fraction with no increase in error rate on audited samples. Add guardrail criteria that must not regress, such as zero leakage of restricted fields, and agree who audits the samples and how disagreements are settled.

Setting an appetite and narrowing the problem

Fixed time, variable scope

Shape Up starts from appetite: decide how much time the problem is worth, then design a solution that fits, instead of estimating an imagined solution. Fixing time and varying scope forces trade-off decisions early. Two further habits apply directly to POCs. Narrow the problem by asking what specifically went wrong in the cases that prompted the request, which often reveals a much smaller target than the original ask. Watch for grab-bags, requests like a redesign or a general assistant with no single problem behind them, and decline to start until one concrete use case is chosen.

Building the prototype fast

Thin UI over real logic

For a prototype or POC, the interface only needs to let a stakeholder put in a real case and see the result with its evidence. Streamlit turns a plain Python script into an interactive web app: widgets such as text inputs, file uploaders and buttons are function calls, and the script reruns from top to bottom on each interaction, with session state and caching for anything that should persist. That keeps the engineer's time on the pipeline itself: data access, prompts, retrieval and output validation. Notebooks and simple chat front ends serve the same purpose.

What to keep and what to throw away

Decide at the start which parts are disposable. The UI usually is. The pipeline function, the prompt and schema definitions, the dataset of customer cases and the evaluation script are not, and should live in a repository with tests from the first day. Keep the pipeline behind a plain function or API boundary so the prototype UI can be replaced by the production surface without rewriting the logic. Log every input, output and intermediate step so that failures seen in demos become test cases.

Running the evaluation loop during a pilot

Levels of evaluation in a customer pilot

Hamel Husain's framing has three levels: cheap assertion-style unit tests run on every change, human and model-graded review of logged traces on a regular cadence, and A/B testing once there are real users. In a pilot, the specific additions are that customer subject matter experts do the human review, so the trace viewer must be simple enough for them to use, and that the pass rate on the agreed dataset is reported to the sponsor each week as the evidence for the exit decision.

Sources

Related lessons

Practice this
Mapping customer workflows and choosing what to automate with AIPython for integrations, services and data transformation