Managing PoC Expectations and Making the Case for an MVP
Applies to: contracting developers delivering a GenAI PoC or any other time-boxed proof of concept · Time to learn: 25 minutes to read, one engagement to practise
A PoC answers one question with evidence. Write the question, the success criteria, and the "not in the PoC" list down at kickoff. Demo the real state of the system, including stubs and failures, next to the eval numbers. Close with a results report that compares criteria against measurements and ends in a clear recommendation: stop, iterate, or go to MVP. When the answer is MVP, translate the gap list into scoped work items with business reasons and effort ranges, and hand it to whoever owns the commercial proposal through the agreed channel. Never quote dates or prices that nobody has agreed.
This page covers the developer side: what you say, show, and write so that the customer's expectations match the evidence. The technical method for running the PoC lives in GenAI Proof of Concept; what production requires lives in AI Productification. This page links to both and does not repeat them. For the surrounding habits see Customer Communication, Contracts, Scope, and Boundaries, and Accountability.
What a PoC is, and what it is not
A PoC reduces uncertainty about one question: can this use case reach a quality bar the business accepts, at a cost and latency it can live with, on data it actually has? It is a measurement exercise that happens to produce software. It is not a small version of the product.
| PoC | MVP | Production / productification | |
|---|---|---|---|
| Goal | Answer a feasibility question with evidence | Put the smallest useful product in front of real users to learn from real use | Run the product reliably, safely, and affordably at scale |
| Data | Representative slice, often a copy, sometimes demo data | Real data for the supported scope, with a data plan | Live data with lineage, retention, and an erasure path |
| Users | Developers, a domain expert, a few stakeholders | A small, named group of real users | All intended users |
| Quality bar | Measured against agreed thresholds on a golden set | Good enough for the chosen users, with known limits stated | Eval gates in CI, online evals, regression protection |
| Security | Minimal: no real credentials in code, data handled per the data plan | Authentication, authorisation, and basic abuse limits for the pilot group | Reviewed against the threat model, guardrails, compliance pack |
| Operations | Runs on a laptop or a single environment; nobody is on call | Basic logging, a known rollback, a named contact for issues | SLOs, alerting, runbooks, on-call model |
| Typical length | 1–6 weeks | Scoped per case from the PoC report | 6–12 weeks after a successful PoC, then a run phase |
The PoC and productification lengths match the service pages (PoC, Productification). The MVP row is deliberately open: its length and content come from the gap list in your PoC report, not from a template.
The word "MVP" is used here in the Lean Startup sense: a version built to begin learning as quickly as possible, then refined through measurement (see Sources). It is a learning vehicle with a deliberately narrow scope, not a half-built production system.
The decision flow
Two gates matter. The first, after the PoC, is decided on the results report. The second, after the MVP, is decided on what real users did. Each gate has three honest outcomes, and "stop" is one of them.
Set expectations at kickoff
Most expectation problems are created in the first meeting and discovered in the last one. Spend the kickoff on four written artefacts.
1. The question and the success criteria
Write the question in one sentence and the criteria as metrics with thresholds and directions, signed off by the customer sponsor before any prompt is written. The method and metric families are in GenAI PoC, step 1. Your job on the people side is to make sure the sponsor can read the criteria and would recognise a pass or a fail without you in the room.
Question: Can the assistant answer supplier-contract questions well enough
to remove the manual first-pass review?
| Metric | Threshold | Measured on |
|-------------------------------|-------------------|-----------------------|
| Answer correctness | at least 85 % | golden set, 120 items |
| p95 latency | at most 4 s | load test |
| Cost per request | at most 0.02 EUR | traces |
Sign-off: sponsor, domain expert, engineering lead. Date: ...
The numbers above are an example format, in line with the examples on the PoC page. Your thresholds come from the customer's business case, not from what you expect to achieve.
2. The "not in the PoC" list
Say out loud what the PoC will not contain, and put it in writing. A good default list:
- Authentication and authorisation beyond a shared test login
- Scale, concurrency, and capacity planning beyond the load test in the plan
- SLAs, availability targets, and support hours
- Security hardening, penetration testing, and compliance documentation
- Monitoring, alerting, and on-call
- Integration with systems other than the ones named in the plan
- Polished UX, branding, and accessibility work
Each item becomes a line in the later gap list, so the list is also the first draft of the MVP conversation. Keep a parking lot for new ideas that surface during the PoC and review it at handover (GenAI PoC, pitfalls). Anything that changes scope goes through the change process in Contracts, Scope, and Boundaries.
3. Demo data versus real data
Agree early which data the PoC uses and what each choice means for the result.
| Data | What it proves | What it does not prove |
|---|---|---|
| Hand-picked demo data | The idea can work in the best case | Anything about typical quality |
| Representative real slice, labelled by the domain expert | Quality on the distribution the customer actually has | Behaviour on data you have not seen yet |
| Live production data | Fit to current reality | Needs a data plan, access approval, and deletion afterwards |
Eval numbers are only meaningful on the second or third row. If the customer can only provide demo data, say in writing that the results will describe the demo data and will not transfer.
4. The decision rule
Agree before the PoC what happens on each outcome. For example: all criteria met leads to an MVP discussion; most met with a clear path leads to a scoped extension; a hard criterion missed with no path leads to stopping. Deciding the rule in advance means the report discusses evidence, not feelings.
The demo trap
A polished demo makes a customer think the work is 90 % done. A working interface suggests a working product, and the customer has no direct view of what is missing: the failure handling, the data pipelines, the security, the monitoring. The gap between demo and product is mostly invisible work.
Counter it by showing the invisible parts on purpose.
How to demo honestly
- State the question and the criteria first. One slide: what we set out to prove and how we measure it.
- Label what is stubbed or mocked. A visible note such as "mocked: customer database; real: model, retrieval, prompts" beats a footnote nobody reads. Say it out loud while you click.
- Show failure cases. Pick real failures from the eval run: a wrong answer, a refusal, a slow request. Explain the cause and whether it is fixable inside the PoC, fixable with more work, or a limit of the approach.
- Show the eval numbers next to the demo. The pass rate per tag (language, difficulty), p95 latency, cost per request. The numbers are the deliverable; the demo illustrates them.
- Use inputs the audience supplies. Let the domain expert type their own hardest question. Record the result as a new golden-set item either way.
- Close with the gap list. Finish on "what production would additionally need", not on the happiest screen.
Words that keep expectations calibrated
| Instead of | Say |
|---|---|
| "It's basically done" | "The core question is answered; production needs the items on the gap list" |
| "It works" | "It passes 87 % of the golden set against an 85 % threshold" |
| "We just need to polish it" | "These five areas are outside the PoC: authentication, scale, monitoring, security review, support model" |
| "Should be quick" | "I will give you an effort range after we have scoped it; I do not want to guess in the room" |
Manage expectations during the PoC
Surprise at the end is a communication failure in the middle. Send a short weekly note, following the habits in Customer Communication.
Week 2 of 3
Results vs criteria
- Correctness: 79 % (threshold 85 %), up from 71 % last week
- p95 latency: 3.1 s (threshold 4 s)
- Cost per request: 0.014 EUR (threshold 0.02 EUR)
What changed: improved retrieval; added 30 hard cases to the golden set
What is blocking: labels for 12 items need the domain expert (needed by Thu)
Risk: correctness may plateau below threshold on scanned documents
Next week: test layout-aware parsing; load test
Decision needed: none this week
Three habits make this work:
- Report against the signed criteria every week, using the same table. Trends are more honest than adjectives.
- Raise a risk the week you see it. "Scanned documents may cap quality" in week 2 is a plan; in week 3 it is bad news.
- Pivot early and in writing. If the evidence says the approach will not reach the bar, propose a change to the approach or the criteria while there is time to test it. A changed criterion needs the same sign-off as the original one.
Close the PoC with a results report
The report is the PoC's main artefact and the document the decision gate uses. Keep it short and evidence-first. It is the "handover report" deliverable on the PoC page.
# PoC results report: <use case>
## 1. Summary
One paragraph and one recommendation: stop / iterate / go to MVP.
## 2. Criteria vs measured
| Metric | Threshold | Measured | Met? | Notes (runs, dataset version) |
|--------|-----------|----------|------|-------------------------------|
## 3. What worked
Approaches and configurations that moved the numbers, with evidence links.
## 4. What did not work
Approaches tried and dropped, and why. Failure analysis by category.
## 5. Risks and limits
Known quality gaps, data issues, cost sensitivity, dependencies.
## 6. What production requires
The gap list (see below): each item, why it matters, rough effort range.
## 7. Recommendation
Stop / iterate / go to MVP, with the reasoning and the proposed MVP scope.
## 8. Open items
Parking-lot ideas, decisions needed, data to delete under the data plan.
Rules for writing it:
- Put numbers in tables and cite the dataset version and the number of runs, so a reader can reproduce them.
- Report misses in the same table as passes. A report that hides a miss loses its value for the passes.
- Separate measured (from the eval suite), observed (seen in a demo or a few runs), and assumed (not tested). Production plans built on assumptions need to say so.
- Keep "what did not work" honest. It saves the customer from paying to rediscover it later.
Make the case for an MVP
If the evidence supports going on, your role is to make the next step concrete. The most persuasive document is the gap list turned into scoped work.
From gap list to scoped proposal
For every gap between the PoC and the MVP (and later, production), write one row.
| Gap | Work item | Why it matters to the business | Rough effort range |
|---|---|---|---|
| No authentication | SSO integration for the pilot group | Only authorised staff may see contract data | Small |
| Quality plateau on scans | Layout-aware document parsing and re-evaluation | Scans are a large share of real documents; below threshold today | Medium |
| No monitoring | Tracing dashboards and a cost alert | Cost and quality drift are visible before users complain | Small |
| Single environment | Staging and production environments as code | Changes are tested before users see them | Medium |
Guidance for each column:
- Gap: use the "not in the PoC" list and the failure analysis. Every row traces back to something the customer has already seen.
- Work item: a deliverable someone can accept or reject, in the style of the productification deliverables.
- Why it matters: one sentence in the customer's language (risk, cost, time, compliance), not in engineering terms.
- Effort range: a range with its main assumption, expressed in relative sizes or person-weeks as your team normally estimates. It is input for whoever scopes the proposal, not a quote.
Then order the rows into an MVP slice (the smallest set that lets real users produce real learning) and a later slice (what production adds). The MVP slice is narrow on purpose: fewer users, fewer features, real data, a named feedback loop. Google's guidance on ML systems makes the same point from the engineering side: keep the first model simple and get the infrastructure right first, and do not be afraid to launch without machine learning in the loop when it is not the bottleneck (see Sources).
Let the evidence sell
- Lead with the customer's own success criteria and the measured result, not with a feature list.
- Show the cost of doing nothing the PoC revealed: the failure categories, the manual effort the use case would remove.
- Point out the hidden maintenance burden honestly. Machine learning systems carry ongoing costs in data dependencies, configuration, and monitoring that a PoC does not exercise; the "Hidden Technical Debt" paper catalogues these (see Sources). Naming them early builds trust and explains why productification exists.
- Offer a choice between slices where one exists, with the trade-offs stated.
Be honest when the answer is "don't proceed"
Sometimes the right report says stop: a hard criterion is unreachable, the economics do not work, or the data is not good enough. Say so plainly, with the evidence and with what would have to change for a retry. A customer who has been told an unwelcome truth once will believe your next "go". Your credibility is a long-term asset, and every inflated recommendation spends it. Google's SRE guidance makes a parallel observation: users build on what a system actually delivers, not on what was promised (see Sources); the same holds for a recommendation.
Hand over to whoever scopes the commercial proposal
Your report and gap list are the technical input to the commercial step. Pass them on through the channel the engagement has agreed, as set out in Contracts, Scope, and Boundaries and the overview, and stay available for technical questions. Within that:
- Do not promise dates or prices yourself. In the room, say "I will give you the technical scope and effort range; the proposal will follow through the agreed route."
- Do not commit to an MVP scope verbally. Agree the technical content of the slice with the customer, then let the proposal formalise it.
- Keep estimates labelled as estimates, with their assumptions, so they cannot be mistaken for commitments.
- Follow-on ideas that are outside the PoC but visible to you belong in the report's open items; see Upsell Opportunities for how to raise them.
Pitfalls to design around
- A demo that outshines the evidence → open every demo with the criteria and close it with the numbers and the gap list.
- Success criteria written after the results → get sign-off before the first prompt, and version the file in the repo.
- Vague "not in the PoC" statements → list authentication, scale, SLAs, hardening, and monitoring by name, in writing.
- Demo data standing in for real data → label the dataset on every results table and say what it does and does not prove.
- Silent scope creep during the PoC → keep a parking lot and route real scope changes through the change process.
- End-of-PoC surprises → send a weekly criteria-versus-measured note, and flag risks the week you see them.
- Estimates that become commitments → give ranges with assumptions, and let the agreed channel issue dates and prices.
- Recommending "go" by default → write the stop and iterate options into the decision rule, and use them when the data says so.
- Gap lists written in engineering terms → attach a business reason to every row.
Checklist
Kickoff
- Question and success criteria signed off by the sponsor, versioned in the repo
- "Not in the PoC" list written and acknowledged (authentication, scale, SLAs, hardening, monitoring, others)
- Data choice agreed and its limits written down (demo, representative slice, live)
- Decision rule agreed for stop, iterate, and go
During the PoC
- Weekly note sent with criteria versus measured, risks, and decisions needed
- Parking lot maintained; scope changes handled through the change process
- Early pivots proposed in writing with the evidence
Demo and close
- Demo labels what is stubbed or mocked, shows failure cases, and shows the eval numbers
- Results report delivered: criteria versus measured, what worked, what did not, risks, what production requires, recommendation
- Measured, observed, and assumed claims kept separate
Making the case
- Gap list converted to rows: gap, work item, business reason, effort range with assumptions
- MVP slice and later slice distinguished
- A "don't proceed" recommendation given plainly where the evidence points that way
- Report and gap list handed to the commercial step through the agreed channel
- No dates or prices promised outside the agreed channel
Sources
- Eric Ries, The Lean Startup, Principles — theleanstartup.com/principles (MVP as the means to begin learning as quickly as possible, then refine through measurement)
- Sculley et al. — Hidden Technical Debt in Machine Learning Systems, NeurIPS (NIPS) 2015 (ongoing maintenance costs and ML-specific risk factors in real-world systems)
- Google — Rules of Machine Learning (Rule 1, "Don't be afraid to launch a product without machine learning"; Rule 4, "Keep the first model simple and get the infrastructure right")
- Google — Site Reliability Engineering, Service Level Objectives (users build on the reality of what you offer, rather than what you say you will supply; avoid absolutes in targets)
- Lieber Dev Training — GenAI Proof of Concept, AI Productification