Chapter 11 — Practice-on-your-own solutions

The one-sentence confirmation-turn prompt. «Before I use this figure, confirm which specific churn measure the 12.4% represents (logo churn, revenue churn, gross-net churn, or something else), the exact date range applied, and which customer segment or scope was included in the calculation.»

The four things to check in the response.

1. Does Copilot name the specific measure? A clean response looks like «12.4% represents net revenue churn calculated as (revenue lost from downgrades and cancellations minus revenue gained from upgrades and expansions) divided by starting revenue.» If the response uses the generic word «churn» without a modifier (logo / revenue / gross / net), the confirmation is incomplete — push back.
2. Does it name the date range? A clean response looks like «calculated over the fiscal Q3 window (Jul 1 — Sep 30, 2026).» If the response says «Q3» without dates, ask which fiscal calendar — company fiscal Q3 and calendar Q3 diverge in most organisations, and the resulting churn figure will differ.
3. Does it name the segment or scope? A clean response looks like «calculated across all paying customers with contracts active on the window start date; excludes trial and freemium customers.» If Copilot doesn’t name the scope, ask — and be ready for the answer «all customers in the customer table,» which may include categories you didn’t mean.
4. Does it hedge or refuse to specify? If Copilot responds with something like «the exact definition depends on how the underlying measure was authored» or «this may vary based on your organisation’s reporting standards,» the response is a soft refusal — the model does not have a defensible answer, and the 12.4% figure is not safe to quote. This is the highest-signal response of the four; treat it as a red flag rather than as a satisfying answer.

Why the confirmation turn matters even when Copilot sounds confident. Three churn measures in the model means three plausible numerators and three plausible denominators, producing up to nine plausible «churn» figures. Copilot picks one on the first turn using its own defaults; those defaults are usually reasonable but rarely match the finance-partner definition. The 0.2–3.0 percentage-point range across the nine candidates is the difference between «churn is on plan» and «we have a retention problem.» A confirmation turn is a fifteen-second discipline that prevents a whole class of «that’s not the churn number we use» conversations downstream.

The downstream discipline. Once Copilot has named the specific measure, date range, and segment, quote all three in the artefact you ship: «Q3 net revenue churn was 12.4% (Jul 1 — Sep 30, paying customers only; excludes trial and freemium).» That format survives reader challenge; «Q3 churn was 12.4%» does not.

The three surviving failure classes (Chapter 11 recap). After Prep, the three classes that still fail are (a) missing entity (the concept the question is about doesn’t exist as a row in any table), (b) missing measure (the entities exist but no column captures the number the question asks for), and (c) missing relationship (the entities and measures exist in separate tables but no join connects them). All three look identical from the outside — Copilot returns either a wrong answer, a hedged answer, or a refusal — but they have different upstream fixes.

The failure class firing here: missing relationship. The tickets entity exists (ticket resolution times are stored). The resolution outcomes exist (whether the ticket was closed successfully). The tools entity exists somewhere in the ecosystem (the AI assistant is a real product with its own event log). What is missing is the join: no field on the ticket row tells you which specific tool or agent action closed that ticket. The AI assistant may have contributed to some resolutions, all resolutions, or none, and the ticket table cannot distinguish.

Why Prep won’t help. Prep is the discipline of cleaning up the data model so Copilot can answer defensible questions. Prep can rename columns, define relationships between existing tables, define measures over existing columns, and constrain the semantic model. What Prep cannot do is invent data that isn’t collected. If the source system never wrote a «closed by tool» field on each ticket resolution event, no amount of downstream Prep in Power BI or Excel or the AI dialog can reconstruct which tickets the AI assistant closed. The data is not merely misformatted; it is not there.

The upstream fix. Two paths, in order of preference:

1. Instrumentation change. Work with the support ops or engineering team that owns the ticket system to add a closed_by_tool field (or equivalent) to the ticket resolution event. Populate it from the resolution workflow — the AI assistant logs a resolution event, the agent-confirmed close logs a different one, and every future ticket carries its own closure attribution. This is the correct fix; it produces defensible attribution data going forward, and it lets every future question of this shape (which tool did what?) be answered directly.
2. Quasi-experimental proxy analysis. If the instrumentation fix is not available in the timeframe the support director needs an answer, run a proxy: compare resolution-time distributions and reopen rates on tickets in the AI-assistant rollout cohort vs a matched pre-rollout or non-rollout cohort. The proxy does not answer «what percentage were resolved because of the AI assistant» — it answers «did the introduction of the AI assistant shift resolution outcomes in the treated cohort compared to a matched untreated cohort.» That is a different question, and the analyst has to be explicit that it is the question being answered.

What to say to the support director. «The exact percentage you asked for is not answerable from the current data because the ticket resolution events don’t record which specific tool closed each ticket. Two paths: (a) I can push for a ‘closed by tool’ field to be added to ticket events, which would answer this question directly for future quarters (about a two-month engineering fix); (b) in the meantime, I can run a quasi-experimental comparison of resolution outcomes for the AI-assistant rollout cohort vs a matched control, which will tell you whether the AI assistant shifted outcomes even though it won’t give a per-ticket attribution. Which path do you want to prioritise?» That response converts the unanswerable question into a director-level choice between two answerable questions, and preserves the analyst’s credibility by naming the data gap instead of guessing at the number.