Chapter 8 — Practice-on-your-own solutions

The Crystallize question set.

1. Audience. Who reads the deck, and what is the highest-status reader in the room? (Pricing team internal review? Product leadership? CFO? Board?)
2. Decision. What decision does the audience need to make after reading the deck? (Extend the price change? Roll it back? Modify it for specific segments? Just report status?)
3. Success criteria. What outcome would you point to and say «the price change worked»? Revenue-neutral? Margin up X%? Churn under Y%? Some blend?
4. Counterfactual. What baseline are we comparing against? «What would have happened without the change» needs a forecast, a control group, or a historical trend extrapolation.
5. Time window. Six months in — but what is the pre-period? January minus six months? Same six months last year? Rolling trailing twelve?
6. Segment lens. Aggregate, or by customer size / product family / geography? Where did we expect variance and where does the PM expect the story to differ?
7. Constraints. What is off-limits? (Named customers, specific competitor comparisons, forward-looking guidance the CFO has not signed off on.)

The two the PM will deflect. Almost certainly (3) success criteria and (4) counterfactual. These are the questions that should have been answered at change-approval time and usually were not — the price change was approved on a hypothesis, and the hypothesis was not translated into a measurable target with a baseline. When the PM was named as the deck owner six months later, they inherited an implicit ask («show it worked») without an operational definition of what «worked» means.

The director’s response. An author would say «OK, I’ll go build the deck against the fields I can answer,» and produce a beautiful set of charts whose meaning collapses under CFO questioning. A director does the opposite: propose specific answers to the two deflected questions, name them as inferred rather than confirmed, and force the PM to either agree or make the decision now.

«For success criteria, I’m proposing three: (a) blended revenue neutral (+/- 2%) vs the pre-change six months; (b) blended margin up at least 3 percentage points; (c) net logo churn under 5% for the segments hit hardest by the change. If you want a different set, let me know now. For the counterfactual, I’m proposing a rolling twelve-month trailing extrapolation as the ‘no-change’ baseline — not perfect, but the best we have without an actual control group. If you want a different baseline (a specific competitor benchmark, a synthetic control on a matched cohort), let me know now. Once we’ve locked these, I’ll go build the deck.»

Why this move is directorial. It converts a spec-gap into a spec by proposing specifics rather than absorbing the ambiguity into the artifact. If the PM agrees, the deck now has a defensible frame. If the PM disagrees, they have to specify the alternative — and either way the deck ships with a criterion the CFO can question but cannot invalidate as post-hoc.

How to read this worked example. This is a reflective problem — the «solution» is a demonstration of the diagnostic move on one plausible case. Your own failure will be different; what matters is that the diagnosis lands on a specific CSAR phase and a specific move that would have caught the failure at that phase.

Worked example — an AI-directed exec dashboard that landed flat in the readout.

The setup. An analyst was asked to build an AI-directed executive dashboard on Q3 services revenue. The BRD named the audience (VP + direct reports), the decision (which service lines to invest in for Q4), and the data source (services revenue by SKU by region). The analyst ran the full CSAR loop against Copilot in Power BI and shipped the dashboard on Wednesday for the Thursday exec readout. The VP’s response after the readout: «This is fine, but I still don’t know what I’m supposed to do with it.»

Phase diagnosis: Crystallize was too shallow. The BRD named the decision («which service lines to invest in for Q4») but never operationalised the decision. What does an investment decision mean in this context? Additional headcount? Marketing dollars? A pricing move? Each of those investments would need a different visualisation: additional headcount needs a capacity-vs-demand chart per line, marketing dollars needs a CAC-vs-LTV chart, pricing needs an elasticity chart. Because the Crystallize phase locked in an ambiguous decision word («invest»), the Scope phase produced a competent-looking dashboard of «services revenue by SKU by region trends» that answered the wrong question — and the VP correctly read it as informative-but-unactionable.

The specific move that would have caught it. At the end of Crystallize, before running Scope, apply the result-then-action test: «When the audience looks at this artifact, what specific action will they take within seven days that they would not have taken without it?» If the answer is vague («they’ll be more informed,» «they’ll ask better questions»), Crystallize is not done. Push back on the requester: «When you say ‘invest,’ are we talking headcount, marketing, or pricing? Each needs a different chart.» Answer the sharpest question and rerun Crystallize before touching the data.

Why not Refine. Refine could not have rescued this dashboard because Refine polishes charts against a spec; a spec that doesn’t operationalise the decision has no ground truth for Refine to iterate against. Perfunctory Refine was the visible symptom (the analyst reported doing two Refine passes and being «satisfied»), but the causal failure was upstream.