feat: consultant-grade Discovery Report — depth, exec summary, UI, conformance + verifier; live-validated on O2C & P2P; 100% coverage - #2
Merged
Conversation
…omy, opportunity operating model, prioritization rationale, metrics framework Deepens the report suite to the prior-engagement bar (per Akhilesh: output was 'very brief, not breaking things down properly'). All depth is genuinely derived and grounding-checked — no fabricated numbers. - models: SystemProfile, FormatPattern, MetricItem; Opportunity gains personas, expected_behaviour, escalation, and three readiness ratings (data/technical/ operational, each 'high|medium|low — reason'); CurrentState gains system_profiles + format_taxonomy; SynthesisContent gains metrics_framework. - render: numbered sections (1.1–1.8); per-system narrative cards; format-taxonomy table; opportunity operating-model block (who uses it / expected behaviour / escalation); colour-coded prioritization-rationale table; metrics framework. - synthesis: emit_synthesis schema + prompt extended for all depth fields; the no-new-numbers rule now explicitly covers metric targets and readiness reasons (directional goals against the verified baseline, never invented %/timeframes). - build: live-parse path wires the new fields; fixture_o2c deepened, every figure still traces to the verified findings (passes validate_synthesis). - tests: +2 (depth present in dataclasses, depth surfaces in HTML). 41 pass; pyrefly clean.
…ion) Closes the live-path test gap before any paid live run. Adds unit + integration tests driving the real contracts offline (scripted fake LLMs through the actual ToolTurn / messages_with_tools interface; real CSV tools for genuine numbers). Coverage: 100% statement + 100% branch across the active pipeline (synthesis, agent_loop, build, tools, docnames, render, verify, refresh, models, assemble, registry). 141 tests. Enforced in CI via fail_under=100 in .coveragerc. Scope (in .coveragerc): omits the real HTTP LLM client and the legacy scripted/linear modules — those are exercised by the live run, not offline. New tests: - test_synthesis: run_synthesis live path (happy / no-emit reprompt / grounding reject-then-fix / give-up / suppress-names), emit-tool schema, allowed_numbers, every validate_synthesis rejection branch, cycle detector. Proves the deepened depth payload survives the grounding gate. - test_agent_loop_orchestration: run_discovery orchestration (activity callback, no-tool reprompt, reject-retry, MAX_TURNS budget) + remaining ground branches. - test_build: live-parse _from_payload (incl. depth fields), NoFixtureForDomain, source-index dedup. - test_tools_errors: dispatch wrapper, every ToolError branch, full _eval_predicate operator matrix, conformance + join_diff edge branches. - test_docnames / test_misc_coverage / test_render_branches: overrides, phrase collapse, assemble folding + verification downgrade, registry indexing + determinism guard, and every renderer optional-field / SVG / source-page arc. Also: docnames._collapse_pairs — replace an always-true dead elif with else. coverage added as a dev dependency; .coverage artifacts gitignored.
…crash)
The emit_synthesis schema and prompt offered 'modernisation' as a valid
opportunity pattern, but OppPattern lacked it — so when the live model chose it
(for 'bring the EDI channel into documented ownership'), the dataclass raised
ValueError and the run fell back to the fixture instead of rendering live content.
- models: add OppPattern.MODERNISATION.
- render: add its _PATTERN_LABEL ('Modernisation').
- tests: regression guard that the schema's pattern enum is a subset of the model
enum (so a schema-valid value can never be model-invalid again), plus _opp()
parses 'modernisation'. 143 tests, 100% coverage, pyrefly clean.
Found by the first live O2C run; verified fixed via free golden replay.
…t trace The first paid live O2C run (agent discovery + live synthesis), saved as a golden cache so it replays offline for free. Live synthesis produced full consultant-grade depth (4 system profiles, 3-type format taxonomy, 5 metrics, per-opportunity personas/behaviour/escalation + 3 readiness ratings) and used the modernisation pattern. All 14 figures independently recomputed from the raw CSVs and confirmed exact: 267 credit + 228 payment-term mismatches, EUR 30,675,000 divergence, EUR 600,000 largest single delta (FR001); EDI 67.3% by count / 66.8% by value (EUR 59,711,399.53); 1,196 unfulfilled EDI (EUR 12,362,493.74), 320 manual / 111 email by comparison; 34 escalations = 23.9% of 142.
…e + visual upgrade Stakeholder review against Akhilesh's prior-engagement bar found the Discovery Report had the depth but read flat and document-only. Akhilesh: build only the Discovery Report (the other docs were formatting references), don't surface gaps in the client output, make it more visual and presentable. This delivers that. New report-quality content (all grounded, optional, defensive parse): - Executive Summary (Report 00) — now the suite landing page / index.html: a headline, code-derived KPI tiles (largest grounded EUR + first %, never model-set), situation/opportunity panels, a value-vs-feasibility chart, top-opportunity cards, and a 'what we read' provenance line. - Use-case summary table on Report 04 (pattern / who / sources / behaviour) before the deep write-ups, mirroring the prior-engagement use-case anatomy. - Target-state 'where this should converge' section on Report 03. - Per-opportunity knowledge_sources + document_formats. Visual upgrade (principal UI/UX) — pure inline SVG, offline-safe, calm palette: - KPI tiles, horizontal impact-ranked bars (Report 02), value-vs-feasibility bubble plot with collision spreading (Reports 00/03), a donut helper, opportunity cards. Less prose, more at-a-glance structure. models/build/synthesis: ExecutiveSummary + target_state + opp source/format fields; schema+prompt extended (optional); fixture deepened. Grounding holds (exec summary, target-state, metric targets stay directional; Report 01 stays factual). 153 tests, 100% coverage, pyrefly clean.
…e/format The first live run of the enriched suite hit a real grounding rejection: the model slipped the word 'exposure' into the factual current-state (report 01), and the generic retry message didn't tell it how to fix it, so it fell back to the fixture. - run_synthesis: 3 attempts (was 2) and a _fix_hint() that gives a targeted remedy per failure kind — for the factual-language case it says exactly which words are banned in report 01 and to move them to pain points / restate neutrally. The next live run recovered and emitted clean. - schema: knowledge_sources + document_formats are now REQUIRED on each opportunity (they were optional and the model skipped them, leaving the use-case table sparse; now reliably populated, like personas/behaviour). - tests: _fix_hint per-error-kind, 3rd-attempt recovery + give-up. 155 tests, 100% coverage, pyrefly clean. Validated live end to end: exec summary, target-state, 4 system profiles, 4 format types, 5 metrics, and every opportunity's sources/formats/personas/readiness all populated; all 12 distinct figures re-recomputed from the raw CSVs and exact.
…+ audit) Live agent discovery + live synthesis of the full enriched Discovery Report: executive summary, current state (4 system profiles + format taxonomy), ranked pain points, opportunity portfolio (use-case summary table + deep write-ups), prioritization rationale, roadmap, metrics framework, target-state. Saved as a golden cache for free offline replay. All 12 figures recomputed from raw CSVs, exact.
Cross-domain proof: same engine, different domain (Procure-to-Pay), ZERO code changes, fully live (no fixture/scripted). Conformance-check diffs PO data vs the written procurement policy. Full enriched suite for client 'Acme Manufacturing'. All 8 figures recomputed from raw exact: 7 single-approval large orders (EUR 557,000 = 12.0% of EUR 4,625,800); 3 maverick (EUR 216,000 = 4.7%); largest EUR 150,000.
anmolg1997
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Takes the discovery suite from "numbers are right but the report is thin" to a consultant-grade, visual-first Discovery Report that meets the prior-engagement bar — validated live on two domains with every figure independently recomputed from the raw data.
1. Result-quality improvements (original scope)
--refreshdiffs a re-run against the prior one (new / resolved / changed)2. Consultant-grade depth (per Akhilesh's feedback: "too brief, not breaking things down")
3. Executive summary + UI/UX upgrade (presentable, less paragraphy)
4. Robustness
OppPatternschema/model mismatch that crashed a live run (modernisation)Anti-fabrication discipline
Code does the math; the LLM does the reasoning. Every client-facing number is a
NumberRefthat must trace to a verified finding; metric targets are directional; Report 01 stays factual. Verified by a grounding gate at synthesis time and by independent recomputation from the raw inputs.Validation (live, both domains)
Tests & quality gate
fail_under = 100in.coveragerc)pyreflyclean on product codeOut of scope (per Akhilesh)
5. Document-grade PDF presentation (visual review vs the prior report's PDF)
Reviewed our rendered PDF page-by-page against Akhilesh's prior report (as a PDF). Content/narrative matched or beat his bar; document presentation was the gap. Closed:
@pageA4,break-inside:avoid) — clean pagination, no mid-element splits; KPI tiles 5-across in print.scripts/make_pdf.py— renders the suite to one print-quality PDF with native running header (title+brand) + footer (page numbers + Confidential).out/o2c-report.pdf,out/p2p-report.pdf.160 tests, 100% coverage, pyrefly clean, grounding intact. Charts are code-derived from grounded findings only (never model-set); P2P charts correctly empty (no multi-channel breakdown in its data).
6. Spec-driven visual system + rename to v1/ (specs/001-report-visual-system)
Renamed
prototype/->v1/(history preserved). Ran a spec-kit flow (constitution -> spec -> plan -> tasks) for a distinctive, chart-rich, print-grade visual system, then implemented in phases:Validated visually page-by-page on both domains, HTML + PDF; chart values re-validated vs raw inputs; full-bleed cover + TOC + page chrome intact; pages fill. 163 tests, 100% coverage, pyrefly clean, grounding + factual guards pass.