← All exercises

Chapter 3 · Demonstration · 8 minutes

Trust, but verify

From Actually Using AI

What this shows

Chapter 3 sets out a four-layer defence against AI’s most consequential failure mode for professionals: confident-sounding facts that aren’t true. The chapter explains the method. This page shows it working — step by step, on a real AI response, catching four different fabrications that any of the layers alone would have missed.

You’re going to watch James, a marketing manager preparing a board paper on remote-work productivity, run AI’s response through the four layers. The response will look credible. It will set out five specific findings, each with a source. Several will be wrong in different ways, and James won’t be able to tell which without doing the work. By the end of the demonstration he’ll have caught what mattered.

The demonstration

James has been asked to prepare a paper for his company’s board on the question “what does the research actually say about remote-work productivity?”. The board is split — half believe the productivity gains are real, half think the early-pandemic numbers were an artefact and have since regressed. They want him to summarise the evidence and make a recommendation.

He’s got a week. He opens AI to get a head start.

The AI’s first response

James asks AI for a summary.

Here’s what comes back:

AI: Here are the most important research findings on remote-work productivity, with sources:

1. Fully remote workers show 13% higher productivity than office workers on equivalent tasks, according to a Stanford study by Nicholas Bloom (2023) published in the American Economic Review. The study followed 1,612 workers at a large travel agency over a nine-month period.

2. Hybrid arrangements (2-3 days remote) produce the highest reported employee satisfaction, with 78% of hybrid workers reporting higher overall job satisfaction than either fully-remote or fully-office workers (McKinsey American Opportunity Survey, 2024).

3. Time-zone-flexible remote work increases output for knowledge-worker teams by an average of 8%, particularly in software development and analytical roles, based on the Microsoft Work Trend Index 2024.

4. Manager-reported productivity gains have not regressed since 2021, with 67% of managers in a Harvard Business Review survey (Updated October 2022) reporting that their teams maintained or exceeded pre-pandemic output levels.

5. Collaboration intensity drops by approximately 15% in fully-remote teams, particularly affecting cross-functional and innovation work — Microsoft Research 2022 study analysing 61,000 employees.

The pattern that emerges from the research: productivity gains for individual focused work are real and sustained, but collaboration costs are also real. This is why most large employers have settled on hybrid arrangements as the operationally sound compromise.

James reads it. It sounds confident, well-structured, and exactly what the board asked for. The pattern in the closing paragraph is the kind of synthesis that would land well in a board paper.

He’s about to start drafting. Then he remembers the four-layer defence from the book.

Layer 1 — force the AI to classify

James pastes the response back to AI and asks for a classification.

The response classifies all five citations as facts — Bloom, McKinsey, Microsoft Work Trend Index, the HBR survey, and the Microsoft Research collaboration study. The judgements and assumptions section is straightforward and adds nothing to the verification problem.

James reads it. The classification looks reasonable. All five citations went into the facts column, which is what he’d have expected — they are, on their face, citations.

Notice what’s happened. AI has confidently classified its own output. The classification doesn’t tell him whether the facts are true — it tells him what AI claims is a fact. That’s a useful distinction, but it’s not verification.

Layer 2 — sources for facts only

James asks AI for verifiable sources for each of the five facts.

Here’s what comes back:

AI: Here are the sources:

1. Stanford 13% productivity study: Bloom, N. (2023). Working from Home Around the Globe. American Economic Review, 113(4), 932-959. The 2023 update to Bloom’s earlier work, expanded sample including UK and US travel agency workers. DOI: 10.1257/aer.20221298

2. McKinsey hybrid satisfaction: McKinsey & Company. (2024). American Opportunity Survey 2024. Available on McKinsey Insights. The 78% figure appears in the section on hybrid work arrangements.

3. Microsoft Work Trend Index 8%: Microsoft. (2024). Work Trend Index 2024: AI at Work. The 8% figure for time-zone-flexible knowledge work appears in the section on team productivity.

4. HBR 67% manager survey: Larson, B. Z., Vroman, S. R., & Makarius, E. E. (2022). A Guide to Managing Your (Newly) Remote Workers. Harvard Business Review. Updated October 2022.

5. Microsoft Research 15% collaboration drop: Yang, L. et al. (2022). The effects of remote work on collaboration among information workers. Nature Human Behaviour, 6(1), 43-54. The study analysed Microsoft email/meeting data from 61,182 employees during 2020.

James now has citations. They look credible — named authors, specific journals, plausible publication dates, DOIs in two cases. If he stopped here, he’d put them in the board paper.

This is the trap. AI in source-providing mode is no better at telling truth from invention than it was in fact-providing mode. It will confidently produce a citation regardless of whether the citation exists.

Layer 3 — spot-check

James doesn’t have time to verify all five. He picks the one his board paper would lean hardest on — the Stanford 13% finding — and clicks through.

The DOI returns a 404 from the American Economic Review website.

He searches Google Scholar for Bloom, Working from Home Around the Globe, 2023. He finds Bloom’s earlier remote-work papers — the 2015 China travel agency study (where the 13% figure originated, for a single firm in a different period) and a 2024 NBER working paper. There is no American Economic Review 2023 paper by that title. The DOI is invented.

The Stanford study, as cited, doesn’t exist. The 13% figure traces back to a real and well-known 2015 paper that the AI has dressed up as a more recent and broader study than it is.

James now knows. One of the five facts is fabricated. He doesn’t know which of the other four are also fabricated. The right discipline is the chapter’s: if you haven’t verified it yourself, it isn’t verified. He has to treat the whole response as unverified until he’s checked the rest, or run the response through another defence layer.

Layer 4 — cross-AI audit

For high-stakes work — and a board paper qualifies — James takes the original AI response, his Layer 1 classifications, and the Layer 2 sources, and pastes them into a different AI tool.

Here’s what comes back:

AI: Looking at the four remaining citations against what I know about these publications, here’s what I can tell you — with the caveat that I can’t access the cited documents in real time and can only flag what looks inconsistent with my training data.

The HBR Larson/Vroman/Makarius piece. The article you’ve been given a citation for is real — A Guide to Managing Your (Newly) Remote Workers is by those three authors. But the date is wrong. To my knowledge, that piece was published in March 2020 at the start of the pandemic, not 2022. The “Updated October 2022” framing looks like AI invention or confabulation — HBR does republish pieces with update notes, but I have no record of that specific one being updated in October 2022. If you cite this piece, check the actual publication date on hbr.org.

The Microsoft Research Yang et al. paper on collaboration. The author, title, and 61,182 employee figure all match what I have on a real Nature Human Behaviour paper from 2022. The finding as stated is broadly consistent with the published research. This citation looks solid, though I’d still verify the exact wording of the 15% claim against the paper itself.

The McKinsey American Opportunity Survey citation. This is the one I’m least able to confirm in either direction. McKinsey does run an American Opportunity Survey series, and a 78% hybrid-satisfaction figure is in the right range for what McKinsey has published on this topic. What I can’t confirm is whether that specific figure appears in the 2024 edition of the AOS, or whether the first AI has taken a real-ish statistic and attached it to the wrong specific document. The two failure modes look almost identical from where I sit. I’d treat this citation as unverified until you’ve actually looked at the cited document.

The Microsoft Work Trend Index 8% figure. Similar uncertainty. The Work Trend Index is real and Microsoft publishes annual editions. The 2024 edition focused heavily on AI adoption at work rather than the team-productivity framing the first AI gave you. The 8% time-zone-flexible knowledge work figure doesn’t match anything I have on the 2024 WTI specifically. It could be from an older Microsoft publication, from a different report entirely, or invented. As cited, I can’t confirm it.

What I’m honest about: I can flag inconsistencies, but I can’t prove a specific number is absent from a 100-page report I haven’t been given. Two of these four — Larson and Yang — I can usefully comment on. The McKinsey and Microsoft Work Trend Index citations need actual verification against the cited documents.

James now has the picture. One citation wholly fabricated (Stanford). One with a fabricated update date that needs the original 2020 source instead (Larson). Two citations the second AI couldn’t fully verify and that he needs to check at source himself (McKinsey, Microsoft Work Trend Index). One citation that looks solid but he should still spot-check (Yang).

He has the makings of a usable board paper if he keeps the Yang citation, finds Larson’s actual 2020 piece, drops or replaces the Stanford citation with Bloom’s actual 2015 paper, and either verifies the McKinsey and Microsoft figures at source or removes them.

What he doesn’t have is the embarrassing version of the paper where three of his five citations turn out to be wrong when a board member checks one.

The point

The four layers work as a funnel, not as a watertight system. What classification exposes, source-checking isolates. What source-checking isolates, cross-auditing challenges. What cross-auditing flags as uncertain, the user verifies at source.

Layer 1 — classification — didn’t catch anything on its own. AI confidently put all five fabricated or partly-real citations into the facts column. What Layer 1 did was create the structure for Layer 2.

Layer 2 — sources — didn’t catch anything on its own either. AI confidently produced citations for everything James asked about, including the ones it had invented. What Layer 2 did was create the target for Layer 3.

Layer 3 — spot-check — caught the wholly fabricated Stanford citation. It would not, on its own, have caught the date-shifted HBR citation or the questionable McKinsey and Microsoft attributions, because James only had time to check one.

Layer 4 — cross-AI audit — caught the HBR date shift and flagged the McKinsey and Microsoft citations as needing verification. It didn’t tell James those latter two were definitely wrong; it told him it couldn’t confirm them and that he had to check at source himself.

That last point matters. Cross-AI audits are filters, not guarantees. A second AI will often catch lazy hallucinations the first one produced, but it can also confidently agree with a fake citation, or be uncertain in cases where the first AI was confidently wrong. The four layers reduce the risk of confident error. They don’t eliminate it. The discipline that does is the chapter’s, all the way through: if you haven’t verified it yourself, it isn’t verified.

Try it yourself

The next time you ask AI for something where facts matter — a paper with citations, a memo with statistics, a presentation with named research — try the four layers.

Ask AI to classify its response into facts, judgements, and assumptions. Then, for the facts column only, ask for sources you could verify independently. Pick the most consequential claim and verify it yourself — the source either exists or it doesn’t, and if it does, it either says what AI claims or it doesn’t. If anything fails, treat the whole response as unverified.

For high-stakes work, paste the original response into a different AI tool and ask it to find what’s wrong. A second AI will typically catch about a third of the errors the first one missed — but it can also confirm errors that look plausible to it, and it can be uncertain on things you’d want certainty about. Treat the second AI’s output the way you’d treat the first AI’s: as input to your own verification, not as a verdict.

The whole process takes ten minutes for a response of this size. The thing it catches might be the difference between a paper your board trusts and a paper your board never asks you to write again.