In August 2026 we ran a simple test: give six AI chatbots — GPT-5, Claude Opus, Claude Sonnet, Grok 4, Perplexity, and DeepSeek — the same prompt asking for plumbing exam practice questions on isometric drawings and drainage systems, then check every single answer against the 2021 International Plumbing Code. Not against our opinion. Against the book.
The six bots produced 72 multiple-choice questions between them. Only 34 survived verification.
The rest failed for reasons that should scare anyone studying for a licensing exam with a chatbot: confidently wrong answer keys, code values from no code that exists, "rules" that are actually old-timer myths, and diagram conventions that would get you marked wrong on a real isometric question.
The wrong answers, with receipts
These aren't nitpicks. Each one is a fact a plumbing exam actually tests, answered wrong with total confidence.
Exhibit 1 — the slope question. One bot keyed the minimum slope for a 3-inch horizontal drain at ¼ inch per foot. IPC Table 704.1 says pipes 2½ inches and smaller get ¼ inch per foot — 3-inch pipe drops to ⅛ inch per foot. This is one of the most commonly tested values in plumbing, and the bot picked the answer that feels right because ¼" is the number everyone half-remembers.
Exhibit 2 — the invented code. Trap-arm distance limits came back matching neither the IPC (Table 909.1: 6 feet for 1½-inch, 8 feet for 2-inch) nor the UPC (42 inches and 60 inches). The bot didn't misquote a code — it blended two codes into a third one that doesn't exist. If you memorize those numbers, there is no exam anywhere you'll be right on.
Exhibit 3 — the jobsite myth. "Two water closets on a 3-inch stack is a code violation" — presented as fact, keyed as the correct answer. IPC Table 710.1(2) permits two water closets on a 3-inch stack. This is a myth that's been argued in supply-house parking lots for decades, and the chatbot picked it up and taught it as code.
Exhibit 4 — the fixture-unit miss. A private water closet keyed at 4 drainage fixture units. IPC Table 709.1: 3. One digit off, worth real points on an exam where DFU totals drive pipe-sizing answers.
Exhibit 5 — reading the drawing wrong. Dashed lines on a DWV isometric keyed as "hidden edges," like a machining blueprint. On plumbing isometrics, dashed lines conventionally indicate vent piping — which is exactly what the exam's drawing questions test. Another set had pressure-reducing valve and gate valve symbols reversed.
And here's the part that makes it dangerous: none of the wrong answers hedged. Every one came with a clean explanation, a confident tone, and formatting that looked exactly like the 34 questions that were right.
The half that was good is the trap
We want to be fair: roughly half the material survived a strict check against the code book. Modern AI models know a lot of plumbing. If the bots were wrong 95% of the time, nobody would use them and there'd be no problem.
The problem is that 47% failed and you cannot tell which 47% it is — not without the code book open, at which point you didn't need the chatbot. Studying from unverified AI questions means roughly every other fact you memorize is either right, subtly off, or from a code that doesn't exist, all delivered in the same confident voice. On a 70-question exam with a 70% cut score, a study source that quietly poisons a third of your facts is the difference between a license and a $100 retake fee plus weeks of lost work.
A note on fairness: models update constantly, so treat this as a snapshot — six mainstream chatbots, same prompt, August 2026, graded against IPC 2021. We're re-running the test with per-model scoring for a full report card, and we'll publish the grades, including on the AI family we use ourselves.
Yes, we use AI too — here's the difference
LicenseExams.com questions are AI-generated. So we're not telling you AI can't write exam questions — we're telling you what it takes to make AI-written questions trustworthy, because we had to build it:
- Grounding. Our generator doesn't answer from memory. It writes questions from the actual extracted text of the code books and references your state lists for the exam — the same editions the exam cites.
- Verification. Every question is then fact-checked against that source text by a separate pass. A question whose answer can't be located in the book doesn't ship. That's the exact check that killed 38 of the 72 chatbot questions above.
- The blind-solve gate. Finally, a solver that has never seen the book attacks each question. If it can get the right answer through test-taking tricks instead of knowledge, the question is rejected and rewritten.
Ask a chatbot for practice questions and you get step zero of three.
If you're prepping for a plumbing, electrical, contractor, or real estate exam, use AI the way it's actually good: explanations, study plans, "explain this concept like I'm new." But when it states a code value, a fixture-unit count, or a slope — verify it against the book, or use practice material that already did. Your license rides on the numbers, and the chatbot doesn't sit for the retake.