In September 2026 we ran the second round of our chatbot test, this time on the Illinois Unlimited Roofing Contractor exam. Same nine-prompt conversation, six models: GPT-6 Astra, Claude Fable 5, Gemini 3 Pro, DeepSeek V4, Grok 4, and Kimi K3. We asked each one what the exam looks like, what the license costs, what the rules say, and then asked for a 20-question practice test. Then we checked every fact and every answer key against the books the exam is actually written from: the CTS Roofer Study Guide, the Illinois Roofing Industry Licensing Act and its rules, OSHA 29 CFR 1926, the NRCA Roofing Manual volumes, and the NRCA Safety Manual.
Here's the headline: five of the six got the shape of the exam right, and four of the six taught at least one wrong number a candidate would memorize.
The real exam is 130 questions, 2 hours 30 minutes, closed book, 70 percent to pass (91 of 130). Sections: Safety 25, Roofing Preparation 20, Steep Slope 35, Low Slope 15, Rules and Regulations 20, Non-Residential 15. Bond is $10,000 for a Limited license and $25,000 for Unlimited. Insurance is $250,000 property damage and $500,000 bodily injury per occurrence. Hold those numbers, because this is where the bots went wrong.
The scorecard
| Model | Facts about the exam and license (7 asked) | Mock: right / wrong / flawed | Worst error |
|---|---|---|---|
| Kimi K3 | 6.5 | 20 / 0 / 0 | Bond quoted at $50,000 unlimited, $25,000 limited |
| DeepSeek V4 | 6.5 | 19 / 0 / 1 | Keyed 2:12 as the accepted shingle minimum |
| Gemini 3 Pro | 6 | 15 / 1 / 3 | Invented an OSHA kettle-extinguisher rule |
| GPT-6 Astra | 5 | 19 / 0 / 1 | Mock contained no tested number at all |
| Claude Fable 5 | 5 | 14 / 3 / 3 | Valley metal, standing seam and shingle slopes all keyed wrong |
| Grok 4 | 3 | 10 / 3 / 4 | 120 questions, 2 hours, computer-based, 84 to pass. All four wrong |
"Flawed" means the key was defensible but the stem or explanation taught something the book contradicts.
The wrong answers, with receipts
The bond. Kimi produced the only 20-for-20 mock of the group and then, in the same conversation, told the user the Unlimited bond is $50,000. The Act says $25,000. That is the one number a candidate has to write on a check, and the best mock of the six got it wrong in the paragraph right above it.
The kettle. Gemini's mock keyed an "OSHA 25-foot extinguisher rule" for asphalt kettles. There is no such OSHA distance. The NRCA Safety Manual says two 10B-rated extinguishers within 30 feet of the kettle. The explanation cited OSHA by part number, which is the part that makes it dangerous: it reads exactly like the questions that were right.
The slopes. Three models keyed 2:12 as the minimum slope for asphalt shingles. 2:12 is the code floor with special underlayment. The NRCA Manual, which the exam draws from, recommends 4:12. Fable went further and keyed standing-seam metal at 3:12 (the code minimum is 1/4:12, and NRCA's structural-panel guidance is 1/2:12) and open valley metal at 36 inches wide (NRCA: 24 inches). Those are three of the most-tested dimensional facts in the Steep Slope section, which is the largest section on the exam.
The whole exam. Grok said 120 questions, 2 hours, computer-based, 84 to pass. Four facts, four wrong. Then it produced a section outline that doesn't exist, a $5,000 penalty that isn't in the Act, and a mock with zero technical roofing content. The real exam draws roughly 85 percent of its items from OSHA and NRCA practice.
The invented rules. Three models told the user they must hold a Limited license for a year before applying for Unlimited. That rule is not in the Act, the administrative rules, or the study guide. Two models produced a full calendar of 2026 test dates; only one sitting was verifiable. Two asserted a separate Chicago roofing license. One cited a form by name that IDFPR doesn't publish.
We'll say the same thing we said about the plumbing round: none of these hedged. Every wrong number arrived with a confident explanation.
The harder problem: the mocks were easy
We also scored each 20-question mock for guessability: how many items can be answered with no roofing knowledge at all, using only the answer choices. It's the same check we run on our own question pools every night.
The real exam is closed book. The question count and the pass line are set so that a roofer who has not studied the Rules section or the OSHA fall-protection numbers will not pass on trade experience alone. So a practice test only helps if it's at least as hard as the real thing.
| Model | Answerable with no roofing knowledge |
|---|---|
| GPT-6 Astra | 17 to 18 of 20 |
| Grok 4 | about 15 of 20 |
| Kimi K3 | 10 to 12 of 20 |
| Claude Fable 5 | about 9 of 20 |
| Gemini 3 Pro | 8 to 9 of 20 |
| DeepSeek V4 | about 7 of 20 |
GPT's mock was the second-most accurate of the six and also nearly useless: an outsider gets 85 percent of it right by reading the answer choices. The longest option wins, the only choice that mentions safety wins, the three distractors are obviously silly. You can score 19 of 20 on that test and walk into the real one without knowing the guardrail load or the asbestos exposure limit. For reference, our own target for a shipped question pool is under 30 percent guessable, and our Illinois roofing pools test at about that.
Credit where it's due
Kimi's mock was fully verified, and it used the right OSHA numbers: the 0.1 fibers-per-cubic-centimeter asbestos limit, the 200-pound guardrail load. DeepSeek's answers on the procedural questions were the most useful in the set: the application deadline, the two test sites, the fee split, and 91 of 130 stated plainly. If you handed a candidate DeepSeek's exam-day answers and Kimi's mock, you'd have most of a study session. You would also have a $50,000 bond figure and a 2:12 shingle rule.
That's the pattern from the plumbing round again. About half of what comes out is good. You can't tell which half without the books open, and if you have the books open you didn't need the bot.
The one we ship
Fable 5 is from the same model family we build our own questions with, and it finished fifth of six on this test: three wrong keys on dimensional facts and a mock delivered as a sandbox link that expired. We said in the plumbing post that we'd grade the family we use ourselves. Here is the grade.
It's also the clearest illustration of why our pipeline exists. A model answering from memory keyed valley metal at 36 inches. Our generator doesn't answer from memory. It writes each question from the extracted text of the NRCA volume the exam cites, a separate pass has to locate the answer in that text before the question can ship, and a third pass that has never seen the book tries to guess its way through. The same model family, gated three times, does not ship a 36-inch valley. Ungated, it wrote one.
What to do with this
Models change monthly, so treat this as a snapshot: six chatbots, one conversation, September 2026, graded against the current Illinois books.
If you're studying for the Illinois roofing exam, or any closed-book trade exam, use a chatbot for what it's good at: explaining a concept, building a study schedule, telling you what a section covers. When it gives you a number (a bond amount, a slope, a distance, a load), treat it as a claim to verify, not a fact to memorize. The test is closed book. The bond check is real money. Neither one cares how confident the explanation sounded.
If you'd rather study from questions that already went through the books, our Illinois Unlimited Roofing practice exam is built the way described above, and there are matching pools for the Residential and Commercial and Industrial licenses. The first questions are free, so you can check our work the same way we checked theirs.