1. Project Context and Goals
Architects, engineers, contractors, and the rest of the AEC industry have been inquiring about how AI is actively impacting our industry and how useful mainstream GenAI is to our practices. Similarly, our developers and clients are doing the same. Whether it’s a query to ChatGPT about Boston’s opt-in stretch code, resources for federal funding information, or a technical PHIUS question, many practitioners I’ve spoken to recently all have the question of how well GenAI actually performs.
Thanks for reading! Subscribe for free to receive new posts and support my work.
This project is an audit of four major AI models:
ChatGPT 5.3
Claude Sonnet 4.6
Microsoft Copilot
Gemini 3 Flash
Each was queried for the same 20 questions across four primary domains:
PHIUS certification and technical knowledge (Q1-5)
Boston and Denver energy stretch codes (Q6-10)
IRA tax credits for green building (Q11-15)
General building science fundamentals (Q16-20)
This was designed with intentional complexity, with some questions being fairly easy to search and answer, and others being more technical, some of which should require professional advice. By asking these questions, I was able to test whether models were able to recognize and correct false premises and find accurate and current information about modern practices and standards.
New accounts via a new email account were created to ensure models weren’t pulling from previous knowledge about me and my practice. Additionally, each model was tested with a fresh chat for each question to attempt to reduce bias and existing knowledge based on previous chats. Once all the responses were generated and input into a database, they were scored based on:
Factual Accuracy (1-3): Was the information provided correct and accurate?
Confidence Calibration (1-3): Did the model appropriately respond or overstate its certainty?
Hallucinations Present (Y/N): Did the model state something completely false?
Hallucination Severity (1-3): If yes, how inaccurate or significant was the error?
Overall Score (1-10): A cumulative score of the first four criteria.
Note: Hallucinations present counted as 1 point, and hallucination severity had to be reversed to accurately score, since a 3 would be bad and 1 would be good, opposite to the factual accuracy and confidence calibration criteria.
2. Methods
Questions were drawn from real-world queries that practitioners and/or clients could bring to an AI tool. These ranged from straightforward questions like “Is all of Massachusetts subject to the opt-in stretch code?” to more technical questions like “[h]ow should a PHIUS CPHC calculate thermal bridging in a wall assembly?” In addition, five of the twenty questions included false statements in the prompt to see whether the LLMs could think critically and correct them. The questions are as follows:
PHIUS
What’s the difference between PHIUS Core and PHIUS+ Certification?
What fenestration documentation is required for PHIUS Revive certification?
Is WUFI Passive required for PHIUS Certification?
How should a PHIUS CPHC calculate thermal bridging in a wall assembly
What are the valid modeling software(s) for PHIUS certification?
Municipality Energy Codes
Does Boston’s Opt-In Stretch Code require PHIUS certification or just PHIUS level performance?
Is all of Massachusetts subject to the opt-in energy stretch code?
If I’m doing a gut renovation in Boston, when does the project trigger full stretch code compliance?
If I’m doing a renovation of my existing commercial building in Denver, when does the project trigger full stretch code compliance?
What are the two compliance pathways for Denver’s Stretch Energy Code?
IRA Tax Credits
Does the IRA actually give me a 30% tax credit for my green building? If so, how do I apply?
Can I stack section 45L credit with state incentives on the same project in New York?
If I have LEED Silver certification, do I automatically qualify for section 45L credits?
How does section 25C credit apply to multifamily retrofits?
My project received a GRRP grant through HUD. Does that impact my IRA tax credit eligibility?
General Building Science
My contractor said we don’t need a vapor barrier if we use closed-cell spray foam. Is this true?
We’re using a lot of insulation in our above-grade walls. Does this mean we can ignore air sealing the wall to have a high-performing home?
My contractor said the only difference between vinyl and fiberglass is price. Is that true?
I can only afford 3 new windows, which ones should I change, or should I put the money into attic insulation instead?
What are the hierarchical energy strategies for buildings?
Each question was also followed up by “[p]lease provide your data sources and what target standard/code you’re basing this off of.”
Evaluating scores was based on an answer key I developed, reflecting how I’d answer each question through my own research and browsing, to ensure consistency in scoring. As a result, there is room for internal bias. However, all “correct” answers were kept brief and to a minimum to reduce bias and to look at each question from a factual perspective rather than opinionated.
Elaborating on the scoring of each LLM response, three of the ratings use a higher-is-better scale, while one uses lower-is-better and was adjusted accordingly:
Factual Accuracy
1 = Incorrect
2 = Mostly correct with some gaps or minor errors
3 = Correct
Confidence Calibration
1 = Overconfident answers given
2 = Somewhat balanced
3 = Well balanced
Hallucination(s) Present
0 = No
1 = Yes
Hallucination Severity (scores were flipped when adding the overall score)
1 = Minor hallucination
2 = Moderate hallucination
3 = Major hallucination
The overall score was a summation of these factors (min of 3, max 10).
As for the data collection and tooling used, responses were structured in a Google Sheet with one row per model-question pair, so there were 80 total rows. A filter sheet was then used to group these responses by model, and a summary sheet was used to compute and simplify the filtering sheet for accessibility and understandability. This document was exported to Microsoft Excel, where all the data visualizations were created.
3. Outcomes
Across the twenty questions, Claude Sonnet 4.6 ranked first with an average score of 9.45/10, followed by Gemini (8.45/10), then Copilot (8.4/10), and finally ChatGPT (8.3/10) (see Fig 1). The gap between Claude and the other models was also prevalent in their hallucination rates and severities. Claude only visibly hallucinated one of the questions (5%) compared to ChatGPT’s 30%, Copilot’s 25%, and Gemini’s 20% (see Fig 2). The hallucination rates in specific was interesting, since most of them occurred around rapidly-changing standards such as the rollout of PHIUS’ METr, the 2026 HUD reauthorization of GRRP from grants to surplus cash loads, and PHIUS’ Revive standards and their 2024 documentation. These are all changes or additions that occurred after most of these models’ training cutoffs and are the exact type of information practitioners and clients would want to quickly search for.
Fig 1:
Fig 2:
When breaking down the average scores by each subject, it was interesting how dramatically performance varied. All four LLMs performed above average regarding IRA and general building science-based questions. They performed slightly worse on energy code questions, which is also likely due to updates that have occurred after training cutoffs or due to difficulties navigating stretch codes. Surprisingly, except for Claude, PHIUS-based questions were extremely difficult for models to answer accurately and correctly. Copilot’s average score on PHIUS questions was the lowest of all sections, scoring 6.2/10, meaning on average, they only obtained 3.2 additional points since the minimum score is 3. This was partially due to Copilot confidently stating outdated (and, as a result, incorrect) information about WUFI Passive and METr.
Fig 3:
I also did a heatmap to show how individual questions performed with each model. Some of the immediate patterns are:
Q1 - What’s the difference between PHIUS Core and PHIUS+ Certification?
This question performed surprisingly poorly. While Claude got it correct, Claude PHIUS’ 2024 standard manual. The other three, however, all used PHIUS’ 2021 standard manual to answer the question.
Q7 - Is all of Massachusetts subject to the opt-in energy stretch code?
This question performed perfectly across the board, which is good since this is a fairly simple and easy question that’s well-covered in documentation from the state and third parties.
Q13 - If I have LEED Silver certification, do I automatically qualify for section 45L credits?
This was another perfectly answered simple question. By looking at the documentation, it’s clear that LEED doesn’t really correlate or have much mention at all in section 45L.
Q17 - We’re using a lot of insulation in our above-grade walls. Does this mean we can ignore air sealing the wall to have a high-performing home?
This was the third and final perfectly answered question. This one of the three was the most challenging, even though it’s still fairly simplistic. However, there aren’t any resources that I could find during desk research that suggested insulation mitigating the need for air sealing in high-performing buildings (thankfully, since that would be incorrect).
Q19 - I can only afford 3 new windows, which ones should I change, or should I put the money into attic insulation instead?
I expected this question to perform fairly poorly. All four models hallucinated a little bit here and there, and in addition, they all came off as overly confident. I marked the answer as generally clients should put the money into attic insulation, but it was building and climate dependent whether or not the windows would be more efficient, and all models suggested potential window replacement ideas.
Fig 4:
4. Reflections
While I was gathering the responses from these models, it became extremely evident that deep experience in building science is currently irreplaceable by AI. A majority of these questions are fairly generic, and even some of those questions were receiving moderate errors from some of these multi-million dollar models. However, AI is also the fastest-growing industry, and models are being updated at extreme rates. With an updated library and the right documentation, like Claude’s, AI could very quickly be able to answer all of these questions with ease.
However, if a model is wrong even 5-10% of the time, and its response is used on a real project, it could lead to huge litigation or financial problems for any given project. Especially in a field like building science, which requires extensive and updated knowledge on certification standards, public policy, energy codes, thermodynamics, and more, creating a good high-performance building is extremely difficult and is something that, at least now, requires “human touch”. If used, the information it provides should definitely be double-checked due to the complexity of the building science landscape.
From a broader urban lens, it goes without saying that the built environment is an ever-changing and extremely crucial part of urban living and existence. For AI to be able to confidently provide incorrect information on modern compliance is a big thing to note. Although this audit focuses on a small corner of the built environment, developers or small-scale homeowners who aren’t AEC practitioners could definitely walk away misinformed, and that could very easily lead to increased cost or timeline burdens to their project.
That being said, it is extremely impressive that these models were able to get so many of these questions close to perfectly correct. The overall average of 8.05/10 is definitely higher than I’d expected it to be, and is definitely an indicator that AI is only going to get better at answering a lot of these types of questions. Looking to the future, these models could heavily benefit from:
In-field expert-level audits before public deployment
Regular re-evaluations and updates tied to code update cycles and standard changes (not just model version changes)
Explicit disclaimers around rapidly changing standards, codes, and certifications.
An engineer or computer scientist working on these models might have just focused on the factual accuracy of these models and whether the model is right or wrong. However, fields like urban planning and building science are far more complex than this. A lot of what happens in these fields requires practitioners to question:
Consequences of actions - if something is slightly incorrect or biased, how and who could it impact on a high and low level.
Importance of critical thinking - if asked a leading question, it’s a practitioner’s job to correct said information and guide others to the correct answers, even if they are narrow (i.e., question 15).
Temporal and spatial sensitivity - it’s a bit more specific to building science than urban planning or design, but things that might work in one climate or ecosystem might be counterintuitive in others. Especially since most high-performance standards are Western and deal with predominantly (but not exclusively) colder climate zones and new-builds, there is a decent percentage of areas where best practices are underdocumented.
Limitations!
It should also be acknowledged that this audit is definitely limited in its validity. A 20-question benchmark with the oversight of one practitioner, who is graduating from their undergraduate degree next week, is a fairly small pool. My personal bias and industry knowledge as a Certified Passive House Consultant and as a soon-to-be graduated urban technologist definitely had an impact (although I did what I could to reduce said bias). I would definitely be interested in doing a more elaborate and large study on something like this as a means of continuing to develop and shape how AI plays a role in the greater AEC landscape.
Additionally, this benchmark only tested one conversation per question with minimal prompt engineering and no follow-up questions. This was a final project for the final class of my undergraduate experience rather than a thesis-level project. As a result, this audit is a snapshot of cold-start performances of these models, which very well could represent the floor of what these models are capable of, rather than the ceiling.
Data & Replication
Below, I’ve pasted the link to the Google Sheet with all of my data (except the visualizations, which were completed in Excel locally). I’ve also pasted in my personal email for any inquiries or questions about this project.
Thank you so much for reading! I had so much fun making this project.
Email: dylaningui@gmail.com