Eye-tracking during building inspections provides insight on how experts think

After a building failure due to natural disasters or poor structural design, safety inspectors must enter a structure to assess the damage before occupants can return. Researchers in the Penn State Department of Architectural Engineering have studied how building inspectors make their safety assessments, by analyzing their gaze patterns with eye-tracking software. Eventually, the eye-tracking data could be used to code autonomous robots, like drones, to conduct building assessments in place of humans.

The researchers’ results were published in Scientific Reports.

“We are looking for a way to capture how an inspector thinks and makes assessments on a building site to understand their intentions and diagnostic choices,” said corresponding author Rebecca Napolitano, assistant professor of architectural engineering. “Studying where they look—and for how long—at certain points on a building can help us do that.”

Researchers tasked 10 architectural engineering graduate students to assess two building facades while wearing Tobii eye-tracking glasses. The glasses have two cameras to measure movements and positioning simultaneously. One camera faces the object, and the other is trained on the user’s eyes to measure pupil dilation and gaze direction.

Study participants were given unlimited time to assess two sections of the facades of two steel-framed buildings in central Pennsylvania. Researchers chose the sites due to the buildings’ various observable issues, including plant growth, peeling paint, degrading bricks, cracks, water cooling and intrusion, and erosion at the foundation.

Participants—who all had some building diagnostics experience and received the same training ahead of the experiment on building inspection—spent varying amounts of time gazing at different points of the facades. Some missed more obvious issues and focused for long periods on smaller issues, such as cracks.

“Some of the issues were more or less obvious, and eventually everyone found most of the same things, but their approaches to the exercise were very different,” Napolitano said. “We learned that people, depending on their backgrounds, have biases when looking at structures.”

Someone accustomed to looking for certain issues, such as water damage, for example, will spend more time on that particular issue, Napolitano said.

The results differed from studies in other fields that have conducted eye-tracking assessments, such as in medical imaging research and psychology, according to the researchers. Experts in those fields were shown to spend the same amount of time on each point of a page, including when encountering abnormalities. The building inspection participants, on the other hand, tended to focus for longer periods on observable issues.

The broad differences in how people assess the buildings present a challenge for coding drones to perform the same inspections, the researchers said.

“It would be hard to take an average of all the data since the study participants had such different results,” Napolitano said. “But it is one small step in understanding how inspectors think, with the long-term goal of creating an algorithm to inform a drone.”

In the future, the researchers plan to carry out another eye-tracking experiment, this time using a 3D stimulus, such as a room inside of a building, as opposed to a building facade, which is considered 2D. Toward that goal, some members of Napolitano’s lab collected sample 3D eye-tracking data inside damaged structures during a visit to Mayfield, Kentucky, after the area’s December 2021 tornado, which the group intends to analyze and code.

“We plan to use the collected field data to create synthetic data, which we can then use to learn more about human-robot relationships,” Napolitano said.

In addition to Napolitano, the co-authors included Muhammad Rakeh Saleem, a Penn State doctoral student in architectural engineering, and Robert Mayne, a math instructor from Chariho Regional High School, Rhode Island, who assisted with the statistical analysis.