Machine Learning and Construction Safety

A CASE STUDY ON USING COMPUTER VISION FOR SAFETY SCREENING IN ENR’S ANNUAL
PHOTO COMPETITION

INTRODUCTION:
CRACKS NOT CATS
As consumers, we often take it for granted that in 1997, IBM’s Watson achieved victory over
typing “cat” into an Internet search engine will Jeopardy champions in 2011, and Google’s
present the results we’re looking for back to us. DeepMind won a game of Go against a
But if we needed to search for useful imagery for professional player in October of 2016. In
our company’s safety or marketing departments, the tradition of these events, this case study
could we find it with the same level of speed and presents the results of an AI compared to
sophistication as cat pictures? What methods are a human to illustrate the potential value of
available for searching through photos and videos machine learning for construction safety.
we use for work? The solution to this problem
lies in applying artificial intelligence technologies
known as machine learning, often used in the SUNDAR PICHAI, CEO of Alphabet
consumer world, to problems in industry. Google Earnings Call 2 :

Over the last five years, amazing progress “Machine learning is a core, transformative way by
has been made in the fields of artificial which we’re rethinking everything we’re doing.”
intelligence and machine learning. Microsoft,
Apple, Facebook, Salesforce and Alphabet
(Google) have all implemented major machine In partnership with Engineering News-Record,
learning initiatives, with excellent results. we present the first-ever 2016 Artificial
As one example, Google Translate’s error Intelligence for Safety Demonstration. We hope
rate dropped by 60% after switching to these results demonstrate the potential value
machine learning models in 2016 1. machine learning holds for construction
Over the last 20 years, improvements in artificial safety, and how these traditionally consumer-
intelligence (AI) have been demonstrated in oriented technologies can now be applied
various public competitions; IBM’s DeepBlue to solving industrial-grade problems.
beat Russian Grandmaster Kasparov in chess

www.smartvid.io | 101 Main Street, 14th Floor, Cambridge, MA 02142

OUR VISION: USING AI machine learning engine, nicknamed VINNIE
(for ‘Very Intelligent Neural Network for Insight
TO HELP INDUSTRY and Evaluation’), then uses speech and image
recognition to analyze and organize project files.
There is a significant amount of photo and video
content created in the field; for example, it’s not As files are uploaded, they are automatically
uncommon to generate at least 1 gigabyte of tagged by their visual and audio content based
media per week on AEC (Architecture, Engineering on what the AI engine, VINNIE, sees and hears
and Construction) projects. The collection, in the content. We call these tags SmartTags as
organization, and retrieval of industrial media no additional effort is required from the user.
is a challenge. But it is also an opportunity. Since VINNIE’s recognition ability is broad, users
can create a list of ‘tag definitions’ to limit its
recognition to only those things that are useful
ImageNet’s popular image recognition to your team. The tagging system also gives
competition shows computers now flexibility for content search and retention – think
beating human accuracy. 4 of a search able to pull thousands of variations
of tags as criteria to show flexible “slices” of your
media database, based on your team’s current
needs. Like safety…

CASE STUDY: THE 2016 AI

FOR CONSTRUCTION SAFETY
CHALLENGE
Engineering News Record, the #1 publication in
the construction and capital projects industry,
holds an annual Year in Construction Photo Contest3
where readers from AEC firms submit thousands
of project images to be selected for publication
in the widely circulated magazine. All images
are judged for artistic merit, and simultaneously
Image recognition (also known as computer
reviewed by a safety expert to ensure that none
vision) has been one of machine learning’s
biggest improvements, and computers can now
categorize millions of images faster and better
than their human rivals. Unfortunately, even VINNIE detecting workers in jobsite photos
(Photo courtesy of ENR)
many new and improved image recognition
tools are not a fit for industrial uses. Much like
Apple’s Siri can recognize “room service” but
doesn’t recognize “Room 1513-A,” Google’s image
recognition can easily categorize “cats,” but
struggles to identify “cracks in concrete.”

The Smartvid.io solution is a media management

platform, designed with the specific needs of
industrial users in mind. The application is project-
centered; industrial photos and videos taken by
team members using “off the shelf” equipment
such as digital cameras, cell phones and GoPros
are all aggregated into one place. Our

www.smartvid.io | 101 Main Street, 14th Floor, Cambridge, MA 02142

depict any hazardous situations. Detection and “wider coverage” of safety checks. This monitoring
prevention of safety risks is one of the most can provide automated flagging of potential
common needs of industry, especially so within issues, giving human experts more time for
AEC. Approximately 1 in 5 American worker careful review.
fatalities occur in construction, with jobsite deaths
in 2015 rising to their highest level in seven years.5

For the 2016 ENR Photo Competition, Smartvid.io VINNIE detecting people missing hard hats
and safety vests. (Photo courtesy of ENR)
partnered with ENR’s editorial staff to hold the
first-ever Artificial Intelligence for Construction
Safety Demonstration. In the spirit of other human
vs. AI demonstrations, VINNIE analyzed all 2016
contest images alongside the traditional human
safety experts. Here’s how it worked:

First, VINNIE automatically scanned photos to for

the presence of human workers. Simply knowing
that people exist in an image is a great way to
flag content a human safety expert needs to
review. Next, the people-containing images were
automatically checked for two specific categories
of safety risks: 1) workers not wearing hard hats
and/or 2) working not wearing safety vests.
CONCLUSION & NEXT STEPS
VINNIE detecting people missing
The results of the AI for Safety Demonstration
safety vests (Photo courtesy of ENR) show the potential value machine learning
has for safety, by being able to weed through
thousands of photos at high speed in order
to flag risk indicators for safety experts. Using
computer vision to spot potential risks certainly
does not remove the need for humans, however;
a safety professional has years of training and
can spot thousands of issues, whereas VINNIE
can currently recognize only a few specific
categories. However, this sort of automated field
monitoring adds leverage to safety efforts as pair
of “computer eyes” that never sleep, identifying
risk with greater speed and accuracy than
otherwise possible.

The results were startling. VINNIE was able to

sort through all 1,080 submissions in under 10
minutes, while the human team required over
4.5 hours. VINNIE correctly detected 446 images
with people, while the human review only found
414. VINNIE also flagged 32 images containing
personnel missing hard hats, and 106 images with
workers missing safety-colored clothing. Human
judges agreed that the implications of using a
tool like VINNIE were better speed, accuracy, and

www.smartvid.io | 101 Main Street, 14th Floor, Cambridge, MA 02142

What comes next? First, we are working to add ENR led the way, contributing anonymous
additional capabilities for VINNIE – beyond photos from 10 prior years of photo
people, hard hats and safety vests – to new competitions to provide material for Vinnie’s
categories of potential risks. training. Additional imagery was provided by
the partners listed below, and others who have
Second, we are working hard to make VINNIE’s decided to remain anonymous. We are grateful
capabilities more accessible by integrating for their help and support.
our Smartvid.io platform into the systems you
already use to store your photo and video
content. Via our integrations with Procore®,
Autodesk’s BIM 360 Field® and other products,
getting your old or ongoing photo and video
content into our platform for VINNIE’s review is
now easier than ever, and can help your safety,
marketing, and operations teams be up and
running quickly.
& OUR ANONYMOUS
PARTNERS
MARTIN LEIK, Regional Safety Director
Suffolk Construction
“We continue to push the boundaries of what is
possible on our projects, so it’s exciting to test
REFERENCES
innovative technologies that could have a positive
impact on the entire construction experience.”
1 D. Castelvecchi, “Deep learning boosts Google
Translate tool,” Nature , 27 9 2016.

Recognition capacity for “People”, “Hard Hats” 2 Seeking Alpha, “Alphabet (GOOG) Q3 2015
and “Safety” categories is available as part of the - Earnings Call Transcript,” 23 10 2015.
Smartvid.io free trial. Go to www.smartvid.io to [Online]. Available: http://seekingalpha.com/
sign up, start screening your photos, and make article/3596706-alphabet-goog-q3-2015-
searching for safety risks as easy as cat pictures. results-earnings-call-transcript.

We invite your feedback at: 3 Engineering News-Record, “Annual Photo

www.smartvid.io/contact as we welcome Contest Archives,” 2016. [Online]. Available:
you to join us on the journey to bring machine http://www.enr.com/photoContest/archives/.
learning to the world of industry.
4 The Economist, “From not working to neural
networking,” The Economist, 25 6 2016.

A BIG “THANK YOU” TO OUR 5 T. Ichniowski, “Construction Fatalities climb

in 2015,” Engineering News-Record , 12 2016.
CONTRIBUTING PARTNERS
In preparation for the AI for Safety
demonstration, VINNIE expanded on previous
safety training by reviewing images provided
by a set of partners who are dedicated to
promoting safety in construction.

www.smartvid.io | 101 Main Street, 14th Floor, Cambridge, MA 02142