Chromium Blog
News and developments from the open source browser project
Fuzzing for Security
Thursday, April 26, 2012
Web browsers are big, complicated pieces of software that are extremely difficult to secure. In the case of Chrome, it’s an even more interesting challenge as we contend with a codebase that evolves at a blisteringly fast pace. All of this means that we need to move very quickly to keep up, and one of the ways we do so is with a scaled out
fuzzing
infrastructure.
Chrome’s fuzzing infrastructure (affectionately named "ClusterFuzz") is built on top of a cluster of several hundred virtual machines running approximately six-thousand simultaneous Chrome instances. ClusterFuzz automatically grabs the most current Chrome
LKGR (Last Known Good Revision)
, and hammers away at it to the tune of around fifty-million test cases a day. That capacity has roughly quadrupled since the system’s inception, and we plan to quadruple it again over the next few weeks.
With that kind of volume, we’d be overloaded if we just automated the test case generation and crash detection. That’s why we’ve automated the entire fuzzing pipeline, including the following processes:
Managing test cases and infrastructure
- To run at maximum capacity we need to generate a constant stream of test cases, distribute them across thousands of Chrome instances running on hundreds of virtual machines, and track the results.
Analyzing crashes
- The only crashes we care about for security purposes are the exploitable ones. So we use
Address Sanitizer
to instrument our Chrome binaries and provide detailed reports on potentially exploitable crashes.
Minimizing test cases
- Fuzzer test cases are often very large files—usually as much as several hundred kilobytes each. So we take the generated test cases and distill them down to the few, essential pieces that actually trigger the crash.
Identifying regressions
- The first step in getting a crash fixed is figuring out where it is and who should fix it. So this phase tracks the crash down to the range of changes that introduced it.
Verifying fixes
- In order to verify when a crash is actually fixed, which we run the open crash cases against each new LKGR build.
In addition to manageability, this level of scale and automation provides a very important additional benefit. By aggressively tracking the Chrome LKGR builds, ClusterFuzz is evolving into a real-time security regression detection capability. To appreciate just what that means, consider that ClusterFuzz has detected 95 unique vulnerabilities since we brought it fully online at the end of last year. In that time, 44 of those vulnerabilities were identified and fixed before they ever had a chance to make it out to a stable release. As we further refine our process and increase our scale, we expect potential security regressions in stable releases to become increasingly less common.
Just like Chrome itself, our fuzzing work is constantly evolving and pushing the state of the art in both scale and techniques. In keeping with Chrome’s
security principles
, we’re helping to make the web safer by upstreaming the security fixes into projects we rely upon, like
WebKit
and
FFmpeg
. As we expand and improve ClusterFuzz, users of those upstream projects will continue to benefit.
Posted by Abhishek Arya and Cris Neckar, Chrome Security Team
Labels
$200K
1
10th birthday
4
abusive ads
1
abusive notifications
2
accessibility
3
ad blockers
1
ad blocking
2
advanced capabilities
1
android
2
anti abuse
1
anti-deception
1
background periodic sync
1
badging
1
benchmarks
1
beta
83
better ads standards
1
billing
1
birthday
4
blink
2
browser
2
browser interoperability
1
bundles
1
capabilities
6
capable web
1
cds
1
cds18
2
cds2018
1
chrome
35
chrome 81
1
chrome 83
2
chrome 84
2
chrome ads
1
chrome apps
5
Chrome dev
1
chrome dev summit
1
chrome dev summit 2018
1
chrome dev summit 2019
1
chrome developer
1
Chrome Developer Center
1
chrome developer summit
1
chrome devtools
1
Chrome extension
1
chrome extensions
3
Chrome Frame
1
Chrome lite
1
Chrome on Android
2
chrome on ios
1
Chrome on Mac
1
Chrome OS
1
chrome privacy
4
chrome releases
1
chrome security
10
chrome web store
32
chromedevtools
1
chromeframe
3
chromeos
4
chromeos.dev
1
chromium
9
cloud print
1
coalition
1
coalition for better ads
1
contact picker
1
content indexing
1
cookies
1
core web vitals
2
csrf
1
css
1
cumulative layout shift
1
custom tabs
1
dart
8
dashboard
1
Data Saver
3
Data saver desktop extension
1
day 2
1
deceptive installation
1
declarative net request api
1
design
2
developer dashboard
1
Developer Program Policy
2
developer website
1
devtools
13
digital event
1
discoverability
1
DNS-over-HTTPS
4
DoH
4
emoji
1
emscriptem
1
enterprise
1
extensions
27
Fast badging
1
faster web
1
features
1
feedback
2
field data
1
first input delay
1
Follow
1
fonts
1
form controls
1
frameworks
1
fugu
2
fund
1
funding
1
gdd
1
google earth
1
google event
1
google io 2019
1
google web developer
1
googlechrome
12
harmful ads
1
html5
11
HTTP/3
1
HTTPS
4
iframes
1
images
1
incognito
1
insecure forms
1
intent to explain
1
ios
1
ios Chrome
1
issue tracker
3
jank
1
javascript
5
lab data
1
labelling
1
largest contentful paint
1
launch
1
lazy-loading
1
lighthouse
2
linux
2
Lite Mode
2
Lite pages
1
loading interventions
1
loading optimizations
1
lock icon
1
long-tail
1
mac
1
manifest v3
2
metrics
2
microsoft edge
1
mixed forms
1
mobile
2
na
1
native client
8
native file system
1
New Features
5
notifications
1
octane
1
open web
4
origin trials
2
pagespeed insights
1
pagespeedinsights
1
passwords
1
payment handler
1
payment request
1
payments
2
performance
20
performance tools
1
permission UI
1
permissions
1
play store
1
portals
3
prefetching
1
privacy
2
privacy sandbox
4
private prefetch proxy
1
profile guided optimization
1
progressive web apps
2
Project Strobe
1
protection
1
pwa
1
QUIC
1
quieter permissions
1
releases
3
removals
1
rlz
1
root program
1
safe browsing
2
Secure DNS
2
security
36
site isolation
1
slow loading
1
sms receiver
1
spam policy
1
spdy
2
spectre
1
speed
4
ssl
2
store listing
1
strobe
2
subscription pages
1
suspicious site reporter extension
1
TCP
1
the fast and the curious
23
TLS
1
tools
1
tracing
1
transparency
1
trusted web activities
1
twa
2
user agent string
1
user data policy
1
v8
6
video
2
wasm
1
web
1
web apps
1
web assembly
2
web developers
1
web intents
1
web packaging
1
web payments
1
web platform
1
web request api
1
web vitals
1
web.dev
1
web.dev live
1
webapi
1
webassembly
1
webaudio
3
webgl
7
webkit
5
WebM
1
webmaster
1
webp
5
webrtc
6
websockets
5
webtiming
1
writable-files
1
yerba beuna center for the arts
1
Archive
2024
Dec
Aug
Jun
May
Apr
Mar
Feb
2023
Nov
Oct
Sep
Aug
Jun
May
Apr
Feb
2022
Dec
Sep
Aug
Jun
May
Apr
Mar
Feb
Jan
2021
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2020
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2019
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2018
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2017
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2016
Dec
Nov
Oct
Sep
Aug
Jun
May
Apr
Mar
Feb
Jan
2015
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2014
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2013
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2012
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2011
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2010
Dec
Nov
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2009
Dec
Nov
Sep
Aug
Jul
Jun
May
Apr
Mar
Feb
Jan
2008
Dec
Nov
Oct
Sep
Feed
Follow @ChromiumDev
Give us feedback in our
Product Forums
.
