This searches a NVD CVE JSON feed for relevant packages and creates vulnerability entries in the same easy-to-consume format that OSV uses.
See the relevant subdirectories for more details specific to that package ecosystem.
CVE entries do not provide an easy mapping to the exact package name used in a package manager, so we dump and compare reference URLs for the package against reference URLs used in the CVE, in addition to some other heuristics to avoid false positives.
Where possible, we try to extract affected version ranges from descriptions (pattern matching) and other fields. We also extract commit hashes from e.g. GitHub links.
Extracted version numbers are cross referenced against the list of actual versions released in the package manager to ensure accuracy.
The intended workflow is for a human to periodically run the tools here against the latest NVD CVE feeds.
(See the PyPI directory for details on how to generate the required pypi input files (pypi_links.json, pypi_versions.json)).
export VULNS_REPO=/path/to/vulns/repo
go run ./cmd/pypi -false_positives $VULNS_REPO/triage/false_positives.yaml \
-nvd_json /path/to/nvdcve-1.1-2021.json \
-pypi_links pypi/pypi_links.json \
-pypi_versions pypi/pypi_versions.json \
-out_dir $VULNS_REPO/vulns
This auto-generates matching <ID>.yaml
files into $VULNS_REPO/vulns
.
However, human intervention will always be required in some cases, when versions
cannot be reliably extracted automatically from CVE data. In these cases a
comment block will be appended to the <ID>.yaml
to indicate that a human
should look at this.
An example notes block will look like:
# <Vulnfeeds Notes>
# Warning: 2017.5.0 is not a valid introduced version
# Warning: 2018.2.0 is not a valid introduced version
# Warning: 3000.0 is not a valid introduced version
These files will have the placeholder ID format <PREFIX>-0000-....
(where
<PREFIX>
is the database ID prefix, e.g. PYSEC
).
Entries can also be manually contributed via a pull request. These entries
should use a placeholder ID of the format <PREFIX>-0000-<anything>
to indicate
that an ID needs to be assigned.
Once vulnerabilities are scraped (or pull requests are merged), we need to assign IDs. Do this by running:
export VULNS_REPO=/path/to/vulns/repo
export PREFIX=PYSEC
go run ./cmd/ids -dir $VULNS_REPO -prefix $PREFIX
This will replace previous placeholder IDs with IDs of the form
<PREFIX>-YEAR-NNN
.
TODO: GitHub Action to do this.
False positives (i.e. CVEs that are incorrectly matched to a package) can be
added into a false_positives.yaml
file and commited as part of the
vulnerability repo for future runs.
The format of this file is newline delimited CVE IDs.
e.g.
ids:
- CVE-2021-1337 # justification
- CVE-2021-1338 # justification
packages:
- blah # justification
...