Dealing with vulnerabilities in IT environments is more important than ever. The media publishes daily articles about new vulnerabilities being discovered, published or exploited. We have now released our own vulnerability database for the public and allow all users to search the database for vulnerability reports and entries.
Continue reading
Not only for penetration testing it is interesting to know, which vulnerabilities exist for a certain software product. Also from the perspective of an IT team it can be useful to quickly obtain information about an employed product version. So far various databases existed for these queries like e.g., https://nvd.nist.gov/vuln/search, https://cvedetails.com or https://snyk.io/vuln
However, during the last years, we could identify several issues with these databases:
Figure: Incorrect vulnerability results for Windows 10
Figure: Keyword search returns a different product than the originally searched for product
This is why we decided to implement our own solution. We considered the following key points:
Upon finding vulnerabilities in products, security researchers commonly register a CVE entry per vulnerability. These CVE entries are given a unique identifier, detailed vulnerability information, as well as a general description.
They can be registered at https://cve.mitre.org and are indexed in the National Vulnerability Database (NVD) in real time (https://cve.mitre.org/about/cve_and_nvd_relationship.html). NIST publishes these data sets publicly and freely, which contain all registered vulnerabilities. We use this data stream as a basis for our own database.
The technical details of the data import and subsequent provisioning are illustrated as follows:
Figure: Overview of the technical components of the vulnerability database
The data sets are organized by year numbers and refreshed daily by NIST. Every night we download the latest files onto our file server.
Afterwards the files are pre-processed to make them compatible with the Elastic Stack Parser. One process that happens here is the expansion of all JSON files: The downloaded files contain JSON objects, however they are often nested, which makes it harder to identify single objects for the parser. We read the JSON and write all object seperators into separate lines. This way we can use a regex ( ‘^{‘ ) to precisely determine, when a new object begins.
Furthermore we strip the file of all unneeded metadata (e.g., autor, version information, etc.), which leaves only the CVE entries in the file as sequential JSON objects.
After the pre-processing, our Logstash parser is able to read the individual lines of the files using the Multiline Codec (https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html). Every time a complete JSON object is read in, Logstash forwards this CVE object to our Elasticsearch instance.
After all CVE entries were read and stored in the Elasticsearch database, we have to understand, which format these entries have and how we can search them for specific products and product vulnerabilities. Our final result is illustrated in the following screenshot: Using unique identifiers, we can return exact vulnerability reports for the queried product version.
Figure: Preview of our vulnerability query frontend
The general format of product versions is specified in the NIST specification. Section 5.3.3 gives a short overview (https://nvlpubs.nist.gov/nistpubs/Legacy/IR/nistir7695.pdf):
cpe:2.3:part:vendor:product_name:version:update:edition:sw_edition:target_sw:target_hw:language:other
A colon is used as a separating character. Asterisk (*) is used as a wildcard symbol.
In our screenshot: “cpe:2.3:o:juniper:junos:17.4r3:*:*:*:*:*:*:*” we can determine that the operating system JunOS of the vendor Juniper in version 17.4r3 is prone to a vulnerability.
Looking at the JSON file, it becomes apparent that there are two formats that are used to store the version number of a vulnerability.
To query the database for specific products, an easy interface to find correct product identifiers is required. We have decided to implement this component using JavaScript Auto-Complete, that displays products and associated CPE identifiers dynamically:
Figure: Autocomplete mechanism of the query frontend
After a choice was made, the vulnerabilities matching the specific product identifier can be queried.
A big advantage that storing vulnerability data in an Elasticsearch database has, is its direct connection to Kibana. Kibana autonomously queries Elasticsearch to generate visualisations from it. In the following we illustrate a selection of visualizations of vulnerability data:
Figure: Amount of registered vulnerabilities per year
Figure: Fractions of the respective risk severity groups per year
We see great potential in using this data for real time statistics on our homepage to provide vulnerability trends which are updated on a daily basis.
Another item on our CVE database roadmap is the implementation of a system that automatically notifies customers of new vulnerabilities, once they are released for a certain CPE identifier. Elasticsearch offers an extensive REST API that allows us to realize this task with the already implemented ELK stack.
Currently we are working on implementing live statistics for our homepage. As soon as this milestone is complete, we will continue with the topic of “Threat Intelligence”. As you can see, we not only focus on the field of penetration testing here at Pentest Factory GmbH, but also have great interest in researching cybersecurity topics and extending our understanding, as well as our service line.
