What to look for in pentests? Quality features explained.

Often, in discussions with new customers, we can see that the market of penetration testing services seems opaque and it is difficult to decide on a service provider. The focus is often on the price of a pentest and other decision criteria are omitted.

With this article, we want to provide you with a basic guide to qualitatively evaluate penetration testing service providers and simplify your decision-making process.

Basic qualification of penetration testers

One issue that arises in pentesting is the issue of experience. Attacking computer systems requires creativity, flexibility, and an understanding of a breadth of technologies and platforms. While several years of experience as a developer or security officer can make it easier to get started, they are still no substitute for practical knowledge of how security mechanisms work and how they can be attacked.

For this reason, we recommend focusing on how many years a penetration tester has been performing tests and what practical qualifications they have. Below we have listed some commonly found qualifications and what knowledge is hereby effectively attested.

As can be seen from the descriptions, of the certifications listed here, only the OSCP demonstrates actual practical knowledge in compromising computer systems. We recommend that you hire testers who have a practical qualification similar to OSCP. To view successful qualification and validity, we recommend asking the service provider for proof (e.g., digital link to Credly, Credential.net, or a scanned copy of a tester’s acquired certificate).

Specialization of penetration testers

As described before, the OSCP certification provides a good reference point to verify essential competencies of a penetration tester. OSCP certification demonstrates knowledge of enumerating and testing individual hosts and services.

However, since modern applications have grown extremely in complexity, we recommend asking the service provider what specializations the individual testers have and having these specializations proven (e.g., certifications, customer references, CVE records). Especially for complex test objects, the tester should be familiar with the technologies and have specialized in the corresponding area. This is especially true in areas such as web applications, API interfaces, Active Directory, mobile application testing, SAP testing and many more.

Offer and scoping

When you request a quote, the quote should be tailored to the application or infrastructure you are testing. To do this, the service provider should find out what the scope of the test object is and, on this basis, make an estimate of how many days of testing are required.

If the service provider does not ask for details about your test object and sends you a quote “blindly”, it may either be that too few days have been calculated in, which means that the application can be tested less deeply or even that certain components are omitted. Alternatively, it can equally happen that too many days are estimated for the test object and you simply have to pay for them, although the test could have been completed in advance.

Tip: If you approach several service providers at the same time (e.g. in a tender), you should describe the test object as precisely as possible (technologies used, typical application functions and processes, number of hosts). This information makes it easier to create an appropriate quote and reduces the likelihood of choosing the wrong test scope or methodology.

Final Report

After a penetration test has been performed, the final report is the key document that records the results of the pentest. Therefore, pay particular attention to the quality of the final report and obtain a sample report in advance of the engagement.

Each finding should include a clear description, with screenshots, of how to identify and exploit the vulnerability so that you or your developers can understand the issue and recreate it if necessary. Also, each finding should include an explanation of what risk has been assigned to the vulnerability and what this assignment is based on (e.g., using risk assessment methods such as CVSS or OWASP).

The report should clearly list all the framework parameters of the test and explain typical W questions, such as:

– When was the test performed (period)?
– What was tested (test scope)?
– What, if anything, was not tested (scope)?
– How was it tested (methodology)?
– Who performed the test (contact person)?
– What risk assessment method was used?
– Which tools, scripts and programs were used?

Ask the service provider for a sample report and compare reports to choose the ideal report format for you. Also be sure to include a management summary that summarizes the test results in non-technical, management-level language. This is especially important because the details of findings are often very technical or complex and can only be understood by technical personnel.

Vulnerability scan versus penetration test

Often the terms vulnerability scan and penetration test get mixed up. A vulnerability scan is an automated procedure by which a program independently or based on certain scan parameters tests the test object for vulnerabilities. No manual testing by a human is performed here.

Be careful when a vulnerability scan is advertised instead of a penetration test. Many vulnerabilities are contextual and can only be identified through manual testing. In addition, vulnerability scanners can return false positive results, which are not actual vulnerabilities.

To test efficiently, one or more automated scans can be part of a penetration test. However, you should ensure that the service provider has a focus on manual verification of results and manual testing of the test object. The automatically generated test results should not be included directly in the report, only after manual verification. Each finding should include a detailed account of how the vulnerability was verified.

Technical and legal basics

Before a penetration test can be legally performed, it is mandatory to obtain the hoster’s permission. If you do not host your application or infrastructure yourself, be sure to ask the hoster for permission to test it. Exceptions to this are some cloud hosters that explicitly allow penetration testing (e.g. Microsoft Azure, Amazon AWS, Google Cloud). Make sure that all approvals have been given before starting the test. The penetration test service provider should raise this issue on its own and be sure to clarify it with you before testing begins.

In order to clearly assign which attacks are carried out by your service provider and which attacks represent a real threat, the service provider should carry out the tests from a fixed IP address. To do this, ask your service provider whether such a static IP address exists and find out in advance of the tests. You can also search your log files for the IP address during the test and get insight into what volume of requests the test generated. Be careful when choosing a service provider should they not use a unique IP address for their testing. In addition, always obtain the contact information of the person performing the technical tests. This way you have the possibility to contact a technical contact person directly in case of problems or technical questions and to get feedback immediately. Furthermore, this allows you to exclude the possibility that a subcontractor was commissioned to carry out the tests in a non-transparent or possibly unofficial manner.

Specific procedure

Web application testing

A penetration test of a web application should follow the public standard test methodology “OWASP”. The OWASP consortium provides procedures for testing all current vulnerability categories. This should definitely be tested.

If you want to test an application that provides a user login and protected user areas, we recommend performing a “grey-box” test. Here, test accounts are provided to the service provider, allowing internal areas behind a login to be tested more efficiently and granularly. Pay attention to whether the service provider suggests this test methodology or, if necessary, actively ask for the test methodology.

If an API interface is to be tested, the service provider should request interface documentation or a collection of sample API requests (e.g. Swagger.json, Postman Collections). Without API documentation, testing APIs is not purposeful because endpoints and parameter names have to be guessed. This can result in important endpoints being overlooked and vulnerabilities not being detected.

IT infrastructure testing

An infrastructure test where multiple host systems are tested for their available services usually consists of several automated scans at the beginning of a test combined with manual test units and a subsequent verification of the scan results.

Active Directory Assessment

Active Directory environments are very dynamic and require specialized knowledge beyond a basic qualification such as OSCP. Make sure the tester of your AD environment has advanced training and certifications in Windows and Active Directory security. These may include, for example, the Certified Red Team Professional (CRTP) or Certified Red Team Expert (CRTE) hands-on certification. However, also many other trainings in the area of Azure AD and Windows environments.