How to evaluate a penetration testing vendor

Choosing a penetration testing vendor is harder than it looks. Every provider claims “expert manual testing,” “actionable reports,” and “continuous coverage.” The words are identical; the work behind them is not. This guide gives you a framework — and the specific questions — to tell them apart.

Start with the outcome you actually need

Before comparing vendors, be clear on why you’re testing. The three most common drivers each favor different capabilities:

• Compliance (SOC 2, ISO 27001, PCI DSS, HIPAA). You need a report an assessor accepts, on a predictable cadence, mapped to controls.
• Risk reduction. You need real vulnerabilities found and fixed — depth and proof matter more than a checklist.
• Customer assurance. Prospects or partners ask for evidence. You need a shareable, credible report and a repeatable program.

Most teams need all three, but the priority order changes which trade-offs are acceptable.

The engagement model is the biggest fork in the road

How the testing is actually delivered matters more than any feature list:

• Automated scanning gives breadth and speed but misses anything requiring judgment — business logic, authorization, chained exploits.
• Traditional consultancy gives deep, point-in-time manual testing, but the report ages the moment you ship your next release.
• Crowdsourced platforms give you a large, rotating pool of testers, priced by credits — great for breadth, less so for a consistent, accountable team.
• PTaaS (Penetration Testing as a Service) combines human-led testing with a platform for continuous, release-aware coverage.

Ask directly: Who does the testing, are they the same people over time, and how do you cover the changes I ship between tests?

The questions that separate real testing from theater

On proof and depth

• Do you provide a reproducible proof-of-exploit for every finding? A finding without proof is a guess. Proof is what makes engineers act instead of argue.
• How much of the test is manual vs. automated? Ask for the ratio and examples of business-logic flaws they’ve found — the class scanners can’t detect.
• Can I see a redacted sample report before signing? If they won’t show you the work product, that’s a signal.

On the team

• What certifications does the bench hold (OSCP, OSWE, CREST, etc.)? Certifications aren’t everything, but a real bench can name theirs.
• Will I have a named engagement lead? Accountability beats a ticket queue.

On the program

• Is retest-to-verified-fix included, or billed separately? A fix you haven’t retested is a hope, not a result.
• How do you handle testing after a significant release? Point-in-time testing leaves most of the year uncovered.
• What integrations do you support (Jira, GitHub, Slack, SIEM, CI/CD)? Findings that live in a PDF don’t get fixed as reliably as findings in your workflow.

On pricing and scope

• Is pricing transparent and predictable? Credit-based models can be flexible but hard to budget; understand what a “retest” or “out-of-scope” finding costs.
• What exactly is in and out of scope? Get the asset list, roles tested, and rules of engagement in writing.

Read the sample report critically

A good report is the clearest signal of quality. Look for:

• An executive summary a non-technical leader can act on.
• Every finding with severity, a CVSS vector, business impact, and specific remediation.
• Reproduction steps and evidence — not just a scanner’s output pasted in.
• Retest results showing what was verified fixed.
• Strategic recommendations, not just a list of bugs.

A simple scoring rubric

Score each vendor 1–5 on: proof-of-exploit discipline, manual depth, team accountability, continuity/coverage between tests, remediation support (retests + integrations), report quality, and pricing transparency. Weight the rows by your priority order from step one. The winner is rarely the cheapest or the one with the longest feature list — it’s the one whose model matches how you actually build and ship.

The one-line test

If you remember nothing else, ask this: “Show me a finding you proved, and tell me how you’d know it’s fixed.” A vendor who can answer clearly — with evidence and a retest — is doing the real work. A vendor who deflects is selling you a scan in a slide deck.