A lot of people talk about AGILE but the normally mean ‘agile’ however when it comes to security testing and penetration testing (to me there is most certainly a difference) we need to be mindful of the different approaches, so we select the right one for the context, scenario, and objectives.

In this post we take a brief look at what we recommend for a range of scenarios and we look at the key differences and what some constraints might mean when it comes to approach selection.

Traditional vs Agile Penetration Testing Approach Selection Matrix

Here’s a high-level matrix based on my experience in this field:

Scenario	Traditional	Agile
No security testing conducted previously	X
No security testing built into release process	X
No process and tooling in environment designed for agile testing	X
Security Testing Conducted Previously but no remedial activities	X
We want to test in a short period of time for assurance/compliance purposes.	X
We penetration tested after the last major release and we want to focus on a specific area.		X
We want to build security testing into our security operations and release process and our SDLC operates on continual/many release model		X

Major Differences

• Traditional Penetrating Outputs a formal Report artefact.
• Agile testing outputs findings into a bug or issue tracking system.

Constraint Considerations

Here are just some considerations we can consider when it comes to constraints, the key point here is that they must be considered and contextualised. These are just some examples:

• If you have a small budget and do not have all the pre-requestees to stand up and integrate an agile testing approach into your processes or pipelines we recommend that traditional testing is conducted.
• If you are required to demonstrate penetration testing for compliance purposes, we recommend that traditional penetration tests are used.
• If your development team is not well versed in security terminology, we recommend that you leverage traditional penetration testing as an initial step.

But wait this is not the only way

Now I am talking about penetration testing. But what if we talk about integrating security testing into the release pipeline and cycle? Now that is another game and an approach, we would recommend but do not think it is a turnkey solution. This requires designing security testing into the process and that requires time, skills, tools, and integration/learning.

Summary

There is a marked difference between styles and the flow and output are different. You need to be mindful of this as it is not as simple as just throwing agile testing into an existing service. How do we know this? We have tried it and the results were not very good (this was with a small window of time so it is possible this would work better over longer period e.g., weeks/months)

Penetration testing scoping and approaches are just a component of a security testing process which aligns to a range of other processes (e.g., design, risk management, change, release etc.) so we strongly advise people do not just think a penetration test is an off the peg item if you are thinking of doing one in an agile manner without putting in the leg work first.