It doesn’t matter if you are a freelancer or the Technical Director of a big team: consistency needs to be one of the pillars of your strategy. You need to follow a set of testing methodologies.

But what does consistency mean in the context of security project management? That all projects are delivered to the same high quality standard. Let me repeat that again:

Consistency means that all projects are delivered to the same high quality standard

Even though that sounds like a simple goal, there are a few parts to it:

• All projects: this means for all of your clients, all the time. It shouldn’t matter if the project team was composed of less experienced people or if this is the 100th test you’re running this year for the same client. All projects matter and nothing will reflect worse in your brand than one of your clients spotting inconsistencies in your approach.
• The same standard: as soon as you have more than one person in the team, they will have different levels of skill, expertise and ability for each type of engagement. Your goal is to ensure that the process of testing is repeatable enough so each person knows the steps that must be taken in each project type. There are plenty of sources that you can base your own testing methodology upon including the Open Source Testing Methodology Manual or the OWASP Testing Guide (for webapps).
• High quality: this is not as obvious as it seems, nobody would think of creating and using a low quality methodology, but in order for a methodology to be useful you need to ensure it is reviewed and updated periodically. You should keep an eye on the security conferences calendar (also a CFP list) and a few industry mailing lists throughout the year and update your methodologies accordingly.

So how do you go about accomplishing these goals?

Building the testing methodology

Store your methodology in a file

We’ve seen this time and again. At some point someone decides that it is time to create or update all the testing methodologies in the organization and time is allocated to create a bunch of Word documents containing the methodologies.

Pros:

• Easy to get the work done
• Easy to assign the task of building the methodology
• Backups are managed by your file sharing solution

Cons:

• Difficult to maintain methodologies up to date.
• Difficult to connect to other tools
• Where is the latest version of the document?
• How do you know when a new version is available?
• How does a new member of the team learn about the location of all the methodologies?
• How do you prevent different testers/teams from using different versions of the document?

Use a wiki

Next alternative is to store you methodology in a wiki.

Pros:

• Easy to get started
• Easy to update content
• Easy to find the latest version of the methodology
• Easier to connect to other tools

Cons:

• Wikis have a tendency to grow uncontrollably and become messy.
• You need to agree on a template for your methodologies, otherwise all of them will have a slightly different structure.
• It is somewhat difficult to know everything that’s in the wiki. Keeping it in good shape requires constant care. For instance, adding content requires adding references to the content in index pages (some times to multiple index pages) and categorizing each page so they are easy to find.
• There is a small overhead for managing the server / wiki software (updates, backups, maintenance, etc.).

Use a tool to manage your testing methodologies

Using a testing methodology management tool like VulnDB or something you create yourself (warning creating your own tools will not always save you time/money).

Pros:

• Unlike wikis, these are purpose-built tools with the goal of managing testing methodologies in mind: information is well structured.
• Easy to update content
• Easy to find the latest version of the methodology
• Easiest to connect to other tools
• There is little overhead involved (if using a 3rd party)

Cons:

• You don’t have absolute control over them (if using a 3rd party).
• With any custom / purpose-built system, there is always a learning curve.
• There is strategic risk involved (if using a 3rd party). Can we trust these guys? Will they be in business tomorrow?

Using the testing methodology

Once you have decided the best way in which to store and manage your testing methodologies the next question to address is: how do you make the process of using your testing methodologies painless enough so you know they will be used every time?

Intellectually we understand that all the steps in our methodology should be performed every time. However, unless there is a convenient way for us to do so, we may end up skipping steps or just ignoring the methodology all together and trusting our good old experience / intuition and just get on with the job at hand. Along the same lines, in bigger teams, it is not enough to say please guys, make sure everyone is using the methodologies. Chances are you won’t have the time to verify everyone is using them so you just have to trust they will.

Freelancers and technical directors alike should focus their attention in removing barriers of adoption. Make the methodologies so easy to use that you’d be wasting time by not using them.

The format in which your methodologies are stored will play a key part in the adoption process. If your methodologies are in Word documents or text files, you need to keep the methodology open while doing your testing and somehow track your progress. This could be easy if your methodologies are structured in a way that lets you start from the top and follow through. However, pentesting is usually not so linear (I like this convergent intelligence vs divergent intelligence post on the subject). As you go along you will notice things and tick off items located in different sections of the methodology.

Even if you store your methodologies in a wiki, the same problem remains. A solution to the progress tracking problem (provided all your wiki-stored methodologies are using a consistent structure) would be to create a tool that extracts the information from the wiki and presents it to the testers in a way they can use (e.g. navigate through the sections, tick-off items as progress is made, etc.). Of course, this involves the overhead of creating (and maintaining) the tool. And then again, it depends on how testers are taking their project notes. If they are using something like Notepad or OneNote, they will have to use at least two different windows: one for the notes and one for following the methodology which isn’t ideal.

In an ideal world you want your methodologies to integrate with the tool you are using for taking project notes. However, as mentioned above, if you are taking your notes using off the shelf note taking applications or text editors you are going to have a hard time integrating. If you are using a collaboration tool like Dradis Pro or some other purpose-built system, then things will be a lot easier. Chances are these tools can be extended to connect to other external tools.

Now you are into something.

If you (or your testers) can take notes and follow a testing methodology without having to go back and forth between different tools, it is very likely you will actually follow the testing methodology.