In the years since automation came to the world of security compliance, it has largely fulfilled its promise to eliminate unreliable and time-consuming manual evidence collection (like taking screenshots—ugh!).
However, with all of the (justified) excitement over the time savings, it’s easy to overlook the details of exactly what evidence each of these tools automatically provides. This post will explore the three main types of evidence that compliance automation tools create and discuss their advantages and disadvantages.
Test-Based Evidence
Prescriptive tools provide test-based evidence. These tools define precisely what they check for and run simple pass/fail tests. If a tool finds what it’s checking for, it will give you a “Pass.” If not, you’ll get a “Fail.” The test result saying you passed is the evidence you present to the auditor.
Let’s explore an example of prescriptive test-based evidence. Say you need to check that all buckets are encrypted. If they are, the tool will tell you that you passed. Great, you can move on. If all buckets are not encrypted, you will fail the test and need to fix the problem. So far, it seems pretty simple, right?
Well, it is, and that’s the main advantage of this type of evidence—but it’s also where we hit the first bump in the “test-based evidence” road. With a simple pass/fail test, the result is all you get. That means you don’t know where the problem lies if you fail.
Since you don’t have the data, you will most likely have to open a ticket for the infrastructure team to determine which buckets are not encrypted and to remedy the situation so that the next time you run the test, you can pass. (Let’s not forget that going back to the relevant system owners and manually sifting through the data for mistakes is exactly what you wanted to avoid by onboarding an automation tool!)
Now, say that the infrastructure team informs you there are indeed unencrypted buckets, but you realize they are out of scope for this specific audit. In other words, you got a false negative. What now? You can’t give the auditor the “fail” as evidence. Unfortunately, there is no straightforward solution for this with test-based evidence.
False positives can easily occur, too. Let’s say your organization’s policy requires all passwords to meet strict criteria: at least 12 characters long, including upper and lower case letters, a special character, and a number. Your tool’s test, however, only checks whether the password is more than eight characters in length (remember, the test is defined by the vendor, not you). Even if all the passwords “pass” the control requirement your tool has decided is important, you have no idea whether they meet your defined policies. This is a false positive.
When you are a smaller company, using the tests pre-defined by a tool may be good enough. But as your organization becomes a little more mature, you will need the ability to configure tests and present more evidence to your auditor.
The last thing to note about this kind of evidence is that there is no guarantee that all auditors will accept a “pass.” Will they trust the test results, knowing that false negatives can occur? What about false positives? Will they be willing to sign off on that?
To Summarize:
- Test-based evidence IPE (Information Provided by the Entity):
- The date and time the vendor conducted the test
- The pass/fail results
- The API call used
- Biggest advantage: Very quick and easy automated compliance evidence. Prescriptive, test-based evidence means you don’t need prior compliance knowledge or expertise and can pass that audit you so desperately need.
- Biggest disadvantage: If your tech stack or compliance program is even slightly complex, the pass/fail tests will provide too many false negatives and false positives to be trustworthy.
Test-Based Evidence 2.0
Many of the problems with prescriptive test-based evidence stem from a lack of context. The next generation of test-based evidence learned from its predecessor's mistakes. Test-based evidence 2.0 provides a PDF of the data the test is based on. Now, you can show the auditor that you passed the test and provide the data to back up your passing grade.
Remember the trouble we had finding the cause of our failing grade with test-based evidence 1.0? The PDF solves that as well. You can look through the data, find what needs fixing, and fix it. Sounds like the perfect solution, right? Well… not quite.
While a PDF is a step up from having no data, it’s static and can’t be customized to meet your needs. It always presents the entire dataset. So when out-of-scope data appears in the PDF, it adds confusion and impacts the test results. It’s not live data that you can slice and dice to make sure you’re only presenting relevant information. And so, like test-based evidence 1.0, you’ll still be prone to false positives and negatives.
To Summarize:
- Test-based evidence 2.0 IPE:
- The date and time the vendor conducted the test and data collection
- The pass/fail result
- A PDF with the data that supports the test result
- The API call used
- Biggest advantage: You get all of the benefits of the first generation of test-based evidence, and the PDF solves many of its problems. Both your organization and the auditor can see what data the test is based on.
- Biggest disadvantage: You’ll still get false positives/negatives and you’ll be forced to work with and present the auditor with a complete dataset, which may include data outside the scope of your compliance program.
Dataset Evidence
The final form of evidence is automatically collected raw data. Not entirely raw as in a JSON file, but a JSON file put into a table for ease of use. There are no automatic pass/fail tests with this form of evidence—but keep reading, and you’ll see why dataset evidence is still the best. (We may be biased since this is our focus at Anecdotes, but we have good evidence to back up our claim—see what we did there?)
The beauty of datasets is that they are incredibly flexible. You can filter data out that’s not in scope for a particular audit, analyze dataset evidence, search for specific information, and review certain data points—really, anything you need.
Instead of tests, dataset evidence operates on rules. Many are predefined by the tool, but you can also create your own. Going back to our earlier password example, you can make a rule requiring passwords to meet your specific criteria (12 characters long and including upper- and lowercase letters, a special character, and a number). If there are any issues, continuous gap detection will proactively tell you exactly what they are so you know exactly what ticket to open and for who.
Additionally, you can use dataset evidence for many use cases outside of passing audits (which is the easy part, believe it or not!). Use it to:
- Understand your compliance and risk posture continuously—not just at audit time
- Automate user access reviews
- Determine whether the mitigation techniques you put in place for risk management are working
To Summarize:
- Dataset evidence IPE:
- The date and time the vendor conducts the data collection
- The JSON file
- The preview table with an item count for validation against the JSON
- A hashing mechanism used for integrity validation
- The API call used
- Biggest advantage: You get all of the advantages of automation with the flexibility to tailor the evidence to meet your specific requirements.
- Biggest disadvantage: There isn’t an automatic pass/fail; you have to do some of the work yourself.
Automation That Works For You
Which of these types of compliance evidence is right for you will depend on where you are in your journey toward compliance maturity. If you’re new to compliance or work for a small company, test-based evidence from a prescriptive solution could be good enough to start. When you manage a mature compliance program, dataset evidence is the way to go.
Whatever the case, don’t be awed by the mere idea of compliance data automation. Do your research and make sure you fully understand what type of automated compliance evidence you can expect from each tool.
About the author
