SANS DevSecOps Survey 2022: 5 Key Takeaways

How DevSecOps and Developer Security Can Reduce Risk, Accelerate Release Velocity, and Save Developers Time

Over the 20+ years I’ve been working in the cybersecurity industry, I’ve been a SANS student, speaker, and sponsor and found it a great resource across many cybersecurity topics. Given how early in the DevSecOps and Developer Security journey we are as an industry, there are very few sources for best practices data sourced from practitioners. So, when we got the opportunity to participate in this year’s report, SANS 2022 DevSecOps Survey: Creating a Culture to Significantly Improve Your Organization’s Security Posture, Deepfactor was all in. The report and webinar dropped last week (9/21/2022) with updated annual data from the last survey.

The 5 Key Takeaways

In this blog, I am going to share 5 key takeaways I had from the report along with some observations of DevSecOps best practices and success criteria from our interactions with Deepfactor customers.

1. Picking the Right DevSecOps Key Performance Indicators (KPIs)

“You can’t improve what you don’t measure.” – Peter Drucker

This classic Peter Drucker quote certainly applies to DevSecOps. When the SANS Survey asked the question, “What are the major KPIs you use to measure the success of your DevSecOps activities?” there were some great KPI options to choose from and I found the changes from 2021 to 2022 data interesting.

I break these KPIs down into three themes that align with the fundamental value proposition of DevSecOps:

Risk Reduction

The DevSecOps impact on risk has several angles reflected in the KPIs above:

• Reducing the total number of vulnerabilities that make it into a production application (Number of Open Security Vulnerabilities). This was top overall KPI, which makes sense. 
• Ensuring that the security testing you are conducting has adequate test coverage (Automated Test Coverage). This KPI increased from 28.4% to 45.1% from 2021 to 2022. I believe this is a reflection of both the ability of tools to perform broader security testing and a greater role played by Quality Assurance teams in the DevSecOps process. 
• Reducing the number of vulnerabilities discovered after deployment into production. Ideally, all known vulnerabilities and risks should be at least identified in Dev and QA before going into production. In some cases, engineering and security teams may decide to use mitigating security controls or accept the risk of shipping the vulnerability depending on the severity and likelihood of compromise.  

Release Velocity

• The time it takes to fix a security vulnerability is critical from two perspectives. The longer it takes to fix a vulnerability after shipping to production, the longer that vulnerability is exposed. In addition, the longer a developer is working on a vulnerability fix, the less time they have to add other enhancements to the application. 
• Change lead time (cycle time to deploy code changes/fixes to production). This metric is more a function of how fast your release cycles are rather than how long it takes to fix the actual vulnerability. Fortunately, the report shows that 61% of respondents are releasing to production weekly or more frequently.

Engineering Time/Cost Savings

• Number of builds failed due to security vulnerabilities found can be a double-edged sword. In a perfect DevSecOps world, all the developers would have been notified of any vulnerabilities or risks during Dev and QA and resolved the issue prior to the build. However, catching the issue at build time and failing the build is often preferable to letting a high severity vulnerability make it to production (also a separate KPI). Failing a build is also the last resort to enforce the security of a build if there is a high-impact vulnerability remaining in the code. 

• False positive rates of reported vulnerabilities is a critical KPI to ensure developers’ buy-in to your DevSecOps program. Ensuring that every alert that a developer is asked to fix is accurate (not a false positive) and prioritized correctly gives developers the confidence to rely on the alerts they get. Conversely, high false positive rates can lead to alert fatigue and developers ignoring automated security alerts. 

• Build time delay imposed by security testing/reviews should be monitored closely. Ideally, your security tooling is automated and integrated into the development and testing environment so that all security and risk information is analyzed in parallel to development and testing. The old approach of scanning all the components of the application at release as a gate will lead to delays and poor performance on this particular KPI.   

• Cost to remediate audit findings actually decreased to 3.6% from an already low 6.6%. While I think this is a good KPI to measure, I suspect the reason it is uncommonly used is that it is difficult to track remediation costs. 

2. The ROI and Time Savings of DevSecOps

According to the survey results, “management buy-in” was the number one factor contributing to DevSecOps security programs’ success. Quantifying the value in terms of time saved by catching security vulnerabilities early in Dev and Test is a core principle of DevSecOps that can be used to build an ROI case for management to support your DevSecOps program and help overcome the lack of management buy-in.

What if you could say, “For each vulnerability we find early in dev and test using DevSecOps, we save an average of X days of engineering time. We have Y number of  security vulnerabilities per year. So, investing Z amount would have a strong xxx% return on investment that pays for itself in less than a year.”?

The problem is that there is very little recent data on the amount of time it takes to resolve an average vulnerability after it is released to production compared to if it is identified during coding, component testing, system acceptance testing, or after being released to production. The chart below, originally from NIST and referenced in this deep source blog, shows the exponential cost savings that can be achieved the earlier a vulnerability is fixed.

One of the fundamental principles of DevSecOps is to find and fix vulnerabilities early in development and testing because it is easier for developers to fix the vulnerability at that point when they are coding that component of the application and while it is fresh in their memory. It is much more difficult to fix a vulnerability for code that was developed months or years earlier, much less if it is a different developer fixing the issue. 

Through the SANS report and webinar, I found data that can help us start to build an example ROI case. Of course, the best ROI business cases will always use your own internal data that is most relevant to your applications and organization.

The first interesting piece of data relating to ROI was in the SANS survey but didn’t actually appear in the report (It would have been a very long report if all 31 charts in the survey were included). The chart below shows the answer to the question, “On Average, how long does it take for your organization to patch/resolve critical security risks/vulnerabilities for systems already in use?

The weighted average number of days it takes to fix a vulnerability according to the SANS survey was 21 days, while the most common number of days was 2-7 days. Of course, this will vary by application and organization. 

During the SANS DevSecOps Report webinar, they conducted a poll on the associated slack channel that asked the following question:

How much more time on average does it take engineering and QA to fix and verify a security vulnerability after it has been released to production vs. fixing it in Dev/QA before the release?

This poll, while not scientific, aligns well with the NIST chart above, with the most common response of “10 times more time” to fix a vulnerability after releasing to production. So, now we have an average time to fix a vulnerability after releasing to production (21 days), the increase/decrease in time spent when fixed in Dev/QA (10X). Now all we need is the average cost of a day’s time for a developer. According to U.S. News and world report, the average Software Developer makes $110,140 per year. Generally, benefits and overhead adds an additional 20%. So, let’s call it $132,168 as an annual cost to the company per developer. The average number of working days in a year is 260, which puts the cost per day of a developer at $508.

[Side note: If you know any good developers who make $110,140, Deepfactor would like to hire them.]

So, here is an example of a basic ROI case for DevSecOps:

• 21 days on average to fix a security vulnerability after it is released to production
• $508 average cost per day for a software developer
• 21x$508=$10,668 to fix each vulnerability
• 21 days/10X reduction in time to fix vulnerabilities in Dev/QA=2.1 days to fix a vulnerability
• 1x$508=$1,067 to fix a vulnerability in Dev/QA
• Assuming 100 security vulnerabilities/year, the cost $1,066,800 without DevSecOps vs. $106,700 with DevSecOps=Cost Savings of $960,100/year
• If it cost $500,000 to implement the tools, processes, and train your developers on DevSecOps practices, the 3-year return on investment would be 3x$960,100=$2,880,300 and the ROI would be $2,880,300/$500,000=576%
• The payback period on the DevSecOps investment would be 190 days.

While your mileage and the variables used above may vary, you can see that there is a compelling business case to be made for DevSecOps. In addition, there are also quantifiable cost savings that can be associated with reducing the risk of a breach, avoiding release delays, and the opportunity cost of developers being able to work on other revenue generating features or applications instead of spending more time fixing security vulnerabilities.

3. DevSecOps Key to Success

According to the Secure DevOps and Misconfigurations 2021 Report by the Cloud Security Alliance, only 30% of security professionals claim to have “fully implemented DevSecOps,” meaning the 70% of security professionals are still working on fully implementing DevSecOps. That’s why I found the following question posed by the SANS 2022 DevSecOps survey interesting:

Again, with so many options, I think it is helpful to break these success factors into three success-factor themes:

Culture and Education

When I talk with Deepfactor customers, shifting the culture of development to view security as part of delivering quality code is one of the most challenging and critical parts of successfully implementing DevSecOps. This requires a combination of both security and engineering management buy-in, individual developer level buy-in, communications between Developers, Security, and Ops teams, and security training that re-enforces secure coding practices.

Key Success Criteria Relating to Culture and Education:

• Securing management buy-in
• Improving communications across Dev, Ops, and Security
• Securing developer/engineering buy-in
• Training developers/engineers in secure coding

For securing developer/engineering buy-in, many industry analysts and customers I’ve spoken with recommend implementing a “Security Champions Program” or “DevSecOps Center of Excellence” as a strategy. These programs recruit a developer from each team to act as a security champion and receive additional training that they can then impart to their peers and be an advocate for security embedded into each engineering team to speak up about security issues. For more reading on how to implement a security champions program, I recommend the following resource:

Forrester Report: Build a Developer Security Champions Program

When it comes to security training, Deepfactor customers have reported the best results through a combination of continuing education courses (See SANS SEC540 Cloud Security and DevSecOps Automation Course) and automated tooling that provides just-in-time alerts when developers implement applications in a way that introduces risk.

Tooling

Tooling for DevSecOps is a critical piece of the puzzle to get DevSecOps right. There are several different types of security tooling required to provide complete coverage and reduce risk. One critical thing to remember about DevSecOps tooling is that it should integrate well into the existing engineering tool chains and not require developers to leave their existing tools. If you use Jira to track and prioritize bugs, then the tooling should integrate with Jira as the way for engineers to consume alerts and resolve issues. Don’t make your developer log into multiple other tools or you risk losing engineering/developer buy-in.

Key Success Criteria Relating to Culture and Education:

• Integrating automated security testing into developer/engineering tool chains and build/deploy workflows.

In the example below, Deepfactor provides a plain english description using an “Unsafe string API,” including detailed resolution information on how to accomplish the same thing in a more secure way, and the stack trace information to pinpoint the code that needs to be changed. This same information can be automatically assigned to an engineer through Jira or other tools using Deepfactor APIs. To learn more about Deepfactor, watch a 20-Minute Speed Demo of Deepfactor.

Automation and Prioritization

One of the biggest challenges in securing developer buy-in is “Alert Fatigue.” With developers struggling to understand and triage “noise” and false positives, they spend more and more time reviewing and investigating alerts that provide a diminishing value to the organization. Overwhelming engineering teams with security alerts often leads developers to ignoring them entirely. In some cases, the flood of information can delay—and even disrupt—remediation efforts as developers struggle with understanding the scope and impact of the issue. We see the problem often in software composition analysis (SCA) tools designed to show a list of all vulnerable dependencies based on scanning a code repository. The solution to “Alert Fatigue” lies in automating the tooling, showing only alerts relevant to individual developers, and prioritizing based on risk and usage.

Key Success Criteria Relating to Culture and Education:

• Automating build/test/deploy/provisioning workloads, and thereby minimizing the time/cost to fix vulnerabilities.

Make sure that your DevSecOps tooling is set up in a way that assigns only the alerts that are relevant to that developer and prioritizes those alerts automatically based on the risk of the vulnerability and usage of that component in the code path. There are often many vulnerabilities that exist in dependencies or components of the application that are never used and therefore should be a lower priority. The Deepfactor approach to prioritizing based on usage is to observe the running application in the QA or test environment and correlate which of the vulnerabilities existing in components that were used when QA is testing the application.

4. Automated Compliance Checking and Enforcement

“So I’ve recently started advising my clients not to write a single security policy without having a means to check it automatically. And this can mean things such as password complexity. Through the whole gamut. So, because we need to be able to inspect what we expect, and the only way that this is going to scale because our companies are not hiring more security engineers at the same pace that they’re hiring developers or employees is we put in a means to be able to do this in an automated manner.”

Kenneth G. Hartman
SANS 2022 DevSecOps Survey: Creating a Culture to Significantly Improve Your Organization’s Security Posture, 9/21/2022

Compliance is sometimes an afterthought when it comes to DevSecOps. Most compliance work is focused on the infrastructure and operational aspect of running applications but there are also compliance implications in the development of an application. Whether it’s SOC 2 Type 2, PCI DSS, ISO 27001, or another compliance standard that is important to your business, put some thought into building compliance checks and enforcement mechanisms into the DevSecOps process.

In the SANS 2022 Survey data, you can see that there was a big jump from 2021 to 2022 in the number of respondents who checked/enforced 100% of their compliance policies. However, that percentage is still only 18.4% of the total that fall into that desirable category. Finding compliance issues in development or QA through DevSecOps will be much less costly to remediate and will avoid any delays or penalties that could result from a bad audit finding or compliance violation in production.

Deepfactor automatically maps all security vulnerabilities found in application code, dependencies, container images, and web/API interfaces to compliance standards and the section of the standard the vulnerability violates using the secure controls framework. With that information, developers and engineering leaders know immediately when a vulnerability alert has a compliance impact. To read more on this topic, visit What Developers Need to Know About the Impact of Compliance Frameworks on Software Development.

5. Most Useful Testing Practices and Tools

The SANS 2022 DevSecOps survey included asking respondents what security testing practices and tools they found useful on a scale of Not Useful, Useful, and Very Useful. There were 26 different security testing practices and tools ranked in the report and I encourage you to review them all on page 12-14 of the SANS report. Here are the top 5 most useful with the percentage of people who ranked the practice/tool very useful or useful based on the survey:

1. Web Application Firewalls (WAF) (84%)
2. Periodic Vulnerability Scanning (83.3%)
3. Secure Coding Training For Developers and Engineers (82.3%
4. Automated State Analysis (82%)
5. Continuous Vulnerability Scanning (79%)

With so many tools and options, I think having this independent view of which tools are most useful is helpful in selecting the tool set and security practices you build your DevSecOps program around.

DevSecOps tooling is one of the areas where Deepfactor can help. Below is a list of the most useful security testing practices and tools from the SANS 2022 Survey that Deepfactor can help address:

Conclusion

I’d like to thank SANS and the authors of the SANS 2022 DevSecOps Survey Chris Edmundson and Kenneth G. Hartman for providing such valuable information to the DevSecOps community so that organizations can apply best practices to fully realize the risk reduction, release velocity, and time/cost savings potential of DevSecOps. We are relatively early in the DevSecOps journey and there is a lot of work to do as an industry to complete the shift left represented by DevSecOps and improve our collective security posture. I hope the key takeaways I summarized from the report provided some helpful insights and was a catalyst to continue researching and implementing DevSecOps in your organization.

If you have any questions or I can help you in any way, please feel free to connect with me on LinkedIn.