Continuous Security

Continuous integration (CI) and continuous delivery (CD) are together the most powerful hidden impact that Agile can have on a product development or engineering organization. Why is it hidden? The benefits of CI/CD are abstract, but the cost of getting the team, skills, and automation in can be daunting. In addition, the work required to get the needed automation in place and working reliably requires dedication to detail to master the tools used and is often viewed as thankless and uninteresting work. The success of “DevOps” has helped to change attitudes around this, but it is still true that many teams are still using manual build and deployment processes. However, investments in CI/CD can be leveraged to provide continuous security as well.

Continuous Security Is an Investment in Freedom

The power of this investment in automation is simple: if you make builds “free”, then you take away the most common reason to avoid them. Building and deploying freely allows the team to see the impact of their changes immediately, both by allowing them to see that the build is broken and by allowing their work to move quickly into the hands of manual or automated testing, where other runtime issues can be discovered as well. This rapid feedback reduces functionality defects and rework by making it possible for product owners and early adopters to see the work early and often. The improved transparency and flow leads to confidence and a better relationship with your internal customers or clients.

In our experience, all of the same arguments can be applied to security as a critical feature of both the code that is built and the infrastructure on which it is deployed. While regulatory or compliance mandates for your product may only be required on a periodic basis (often annually or monthly, or perhaps with each release you deploy to production), there is no better way to pass that audit than to already know that you expect to pass.

While passing audits is an important motivator, the real win is to sleep well between the audits because you are confident in having continuous processes in place that ensure that:

• Newly created vulnerabilities are remediated by the development team before the code gets deployed to production, and
• Newly identified vulnerabilities in your operating system, infrastructure, platforms, and application code are being raised and remediated as fast as needed based on the severity of the vulnerability.

Integrating Continuous Security

Here are a number of steps your team can take to integrate a continuous security mindset into their daily lives. Together they combine to improve the security awareness of your entire stack from the servers, virtual machines, containers and networking that hosts either your application servers or your container cluster, as well as the code you build that runs in these environments. The best approach is to start with your lower development and test environments until you are confident that your applications are working properly in those environments, and then you can take the big leap to migrating your production environment.

• Start with a hardened Center for Information Security (CIS) Level 1 or higher base image—let them ensure that your operating system has a minimum set of services and is routinely patched. Use this base image to generate your own “golden” image that can be patched as needed in between vendor updates, and can use scripting to add or change any remaining OS issues that arise from your scans.
• Use automation to create and deploy your application containers using “infrastructure as code.” The best way to ensure that your stack is hardened and robust is to recreate it from “golden” container images.
• Automate the scanning of your container images in your CI/CD/CS pipelines so that every build will fail if vulnerabilities are found.
  Schedule routine scanning of your golden container images to make sure new vulnerabilities reported are caught quickly and can be fixed in your golden images with minimal delay.
• Build static code scanning into your CI/CD pipeline. If it is too time consuming to do on every build, then schedule a build to run overnight each night that will fail if new vulnerabilities are found.
• Consider also using dynamic scanning tools to help prioritize your library and code vulnerabilities by factoring in the frequency with which these code paths and libraries are executed by automated or manual testing. Typically these tools have too much impact on performance to be used in production at scale, but deploy them in all your lower environments where manual, automated, and user testing is happening.
• If your cloud vendor provides this platform as a service and has the needed certifications and attestations for it, use their container hosting infrastructure to manage scale up/scale down and rolling builds to ensure that all containers in use are based on the latest known good code and operating system builds.

Prioritizing and Managing Alerts

Now that you have the basic automation and instrumentation in place, you have created DevSecOps pipelines that drive alerting and cooperation between your DevOps and Development teams to work together to react and respond to problems. However, you still face a prioritization challenge: These tools are going to provide a lot of frequent reminders of what issues need to be addressed to your engineering teams, but without proper prioritization they could still go unaddressed. Both product owners and developers would rather spend time building cool new features than addressing security issues, so you need to consider how to get the visibility and governance that you need to “trust but verify”.

This is where a Security Information and Event Management (SIEM) tool or managed service will help. The most difficult but crucial step is to have your DevSecOps team work through integrating all of these log and event sources into your SIEM. This has both operational and visibility benefits. Using a SIEM allows you to consolidate all of the information coming from these various sources into one location where they can be triaged and dispatched. You will want to centralize and automate these steps, because any process that depends on humans to have the discipline to routinely check numerous vulnerability sources to look for trouble is eventually going to disappoint you. Here’s a high-level overview of how all of these elements come together:

With this SIEM, you also now have access to alerts yourself that can help you focus the team on high-severity issues quickly as they arise, and to reports that can summarize the overall security readiness of the product and team. I have used this combined with other functionality and stress test results to help the team make the decision about whether a new release is ready to get promoted to production. The good news is that in most situations, the alerts have kept you apprised of problems as they arise and the team has already reacted to address high-severity problems before the review, but don’t allow this to let your guard down—contrary to the popular maxim, in this case a pot boils best if it is watched.

Continuous Balance

One last piece of advice: You will find that it is easy for you and your team to get discouraged by the amount of feedback you are now receiving about your security posture, and it can seem daunting to climb the mountain. You need to be aware that this is happening and can cause the team to reject what they are seeing because it’s overwhelming to try to achieve what your security policies suggest should be enforced as the standard. With that standard in mind as the goal, I have a couple of suggestions for how to manage through it:

• Don’t divert too much of your product team’s velocity to security unless you know it is absolutely necessary. You may be in a situation where you have to pass an audit or there are severe consequences, and this might be an example where you have to focus a lot of the team’s energy on resolving those findings, but keep in mind that most audit processes include a process for demonstrating that the issue does not have the risk or business impact that is suggested and an exception can be made. In general, try not to sustain more than about 10 percent of your sprint velocity on tackling the known vulnerabilities, and use severity to order the backlog so that you know you are always working on the highest-risk items.
• Assuming your application is already in use in production today, keep in mind that you will lose the support of your team if you tell them that they can’t ship again until the product meets your security policy. Instead, ask them with each release, “Is this release more secure than what is in production today, and how do you know?” If they can answer this question with a confident “Yes!” and provide you with the supporting data to prove it, then you now have a continuous improvement process in place that will eventually give you what you want.

The total security of a system involves so much more than what was discussed in this article—there are all of the usual considerations about encryption of data over the wire or wireless networks and at rest, firewalls, penetration testing, passwords in code or in the clear, etc. In my experience, this is usually where teams start, and they usually have addressed the most serious of these concerns before Continuous Security makes sense as the right next investment to make. Start there if you are not already certain these things are in control, but come back to Continuous Security when they are.