It’s been almost a decade since Docker and containers changed the landscape of modern software delivery. Early in their evolution, there was a lot of fear around container security because they didn’t have a lot of the familiar architectural security elements when compared to VMs. Today, mature tooling and processes along with a widespread understanding of the risks and mitigation have persuaded many holdouts to adopt containers as another tool in their infrastructure strategies.
This blog takes an updated look at the most mature and popular tools in the various stages of the container hardening process, giving you 7 strategies and related tools to try as you head into 2023. The strategies covered are:
- Container Image Creation
- Minimal Containers
- Container Image Scanning
- Policy Enforcement
- Container Encryption
- Container Signing
- Host Operating System Vulnerability Scanning
1. Container Image Creation
Container creation requires an understanding of the software packages, users, and environment settings that are necessary to deploy a workload. Container builds use a manifest of instructions for building containers, which are read and executed sequentially to generate a new container.
A variety of tools can produce containers:
Packer is a favorite of mine and I’ll use it as an example. It builds containers using the Docker builder plugin. An advantage of using Packer is having a common workflow for building operating system golden images as well as containers. Packer also has a "post-processor" capability. The post-processor creates a framework for container tagging, encrypting, signing, and container scanning. Packer can then push to artifact repositories after the build and post-processing steps.
Once the build step is complete, a container is pushed to an approved repository. To push to approved repositories, users should have an authentication and authorization step to grant access to a container repository. The credentials for accessing container repositories can then be stored in a secrets manager.
2. Minimal Containers
A security benefit of using containers is having a limited number of operating system packages inside a container. Containers with a precise set of software packages decrease the number of possible security vulnerabilities in a container. In the last few years, there has been an increase in the number of minimal Linux distributions for containerization, and you may want to consider adopting some of them if you’re looking for more ways to improve your container security posture.
A few of the container images that use very minimal operating systems with just enough packages installed to run include:
Another way to limit the size of containers and the number of unnecessary software packages is to do multi-stage builds. A container that serves a Golang REST API would have two stages in the multi-stage build:
The primary stage would be labeled as the compile stage. The compile stage needs an operating environment with enough packages to build the Go binary.
The second stage would use a minimal base operating system and the Go binary created in the compile stage.
Minimal containers, whether they’re ready out of the box or homegrown via multi-stage builds, minimize your threat surface because there are fewer packages and operating system components to exploit.
3. Container Image Scanning
Scanning container images is a critical security step in the container lifecycle. You can run and enforce image scans at different stages of the software delivery lifecycle. Scans can execute…
- Before an image is pushed to a repository
- In the image repository
- Before running in an orchestration environment like Kubernetes
There are many popular scanning tools to choose from. Some of the popular options include:
- Prisma Cloud
- Snyk
- Trivy (GitLab's default container scanner)
- Grype
If you want to scan a container before an image is pushed to a repository, an image build tool like Packer can do this during the build process, scanning images and checking for container vulnerabilities.
If you want to scan a container when it’s in the image repository, tools like Sysdig and Jfrog are able to scan a container repository and monitor security drift as containers become stale and new common vulnerability and exposure reports are released on containers.
Scanning tools can also continuously monitor web feeds for new vulnerabilities and send alerts when an image is out of compliance. At the enterprise level, there are tools that go even further and automate base-image upgrades when they find vulnerabilities.
Terraform, the most commonly used infrastructure as a code provisioning tool can integrate with container scanning tools to ensure that images have been scanned before being deployed onto specific architecture. And if your company runs the commercial versions of Terraform, they can use native or third-party tools and guardrails to automatically scan and enforce rules on container images before running in an orchestration environment (e.g. Kubernetes, Docker Swarm, and Nomad).
4. Policy Enforcement
Policy checks are a great way to enforce security standards on a system as part of the software delivery lifecycle. In order to add a check before allowing a service to be run in Kubernetes, secure environments require additional gate checks called admission controllers.
The admission controller can scan containers for vulnerabilities and check for compliance at runtime. These gate checks should be codified with policy as code (PaC). Just like infrastructure as code, policies can be tracked, version controlled, and reused like modules or functions in programming languages. Example policies could include:
- Validation of container signatures for Software Bill of Materials
- Enforcing approved container registry whitelists
- Container scan severity thresholds - (Prohibit the deployment of services that have high severity CVEs in packages)
- Enforce the use of necessary tags or labels on services
- Ensuring container applications are not running as the root user
Look for a policy as code or other guardrail features in the tools you use. For example, if you want to run checks and enforce them on containers before they’re deployed to an orchestrator, use a native feature like admission controllers in Kubernetes or Sentinel policy as code in Nomad (enterprise version). Terraform can also use Sentinel (enterprise version) to enforce policy before deploying a container onto a single instance server.
5. Container Encryption
Encryption is an additional security measure with an authentication and authorization step before the manipulation of a container. When using encryption tools for containers, the container itself is encrypted to include all the internal files and the metadata associated with the container. This encryption happens at the build step before being pushed to an external repository. Typical key encryption formats are: JWE (RFC7516), PGP (RFC4880), and PKCS7.
The encryption key can be stored in a secrets manager. Vault is getting the broadest adoption but there are other options here as well:
- Azure Key Vault
- AWS Secrets Manager
- GCP Secrets Management
These secret managers can also add an authentication step to encrypt and decrypt containers. CI/CD platforms, container scanning tools, and other services also need access to the encryption key stored in the secrets manager, so make sure it has a broad integration ecosystem. Vault, for example, integrates with open-source and commercial tools to add an encryption and decryption step into the software delivery lifecycle. Skopeo (a tool for moving encrypted image repositories), Singularity (for building and running encrypted containers), and Containerd's Imgcrypt (to encrypt and decrypt Open Container Initiative (OCI) containers) are examples of tools that can add a cryptographic workflow in your container image pipeline.
When importing containers from outside sources, it’s important to have tools that can encrypt them and keep them up to date with vulnerability patches. Running container scans and encrypting containers before they are pushed to trusted sources are two great ways to continuously secure your software supply chain and image repository.
To run a container in an orchestration pipeline, the service needs to authenticate and be authorized to access the decryption key. This additional step in the pipeline adds file encryption and provides least-privilege access to the container at runtime.
6. Container Signing
Securing the software supply chain is critical for defending customer data, especially with the industry’s ubiquitous use of open-source software. Many organizations like SLSA and OpenSSF’s governing boards have created frameworks for measuring the security and integrity of the software supply chain. Containers can have digital signatures applied to commits as metadata with tools like Sigstore's cosign.
Additional metadata like the author of the commit or vulnerability scan results can be used as a validation gate check before deploying to a container orchestration platform. The commit signature can use a self-signed key with cosign to verify the authenticity of the steps or allow integrations with external public key infrastructure (PKI) servers. In PKI workflows, Vault can act as a certificate authority and generate PKI certificates programmatically.
The benefit of using Vault certificates for signing a commit is the authentication process. Authenticating before each commit affirms that the human or machine executing the signing process is authorized to sign container images. Dynamic, short-lived PKI certificates shorten the threat window in which an adversary might gain access to a key and use it to inject bad code or improperly sign a container.
Tools like Sigstore's Rekor add a timestamp of the signature verification to an immutable ledger. This ledger can be used to validate the signature authenticity even after the short-lived certificate has expired.
7. Host Operating System Vulnerability Scanning
Vulnerabilities in the host operating system provide threat vectors to the containers running on the host system. Hence, the host operating system running the container needs to be monitored and scanned for vulnerabilities. OS scanning tools have been around for ages but some of the popular commercial and open-source host vulnerability scanners are:
- Palo Alto’s Prisma Cloud
- Snyk
- OpenSCAP
- OpenVaS
- NMAP
It’s good security hygiene to keep operating systems up to date and secure but it can be a pain to manually update hosts consistently. One way to speed this up is to automate the process and treat your host systems like cattle vs. pets.
Container orchestration platforms allow the host operating system to become commoditized and much easier to cull. If configured correctly, host operating systems can be shut down and containers will ultimately find a new home on an available server in the cluster with the appropriate amount of CPU, memory, and disk.
As operating systems are scanned and CVEs are released, automated triggers from scanning tools can kick off a toolchain, like Packer (image builder) and Terraform (infra provisioning), to quickly patch an entire cluster of servers with no downtime for running services. Finding a workflow to get patching quickly will ensure there aren’t any underlying attack vectors allowing access to the container from the underlying operating system.
Final Notes
Container security is a must in modern cloud architecture. Hopefully, this list of current tools will help improve your process for securing the creation, signing, encrypting, verifying, validating, provisioning, connecting, and running of containers.
More strategies?
Did I miss anything? Let me know in the comments!
For more information, check out the following resources: