Container Security: Image Scanning, Non-Root Execution, and Pod Security Standards

The majority of teams approach container security by simply taking it down to a checklist. Once the photo is scanned, check the box, skip to the following image. However, for those who have worked on any real Kubernetes cluster, you will immediately see the posture of the base image that you installed 3 months ago fade away when you now see a new CVE.

This is not a beginner is overview. For most of the RAM and disk space, it’s an in-depth look at the state of container security today in the three key areas: image scanning, non-root execution, and Pod Security Standards. But, and the most important, where each of these is going.

The State of Container Image Scanning – Solid Foundation, Shifting Expectations

There has been a long time in which to develop image scanning and most teams will have something in place. Useful tools such as the Trivy, Clair, Anchore and Harbor are well-established, well-published, and fit seamlessly into CI/CD pipelines.

The conventional approach is as follows:

• Scan at build time to prevent issues from being committed.
• Scan on push to the registry to prevent non-conforming images
• Do repeated scans within the registry – new CVEs are discovered after deployment.

A place where many teams fail and that is the last point. In my experience, most pipelines take care to gate on the build-time scan and do not modify registry images for months. This is not a safe position, it’s a false sense of safety.

What Scanners Actually Catch (and What They Miss)

Scanners are effective at detecting known packages and libraries for OS packages that contain CVEs. They can highlight obsolete base images, mal-configured layers, policy violations (such as exposed ports or using root) and a whole lot more, including things that are actually baked in the dockerfile.

Instead, they do not catch them:

• Runtime exploits- takes from behaviour, not static code.
• A misconfigured RBAC and/or weak admission control
• Vulnerabilities in YOUR application code (not typing them on the command line!)

This is explained in DevSecOps guideline by OWASP (Container Vulnerability Scanning): scan results have no value unless combined with an actual DevSecOps remediation workflow. If there’s no triage, prioritisation, and perhaps fix cycle, then it’s merely generating reports no one takes any actions on.

The Shift Toward Supply-Chain-Aware Scanning

It’s a place where everything is actually changing. Pure CVE scanning is transitioning to full supply-chain security, where software Bills of Materials (SBOMs) and image signing using tools such as Cosign are joining part of the CVE scanning process.

The most common pattern in real world production spaces:

• No image types can run without signature and verification (only signed and verified images) (at admission)
• All pictures to be used have to be from approved and internal registries.
• The results of those scans are compared with runtime information obtained by means of telemetry to provide real world relevance, rather than CVSS score priority.

This is not just theory – admission admission tools such as Kyverno and OPA Gatekeeper are now able to do admission enforcement, and teams are deploying them. For anyone that is working with Kubernetes Security on any level, admission layer is a must.

Non-Root Execution – Still Ignored More Than It Should Be

There are a number of things people know they should not do, but still do; such as running containers as root. Typically for it to be easier. The base image runs as the root user, the app requires some file paths, no one wants to debug permission issues, etc. Hence, there is a lack of securityContext setting.

This is a problem I see over and over in internal audits, not because engineers don’t know better, but because it’s the way it is, and there’s nothing to force them off their feet.
How Non-Root Enforcement Actually Works is a book by Don’t Root.

How Non-Root Enforcement Actually Works

At the Dockerfile level, these steps create an unprivileged user and switch to it:

RUN addgroup --system appgroup && adduser --system --ingroup appgroup appuser
USER appuser

At the Kubernetes level, you enforce it through securityContext:

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  runAsGroup: 3000
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
      - ALL

The focus on the capabilities.drop: ALL line is more important than often realised. The capabilities of Linux are finely-grained privileges, such as “NET_BIND_SERVICE”, “SYS_ADMIN”, or “CHOWN”. Severe on the fact of it is that once the capabilities are dropped by default and only added back when they’re truly required by the workload, you’re applying actual least-privilege logic.

Almost all guides will say more about dropping than on anything else. Because of my experience in real life, my workloads were few, typically one to two capabilities, and most never use any capabilities. This is a tedious audit but worthwhile on a per service basis.

What “Cluster-Wide Enforcement” Looks Like in Practice

Thinking around non-root isn’t only “the way to deploy to make one’s configuration.” Making it a hard requirement at the namespace level so that teams cannot avoid it if they don’t want to explicitly.

This is where Pod Security Standards fit in – interrelated with the next section. But, the cultural shift is important as well: non-root templates should be used for new workloads, rather than something that a security team will chase down after the fact.

If you’re wondering about how this relates to network level controls, you might want to read along with Network Policies, which is covered in the Kubernetes Network Security documentation, and is a separate layer on top of workload hardening, but still an effective complement to it.

Container Security and Pod Security Standards — The Enforcement Layer That Actually Has Teeth

Pod Security Standards is a more powerful alternative, replacing the old PodSecurityPolicy (PSP) which was difficult to make to work properly. As of Kubernetes 1.25, PSP has been completely removed. PSS offers you 3 profiles:

Labeling Namespaces and Understanding Enforcement Modes

Labels are used within PSS to create its namespaces:

labels:
  pod-security.kubernetes.io/enforce: restricted
  pod-security.kubernetes.io/audit: restricted
  pod-security.kubernetes.io/warn: restricted

There are 3 modes: Enforce: Drops non-compliant pods, Audit: Logs violations and does not drop pods, Warn: Warnings are displayed to the user. The process of enabling starts with audit in production, and then enforcing when workloads are updated.

Restricted Profile requires:

• Non-root user execution
• No privilege escalation
• Dropping all of, and a few being added to the capabilities (note the additions are for specific purposes only)
• It is highly recommended to use a read-only root filesystem.Use of a read-only root filesystem encouraged.
• Seeking a host path volume failed to return any host path volumes for sensitive paths.
• No privileged containers

The results of my testing with a cluster of dedicated test site machines with mixed workloads: approximately 30-40% of the deployments fail the ‘restricted’ profile on the initial pass; most of these require to have either a runAsNonRoot attribute or a NET_BIND_SERVICE attribute for services that bind to port 80. Both are answerable, but both call for intentional adjustment(s).

PSS + Policy Engines – Why You Need Both

You will find a good baseline of information in the PSS, but it does not go as deep as you’d like. Not possible to impose “images must be drawn from our internal registry” using PSS. Namespace labels do not enforce resource limits, required labels or runtime classes.

This is where Kyverno and OPA Gatekeeper can help. These policy engines enable the authoring of custom admission rules in code and, in an increasing fashion, teams are using them to manage their policies using the same review process as for their applications — hence, GitOps.

The material inside a layered stack has the following form:

• Enforcement of the workload hardening profile at the namespacer level by PSS
• Kyverno/Gatekeeper – custom rules for registry trust, resource quotas, labels, naming conventions
• Use Image scanning to continuously scan images through CI/CD gates and registry.Leverage image scanning to continuously scan images through CI/CD gates and registry.
• Monitoring at runtime – Falco, various tools that look for things that are abnormal after things go online.

Lack of Cluster-Level Policy Enforcement (K04) is one of the common concerns in real-world Kubernetes clusters, as highlighted by OWASP’s Kubernetes Top Ten. The integration of PSS and a policy engine directly fills that gap.

This is related to larger (cluster) design principles as well. The 4Cs of Kubernetes Security – Cloud, Cluster, Container, Code is worth reading in its entirety as it is a good model to understand the place of each of these elements in the overall stack.

My Take on the Gaps Most Teams Still Have

It’s easy to “talk the talk” when it comes to this. The gaps in practice are more specific.

Gap #1: Leaping without a remediation process. Teams look at images and only notice 200 medium-severity CVEs, have no knowledge of which of these are actually a risk in their environment and effectively ignore the results. The problem area for scanning programs is not just with prioritizing by CVSS, but by prioritizing by exploitability.

Gap 2: Non-root set at pod level but not at container level. For kubernetes: runAsNonRoot can be explicitly defined in the pod spec, but each individual container can override it. A container may be running as root even if it is not configured as such, if you are not also setting it as such.

Gap 3: PSS was set with a baseline in production because of the restriction of “broke things”. This is the most prevailing same one. While baseline is good, it is not mandatory non-root but rather it simply prevents known paths for privilege escalation. Baseline is a good starting point but it’s not the end of the road when it comes to workload hardening.

Gap 4: Lack of a strategy for secrets. Even hardened workloads can be breached when secrets are not properly managed. As you put together a complete security posture, Secrets Management in Kubernetes should be a logical next step, following image scanning and PSS.

A Practical Three-Phase Rollout

A sensible starting framework for teams based on inconsistent or mixed baseline is:

The phase 1 is getting the basics right.Phase 1 is about getting this all settled.

• Fails builds on critical/high CVEs; Deploy Trivy in CI.
• Enable the default securityContext that will be assigned to new workloads (non-root, drop ALL capabilities, read-only root filesystem)
• All namespaces with some level of enforcement, and audit for restricted.

In Phase 2, harden and automate.In Phase 2, harden and automate.

• Move a production’s namespaces into a restricted phase.Move production namespaces into a restricted phase.
• Install Kyverno policies to enforce trusted registries and privileged containers disallowing
• Enlist “Registry Scanning” to continuously monitor images after deployment.

Phase 3 is a follow-on to Phases 1 and 2, in which compliance and monitoring are tied in.

• For PSS maps to CIS Benchmark controls and NIST guidance
• Be sure to add runtime monitoring (Falco or another) to detect what static controls cannot
• Implement a feedback loop: “input” should be CVEs found to address first at runtime

Two external references from books to be found that can be trusted to strengthen their beliefs.
If you want to explore this subject further and would like to read more authoritative and depth articles, then the two resources below are worth your while:

• OWASP Kubernetes Top Ten
• Kubernetes Official Pod Security Standards Documentation

Wrapping Up

Container Security using Image Scanning, Non-Root Execution, and Pod Security Standards isn’t simply a problem to be solved; it’s three different layers, each of which has its own maturity curve.

Picture scanning has become a fundamental element today, and it will continue to develop into supply-chain consciousness. There is a good understanding of non-root execution and it is inconsistent in its use. Pod Security Standards works best in conjunction with policy engines and rollout plans for the configurations of existing workloads, providing an opportunity to enforce with real power.

It isn’t always the teams with the most tools who are right. They have definite defaults which are enforced at the platform level, and a remediation process which actually gets executed. From there, work outwards.

Pranay Sai Aduvala

I’m a technology writer with a passion for AI and digital marketing. I create engaging and useful content that bridges the gap between complex technology concepts and digital technologies. My writing makes the process easy and curious. and encourage participation I continue to research innovation and technology. Let’s connect and talk technology!