Threat model and security principles

• Shift-left: detect and prevent issues during build and CI/CD.
• Least privilege: for images, containers, Kubernetes RBAC, and network policies.
• Defense in depth: multiple controls at build, deploy, runtime, and network layers.
• Immutable artifacts: build once, sign, and promote artifacts to environments.
• Observability and fast response: logging, auditing, and runtime detection.

Build-time and supply chain security

Fix many issues before runtime by securing source, dependencies, and images.

1) Harden Dockerfiles and images

Examples and best practices:

• Use minimal base images (e.g., distroless, Alpine) to reduce attack surface.
• Specify explicit versions for base images and packages.
• Run processes as non-root user inside container.
• Remove build-time tools from final image (multi-stage builds).
• Pin images by digest when deploying (immutable reference).

Example Dockerfile (multi-stage, non-root)
FROM golang:1.20-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGOENABLED=0 GOOS=linux GOARCH=amd64 go build -o app ./cmd/service


FROM gcr.io/distroless/static-debian11
COPY --from=builder /app/app /app/app
USER 10001
ENTRYPOINT ["/app/app"]
    

Example: scan built image with Trivy in CI
trivy image --severity HIGH,CRITICAL --exit-code 1 --no-progress myrepo/myapp:sha256-...
    

Sign an OCI image with cosign
cosign sign --key cosign.key ghcr.io/org/myapp@sha256:...
Verify before deployment
cosign verify ghcr.io/org/myapp@sha256:...
    

Example pipeline stages

Checkout & lint -> 2. Unit tests -> 3. Dependency scan -> 4. Build image ->
Image scan (fail on high/critical) -> 6. Sign image -> 7. Push -> 8. Deploy with gate

    

OPA/Gatekeeper sample constraint snippet (conceptual)
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
name: k8sdenyprivileged
spec:
crd:
spec:
names:
kind: K8sDenyPrivileged
targets:

target: admission.k8s.gatekeeper.sh

rego: |
package k8sdenyprivileged
denymsg {
input.review.object.spec.containers[].securityContext.privileged == true
msg = "Privileged containers are not allowed"
}
    

Logging, auditing, and incident response

Observability is critical for detection and fast response.

• Export Kubernetes audit logs to a centralized location and monitor for privilege escalations or abnormal API calls.
• Collect and aggregate container logs and metrics (Prometheus, Grafana, ELK/EFK) and create alerts for anomalous patterns.
• Use EDR and runtime anomaly detection to detect suspicious behavior inside containers.
• Have a playbook: isolate workloads (network policy or node quarantine), revoke compromised credentials, roll back to signed images, and investigate using audit data.

Practical checks and quick wins

Actionable checklist you can apply quickly:

1. Enable image scanning in CI and block builds with high/critical findings.
2. Sign images after build and verify signatures before deploy.
3. Run containers as non-root and set read-only root filesystems.
4. Enforce NetworkPolicies to limit cross-service communication.
5. Use a secrets manager with short-lived credentials and avoid storing secrets in plaintext in git or images.
6. Restrict Kubernetes API access with least-privilege RBAC and enable audit logging.
7. Install runtime detection (e.g., Falco) to alert on suspicious container behavior.

Example workflows — from build to runtime

Two short workflow examples that tie recommendations together.

CI/CD pipeline flow

Build -> Static tests -> Dependency scan -> Build image -> Trivy scan -> Sign image (cosign) -> Push to registry -> Deploy via GitOps (ArgoCD/Flux) which verifies signature and image digest -> Post-deploy runtime policy checks (OPA) and runtime monitoring (Falco).

Compromise containment scenario

If a container is suspected compromised: (1) Use NetworkPolicy or service mesh to isolate the pod/namespace, (2) scale the pod down or cordon the node, (3) revoke any active credentials/keys used by that workload, (4) promote a signed clean image (rollback), (5) analyze audit logs and runtime alerts to determine root cause.

Common pitfalls to avoid

• Relying solely on images being "small" — even small images can contain vulnerable libraries.
• Leaving default service accounts with broad permissions in namespaces.
• Baking secrets into images or storing them in repo or plain Kubernetes Secrets without encryption.
• Assuming network isolation exists by default — you must explicitly define NetworkPolicies.
• Not enforcing signature verification or image provenance at deploy time.