28. Security Best Practices#
flowchart TB
subgraph Supply["Supply Chain Security"]
SBOM[SBOM Generation<br/>Syft]
Sign[Image Signing<br/>Cosign / Sigstore]
Verify[Admission Verification<br/>Kyverno / Connaisseur]
end
subgraph Image["Image Security"]
Minimal[Minimal Base Image<br/>distroless / scratch]
Scan[Vulnerability Scanning<br/>Trivy / Grype]
NoRoot[Non-root User]
end
subgraph Container["Container Runtime Security"]
RO[Read-only Filesystem]
Seccomp[Seccomp Profile]
AppArmor[AppArmor Profile]
NoCap[Drop ALL Capabilities]
end
subgraph K8s["Kubernetes Security"]
RBAC[RBAC<br/>Least Privilege]
PSS[Pod Security Standards<br/>restricted]
NetPol[Network Policy<br/>Default Deny]
OPA[OPA Gatekeeper<br/>Policy Enforcement]
end
subgraph PodAuth["Pod AuthN/AuthZ"]
SA[ServiceAccount<br/>Token Projection]
IRSA[IAM Roles for<br/>Service Accounts]
OIDC_P[OIDC Identity<br/>Provider]
end
subgraph Secrets["Secrets Management"]
Vault[HashiCorp Vault]
Sealed[Sealed Secrets]
ESO[External Secrets Operator]
end
Supply --> Image --> Container --> K8s --> PodAuth --> Secrets
Container Security#
Container security starts at build time. A secure container image minimizes the attack surface and runs with the least privileges necessary.
Non-root User#
Never run containers as root. Create a dedicated user in the Dockerfile:
# Python example
FROM python:3.12-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
FROM python:3.12-slim
RUN groupadd -r app && useradd -r -g app -d /app -s /sbin/nologin app
WORKDIR /app
COPY --from=builder /install /usr/local
COPY --chown=app:app . .
USER app
EXPOSE 8080
CMD ["python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]
# Go example โ scratch image (no shell, no user database)
FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o /server .
FROM scratch
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /server /server
USER 65534:65534 # nobody user
EXPOSE 8080
ENTRYPOINT ["/server"]
Read-only Filesystem and Security Context#
apiVersion: apps/v1
kind: Deployment
metadata:
name: secure-app
spec:
replicas: 3
selector:
matchLabels:
app: secure-app
template:
metadata:
labels:
app: secure-app
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1001
runAsGroup: 1001
fsGroup: 1001
seccompProfile:
type: RuntimeDefault
containers:
- name: app
image: registry.example.com/secure-app:1.0.0
ports:
- containerPort: 8080
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
# Only add specific capabilities if absolutely needed:
# capabilities:
# add: ["NET_BIND_SERVICE"] # for binding to ports < 1024
volumeMounts:
- name: tmp
mountPath: /tmp
- name: cache
mountPath: /app/.cache
resources:
requests:
cpu: "100m"
memory: "128Mi"
limits:
cpu: "500m"
memory: "256Mi"
volumes:
- name: tmp
emptyDir:
sizeLimit: 100Mi
- name: cache
emptyDir:
sizeLimit: 50Mi
automountServiceAccountToken: false # disable if not needed
Seccomp and AppArmor Profiles#
# Custom seccomp profile (restrict syscalls)
# Place at /var/lib/kubelet/seccomp/profiles/restricted.json on each node
{
"defaultAction": "SCMP_ACT_ERRNO",
"architectures": ["SCMP_ARCH_X86_64"],
"syscalls": [
{
"names": [
"accept4", "access", "arch_prctl", "bind", "brk", "clone",
"close", "connect", "epoll_create1", "epoll_ctl", "epoll_pwait",
"execve", "exit", "exit_group", "fcntl", "fstat", "futex",
"getdents64", "getpid", "getsockname", "getsockopt", "listen",
"madvise", "mmap", "mprotect", "munmap", "nanosleep", "newfstatat",
"openat", "pipe2", "pread64", "read", "recvfrom", "rt_sigaction",
"rt_sigprocmask", "rt_sigreturn", "sched_getaffinity", "sched_yield",
"sendto", "set_robust_list", "set_tid_address", "setsockopt",
"sigaltstack", "socket", "tgkill", "write", "writev"
],
"action": "SCMP_ACT_ALLOW"
}
]
}
# Pod using custom seccomp profile
spec:
securityContext:
seccompProfile:
type: Localhost
localhostProfile: profiles/restricted.json
containers:
- name: app
# AppArmor annotation (per-container)
# Note: AppArmor uses annotations, not securityContext
# AppArmor annotation
metadata:
annotations:
container.apparmor.security.beta.kubernetes.io/app: runtime/default
Pod Security Standards#
Kubernetes Pod Security Standards (PSS) replace the deprecated PodSecurityPolicy. They define three levels: privileged, baseline, and restricted.
# Enforce restricted policy on namespace
apiVersion: v1
kind: Namespace
metadata:
name: production
labels:
# Enforce: reject pods that violate the policy
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: latest
# Audit: log violations but allow
pod-security.kubernetes.io/audit: restricted
pod-security.kubernetes.io/audit-version: latest
# Warn: show warnings to users
pod-security.kubernetes.io/warn: restricted
pod-security.kubernetes.io/warn-version: latest
Compliant Pod (Passes restricted Level)#
apiVersion: v1
kind: Pod
metadata:
name: compliant-pod
namespace: production
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1001
fsGroup: 1001
seccompProfile:
type: RuntimeDefault
containers:
- name: app
image: registry.example.com/myapp:1.0.0
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
capabilities:
drop:
- ALL
ports:
- containerPort: 8080
resources:
limits:
cpu: "500m"
memory: "256Mi"
requests:
cpu: "100m"
memory: "128Mi"
automountServiceAccountToken: false
Verify PSS Compliance#
# Dry-run to check if a pod would be admitted
kubectl label --dry-run=server --overwrite ns production \
pod-security.kubernetes.io/enforce=restricted
# Check existing violations in a namespace
kubectl get pods -n production -o json | \
kubectl apply --dry-run=server -f - 2>&1 | grep -i "forbidden"
RBAC (Role-Based Access Control)#
Follow the principle of least privilege. Never grant cluster-admin to applications or developers.
# Namespace-scoped Role: read-only access to pods and logs
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: pod-reader
namespace: production
rules:
- apiGroups: [""]
resources: ["pods", "pods/log", "pods/status"]
verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
resources: ["deployments", "replicasets"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["events"]
verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: dev-pod-reader
namespace: production
subjects:
- kind: Group
name: developers
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
---
# Deployer role: can manage deployments but not secrets
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: deployer
namespace: production
rules:
- apiGroups: ["apps"]
resources: ["deployments", "replicasets"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: [""]
resources: ["pods", "services", "configmaps"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["pods/exec", "pods/portforward"]
verbs: [] # explicitly deny
---
# Service account for CI/CD (scoped to specific namespace)
apiVersion: v1
kind: ServiceAccount
metadata:
name: ci-deployer
namespace: production
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: ci-deployer-binding
namespace: production
subjects:
- kind: ServiceAccount
name: ci-deployer
namespace: production
roleRef:
kind: Role
name: deployer
apiGroup: rbac.authorization.k8s.io
RBAC Audit Commands#
# Check who can perform an action
kubectl auth can-i create deployments --namespace production --as developer@example.com
kubectl auth can-i delete secrets --namespace production --as system:serviceaccount:production:ci-deployer
# List all roles and bindings
kubectl get roles,rolebindings -n production
kubectl get clusterroles,clusterrolebindings
# Find overly permissive roles
kubectl get clusterrolebindings -o json | \
jq '.items[] | select(.roleRef.name == "cluster-admin") | .subjects[]'
NetworkPolicy#
Network policies implement microsegmentation โ default deny all traffic, then explicitly allow only whatโs needed.
# 1. Default deny ALL ingress and egress in the namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: production
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
# 2. Allow DNS resolution (required for all pods)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
namespace: production
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP
---
# 3. Allow ingress-nginx โ frontend
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-ingress-to-frontend
namespace: production
spec:
podSelector:
matchLabels:
app: frontend
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: ingress-nginx
ports:
- port: 8080
protocol: TCP
---
# 4. Allow frontend โ api
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-frontend-to-api
namespace: production
spec:
podSelector:
matchLabels:
app: api
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
ports:
- port: 8080
protocol: TCP
---
# 5. Allow api โ database
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-api-to-db
namespace: production
spec:
podSelector:
matchLabels:
app: postgres
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: api
ports:
- port: 5432
protocol: TCP
---
# 6. Allow api egress to external APIs (specific CIDR)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-api-external
namespace: production
spec:
podSelector:
matchLabels:
app: api
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
ports:
- port: 443
protocol: TCP
OPA Gatekeeper#
OPA (Open Policy Agent) Gatekeeper enforces custom policies as admission webhooks. It prevents non-compliant resources from being created.
# Install Gatekeeper
helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts
helm install gatekeeper gatekeeper/gatekeeper \
-n gatekeeper-system --create-namespace \
--set replicas=3 \
--set audit.replicas=2
Constraint Template: Require Resource Limits#
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
name: k8srequiredresources
spec:
crd:
spec:
names:
kind: K8sRequiredResources
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8srequiredresources
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
not container.resources.limits.cpu
msg := sprintf("Container '%v' must have CPU limits", [container.name])
}
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
not container.resources.limits.memory
msg := sprintf("Container '%v' must have memory limits", [container.name])
}
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
not container.resources.requests.cpu
msg := sprintf("Container '%v' must have CPU requests", [container.name])
}
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
not container.resources.requests.memory
msg := sprintf("Container '%v' must have memory requests", [container.name])
}
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredResources
metadata:
name: require-resource-limits
spec:
match:
kinds:
- apiGroups: ["apps"]
kinds: ["Deployment", "StatefulSet", "DaemonSet"]
namespaces: ["production", "staging"]
enforcementAction: deny
Constraint Template: Block Latest Tag#
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
name: k8sblocklatesttag
spec:
crd:
spec:
names:
kind: K8sBlockLatestTag
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8sblocklatesttag
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
endswith(container.image, ":latest")
msg := sprintf("Container '%v' uses ':latest' tag โ use a specific version", [container.name])
}
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
not contains(container.image, ":")
msg := sprintf("Container '%v' has no tag โ use a specific version", [container.name])
}
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sBlockLatestTag
metadata:
name: block-latest-tag
spec:
match:
kinds:
- apiGroups: ["apps"]
kinds: ["Deployment", "StatefulSet"]
namespaces: ["production"]
Constraint Template: Require Labels#
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
name: k8srequiredlabels
spec:
crd:
spec:
names:
kind: K8sRequiredLabels
validation:
openAPIV3Schema:
type: object
properties:
labels:
type: array
items:
type: string
targets:
- target: admission.k8s.gatekeeper.sh
rego: |
package k8srequiredlabels
violation[{"msg": msg}] {
provided := {label | input.review.object.metadata.labels[label]}
required := {label | label := input.parameters.labels[_]}
missing := required - provided
count(missing) > 0
msg := sprintf("Missing required labels: %v", [missing])
}
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
name: require-standard-labels
spec:
match:
kinds:
- apiGroups: ["apps"]
kinds: ["Deployment"]
parameters:
labels:
- "app.kubernetes.io/name"
- "app.kubernetes.io/version"
- "app.kubernetes.io/managed-by"
- "owner"
Image Scanning with Trivy#
# Scan a container image
trivy image --severity HIGH,CRITICAL registry.example.com/myapp:1.0.0
# Scan and fail on critical vulnerabilities
trivy image --exit-code 1 --severity CRITICAL --ignore-unfixed myapp:latest
# Scan filesystem (source code dependencies)
trivy fs --severity HIGH,CRITICAL .
# Scan Kubernetes cluster
trivy k8s --report summary cluster
# Scan a Helm chart
trivy config ./chart/
# Generate SBOM
trivy image --format spdx-json --output sbom.json myapp:1.0.0
# CI integration โ GitHub Actions
- name: Scan image with Trivy
uses: aquasecurity/trivy-action@master
with:
image-ref: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
format: 'sarif'
output: 'trivy-results.sarif'
severity: 'CRITICAL,HIGH'
exit-code: '1'
ignore-unfixed: true
- name: Upload scan results to GitHub Security
uses: github/codeql-action/upload-sarif@v3
if: always()
with:
sarif_file: 'trivy-results.sarif'
Secrets Management#
Sealed Secrets (GitOps-friendly)#
Sealed Secrets encrypts secrets so they can be safely stored in Git.
# Install sealed-secrets controller
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system
# Install kubeseal CLI
brew install kubeseal
# Create a regular secret, then seal it
kubectl create secret generic db-credentials \
--from-literal=username=admin \
--from-literal=password=s3cur3p@ss \
--dry-run=client -o yaml | \
kubeseal --format yaml > sealed-db-credentials.yaml
# The sealed secret can be committed to Git safely
cat sealed-db-credentials.yaml
# sealed-db-credentials.yaml (safe to commit to Git)
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
name: db-credentials
namespace: production
spec:
encryptedData:
username: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq...
password: AgCtr8OJSWK+PiTySYZZA9rO43cGDEq...
template:
metadata:
name: db-credentials
namespace: production
External Secrets Operator (Vault / AWS / GCP)#
# Install External Secrets Operator
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets -n external-secrets --create-namespace
# SecretStore pointing to HashiCorp Vault
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend
namespace: production
spec:
provider:
vault:
server: "https://vault.example.com"
path: "secret"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "production-role"
serviceAccountRef:
name: vault-auth
---
# ExternalSecret โ syncs Vault secret to K8s Secret
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: db-credentials
namespace: production
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend
kind: SecretStore
target:
name: db-credentials
creationPolicy: Owner
data:
- secretKey: username
remoteRef:
key: production/database
property: username
- secretKey: password
remoteRef:
key: production/database
property: password
Supply Chain Security#
SBOM Generation and Image Signing#
# Generate SBOM with Syft
syft registry.example.com/myapp:1.0.0 -o spdx-json > sbom.spdx.json
syft registry.example.com/myapp:1.0.0 -o cyclonedx-json > sbom.cdx.json
# Sign image with Cosign (keyless โ uses Sigstore/Fulcio)
cosign sign registry.example.com/myapp:1.0.0
# Sign with a key pair
cosign generate-key-pair
cosign sign --key cosign.key registry.example.com/myapp:1.0.0
# Verify signature
cosign verify --key cosign.pub registry.example.com/myapp:1.0.0
# Attach SBOM to image
cosign attach sbom --sbom sbom.spdx.json registry.example.com/myapp:1.0.0
Kyverno Policy: Require Signed Images#
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: verify-image-signature
spec:
validationFailureAction: Enforce
background: false
webhookTimeoutSeconds: 30
rules:
- name: verify-cosign-signature
match:
any:
- resources:
kinds:
- Pod
verifyImages:
- imageReferences:
- "registry.example.com/*"
attestors:
- entries:
- keys:
publicKeys: |-
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
-----END PUBLIC KEY-----
Security Checklist#
Category |
Check |
Priority |
|---|---|---|
Image |
Use minimal base image (distroless/scratch) |
High |
Image |
Scan for CVEs in CI pipeline |
Critical |
Image |
No secrets baked into image layers |
Critical |
Image |
Pin image digests in production |
High |
Container |
Run as non-root |
Critical |
Container |
Read-only root filesystem |
High |
Container |
Drop ALL capabilities |
High |
Container |
Set seccomp profile to RuntimeDefault |
Medium |
K8s |
RBAC with least privilege |
Critical |
K8s |
Pod Security Standards: restricted |
High |
K8s |
NetworkPolicy: default deny |
High |
K8s |
Disable automountServiceAccountToken |
Medium |
K8s |
Resource limits on all containers |
High |
Secrets |
External secrets management (Vault/ESO) |
High |
Secrets |
Rotate secrets regularly |
Medium |
Supply Chain |
Sign images with Cosign |
Medium |
Supply Chain |
Generate and store SBOM |
Medium |
Audit |
Enable K8s audit logging |
High |
Audit |
Monitor with Falco for runtime threats |
Medium |