Kubernetes Secrets Management with SOPS and Age

Storing Kubernetes secrets in Git repositories without encryption is one of the most common security mistakes in cloud-native deployments. While Kubernetes Secrets use base64 encoding, this provides zero cryptographic protection — anyone with repository access can decode them instantly. SOPS (Secrets OPerationS) combined with Age encryption solves this problem elegantly, allowing you to store encrypted secrets directly in Git while maintaining full compatibility with GitOps workflows like ArgoCD and FluxCD.

Prerequisites

• A running Kubernetes cluster (v1.24+) with kubectl configured
• sops v3.8+ installed
• age v1.1+ installed
• Basic familiarity with Kubernetes Secrets and YAML manifests
• Git repository for your Kubernetes manifests
• Optional: ArgoCD or FluxCD for GitOps integration

Understanding Kubernetes Secrets Risks

Kubernetes Secrets are often misunderstood as a secure storage mechanism. In reality, they have significant limitations that you must understand before designing your secrets strategy.

Base64 Is Not Encryption

A standard Kubernetes Secret stores values as base64-encoded strings:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  username: YWRtaW4=
  password: cDRzc3cwcmQxMjM=

Decoding these values is trivial:

echo "cDRzc3cwcmQxMjM=" | base64 -d
# Output: p4ssw0rd123

Anyone who clones your repository or gains read access to etcd can extract every secret in your cluster.

Etcd Storage Concerns

By default, Kubernetes stores Secrets unencrypted in etcd. While you can enable encryption at rest with an EncryptionConfiguration, this only protects the etcd data files — not the API server responses or Git-stored manifests. You need a solution that encrypts secrets before they enter your version control system.

The Git Problem

GitOps requires that your desired cluster state lives in Git. But committing plaintext secrets to Git means:

• Every developer with repo access sees production credentials
• Secret history persists in Git forever, even after deletion
• Leaked repositories expose every secret ever committed
• Compliance frameworks (SOC 2, PCI-DSS) prohibit plaintext credentials in source control

Installing SOPS and Age

Linux

# Install Age
sudo apt-get install age
# Or from GitHub releases
curl -LO https://github.com/FiloSottile/age/releases/download/v1.2.0/age-v1.2.0-linux-amd64.tar.gz
tar xzf age-v1.2.0-linux-amd64.tar.gz
sudo mv age/age age/age-keygen /usr/local/bin/

# Install SOPS
curl -LO https://github.com/getsops/sops/releases/download/v3.9.4/sops-v3.9.4.linux.amd64
sudo mv sops-v3.9.4.linux.amd64 /usr/local/bin/sops
sudo chmod +x /usr/local/bin/sops

macOS

brew install sops age

Verify both installations:

sops --version
# sops 3.9.4
age --version
# v1.2.0

Encrypting Secrets with SOPS and Age

Generate an Age Key Pair

age-keygen -o age-key.txt
# Public key: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p

Store the private key securely. The public key is safe to share and commit to your repository:

# Store the key where SOPS can find it
mkdir -p ~/.config/sops/age
mv age-key.txt ~/.config/sops/age/keys.txt

# Or set the environment variable
export SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt

Create .sops.yaml Configuration

Create a .sops.yaml file in your repository root to define encryption rules:

creation_rules:
  # Encrypt all files in the secrets/ directory
  - path_regex: secrets/.*\.yaml$
    age: >-
      age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p

  # Encrypt staging secrets with a different key
  - path_regex: envs/staging/secrets/.*\.yaml$
    age: >-
      age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p,
      age1lzd99uca0lqtmgahfmxj4gvr2fcswcaxmxnz30fwcmm22hjvrzrqsnqxsl

  # Different key for production
  - path_regex: envs/production/secrets/.*\.yaml$
    age: >-
      age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p,
      age1an5quvgyl8uny5mkmgkzpnpe9wuufrl2vvmqsht4xp3k96s608q0eamcl

Multiple recipients (comma-separated) enable team access — any listed key can decrypt the file.

Encrypt a Kubernetes Secret

Start with your plaintext secret manifest:

# secrets/db-credentials.yaml
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: production
type: Opaque
stringData:
  username: admin
  password: "s3cur3-pr0d-p@ssw0rd"
  connection-string: "postgresql://admin:s3cur3-pr0d-p@ssw0rd@db.internal:5432/myapp"

Encrypt it with SOPS:

sops --encrypt --in-place secrets/db-credentials.yaml

The encrypted file preserves the YAML structure but encrypts values:

apiVersion: v1
kind: Secret
metadata:
    name: db-credentials
    namespace: production
type: Opaque
stringData:
    username: ENC[AES256_GCM,data:k8mN3w==,iv:abc...,tag:def...,type:str]
    password: ENC[AES256_GCM,data:dGhpcyBpcyBl...,iv:ghi...,tag:jkl...,type:str]
    connection-string: ENC[AES256_GCM,data:bG9uZ2VyIHN0cmluZw==...,iv:mno...,tag:pqr...,type:str]
sops:
    age:
        - recipient: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            ...
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2026-02-17T10:00:00Z"
    version: 3.9.4

Keys (apiVersion, kind, metadata) remain readable while values are encrypted. This means git diff shows which secrets changed without revealing the actual values.

Decrypt and Apply

# Decrypt and apply in one command
sops --decrypt secrets/db-credentials.yaml | kubectl apply -f -

# Or decrypt to a temporary file
sops --decrypt secrets/db-credentials.yaml > /tmp/secret.yaml
kubectl apply -f /tmp/secret.yaml
rm -f /tmp/secret.yaml

Integrating with GitOps Workflows

ArgoCD with KSOPS

KSOPS is a Kustomize plugin that decrypts SOPS-encrypted files during ArgoCD sync operations.

Install KSOPS in your ArgoCD repo server:

# argocd-repo-server patch
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
  namespace: argocd
spec:
  template:
    spec:
      containers:
        - name: argocd-repo-server
          env:
            - name: SOPS_AGE_KEY
              valueFrom:
                secretKeyRef:
                  name: sops-age-key
                  key: age-key.txt
            - name: XDG_CONFIG_HOME
              value: /.config
          volumeMounts:
            - mountPath: /.config/kustomize/plugin/viaduct.ai/v1/ksops
              name: custom-tools
      initContainers:
        - name: install-ksops
          image: viaductoss/ksops:v4.3.2
          command: ["/bin/sh", "-c"]
          args:
            - cp /usr/local/bin/ksops /.config/kustomize/plugin/viaduct.ai/v1/ksops/ksops
          volumeMounts:
            - mountPath: /.config/kustomize/plugin/viaduct.ai/v1/ksops
              name: custom-tools
      volumes:
        - name: custom-tools
          emptyDir: {}

Create a KSOPS generator in your Kustomize overlay:

# secret-generator.yaml
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: secret-generator
files:
  - secrets/db-credentials.yaml
  - secrets/api-keys.yaml

FluxCD Native Integration

FluxCD has built-in SOPS support. Create a decryption secret and configure your Kustomization:

# Create the Age key secret in the flux-system namespace
kubectl create secret generic sops-age \
  --namespace=flux-system \
  --from-file=age.agekey=~/.config/sops/age/keys.txt

# flux-kustomization.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: my-app
  namespace: flux-system
spec:
  interval: 10m
  path: ./envs/production
  prune: true
  sourceRef:
    kind: GitRepository
    name: my-app
  decryption:
    provider: sops
    secretRef:
      name: sops-age

FluxCD automatically decrypts any SOPS-encrypted files in the specified path during reconciliation.

SOPS+Age vs Sealed Secrets vs Vault vs External Secrets

Feature	SOPS + Age	Sealed Secrets	HashiCorp Vault	External Secrets
Encryption location	Client-side	Controller-side	Server-side	Server-side
Git-friendly	Yes (encrypted YAML)	Yes (custom resource)	No (references only)	No (references only)
Infrastructure needed	None	Cluster controller	Vault server	Operator + backend
Key management	Age key files	Cluster certificate	Vault policies	Varies by backend
Multi-environment	.sops.yaml rules	Per-cluster certs	Namespaces/policies	Multiple stores
GitOps integration	ArgoCD/Flux plugins	Native K8s	CSI driver/injector	Operator sync
Rotation	Re-encrypt with SOPS	Reseal required	Dynamic secrets	Backend-dependent
Complexity	Low	Low	High	Medium
Offline capable	Yes	No	No	No
Best for	Small-medium teams	Single clusters	Enterprise/compliance	Multi-cloud

SOPS+Age excels when you want simplicity, offline capability, and true GitOps without additional infrastructure.

Real-World Scenario

You manage a multi-environment Kubernetes deployment across development, staging, and production clusters. Your team of eight engineers uses ArgoCD for GitOps, and you need a secrets solution that:

• Lets developers create and update secrets without cluster access
• Keeps encrypted secrets in Git for auditability
• Uses different encryption keys per environment
• Allows key rotation without re-deploying every secret

Here is how you would structure your repository:

k8s-manifests/
├── .sops.yaml
├── base/
│   ├── deployment.yaml
│   └── service.yaml
├── envs/
│   ├── dev/
│   │   ├── kustomization.yaml
│   │   ├── secret-generator.yaml
│   │   └── secrets/
│   │       └── app-secrets.yaml    # encrypted with dev key
│   ├── staging/
│   │   ├── kustomization.yaml
│   │   ├── secret-generator.yaml
│   │   └── secrets/
│   │       └── app-secrets.yaml    # encrypted with staging key
│   └── production/
│       ├── kustomization.yaml
│       ├── secret-generator.yaml
│       └── secrets/
│           └── app-secrets.yaml    # encrypted with prod key

Each environment has its own Age key pair. Developers hold the dev key, team leads hold dev+staging, and only the CI/CD pipeline holds the production key. The .sops.yaml file routes encryption to the correct key based on file path.

Gotchas and Edge Cases

Key rotation requires re-encryption. When you rotate an Age key, you must decrypt every file with the old key and re-encrypt with the new one. SOPS provides sops updatekeys for this, but test it first:

# Update keys for a single file (uses .sops.yaml rules)
sops updatekeys secrets/db-credentials.yaml

# Batch re-encrypt all secrets
find . -name "*.yaml" -path "*/secrets/*" -exec sops updatekeys {} \;

Partial encryption with encrypted_regex. By default, SOPS encrypts all values. Use encrypted_regex in .sops.yaml to encrypt only specific keys:

creation_rules:
  - path_regex: secrets/.*\.yaml$
    encrypted_regex: "^(data|stringData)$"
    age: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p

Binary secrets need base64 encoding first. SOPS operates on text files. For binary secrets (TLS certificates, keystores), base64-encode them before encrypting:

# Encode the certificate, then let SOPS encrypt the base64 string
cat tls.crt | base64 -w0 > tls.crt.b64

Never commit the Age private key. Add it to .gitignore immediately:

echo "age-key.txt" >> .gitignore
echo "*.agekey" >> .gitignore

SOPS metadata section grows with recipients. Each additional Age recipient adds ~300 bytes of metadata. With many recipients, consider using a shared team key distributed through a secure channel rather than individual keys.

Troubleshooting

Error: could not find common decryption key The Age private key is not available. Verify the key file location:

# Check if the key file exists
ls -la ~/.config/sops/age/keys.txt

# Or check the environment variable
echo $SOPS_AGE_KEY_FILE

# Verify the key matches the recipient
grep "public key" ~/.config/sops/age/keys.txt

Error: failed to decrypt The file was encrypted with a different Age key. Check which key was used:

sops --decrypt --verbose secrets/db-credentials.yaml 2>&1 | grep "recipient"

ArgoCD sync fails with KSOPS errors Ensure the KSOPS init container completed successfully and the Age key secret exists:

kubectl logs deployment/argocd-repo-server -n argocd -c install-ksops
kubectl get secret sops-age-key -n argocd

FluxCD Kustomization stuck in Not Ready Check the Kustomize controller logs:

kubectl logs deployment/kustomize-controller -n flux-system | grep -i sops

Common cause: the sops-age secret is in the wrong namespace or has the wrong key name (must be age.agekey).

Encrypted file shows mac mismatch error The file was modified after encryption without using sops --set or sops edit. Re-encrypt from the plaintext source:

sops --decrypt secrets/db-credentials.yaml > /tmp/plain.yaml
sops --encrypt /tmp/plain.yaml > secrets/db-credentials.yaml
rm -f /tmp/plain.yaml

Summary

• Kubernetes Secrets use base64 encoding, not encryption — they are not secure by default
• SOPS encrypts YAML values while keeping keys readable, enabling meaningful Git diffs
• Age provides simpler key management than PGP with no configuration overhead
• The .sops.yaml file defines per-path encryption rules for multi-environment setups
• ArgoCD integrates via KSOPS plugin; FluxCD has native SOPS decryption support
• Key rotation requires re-encrypting all affected files with sops updatekeys
• Always store Age private keys outside the repository and in .gitignore
• For enterprise environments requiring dynamic secrets, consider HashiCorp Vault instead

Related Articles

• Kubernetes RBAC Best Practices
• GitOps with ArgoCD and Kubernetes
• HashiCorp Vault on Kubernetes
• Securing Container Images in CI/CD Pipelines
• Kubernetes Network Policies Deep Dive