HashiCorp Vault solves one of the most persistent problems in modern infrastructure: where do you safely store and distribute secrets? Hardcoded credentials in config files, environment variables committed to git, and shared password spreadsheets are responsible for a disproportionate share of breaches. HashiCorp Vault provides a unified API for secrets management, dynamic credential generation, encryption as a service, and fine-grained access control — all with a complete audit trail. In this guide you will learn how Vault works, how to deploy it, and how to integrate it with Kubernetes and Terraform in a real production workflow.

Prerequisites

  • Linux or macOS host (or a Kubernetes cluster for the cloud-native path)
  • Basic familiarity with the command line and YAML configuration
  • Docker (optional, for a quick dev server)
  • A Kubernetes cluster with kubectl access (for the Kubernetes integration section)
  • Terraform 1.x installed (for the Terraform provider section)
  • Understanding of TLS certificates is helpful but not required

How HashiCorp Vault Works

At its core, Vault is a process that exposes an HTTP API. Everything — reading secrets, authenticating, managing policies — goes through that API. Vault stores its data in a storage backend (Integrated Storage based on Raft is the recommended default) and encrypts everything at rest using a master key that never leaves memory.

When Vault starts, it is sealed: it holds the encrypted data but cannot decrypt it because the master key is split across multiple unseal key shares using Shamir’s Secret Sharing. Vault becomes operational only after the minimum quorum of key shares is provided. This design means that even if someone steals the storage backend, they cannot read any secrets without the unseal keys.

Key concepts

  • Secrets engines — plugins that generate or store secrets. The KV engine stores static key/value pairs. The Database engine generates dynamic credentials. The PKI engine issues X.509 certificates.
  • Auth methods — how clients prove identity to Vault. Options include AppRole (for machines), Kubernetes service accounts, AWS IAM, LDAP, GitHub, and OIDC.
  • Policies — HCL files that grant read, write, or admin access to specific paths in Vault.
  • Leases and TTLs — every secret has a lease; when the lease expires, Vault revokes it. Applications must renew or re-request credentials before expiry.
  • Audit devices — Vault can write every request and response to a log file, syslog, or socket. Audit is tamper-evident and cryptographically verified.

Installing and Initializing Vault

For a production-grade deployment, HashiCorp recommends running Vault in High Availability (HA) mode with Integrated Storage across three or five nodes. For learning purposes, a single dev server is sufficient.

Quick dev server (not for production):

vault server -dev

This starts an in-memory Vault, auto-unseals it, and prints a root token. Data is lost when the process exits.

Production single-node with Integrated Storage:

Create /etc/vault.d/vault.hcl:

ui = true

storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-node-1"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_cert_file = "/opt/vault/tls/vault.crt"
  tls_key_file  = "/opt/vault/tls/vault.key"
}

api_addr     = "https://vault.example.com:8200"
cluster_addr = "https://vault.example.com:8201"

Start the service, then initialize it:

vault operator init -key-shares=5 -key-threshold=3

This outputs five unseal keys and one root token. Store these in separate secure locations immediately — losing enough key shares means losing access to all your data permanently. Unseal with three of the five keys:

vault operator unseal <key-1>
vault operator unseal <key-2>
vault operator unseal <key-3>

Dynamic Secrets: The Core Advantage

Static credentials (a database password that never changes) are a liability. If they leak, the window of exposure is indefinite. Dynamic secrets close that window.

Enable the database secrets engine and configure it for PostgreSQL:

vault secrets enable database

vault write database/config/myapp-db \
  plugin_name=postgresql-database-plugin \
  allowed_roles="myapp-role" \
  connection_url="postgresql://{{username}}:{{password}}@db.example.com:5432/myapp" \
  username="vault" \
  password="vault-superuser-password"

vault write database/roles/myapp-role \
  db_name=myapp-db \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

Now any application or CI job with the right Vault policy can run:

vault read database/creds/myapp-role

Vault connects to PostgreSQL, creates a uniquely-named user, and returns credentials that expire in one hour. When the lease expires (or the application calls vault lease revoke), Vault drops the database user. The application never holds a long-lived password.

PKI Secrets Engine for Internal Certificates

Managing TLS certificates manually is error-prone and rarely audited. Vault’s PKI secrets engine acts as an internal Certificate Authority, issuing short-lived certificates on demand.

vault secrets enable pki
vault secrets tune -max-lease-ttl=87600h pki

# Generate root CA (store the cert externally, keep offline if possible)
vault write -field=certificate pki/root/generate/internal \
  common_name="example.com" \
  ttl=87600h > CA_cert.crt

# Configure CRL and issuing URLs
vault write pki/config/urls \
  issuing_certificates="https://vault.example.com:8200/v1/pki/ca" \
  crl_distribution_points="https://vault.example.com:8200/v1/pki/crl"

# Enable intermediate CA (best practice for production)
vault secrets enable -path=pki_int pki
vault write -format=json pki_int/intermediate/generate/internal \
  common_name="example.com Intermediate Authority" | jq -r '.data.csr' > pki_int.csr

vault write -format=json pki/root/sign-intermediate csr=@pki_int.csr \
  format=pem_bundle ttl="43800h" | jq -r '.data.certificate' > intermediate.cert.pem

vault write pki_int/intermediate/set-signed certificate=@intermediate.cert.pem

# Create a role for issuing server certs
vault write pki_int/roles/example-dot-com \
  allowed_domains="example.com" \
  allow_subdomains=true \
  max_ttl="72h"

Any service that needs a TLS certificate requests one with:

vault write pki_int/issue/example-dot-com common_name="api.example.com" ttl="24h"

Certificates valid for 24 hours eliminate the risk of forgotten certificate rotations.

Kubernetes Integration with Vault Agent Injector

Kubernetes Secrets are base64-encoded and stored in etcd. Without additional controls, any user with kubectl get secret access can read them. Vault’s Agent Injector keeps secrets out of Kubernetes entirely.

Install with Helm:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault \
  --set "injector.enabled=true" \
  --set "server.dev.enabled=false"

Enable and configure Kubernetes auth in Vault:

vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Create a policy and role:

vault policy write myapp - <<EOF
path "secret/data/myapp/*" {
  capabilities = ["read"]
}
EOF

vault write auth/kubernetes/role/myapp \
  bound_service_account_names=myapp \
  bound_service_account_namespaces=production \
  policies=myapp \
  ttl=1h

Annotate your pod to inject secrets at runtime:

annotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/role: "myapp"
  vault.hashicorp.com/agent-inject-secret-config.env: "secret/data/myapp/config"
  vault.hashicorp.com/agent-inject-template-config.env: |
    {{- with secret "secret/data/myapp/config" -}}
    export DB_PASSWORD="{{ .Data.data.db_password }}"
    {{- end }}

The Vault Agent sidecar authenticates, fetches the secret, writes it to a shared in-memory volume, and renews it before expiry. The application reads a file — it never makes a Vault API call.

Vault vs. Alternatives: Secrets Management Comparison

FeatureHashiCorp VaultAWS Secrets ManagerAzure Key VaultKubernetes Secrets
Dynamic secretsYes (database, cloud, PKI)Limited (RDS only)NoNo
Multi-cloudYesAWS onlyAzure onlyYes (with CSI)
PKI / CAFull CA with CRLNoYes (limited)No
Audit loggingBuilt-in, tamper-evidentCloudTrailAzure MonitorNeeds add-on
Self-hostedYesNoNoYes
CostFree (Community)Per secret + API callPer operationFree
Kubernetes nativeAgent Injector / VSOCSI driverCSI driverNative
Learning curveHighLowLowVery low

Vault is the right choice when you need vendor-neutral, auditable secrets management with dynamic credential generation across multiple platforms. AWS Secrets Manager wins when you are 100% AWS-native and simplicity is the priority.

Real-World Scenario: Securing a Multi-Tier Application

You have a production Kubernetes cluster running a Node.js API, a PostgreSQL database, and a Redis cache. Your current setup stores the database password as a Kubernetes Secret and the Redis password in a ConfigMap. A recent internal audit flagged both as compliance risks.

Here is the migration path:

  1. Deploy Vault in HA mode (3 nodes) alongside the cluster using Helm. Use Integrated Storage with a Raft quorum.
  2. Enable dynamic secrets for PostgreSQL. Remove the static PGPASSWORD environment variable. The Vault Agent Injector writes a fresh credential file every 55 minutes (before the 1-hour TTL expires).
  3. Store the Redis password in Vault KV v2 at secret/data/production/redis. The Agent Injector injects it as /vault/secrets/redis.env sourced by the application’s entrypoint script.
  4. Enable PKI for the internal TLS between the API and the database. Certificates rotate every 24 hours automatically.
  5. Configure audit logging to write to a syslog target forwarded to your SIEM. Every secret access is now tracked with client identity, source IP, and timestamp.
  6. Integrate with Terraform using the Vault provider to manage policies and roles as code, version-controlled in your infrastructure repository.

Result: no secrets in Kubernetes etcd, no static credentials, full audit trail, and automatic rotation — all without changing a line of application code.

Gotchas and Edge Cases

Seal after restart. Vault reseals itself on every process restart. You need an automated unseal mechanism in production. Options are HashiCorp’s own Auto Unseal (using AWS KMS, GCP Cloud KMS, or Azure Key Vault as the key source), or running a Vault HSM integration. Plan for this before deploying to production.

Lease explosion. If you generate dynamic secrets for thousands of short-lived CI jobs and never explicitly revoke them, Vault’s lease store grows without bound and eventually degrades performance. Always call vault lease revoke at the end of a job, or use Vault Agent which handles renewal and revocation automatically.

Root token should be revoked. The root token generated during vault operator init has unrestricted access. After initial setup, generate a time-limited root token for emergency use and revoke the original: vault token revoke <root-token>.

Storage backend performance. Integrated Storage (Raft) performs well for most workloads but is sensitive to disk I/O latency. On cloud VMs, use SSD-backed volumes. Avoid NFS or network-attached storage for the Raft data directory.

Namespace confusion. Vault Enterprise namespaces are logical isolation units (like separate Vault instances). Vault Community Edition has no namespaces. If you test with Enterprise and deploy Community, paths that worked before will not exist.

Clock skew. Several auth methods (AWS IAM, Kubernetes, JWT) validate signed tokens with a timestamp. If server clocks drift more than a few seconds, authentication fails silently. Ensure NTP is configured on all Vault nodes.

Troubleshooting

“Error initializing: Post https://127.0.0.1:8200/v1/sys/init: x509: certificate signed by unknown authority” Set VAULT_SKIP_VERIFY=true for testing only, or add your CA cert to the system trust store and set VAULT_CACERT=/path/to/ca.crt.

“Error making API request: Code: 503. Errors: Vault is sealed.” Vault has restarted and is waiting for unseal keys. Run vault operator unseal with the required number of key shares, or verify your Auto Unseal configuration.

Vault Agent fails to authenticate on Kubernetes with “permission denied” Check that the pod’s ServiceAccount name and namespace exactly match bound_service_account_names and bound_service_account_namespaces in the Vault Kubernetes role. Also verify the Vault policy grants the paths being requested.

“context deadline exceeded” when Vault reads from the database The database plugin cannot reach the database host. Check the connection_url in database/config/<name>, verify network connectivity and firewall rules between Vault and the database, and ensure the Vault superuser has CREATEROLE privileges.

Secrets not updating in a running pod after Vault Agent renewal By default, Vault Agent renders templates to files. The application must watch the file for changes or restart to pick up new credentials. Use the command block in the Vault Agent template stanza to trigger a kill -HUP or similar signal.

Summary

  • HashiCorp Vault is the industry standard for vendor-neutral, auditable secrets management across cloud and on-premises environments.
  • The KV v2 engine handles static secrets; the Database and PKI engines generate short-lived dynamic credentials and certificates that auto-revoke.
  • Vault Agent decouples your application from the Vault API — it handles authentication, secret fetching, template rendering, renewal, and revocation.
  • Kubernetes integration via the Agent Injector or Vault Secrets Operator eliminates static Kubernetes Secrets and etcd exposure.
  • Every operation through Vault is captured in audit logs, giving you a complete, tamper-evident record for compliance.
  • Plan for Auto Unseal from day one to avoid operational disruptions when Vault nodes restart.
  • Dynamic secrets dramatically shrink the blast radius of a credential leak — a one-hour database credential is far less dangerous than a password that has been valid for three years.