What is erasure coding in MinIO and why does it matter?

Erasure coding splits objects into data and parity shards distributed across drives. MinIO can survive up to half the drives failing per erasure set without data loss — far more resilient than RAID-5 with better storage efficiency.

How do I run MinIO in distributed mode?

Launch identical minio server commands on all nodes, listing every node's URL and drives. All nodes must share the same MINIO_ROOT_USER/PASSWORD, same MinIO version, and be reachable over the network before the cluster forms.

Does MinIO support encryption at rest?

Yes. SSE-S3 uses MinIO's internal KMS for transparent per-object encryption. SSE-KMS integrates HashiCorp Vault or AWS KMS for externally managed keys with envelope encryption and key rotation.

Can MinIO send notifications when objects are uploaded or deleted?

Yes. MinIO bucket notifications support webhook, AMQP (RabbitMQ), Kafka, Redis, PostgreSQL, MySQL, and NATS targets. Configure the target in the MinIO environment, then enable notifications per bucket via mc or the Console.

MinIO: Complete Self-Hosted S3-Compatible Object Storage Guide

TL;DR — Quick Summary

Complete MinIO guide: architecture, distributed mode, erasure coding, IAM, SSE, bucket notifications, replication, and production Docker Compose deployment.

MinIO is a high-performance, Kubernetes-native object storage system that speaks the full AWS S3 API. Where the existing MinIO setup guide covers basic installation and troubleshooting, this guide goes deeper: distributed architecture with erasure coding, built-in IAM, server-side encryption, bucket notifications, cross-site replication, and a production Docker Compose stack with nginx. Every S3-compatible tool — boto3, rclone, Terraform, the aws CLI — works with MinIO by changing a single endpoint URL.

Prerequisites

Linux server (Ubuntu 22.04+ or RHEL 9+), or Docker 24+.
At least 8 GB RAM and dedicated drives (not the OS disk) for production.
Ports 9000 (API) and 9001 (Console) accessible.
For distributed mode: identical hardware across nodes, DNS or /etc/hosts resolving every node name, and synchronized clocks (NTP).

MinIO Architecture: Erasure Coding and Bitrot Protection

MinIO stores objects using erasure coding (Reed-Solomon). When you write an object, MinIO splits it into N data shards and M parity shards — collectively called an erasure set. A 16-drive erasure set with EC:8 gives you 8 data + 8 parity shards: MinIO survives losing any 8 drives simultaneously without data loss.

Key architectural concepts:

Erasure set — the unit of redundancy, typically 4–16 drives. MinIO selects the largest power of 2 that fits your drive count.
Server pool — a group of nodes that form a single namespace. Add pools online to expand capacity without downtime.
Bitrot protection — every shard is checksummed (HighwayHash). Reads verify checksums; corrupted shards are healed from parity automatically.
Inline data encryption — objects are encrypted at the shard level before being written to disk, not at the volume level.
Read quorum / Write quorum — reads need N/2 shards, writes need N/2 + 1. Below quorum, MinIO returns errors rather than corrupt data.

Installation Methods

Method 1: Binary + systemd (Recommended for Bare Metal)

# Download binary
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
sudo mv minio /usr/local/bin/

# Dedicated user and data directory
sudo useradd -r minio-user -s /sbin/nologin
sudo mkdir -p /data/minio
sudo chown minio-user:minio-user /data/minio

Environment file /etc/default/minio:

MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=StrongPassword123!
MINIO_VOLUMES="/data/minio"
MINIO_OPTS="--console-address :9001"

Systemd unit /etc/systemd/system/minio.service:

[Unit]
Description=MinIO Object Storage
After=network-online.target
Wants=network-online.target

[Service]
User=minio-user
Group=minio-user
EnvironmentFile=/etc/default/minio
ExecStart=/usr/local/bin/minio server $MINIO_VOLUMES $MINIO_OPTS
Restart=always
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now minio

Method 2: Docker

docker run -d \
  --name minio \
  -p 9000:9000 -p 9001:9001 \
  -e MINIO_ROOT_USER=minioadmin \
  -e MINIO_ROOT_PASSWORD=StrongPassword123! \
  -v /data/minio:/data \
  quay.io/minio/minio server /data --console-address ":9001"

Method 3: Kubernetes Operator

kubectl apply -f https://raw.githubusercontent.com/minio/operator/master/deploy/namespace.yaml
helm install minio-operator minio/operator --namespace minio-operator

The Operator manages Tenant custom resources, each representing an independent MinIO cluster with its own storage, TLS certificates, and IAM.

Distributed Mode: Multi-Node Multi-Drive

Distributed MinIO requires launching the same command on every node simultaneously. All nodes must share identical credentials and MinIO version.

For a 4-node cluster with 4 drives each (16 drives total, EC:8):

# Run on ALL 4 nodes — identical command
export MINIO_ROOT_USER=minioadmin
export MINIO_ROOT_PASSWORD=StrongPassword123!

minio server \
  http://minio-node{1...4}/data/{1...4} \
  --console-address ":9001"

MinIO uses brace expansion — minio-node{1...4} expands to minio-node1, minio-node2, minio-node3, minio-node4. Each node must be resolvable via DNS or /etc/hosts.

Adding a server pool to an existing cluster:

minio server \
  http://minio-node{1...4}/data/{1...4} \
  http://minio-node{5...8}/data/{1...4} \
  --console-address ":9001"

Pools are independent erasure sets that share the same namespace. Objects are distributed across pools by a weighted placement algorithm.

MinIO Console and mc CLI

MinIO Console (Web UI)

The Console runs on port 9001 and covers all operational tasks:

Buckets — create, configure versioning, lifecycle, encryption, notifications.
Identity — users, groups, service accounts, IAM policies.
Monitoring — real-time metrics, drive health, erasure set status.
Logs — audit log viewer, server log streaming.

mc CLI Reference

# Install
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc && sudo mv mc /usr/local/bin/

# Configure alias
mc alias set myminio https://minio.example.com minioadmin StrongPassword123!

Command	Description
`mc mb myminio/backups`	Create bucket
`mc cp file.tar.gz myminio/backups/`	Upload file
`mc mirror /local/dir myminio/backups/dir/`	Sync directory
`mc ls myminio/backups`	List objects
`mc cat myminio/backups/config.json`	Stream object to stdout
`mc rm myminio/backups/old.tar.gz`	Delete object
`mc version enable myminio/backups`	Enable versioning
`mc policy set download myminio/public`	Make bucket public-read
`mc admin info myminio`	Cluster status
`mc admin prometheus generate myminio`	Prometheus scrape config

Bucket Management: Versioning, Lifecycle, and Object Locking

Versioning

mc version enable myminio/my-bucket
# Every PUT creates a new version; DELETEs insert a delete marker

Lifecycle Rules (ILM)

# Expire non-current versions after 30 days
mc ilm add --expiry-days 30 --noncurrentversion-expiration-days 30 myminio/my-bucket

# Transition objects to a cold tier after 90 days (requires MinIO Tiering license)
mc ilm add --transition-days 90 --transition-storage-class COLD myminio/my-bucket

Object Locking (WORM)

# Enable at bucket creation — cannot be disabled later
mc mb --with-lock myminio/compliance-bucket

# Set default retention: COMPLIANCE mode, 7 years
mc retention set --default COMPLIANCE "7y" myminio/compliance-bucket

COMPLIANCE mode locks objects even from the root user. GOVERNANCE mode allows root to override. Use COMPLIANCE for regulatory WORM requirements.

Identity and Access Management

MinIO’s IAM is a subset of AWS IAM. Policies are JSON documents with Version, Statement, Effect, Action, and Resource fields.

Custom Policy Example

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::backups",
        "arn:aws:s3:::backups/*"
      ]
    }
  ]
}

mc admin policy create myminio readonly-backups policy.json
mc admin user add myminio appuser secretkey123
mc admin policy attach myminio readonly-backups --user appuser

Service Accounts

# Create a service account (scoped access key) for an application
mc admin user svcacct add myminio appuser
# Returns: AccessKey + SecretKey for application config

LDAP / Active Directory Integration

export MINIO_IDENTITY_LDAP_SERVER_ADDR="ldap.corp.example.com:389"
export MINIO_IDENTITY_LDAP_LOOKUP_BIND_DN="cn=minio,dc=corp,dc=example,dc=com"
export MINIO_IDENTITY_LDAP_LOOKUP_BIND_PASSWORD="ldappassword"
export MINIO_IDENTITY_LDAP_USER_DN_SEARCH_BASE_DN="ou=users,dc=corp,dc=example,dc=com"
export MINIO_IDENTITY_LDAP_USER_DN_SEARCH_FILTER="(uid=%s)"

OpenID Connect (Keycloak, Okta, Azure AD)

export MINIO_IDENTITY_OPENID_CONFIG_URL="https://keycloak.example.com/realms/myrealm/.well-known/openid-configuration"
export MINIO_IDENTITY_OPENID_CLIENT_ID="minio"
export MINIO_IDENTITY_OPENID_CLIENT_SECRET="secret"
export MINIO_IDENTITY_OPENID_CLAIM_NAME="policy"

Map the policy claim in your OIDC token to a MinIO IAM policy name, and users inherit their permissions from the IdP.

Server-Side Encryption

SSE-S3 (Internal KMS)

# Single-key internal encryption — simplest option
export MINIO_KMS_SECRET_KEY="my-minio-key:bXltaW5pb2tleXBhc3N3b3JkMTIzNDU2Nzg="
# Enable default encryption on a bucket
mc encrypt set sse-s3 myminio/my-bucket

SSE-KMS with HashiCorp Vault

export MINIO_KMS_KES_ENDPOINT="https://kes.example.com:7373"
export MINIO_KMS_KES_KEY_NAME="minio-default-key"
export MINIO_KMS_KES_CERT_FILE="/etc/minio/kes-client.crt"
export MINIO_KMS_KES_KEY_FILE="/etc/minio/kes-client.key"
export MINIO_KMS_KES_CAPATH="/etc/minio/kes-ca.crt"

# Enable SSE-KMS default encryption on a bucket
mc encrypt set sse-kms minio-default-key myminio/sensitive-bucket

MinIO uses KES (Key Encryption Service) as a bridge between MinIO and external KMS providers (Vault, AWS KMS, Azure Key Vault, GCP KMS). Each object gets a unique data encryption key (DEK) wrapped by the master key via envelope encryption.

Bucket Notifications

MinIO fires events on object lifecycle (PUT, DELETE, GET, etc.) to external targets.

Configure a Webhook Target

mc admin config set myminio notify_webhook:1 \
  endpoint="https://hooks.example.com/minio" \
  auth_token="Bearer mytoken"
mc admin service restart myminio

# Enable notifications on a bucket
mc event add myminio/my-bucket arn:minio:sqs::1:webhook \
  --event put,delete

Configure a Kafka Target

mc admin config set myminio notify_kafka:1 \
  brokers="kafka1.example.com:9092" \
  topic="minio-events" \
  tls=off
mc admin service restart myminio
mc event add myminio/logs-bucket arn:minio:sqs::1:kafka --event put

Supported targets: notify_webhook, notify_amqp (RabbitMQ), notify_kafka, notify_redis, notify_postgres, notify_mysql, notify_nats, notify_elasticsearch.

Site Replication and Bucket Replication

Site-to-Site Replication (Active-Active)

# Set up aliases for both sites
mc alias set site-a https://minio-a.example.com adminA passA
mc alias set site-b https://minio-b.example.com adminB passB

# Enable site replication — both sites must have admin access
mc admin replicate add site-a site-b

Site replication mirrors all buckets, IAM, and metadata between two MinIO clusters. Both sites serve reads and writes; changes replicate asynchronously.

Bucket-Level Replication (Disaster Recovery)

# Source and destination buckets must have versioning enabled
mc version enable site-a/important-bucket
mc version enable site-b/important-bucket-replica

# Create replication rule
mc replicate add site-a/important-bucket \
  --remote-bucket https://adminB:passB@minio-b.example.com/important-bucket-replica \
  --replicate "delete,delete-marker,existing-objects"

Monitoring with Prometheus and Grafana

# Generate Prometheus scrape config
mc admin prometheus generate myminio > /etc/prometheus/minio.yml

Key MinIO metrics exported:

Metric	Description
`minio_cluster_capacity_raw_total_bytes`	Total raw storage
`minio_cluster_capacity_usable_free_bytes`	Usable free space
`minio_cluster_nodes_online_total`	Online node count
`minio_cluster_drive_offline_total`	Offline drive count
`minio_s3_requests_total`	Request rate by type
`minio_s3_errors_total`	Error rate

MinIO’s official Grafana dashboard ID is 13502 — import it directly from grafana.com.

MinIO vs Alternatives

Feature	MinIO	Ceph RGW	SeaweedFS	GlusterFS	AWS S3
S3 API	Full	Full	Partial	No	Full (canonical)
Erasure coding	Yes (native)	Yes (via RADOS)	Yes (Reed-Solomon)	No	Managed
Setup complexity	Low	Very high	Low	Medium	None
Kubernetes-native	Yes	Partial	Partial	No	N/A
Object locking (WORM)	Yes	Yes	No	No	Yes
Bucket notifications	Yes	Limited	No	No	Yes (SNS)
Site replication	Yes (active-active)	Yes	Manual	No	Yes (CRR)
IAM / OIDC	Yes	Yes	No	No	Yes (full IAM)
Inline encryption	SSE-S3/KMS	SSE-S3/KMS	SSE	No	SSE-S3/KMS
Best for	S3 workloads	Scale-out all types	High file count	Shared filesystem	Managed cloud

Production Docker Compose with nginx

# docker-compose.yml
services:
  minio:
    image: quay.io/minio/minio:latest
    container_name: minio
    restart: unless-stopped
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: "${MINIO_ROOT_USER}"
      MINIO_ROOT_PASSWORD: "${MINIO_ROOT_PASSWORD}"
      MINIO_KMS_SECRET_KEY: "${MINIO_KMS_SECRET_KEY}"
    volumes:
      - minio_data:/data
    healthcheck:
      test: ["CMD", "mc", "ready", "local"]
      interval: 30s
      timeout: 10s
      retries: 3
    networks:
      - minio_net

  nginx:
    image: nginx:alpine
    container_name: minio-nginx
    restart: unless-stopped
    ports:
      - "443:443"
      - "9001:9001"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - ./certs:/etc/nginx/certs:ro
    depends_on:
      minio:
        condition: service_healthy
    networks:
      - minio_net

volumes:
  minio_data:

networks:
  minio_net:
    driver: bridge

nginx configuration (nginx.conf):

events { worker_connections 1024; }

http {
  upstream minio_api {
    server minio:9000;
  }
  upstream minio_console {
    server minio:9001;
  }

  server {
    listen 443 ssl;
    server_name s3.example.com;
    ssl_certificate /etc/nginx/certs/fullchain.pem;
    ssl_certificate_key /etc/nginx/certs/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    client_max_body_size 0;
    proxy_buffering off;

    location / {
      proxy_pass http://minio_api;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
    }
  }

  server {
    listen 9001 ssl;
    server_name s3.example.com;
    ssl_certificate /etc/nginx/certs/fullchain.pem;
    ssl_certificate_key /etc/nginx/certs/privkey.pem;

    location / {
      proxy_pass http://minio_console;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
    }
  }
}

Gotchas and Edge Cases

Clock skew — S3 signatures expire if the client clock differs from the server by more than 15 minutes. Keep all nodes and clients NTP-synchronized.
Drive count must form valid erasure sets — MinIO requires 4, 6, 8, 10, 12, 14, or 16 drives per erasure set. An odd or prime count will fail at startup.
Object locking is irreversible — You cannot disable object locking on a bucket once enabled. Create compliance-locked buckets only when required.
Distributed mode: all nodes start together — MinIO will not start until it can contact a quorum of nodes. Node startup order matters during cluster initialization.
force_path_style: true for non-AWS endpoints — AWS SDK defaults to virtual-hosted style (bucket.endpoint). Non-AWS MinIO deployments require path-style (endpoint/bucket).
Console WebSocket — nginx must pass Upgrade and Connection headers for the Console’s real-time metrics to work.

Summary

MinIO provides a complete S3-compatible object storage stack self-hosted — erasure coding, IAM, SSE, notifications, and replication.
Run single-node for dev/small deployments; distributed mode with server pools for production scale-out.
Use SSE-KMS + KES for regulated workloads requiring external key management.
Object locking (WORM) in COMPLIANCE mode satisfies SEC 17a-4, FINRA, and similar regulations.
Monitor with Prometheus + Grafana dashboard 13502; set alerts on minio_cluster_drive_offline_total.
The Docker Compose + nginx stack provides TLS termination and a production-ready starting point.