minio/docs/kms

KMS Guide

MinIO uses a key-management-system (KMS) to support SSE-S3. If a client requests SSE-S3, or auto-encryption is enabled, the MinIO server encrypts each object with an unique object key which is protected by a master key managed by the KMS.

MinIO still provides native Hashicorp Vault support. However, this feature is deprecated and may be removed in the future. Therefore, we strongly recommend to use the architecture and KMS Guide below. If you have to maintain a legacy MinIO-Vault deployment you can find the legacy documentation here.

Architecture and Concepts

The KMS decouples MinIO as an application-facing storage system from the secure key storage and may be managed by a dedicated security team. MinIO supports commonly-used KMS implementations, like Hashicorp Vault via our KES project. KES makes it possible to scale your KMS horizontally with your storage infrastructure (MinIO clusters). In general, the MinIO-KMS infrastructure looks like this:

     ┌─────────┐         ┌────────────┐         ┌─────────┐  
     │  MinIO  ├─────────┤ KES Server ├─────────┤   KMS   │ 
     └─────────┘         └────────────┘         └─────────┘  

When you scale your storage infrastructure to multiple MinIO clusters your architecture should look like this:

    ┌────────────┐
    │ ┌──────────┴─┬─────╮          ┌────────────┐
    └─┤ ┌──────────┴─┬───┴──────────┤ ┌──────────┴─┬─────────────────╮
      └─┤ ┌──────────┴─┬─────┬──────┴─┤ KES Server ├─────────────────┤
        └─┤   MinIO    ├─────╯        └────────────┘            ┌────┴────┐
          └────────────┘                                        │   KMS   │
                                                                └─────────┘

Observe that all MinIO clusters only have a connection to "their own" KES instance and no direct access to Vault (as one possible KMS implementation). Each KES instance will handle all encrypton/decryption requests made by "its" MinIO cluster such that the central KMS implementation does not have to handle a lot of traffic. Instead, each KES instance will use the central KMS implementation as secure key store and fetches the required master keys from it.

Get Started Guide

In the subsequent sections this guide shows how to setup a MinIO-KMS deployment with Hashicorp Vault as KMS implementation. Therefore, it shows how to setup and configure:

• A Vault server as central key store.
• A KES server instance as middleware between MinIO and Vault.
• The MinIO instance itself.

Please note that this guide uses self-signed certificates for simplicity. In a production deployment you should use X.509 certificates issued by a "public" (e.g. Let's Encrypt) or your organization-internal CA.

This guide shows how to set up three different servers on the same machine:

• The Vault server as https://127.0.0.1:8200
• The KES server as https://127.0.0.1:7373
• The MinIO server as https://127.0.0.1:9000

1 Prerequisites

Install MinIO, KES and Vault. For MinIO take a look at the MinIO quickstart guide. Then install KES and download the latest Vault binary for your OS and platform.

2 Generate TLS certificates

Since KES sends object encryption keys to MinIO and Vault sends master keys (used to encrypt the object encryption keys) to KES we absolutely need TLS connections between MinIO, KES and Vault. Therefore, we need to generate at least two TLS certificates.

2.1 Generate a TLS certificate for Vault

To generate a new private key for Vault's certificate run the following openssl command:

openssl ecparam -genkey -name prime256v1 | openssl ec -out vault-tls.key

Then generate a new TLS certificate for the private/public key pair via:

openssl req -new -x509 -days 365 \
    -key vault-tls.key \
    -out vault-tls.crt \
    -subj "/C=/ST=/L=/O=/CN=localhost" \
    -addext "subjectAltName = IP:127.0.0.1"

You can ignore output messages like: req: No value provided for Subject Attribute C, skipped. OpenSSL just tells you that you haven't specified a country, state, a.s.o for the certificate subject. You may want to adjust the X.509 subject (-subj parameter) and subject alternative name (SAN). Note that this is a self-signed certificate. For production deployments this certificate should be issued by a CA.

2.2 Generate a TLS certificate for KES

To generate a new private key for KES's certificate run the following openssl command:

openssl ecparam -genkey -name prime256v1 | openssl ec -out kes-tls.key

Then generate a new TLS certificate for the private/public key pair via:

openssl req -new -x509 -days 365 \
    -key kes-tls.key \
    -out kes-tls.crt \
    -subj "/C=/ST=/L=/O=/CN=localhost" \
    -addext "subjectAltName = IP:127.0.0.1"

You can ignore output messages like: req: No value provided for Subject Attribute C, skipped. OpenSSL just tells you that you haven't specified a country, state, a.s.o for the certificate subject. You may want to adjust the X.509 subject (-subj parameter) and subject alternative name (SAN). Note that this is a self-signed certificate. For production deployments this certificate should be issued by a CA.

2.3 Generate a TLS certificate for MinIO (Optional)

This step is optional. However, we recommend to up/download your S3 objects via TLS - especially when they should be encrypted at the storage backend with a KMS.

Checkout the MinIO TLS guide for configuring MinIO and TLS.

3 Set up Vault

On unix-like systems, Vault uses the mlock syscall to prevent the OS from writing in-memory data to disk (swapping). Therefore, you should give the Vault executable the ability to use the mlock syscall without running the process as root. To do so run:

sudo setcap cap_ipc_lock=+ep $(readlink -f $(which vault))

Then create the Vault config file:

cat > vault-config.json <<EOF
{
  "api_addr": "https://127.0.0.1:8200",
  "backend": {
    "file": {
      "path": "vault/file"
    }
  },
  "default_lease_ttl": "168h",
  "max_lease_ttl": "720h",
  "listener": {
    "tcp": {
      "address": "0.0.0.0:8200",
      "tls_cert_file": "vault-tls.crt",
      "tls_key_file": "vault-tls.key",
      "tls_min_version": "tls12"
    }
  }
}
EOF

Note that we run Vault with a file backend. For high-availability you may want to use a different backend - like etcd or consul.

Finally, start the Vault server via:

vault server -config vault-config.json

3.1 Initialize and unseal Vault

In a separate terminal window set the VAULT_ADDR env. variable to your Vault server:

export VAULT_ADDR='https://127.0.0.1:8200'

Further, you may want to run export VAULT_SKIP_VERIFY=true if Vault uses a self-signed TLS certificate. When Vault serves a TLS certificate that has been issued by a CA that is trusted by your machine - e.g. Let's Encrypt - then you don't need to run this command.

Then initialize Vault via:

vault operator init

Vault will print n (5 by default) unseal key shares of which at least m (3 by default) are required to regenerate the actual unseal key to unseal Vault. Therefore, make sure to remember them. In particular, keep those unseal key shares at a secure and durable location.

You should see some output similar to:

Unseal Key 1: eyW/+8ZtsgT81Cb0e8OVxzJAQP5lY7Dcamnze+JnWEDT
Unseal Key 2: 0tZn+7QQCxphpHwTm6/dC3LpP5JGIbYl6PK8Sy79R+P2
Unseal Key 3: cmhs+AUMXUuB6Lzsvgcbp3bRT6VDGQjgCBwB2xm0ANeF
Unseal Key 4: /fTPpec5fWpGqWHK+uhnnTNMQyAbl5alUi4iq2yNgyqj
Unseal Key 5: UPdDVPto+H6ko+20NKmagK40MOskqOBw4y/S51WpgVy/

Initial Root Token: s.zaU4Gbcu0Wh46uj2V3VuUde0

Vault is initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

Now, set the env. variable VAULT_TOKEN to the root token printed by the command before:

export VAULT_TOKEN=s.zaU4Gbcu0Wh46uj2V3VuUde0

Then use any of the previously generated key shares to unseal Vault.

vault operator unseal eyW/+8ZtsgT81Cb0e8OVxzJAQP5lY7Dcamnze+JnWEDT
vault operator unseal 0tZn+7QQCxphpHwTm6/dC3LpP5JGIbYl6PK8Sy79R+P2
vault operator unseal cmhs+AUMXUuB6Lzsvgcbp3bRT6VDGQjgCBwB2xm0ANeF

Once you have submitted enough valid key shares Vault will become unsealed and able to process requests.

3.2 Enable Vault's K/V backend

The cryptographic master keys (but not the object encryption keys) will be stored at Vault. Therefore, we need to enable Vault's K/V backend. To do so run:

vault secrets enable kv

3.3 Enable AppRole authentication

Since we want connect one/multiple KES server to Vault later, we have to enable AppRole authentication. To do so run:

vault auth enable approle

3.4 Create an access policy for the K/V engine

The following policy determines how an application (i.e. KES server) can interact with Vault.

cat > minio-kes-policy.hcl <<EOF
path "kv/minio/*" {
  capabilities = [ "create", "read", "delete" ]
}

EOF

Observe the path-prefix minio in kv/minio/*. This prefix ensures that the KES server can only read from and write to entries under minio/* - but not under some-app/*. How to separate domains on the K/V engine depends on your infrastructure and security requirements.

Then we upload the policy to Vault:

vault policy write minio-key-policy ./minio-kes-policy.hcl

3.5 Create an new AppRole ID and bind it to a policy

Now, we need to create a new AppRole ID and grant that ID specific permissions. The application (i.e. KES server) will authenticate to Vault via the AppRole role ID and secret ID and is only allowed to perform operations granted by the specific policy.

So, we first create a new role for our KES server:

vault write auth/approle/role/kes-role token_num_uses=0  secret_id_num_uses=0  period=5m

Then we bind a policy to the role:

vault write auth/approle/role/kes-role policies=minio-key-policy

Finally, we request an AppRole role ID and secret ID from Vault.
First, the role ID:

vault read auth/approle/role/kes-role/role-id 

Then the secret ID:

vault write -f auth/approle/role/kes-role/secret-id

We are only interested in the secret_id - not in the secret_id_accessor.

4 Set up KES

Similar to Vault, KES uses the mlock syscall on linux systems to prevent the OS from writing in-memory data to disk (swapping). Therefore, you should give the KES executable the ability to use the mlock syscall without running the process as root. To do so run:

sudo setcap cap_ipc_lock=+ep $(readlink -f $(which kes))

4.1 Create an identity for MinIO

Each user or application must present a valid X.509 certificate when connecting to the KES server (mTLS). The KES server will accept/reject the connection attempt and applies policies based on the certificate.

Therefore, each MinIO cluster needs a X.509 TLS certificate for client authentication. You can create a (self-signed) certificate by running:

kes tool identity new MinIO --key=minio.key --cert=minio.cert --time=8760h

Note that MinIO is the subject name. You may choose a more appropriate name for your deployment scenario. Also, for production deployments we recommend to get a TLS certificate for client authentication that has been issued by a CA.

To get the identity of a X.509 certificate run:

kes tool identity of minio.cert

This command works with any (valid) X.509 certificate - regardless how it has been created - and produces an output similar to:

Identity: dd46485bedc9ad2909d2e8f9017216eec4413bc5c64b236d992f7ec19c843c5f

4.2 Create the KES config file

Now, we can create the KES config file and start the KES server.

# The TCP address (ip:port) for the KES server to listen on.
address: 0.0.0.0:7373

tls:
  key:  kes-tls.key
  cert: kes-tls.crt

policy:
  minio:
    paths:
    - /v1/key/create/minio-*
    - /v1/key/generate/minio-*
    - /v1/key/decrypt/minio-*
    identities:
    - dd46485bedc9ad2909d2e8f9017216eec4413bc5c64b236d992f7ec19c843c5f

cache:
  expiry:
    any:    5m0s 
    unused: 20s 

keys:
  vault:
    endpoint: https://127.0.0.1:8200  # The Vault endpoint - i.e. https://127.0.0.1:8200
    prefix:   minio                   # The domain resp. prefix at Vault's K/V backend 

    approle:
      id:     ""    # Your AppRole Role ID 
      secret: ""    # Your AppRole Secret ID
      retry:  15s   # Duration until the server tries to re-authenticate after connection loss.

    tls:
      ca: vault-tls.crt  # Since we use self-signed certificates
    
    status:
      ping: 10s

Please change the value of identities in the policy section to the identity of your minio.cert. Also, insert the AppRole role ID and secret ID that you have created previously during the Vault setup. You can find a documented config file with all available parameters here.

Finally, start the KES server via:

kes server --config=kes-config.yaml --mlock --root=disabled --auth=off

Note that we effectively disable the special root identity since we don't need it. For more information about the KES access control model and identities checkout: KES Concepts. Further, note that we disable --auth=off since the client X.509 certificate is a self-signed certificate.

4.3 Create a new master key

Before we can proceed with the MinIO setup we need to create a new master key. Therefore we use the MinIO identity and the KES CLI.

In a new terminal window become the MinIO identity via:

export KES_CLIENT_TLS_KEY_FILE=minio.key
export KES_CLIENT_TLS_CERT_FILE=minio.cert

Then create the master key by running:

kes key create minio-key-1 -k

The -k flag is only required since we use self-signed certificates. Also, observe that based on the server config file the MinIO identity is only allowed to create/use master keys that start with minio-. So, trying to create a key e.g. kes key create my-key-1 -k will fail with a prohibited by policy error.

5 Set up the MinIO server

The MinIO server will need to know the KES server endpoint, its mTLS client certificate for authentication and authorization and the default master key name.

export MINIO_KMS_KES_ENDPOINT=https://localhost:7373
export MINIO_KMS_KES_KEY_FILE=minio.key
export MINIO_KMS_KES_CERT_FILE=minio.cert
export MINIO_KMS_KES_KEY_NAME=minio-key-1
export MINIO_KMS_KES_CA_PATH=kes-tls.crt

The MINIO_KMS_KES_CAPATH is only required since we use self-signed certificates.

Optionally, enable auto-encryption to encrypt uploaded objects automatically:

export MINIO_KMS_AUTO_ENCRYPTION=on

For more information about auto-encryption see: Appendix A

Then start the MinIO server:

export MINIO_ACCESS_KEY=minio
export MINIO_SECRET_KEY=minio123

minio server ~/export

Appendix A - Auto-Encryption

Optionally, you can instruct the MinIO server to automatically encrypt all objects with keys from the KES server - even if the client does not specify any encryption headers during the S3 PUT operation.

Auto-Encryption is especially useful when the MinIO operator wants to ensure that all data stored on MinIO gets encrypted before it's written to the storage backend.

To enable auto-encryption set the environment variable to on:

export MINIO_KMS_AUTO_ENCRYPTION=on

Note that auto-encryption only affects requests without S3 encryption headers. So, if a S3 client sends e.g. SSE-C headers, MinIO will encrypt the object with the key sent by the client and won't reach out to the KMS.

To verify auto-encryption, use the mc command:

mc cp test.file myminio/crypt/
test.file:              5 B / 5 B  ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓  100.00% 337 B/s 0s
mc stat myminio/crypt/test.file
Name      : test.file
...
Encrypted :
  X-Amz-Server-Side-Encryption: AES256

Appendix B - Specify a master key

Instead of a proper KMS setup you can also test MinIO encryption using a KMS master key. A single master key via env. variable is for testing purposes only and not recommended for production deployments.

A KMS master key consists of a master-key ID (CMK) and the 256 bit master key encoded as HEX value separated by a :. A KMS master key can be specified directly using:

export MINIO_KMS_MASTER_KEY=minio-demo-key:6368616e676520746869732070617373776f726420746f206120736563726574

Please use your own master key. A random master key can be generated using e.g. this command on Linux/Mac/BSD systems:

head -c 32 /dev/urandom | xxd -c 32 -ps

---

Alternatively, you may pass a master key as a Docker secret.

echo "my-minio-key:6368616e676520746869732070617373776f726420746f206120736563726574" | docker secret create kms_master_key

To use another secret name, follow the instructions above and replace kms_master_key with your custom names (e.g. my_kms_master_key). Then, set the MINIO_KMS_MASTER_KEY_FILE environment variable to your secret name:

export MINIO_KMS_MASTER_KEY_FILE=my_kms_master_key

Explore Further

• Use mc with MinIO Server
• Use aws-cli with MinIO Server
• Use s3cmd with MinIO Server
• Use minio-go SDK with MinIO Server
• The MinIO documentation website