minio/docs/orchestration/kubernetes-yaml

Cloud Native Deployment of Minio on Kubernetes

Table of Contents

• Prerequisites

• Minio Standalone Server Deployment

  • Standalone Quickstart
  • Create Persistent Volume Claim
  • Create Deployment
  • Create LoadBalancer Service
  • Update existing Minio Deployment
  • Resource cleanup

• Minio Distributed Server Deployment

  • Distributed Quickstart
  • Create Minio Headless Service
  • Create Minio Statefulset
  • Create LoadBalancer Service
  • Update existing Minio StatefulSet
  • Deploying on cluster nodes with local host path
  • Resource cleanup

• Minio GCS Gateway Deployment

  • GCS Gateway Quickstart
  • Create GCS Credentials Secret
  • Create Minio GCS Gateway Deployment
  • Create Minio LoadBalancer Service
  • Update Existing Minio GCS Deployment
  • Resource cleanup

Prerequisites

To run this example, you need Kubernetes version >=1.4 cluster installed and running, and that you have installed the kubectl command line tool in your path. Please see the getting started guides for installation instructions for your platform.

Minio Standalone Server Deployment

The following section describes the process to deploy standalone Minio server on Kubernetes. The deployment uses the official Minio Docker image from Docker Hub.

This section uses following core components of Kubernetes:

• Pods
• Services
• Deployments
• Persistent Volume Claims

Standalone Quickstart

Run the below commands to get started quickly

kubectl create -f https://github.com/minio/minio/blob/master/docs/orchestration/kubernetes-yaml/minio-standalone-pvc.yaml?raw=true
kubectl create -f https://github.com/minio/minio/blob/master/docs/orchestration/kubernetes-yaml/minio-standalone-deployment.yaml?raw=true
kubectl create -f https://github.com/minio/minio/blob/master/docs/orchestration/kubernetes-yaml/minio-standalone-service.yaml?raw=true

Create Persistent Volume Claim

Minio needs persistent storage to store objects. If there is no persistent storage, the data stored in Minio instance will be stored in the container file system and will be wiped off as soon as the container restarts.

Create a persistent volume claim (PVC) to request storage for the Minio instance. Kubernetes looks out for PVs matching the PVC request in the cluster and binds it to the PVC automatically.

This is the PVC description.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  # This name uniquely identifies the PVC. This is used in deployment.
  name: minio-pv-claim
spec:
  # Read more about access modes here: http://kubernetes.io/docs/user-guide/persistent-volumes/#access-modes
  accessModes:
    # The volume is mounted as read-write by a single node
    - ReadWriteOnce
  resources:
    # This is the request for storage. Should be available in the cluster.
    requests:
      storage: 10Gi

Create the PersistentVolumeClaim

kubectl create -f https://github.com/minio/minio/blob/master/docs/orchestration/kubernetes-yaml/minio-standalone-pvc.yaml?raw=true
persistentvolumeclaim "minio-pv-claim" created

Create Minio Deployment

A deployment encapsulates replica sets and pods — so, if a pod goes down, replication controller makes sure another pod comes up automatically. This way you won’t need to bother about pod failures and will have a stable Minio service available.

This is the deployment description.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  # This name uniquely identifies the Deployment
  name: minio
spec:
  strategy:
    # Specifies the strategy used to replace old Pods by new ones
    # Refer: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#strategy
    type: Recreate
  template:
    metadata:
      labels:
        # This label is used as a selector in Service definition
        app: minio
    spec:
      # Volumes used by this deployment
      volumes:
      - name: data
        # This volume is based on PVC
        persistentVolumeClaim:
          # Name of the PVC created earlier
          claimName: minio-pv-claim
      containers:
      - name: minio
        # Volume mounts for this container
        volumeMounts:
        # Volume 'data' is mounted to path '/data'
        - name: data 
          mountPath: "/data"
        # Pulls the lastest Minio image from Docker Hub
        image: minio/minio:RELEASE.2018-05-04T23-13-12Z
        args:
        - server
        - /data
        env:
        # Minio access key and secret key
        - name: MINIO_ACCESS_KEY
          value: "minio"
        - name: MINIO_SECRET_KEY
          value: "minio123"
        ports:
        - containerPort: 9000
        # Readiness probe detects situations when Minio server instance
        # is not ready to accept traffic. Kubernetes doesn't forward
        # traffic to the pod till readiness checks fail.
        readinessProbe:
          httpGet:
            path: /minio/health/ready
            port: 9000
          initialDelaySeconds: 120
          periodSeconds: 20
        # Liveness probe detects situations where Minio server instance
        # is not working properly and needs restart. Kubernetes automatically
        # restarts the pods if liveness checks fail.
        livenessProbe:
          httpGet:
            path: /minio/health/live
            port: 9000
          initialDelaySeconds: 120
          periodSeconds: 20

Create the Deployment

kubectl create -f https://github.com/minio/minio/blob/master/docs/orchestration/kubernetes-yaml/minio-standalone-deployment.yaml?raw=true
deployment "minio-deployment" created

Create Minio Service

Now that you have a Minio deployment running, you may either want to access it internally (within the cluster) or expose it as a Service onto an external (outside of your cluster, maybe public internet) IP address, depending on your use case. You can achieve this using Services. There are 3 major service types — default type is ClusterIP, which exposes a service to connection from inside the cluster. NodePort and LoadBalancer are two types that expose services to external traffic.

In this example, we expose the Minio Deployment by creating a LoadBalancer service. This is the service description.

apiVersion: v1
kind: Service
metadata:
  # This name uniquely identifies the service
  name: minio-service
spec:
  type: LoadBalancer
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
  selector:
    # Looks for labels `app:minio` in the namespace and applies the spec
    app: minio

Create the Minio service

kubectl create -f https://github.com/minio/minio/blob/master/docs/orchestration/kubernetes-yaml/minio-standalone-service.yaml?raw=true
service "minio-service" created

The LoadBalancer service takes couple of minutes to launch. To check if the service was created successfully, run the command

kubectl get svc minio-service
NAME            CLUSTER-IP     EXTERNAL-IP       PORT(S)          AGE
minio-service   10.55.248.23   104.199.249.165   9000:31852/TCP   1m

Update existing Minio Deployment

You can update an existing Minio deployment to use a newer Minio release. To do this, use the kubectl set image command:

kubectl set image deployment/minio-deployment minio=<replace-with-new-minio-image>

Kubernetes will restart the deployment to update the image. You will get a message as shown below, on successful update:

deployment "minio-deployment" image updated

Standalone Resource cleanup

You can cleanup the cluster using

kubectl delete deployment minio \
&&  kubectl delete pvc minio-pv-claim \
&& kubectl delete svc minio-service

Minio Distributed Server Deployment

The following document describes the process to deploy distributed Minio server on Kubernetes. This example uses the official Minio Docker image from Docker Hub.

This example uses following core components of Kubernetes:

• Pods
• Services
• Statefulsets

Distributed Quickstart

Run the below commands to get started quickly

kubectl create -f https://github.com/minio/minio/blob/master/docs/orchestration/kubernetes-yaml/minio-distributed-headless-service.yaml?raw=true
kubectl create -f https://github.com/minio/minio/blob/master/docs/orchestration/kubernetes-yaml/minio-distributed-statefulset.yaml?raw=true
kubectl create -f https://github.com/minio/minio/blob/master/docs/orchestration/kubernetes-yaml/minio-distributed-service.yaml?raw=true

Create Minio Headless Service

Headless Service controls the domain within which StatefulSets are created. The domain managed by this Service takes the form: $(service name).$(namespace).svc.cluster.local (where “cluster.local” is the cluster domain), and the pods in this domain take the form: $(pod-name-{i}).$(service name).$(namespace).svc.cluster.local. This is required to get a DNS resolvable URL for each of the pods created within the Statefulset.

This is the Headless service description.

apiVersion: v1
kind: Service
metadata:
  name: minio
  labels:
    app: minio
spec:
  clusterIP: None
  ports:
    - port: 9000
      name: minio
  selector:
    app: minio

Create the Headless Service

$ kubectl create -f https://github.com/minio/minio/blob/master/docs/orchestration/kubernetes-yaml/minio-distributed-headless-service.yaml?raw=true
service "minio" created

Create Minio Statefulset

A StatefulSet provides a deterministic name and a unique identity to each pod, making it easy to deploy stateful distributed applications. To launch distributed Minio you need to pass drive locations as parameters to the minio server command. Then, you’ll need to run the same command on all the participating pods. StatefulSets offer a perfect way to handle this requirement.

This is the Statefulset description.

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  # This name uniquely identifies the StatefulSet
  name: minio
spec:
  serviceName: minio
  replicas: 4
  selector:
    matchLabels:
      app: minio # has to match .spec.template.metadata.labels
  template:
    metadata:
      labels:
        app: minio # has to match .spec.selector.matchLabels
    spec:
      containers:
      - name: minio
        env:
        - name: MINIO_ACCESS_KEY
          value: "minio"
        - name: MINIO_SECRET_KEY
          value: "minio123"
        image: minio/minio:RELEASE.2018-05-04T23-13-12Z
        args:
        - server
        - http://minio-0.minio.default.svc.cluster.local/data
        - http://minio-1.minio.default.svc.cluster.local/data
        - http://minio-2.minio.default.svc.cluster.local/data
        - http://minio-3.minio.default.svc.cluster.local/data
        ports:
        - containerPort: 9000
        # These volume mounts are persistent. Each pod in the PetSet
        # gets a volume mounted based on this field.
        volumeMounts:
        - name: data
          mountPath: /data
        # Liveness probe detects situations where Minio server instance
        # is not working properly and needs restart. Kubernetes automatically
        # restarts the pods if liveness checks fail.
        livenessProbe:
          httpGet:
            path: /minio/health/live
            port: 9000
          initialDelaySeconds: 120
          periodSeconds: 20
  # These are converted to volume claims by the controller
  # and mounted at the paths mentioned above.
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi

Create the Statefulset

$ kubectl create -f https://github.com/minio/minio/blob/master/docs/orchestration/kubernetes-yaml/minio-distributed-statefulset.yaml?raw=true
statefulset "minio" created

Create Minio Service

Now that you have a Minio statefulset running, you may either want to access it internally (within the cluster) or expose it as a Service onto an external (outside of your cluster, maybe public internet) IP address, depending on your use case. You can achieve this using Services. There are 3 major service types — default type is ClusterIP, which exposes a service to connection from inside the cluster. NodePort and LoadBalancer are two types that expose services to external traffic.

In this example, we expose the Minio Deployment by creating a LoadBalancer service. This is the service description.

apiVersion: v1
kind: Service
metadata:
  name: minio-service
spec:
  type: LoadBalancer
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
  selector:
    app: minio

Create the Minio service

$ kubectl create -f https://github.com/minio/minio/blob/master/docs/orchestration/kubernetes-yaml/minio-distributed-service.yaml?raw=true
service "minio-service" created

The LoadBalancer service takes couple of minutes to launch. To check if the service was created successfully, run the command

$ kubectl get svc minio-service
NAME            CLUSTER-IP     EXTERNAL-IP       PORT(S)          AGE
minio-service   10.55.248.23   104.199.249.165   9000:31852/TCP   1m

Update existing Minio StatefulSet

You can update an existing Minio StatefulSet to use a newer Minio release. To do this, use the kubectl patch statefulset command:

kubectl patch statefulset minio --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"<replace-with-new-minio-image>"}]'

On successful update, you should see the output below

statefulset "minio" patched

Then delete all the pods in your StatefulSet one by one as shown below. Kubernetes will restart those pods for you, using the new image.

kubectl delete minio-0

Resource cleanup

You can cleanup the cluster using

kubectl delete statefulset minio \
&&  kubectl delete svc minio \
&& kubectl delete svc minio-service

Deploying on cluster nodes with local host path

If your cluster does not have a storage solution or PV abstraction, you must explicitly define what nodes you wish to run Minio on, and define a homogeneous path to a local fast block device available on every host.

This must be changed in the example daemonset: minio-distributed-daemonset.yaml

Specifically the hostpath:

        hostPath:
          path: /data/minio/

And the list of hosts:

        - http://hostname1:9000/data/minio
        - http://hostname2:9000/data/minio
        - http://hostname3:9000/data/minio
        - http://hostname4:9000/data/minio

Once deployed, tag the defined host with the minio-server=true label:

kubectl label node hostname1  -l minio-server=true
kubectl label node hostname2  -l minio-server=true
kubectl label node hostname3  -l minio-server=true
kubectl label node hostname4  -l minio-server=true

Minio GCS Gateway Deployment

The following section describes the process to deploy Minio GCS Gateway on Kubernetes. The deployment uses the official Minio Docker image from Docker Hub.

This section uses following core components of Kubernetes:

• Secrets
• Services
• Deployments

GCS Gateway Quickstart

Create the Google Cloud Service credentials file using the steps mentioned here.

Use the path of file generated above to create a Kubernetes secret.

kubectl create secret generic gcs-credentials --from-file=/path/to/gcloud/credentials/application_default_credentials.json

Then download the minio-gcs-gateway-deployment.yaml file

wget https://github.com/minio/minio/blob/master/docs/orchestration/kubernetes-yaml/minio-gcs-gateway-deployment.yaml?raw=true

Update the section gcp_project_id with your GCS project ID. Then run

kubectl create -f minio-gcs-gateway-deployment.yaml
kubectl create -f https://github.com/minio/minio/blob/master/docs/orchestration/kubernetes-yaml/minio-gcs-gateway-service.yaml?raw=true

Create GCS Credentials Secret

A secret is intended to hold sensitive information, such as passwords, OAuth tokens, and ssh keys. Putting this information in a secret is safer and more flexible than putting it verbatim in a pod definition or in a docker image.

Create the Google Cloud Service credentials file using the steps mentioned here.

Use the path of file generated above to create a Kubernetes secret.

kubectl create secret generic gcs-credentials --from-file=/path/to/gcloud/credentials/application_default_credentials.json

Create Minio GCS Gateway Deployment

A deployment encapsulates replica sets and pods — so, if a pod goes down, replication controller makes sure another pod comes up automatically. This way you won’t need to bother about pod failures and will have a stable Minio service available.

Minio Gateway uses GCS as its storage backend and need to use a GCP projectid to identify your credentials. Update the section gcp_project_id with your GCS project ID. This is the deployment description.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  # This name uniquely identifies the Deployment
  name: minio-deployment
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        # Label is used as selector in the service.
        app: minio
    spec:
      # Refer to the secret created earlier
      volumes:
      - name: gcs-credentials
        secret:
          # Name of the Secret created earlier
          secretName: gcs-credentials
      containers:
      - name: minio
        # Pulls the default Minio image from Docker Hub
        image: minio/minio:RELEASE.2018-05-04T23-13-12Z
        args:
        - gateway
        - gcs
        - gcp_project_id
        env:
        # Minio access key and secret key
        - name: MINIO_ACCESS_KEY
          value: "minio"
        - name: MINIO_SECRET_KEY
          value: "minio123"
        # Google Cloud Service uses this variable
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: "/etc/credentials/application_default_credentials.json"
        ports:
        - containerPort: 9000
        # Mount the volume into the pod
        volumeMounts:
        - name: gcs-credentials
          mountPath: "/etc/credentials"
          readOnly: true

Create the Deployment

kubectl create -f https://github.com/minio/minio/blob/master/docs/orchestration/kubernetes-yaml/minio-gcs-gateway-deployment.yaml?raw=true
deployment "minio-deployment" created

Create Minio LoadBalancer Service

Now that you have a Minio deployment running, you may either want to access it internally (within the cluster) or expose it as a Service onto an external (outside of your cluster, maybe public internet) IP address, depending on your use case. You can achieve this using Services. There are 3 major service types — default type is ClusterIP, which exposes a service to connection from inside the cluster. NodePort and LoadBalancer are two types that expose services to external traffic.

In this example, we expose the Minio Deployment by creating a LoadBalancer service. This is the service description.

apiVersion: v1
kind: Service
metadata:
  name: minio-service
spec:
  type: LoadBalancer
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
  selector:
    app: minio

Create the Minio service

kubectl create -f https://github.com/minio/minio/blob/master/docs/orchestration/kubernetes-yaml/minio-gcs-gateway-service.yaml?raw=true
service "minio-service" created

The LoadBalancer service takes couple of minutes to launch. To check if the service was created successfully, run the command

kubectl get svc minio-service
NAME            CLUSTER-IP     EXTERNAL-IP       PORT(S)          AGE
minio-service   10.55.248.23   104.199.249.165   9000:31852/TCP   1m

Update Existing Minio GCS Deployment

You can update an existing Minio deployment to use a newer Minio release. To do this, use the kubectl set image command:

kubectl set image deployment/minio-deployment minio=<replace-with-new-minio-image>

Kubernetes will restart the deployment to update the image. You will get a message as shown below, on successful update:

deployment "minio-deployment" image updated

GCS Gateway Resource Cleanup

You can cleanup the cluster using

kubectl delete deployment minio-deployment \
&&  kubectl delete secret gcs-credentials