How to Resize a Kubernetes StatefulSet’s Volumes

Kubernetes StatefulSets are used to deploy stateful applications inside your cluster. Each Pod in the StatefulSet can access local persistent volumes that stick to it even after it’s rescheduled. This allows Pods to maintain individual state that’s separate from their neighbors in the set.

Unfortunately these volumes come with a big limitation: Kubernetes doesn’t provide a way to resize them from the StatefulSet object. The spec.resources.requests.storage property of the StatefulSet’s volumeClaimTemplates field is immutable, preventing you from applying any capacity increases you require. This article will show you how to workaround the problem.

Creating a StatefulSet

Copy this YAML and save it to ss.yaml:

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  selector:
    app: nginx
  ports:
  - name: nginx
    port: 80
  clusterIP: None
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 3
  serviceName: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - name: web
          containerPort: 80
        volumeMounts:
        - name: data
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 1Gi

Apply the YAML to your cluster with Kubectl:

$ kubectl apply -f ss.yaml
service/nginx created
statefulset.apps/nginx created

You’ll need a storage class and provisioner in your cluster to run this example. It creates a StatefulSet that runs three replicas of an NGINX web server.

While this isn’t representative of when StatefulSets should be used, it’s adequate as a demo of the volume problems you can face. A volume claim with 1 Gi of storage is mounted to NGINX’s data directory. Your web content could outgrow this relatively small allowance as your service scales. However trying to modify the volumeClaimTemplates.spec.resources.requests.storage field to 10Gi will report the following error when you run kubectl apply:

$ kubectl apply -f ss.yaml
service/nginx unchanged
The StatefulSet "nginx" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

This occurs because almost all the fields of a StatefulSet’s manifest are immutable after creation.

Manually Resizing StatefulSet Volumes

You can bypass the restriction by manually resizing the persistent volume claim (PVC). You’ll then need to recreate the StatefulSet to release and rebind the volume from your Pods. This will trigger the actual volume resize event.

First use Kubectl to find the PVCs associated with your StatefulSet:

$ kubectl get pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES
data-nginx-0   Bound    pvc-ccb2c835-e2d3-4632-b8ba-4c8c142795e4   1Gi        RWO         
data-nginx-1   Bound    pvc-1b0b27fe-3874-4ed5-91be-d8e552e515f2   1Gi        RWO         
data-nginx-2   Bound    pvc-4b7790c2-3ae6-4e04-afee-a2e1bae4323b   1Gi        RWO

There are three PVCs because there are three replicas in the StatefulSet. Each Pod gets its own individual volume.

Now use kubectl edit to adjust the capacity of each volume:

$ kubectl edit pvc data-nginx-0

The PVC’s YAML manifest will appear in your editor. Find the spec.resources.requests.storage field and change it to your new desired capacity:

# ...
spec:
  resources:
    requests:
      storage: 10Gi
# ...

Save and close the file. Kubectl should report that the change has been applied to your cluster.

persistentvolumeclaim/data-nginx-0 edited

Now repeat these steps for the StatefulSet’s remaining PVCs. Listing your cluster’s persistent volumes should then show the new size against each one:

$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               
pvc-0a0d0b15-241f-4332-8c34-a24b61944fb7   10Gi       RWO            Delete           Bound    default/data-nginx-2
pvc-33af452d-feff-429d-80cd-a45232e700c1   10Gi       RWO            Delete           Bound    default/data-nginx-0
pvc-49f3a1c5-b780-4580-9eae-17a1f002e9f5   10Gi       RWO            Delete           Bound    default/data-nginx-1

The claims will maintain the old size for now:

$ kubectl get pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES
data-nginx-0   Bound    pvc-33af452d-feff-429d-80cd-a45232e700c1   10Gi       RWO         
data-nginx-1   Bound    pvc-49f3a1c5-b780-4580-9eae-17a1f002e9f5   10Gi       RWO         
data-nginx-2   Bound    pvc-0a0d0b15-241f-4332-8c34-a24b61944fb7   10Gi       RWO

This is because the volume can’t be resized while Pods are still using it.

Recreating the StatefulSet

Complete the resize by releasing the volume claim from the StatefulSet that’s holding it. Delete the StatefulSet but use the orphan cascading mechanism so its Pods remain in your cluster. This will help minimize downtime.

$ kubectl delete statefulset --cascade=orphan nginx
statefulset.apps "nginx" deleted

Next edit your original YAML file to include the new volume size in the spec.resources.requests.storage file. Then use kubectl apply to recreate the StatefulSet in your cluster:

$ kubectl apply -f ss.yaml
service/nginx unchanged
statefulset.apps/nginx created

The new StatefulSet will assume ownership of the previously orphaned Pods because they’ll already meet its requirements. The volumes may get resized at this point but in most cases you’ll have to manually initiate a rollout that restarts your Pods:

$ kubectl rollout restart statefulset nginx

The rollout proceeds sequentially, targeting one Pod at a time. This ensures your service remains accessible throughout.

Now your PVCs should show the new size:

$ kubectl get pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES
data-nginx-0   Bound    pvc-33af452d-feff-429d-80cd-a45232e700c1   10Gi       RWO         
data-nginx-1   Bound    pvc-49f3a1c5-b780-4580-9eae-17a1f002e9f5   10Gi       RWO         
data-nginx-2   Bound    pvc-0a0d0b15-241f-4332-8c34-a24b61944fb7   10Gi       RWO

Try connecting to one of your Pods to check the increased capacity is visible from within:

$ kubectl exec -it nginx-0 bash
root@nginx-0:/# df -h /usr/share/nginx/html
Filesystem                                                                Size  Used Avail Use% Mounted on
/dev/disk/by-id/scsi-0DO_Volume_pvc-33af452d-feff-429d-80cd-a45232e700c1  9.9G  4.5M  9.4G   1% /usr/share/nginx/html

The Pod’s reporting the expected 10 Gi of storage.

Summary

Kubernetes StatefulSets let you run stateful applications in Kubernetes with persistent storage volumes that are scoped to individual Pods. However the flexibility this permits ends when you need to resize one of your volumes. This is a missing feature which currently requires several manual steps to be completed in sequence.

The Kubernetes maintainers are aware of the issue. There’s an open feature request to develop a solution which should eventually let you initiate volume resizes by editing a StatefulSet’s manifest. This will be much quicker and safer than the current situation.

One final caveat is that volume resizes are dependent on a storage driver that permits dynamic expansion. This feature only became generally available in Kubernetes v1.24 and not all drivers, Kubernetes distributions, and cloud platforms will support it. You can check whether yours does by running kubectl get sc and looking for true in the ALLOWVOLUMEXPANSION column of the storage driver you’re using with your StatefulSets.