Skip to main content

Volume Eviction

Volume eviction/migration is one of most important funcions in the HwameiStor, especially in the production environment. HwameiStor will keep the data in the volume when migrate it.

Once a Kubernetes node or pod is evicted by the system for any reason, HwameiStor will detect all the volume replicas located on this node, and automatically migrate them to other avaliable nodes. So that, the evicted pods can be rescheduled to another node and continue to run.

Node Eviction​

In a Kubernetes cluster, a node can be drained by using the following procedure. So that, all the pods and volume replicas on this node will be evicted, and then continue the services on other avaliable nodes.

kubectl drain k8s-node-1 --ignore-daemonsets=true

Check if all the volumes' migration complete or not by running:

kubectl get LocalStorageNode k8s-node-1 -o yaml

The output may look like:

apiVersion: hwameistor.io/v1alpha1
kind: LocalStorageNode
metadata:
  creationTimestamp: "2022-10-11T07:41:58Z"
  generation: 1
  name: k8s-node-1
  resourceVersion: "6402198"
  uid: c71cc6ac-566a-4e0b-8687-69679b07471f
spec:
  hostname: k8s-node-1
  storageIP: 10.6.113.22
  topogoly:
    region: default
    zone: default
status:
  ...
  pools:
    LocalStorage_PoolHDD:
      class: HDD
      disks:
      - capacityBytes: 17175674880
        devPath: /dev/sdb
        state: InUse
        type: HDD
      freeCapacityBytes: 16101933056
      freeVolumeCount: 999
      name: LocalStorage_PoolHDD
      totalCapacityBytes: 17175674880
      totalVolumeCount: 1000
      type: REGULAR
      usedCapacityBytes: 1073741824
      usedVolumeCount: 1
      volumeCapacityBytesLimit: 17175674880
      # ** make sure volumes is empty ** #
      volumes:  
  state: Ready

Check if there is any volume replica still located in the evicted node by running:

kubectl get localvolumereplica

The output may look like:

NAME                                              CAPACITY     NODE         STATE   SYNCED   DEVICE                                                                  AGE
pvc-1427f36b-adc4-4aef-8d83-93c59064d113-957f7g   1073741824   k8s-node-3   Ready   true     /dev/LocalStorage_PoolHDD-HA/pvc-1427f36b-adc4-4aef-8d83-93c59064d113   20h
pvc-1427f36b-adc4-4aef-8d83-93c59064d113-qlpbmq   1073741824   k8s-node-2   Ready   true     /dev/LocalStorage_PoolHDD-HA/pvc-1427f36b-adc4-4aef-8d83-93c59064d113   30m
pvc-6ca4c0d4-da10-4e2e-83b2-19cbf5c5e3e4-scrxjb   1073741824   k8s-node-2   Ready   true     /dev/LocalStorage_PoolHDD/pvc-6ca4c0d4-da10-4e2e-83b2-19cbf5c5e3e4      30m
pvc-f8f017f9-eb09-4fbe-9795-a6e2d6873148-5t782b   1073741824   k8s-node-2   Ready   true     /dev/LocalStorage_PoolHDD-HA/pvc-f8f017f9-eb09-4fbe-9795-a6e2d6873148   30m

In some cases, user doesn't want to migrate the volumes when draining a node. User can add a label into the node before draining it.

kubectl label node k8s-node-1 hwameistor.io/eviction=disable

Pod Eviction​

When a Kubernetes node is overloaded, it will evict some low-priority pods to recycle system's resources to keep other pods safe. HwameiStor will detect the evicted pod and migrate the associated volumes to another available node. So that, the pod can continue to run on it.

Pod Migration​

The migration can be pro-actively triggered on the pod and associated HwameiStor volume by using either one of following methods.

  • Method #1

    kubectl label pod mysql-pod hwameistor.io/eviction=start
    kubectl delete pod mysql-pod

  • Method #2

    $ cat << EOF | kubectl apply -f -
    apiVersion: hwameistor.io/v1alpha1
    kind: LocalVolumeMigrate
    metadata:
      name: migrate-pvc-6ca4c0d4-da10-4e2e-83b2-19cbf5c5e3e4
    spec:
      sourceNode: k8s-node-1
      targetNodesSuggested: 
      - k8s-node-2
      - k8s-node-3
      volumeName: pvc-6ca4c0d4-da10-4e2e-83b2-19cbf5c5e3e4
      migrateAllVols: true
    EOF

    $ kubectl delete pod mysql-pod