Overview
This page delves into the practicalities of cloning virtual machines in a Kubernetes/KubeVirt cluster using the VastCSI driver.
Our key objectives include testing the VM's cloning functionality in the KubeVirt setup, subjecting the cluster to stress tests involving simultaneous cloning and PVC/snapshots deletion, and finally, providing a detailed summary of the specific environment and commands used for future reference.
Let's kick off with a brief rundown of KubeVirt and the technical components in play
Important Note:
Environment Building Blocks
KubeVirt | CDI | DataVolume | VM |
KubeVirt is an extension for Kubernetes that enables the management and orchestration of virtual machines (VMs) alongside containerized workloads. It allows users to run and manage traditional VMs and Kubernetes workloads in a unified environment, providing flexibility and ease of use for diverse workloads. | The Containerized Data Importer (CDI) project provides facilities for enabling Persistent Volume Claims (PVCs) to be used as disks for KubeVirt VMs by way of DataVolumes. We will import disk images from a web server (Fedora, CentOS) to DataVolumes.
| In KubeVirt, a | A virtual machine managed by KubeVirt within a Kubernetes cluster. It runs as a pod, leveraging Kubernetes infrastructure for orchestration and sharing the same underlying host resources. Can be accessed as console using either |
Assuming the environment is ready for functionality and the stress test is complete, please proceed to the next section.
If you're planning to install Env from scratch, you might want to start by looking at Appendix A - Setup.
#1 - Clone an RW PVC from a snapshot → Attach to a new vm
Flow
Import a dataVolume from an image into a "golden image" PVC named Fedora.
ReadWriteMany - meaning many VMs can use the same PVC.
ReadWriteOnce - PVCs can be accessed by only one VM at a time. → Since this is a golden image, and we want to make some modifications such as ssh keys, root password, and add some packages, we'll use this option. Then clone the PVC N times (per VMs).
Clone golden PVC, PVC per vm.
Create the VM’s pre PVC - write data to the VM.
Create a snapshot from a PVC
Restore a writable (ReadWriteMany) from the snapshot to a new PVC
Attach a new VM to the newly created RW PVC
Functionality Tests
The procedure involves creating a virtual machine (VM) from a Persistent Volume Claim (PVC) associated with a golden image. Subsequently, the data is written internally within the VM. A snapshot of its PVC is created, resulting in a new VM derived from it. The objective is to determine the image's writability and confirm the persistence of previously written data.
Important Notes:
|
Method
First, let’s import a Fedora image from a dataVolume to a PVC using the following YAML, which includes:
kind: DataVolume
storageClassName: vastdata-filesystem (csi storageClass name)
access mode: ReadOnlyMany (keep it immutable, next we'll create RW pvc)
PVC size: 30G.
Writable YAML template
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
name: "fedora"
annotations:
cdi.kubevirt.io/storage.bind.immediate.requested: "true"
spec:
source:
http:
url: "https://download.fedoraproject.org/pub/fedora/linux/releases/37/Cloud/x86_64/images/Fedora-Cloud-Base-37-1.7.x86_64.raw.xz"
pvc:
storageClassName: vastdata-filesystem
accessModes:
- ReadOnlyMany
resources:
requests:
storage: "30Gi"Applying the data volume will create a PVC.
A temporary pod with the prefix importer will be created; its job is to import the image to a PVC.
We can check the status. The new data volume named Fedora isn't ready until the job is done.
root@v55lg2:/opt/LLNL/datavolumes# kubectl apply -f dv_fedora.yaml
datavolume.cdi.kubevirt.io/fedora created
root@v55lg2:/opt/LLNL/datavolumes# kubectl get pods
NAME READY STATUS RESTARTS AGE
importer-prime-68d84758-a49e-4996-88b6-9ac52a36a1ee 1/1 Running 0 3m22s
root@v55lg2:/opt/LLNL/datavolumes# kubectl get datavolumes.cdi.kubevirt.io
NAME PHASE PROGRESS RESTARTS AGE
fedora ImportInProgress 66.77% 3m4sMonitor the process by checking the importer pod logs.
When complete, the Fedora datavolume and PVC are ready to use.
root@v55lg2:/opt/LLNL/datavolumes# kubectl logs -f importer-prime-68d84758-a49e-4996-88b6-9ac52a36a1ee
I0122 12:32:46.146124 1 importer.go:103] Starting importer
I0122 12:32:46.147086 1 importer.go:176] begin import process
I0122 12:32:47.919697 1 data-processor.go:356] Calculating available size
I0122 12:32:47.920485 1 data-processor.go:368] Checking out file system volume size.
I0122 12:32:47.921197 1 data-processor.go:376] Request image size not empty.
I0122 12:32:47.921221 1 data-processor.go:381] Target size 30Gi.
I0122 12:32:48.061179 1 data-processor.go:255] New phase: TransferDataFile
I0122 12:32:48.063593 1 util.go:194] Writing data...
I0122 12:32:49.061893 1 prometheus.go:75] 0.02
I0122 12:32:52.063418 1 prometheus.go:75] 0.75
I0122 12:32:53.064514 1 prometheus.go:75] 1.46
I0122 12:32:55.076411 1 prometheus.go:75] 2.50
...
I0122 12:37:36.362152 1 prometheus.go:75] 99.97
I0122 12:37:37.362659 1 prometheus.go:75] 99.98
I0122 12:37:48.369059 1 prometheus.go:75] 99.99
I0122 12:37:49.370083 1 prometheus.go:75] 100.00
I0122 12:37:52.526490 1 data-processor.go:255] New phase: Resize
E0122 12:37:52.714510 1 prlimit.go:155] failed to kill the process; os: process already finished
W0122 12:37:52.714615 1 data-processor.go:338] Available space less than requested size, resizing image to available space 30440161280.
I0122 12:37:52.714629 1 data-processor.go:349] Expanding image size to: 30440161280
E0122 12:37:52.814279 1 prlimit.go:155] failed to kill the process; os: process already finished
I0122 12:37:52.814300 1 data-processor.go:261] Validating image
E0122 12:37:52.869491 1 prlimit.go:155] failed to kill the process; os: process already finished
I0122 12:37:52.870161 1 data-processor.go:255] New phase: Complete
I0122 12:37:52.870447 1 importer.go:223] Import Complete
root@v55lg2:/opt/LLNL/datavolumes# kubectl get datavolumes.cdi.kubevirt.io
NAME PHASE PROGRESS RESTARTS AGE
fedora Succeeded 100.0% 6m10s
root@v55lg2:/opt/LLNL/datavolumes# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
fedora Bound pvc-1d8ca312-f0f6-4427-b3de-ddde9f0e2699 30Gi ROX vastdata-filesystem 6m31sWe created this "golden" PVC with RO mode.
Clone a PVC
Now it's time to clone this PVC into a new "golden-pvc", and modify it as we see fit. In this case, I created a VM with the following tweaks:
Filled its belly with 20 GB of data to test some real workload while cloning
run's elbencho benchmark @boot to continuously generate some I/O as if it is a VDI, for example
installed some required packages
Modify the ssh conf and root password
We'll use this PVC to create additional PVCs, one for each VM, but first, let's start by cloning the PVC Fedora → golden-fedora (rwm).
Note that now we're using accessMode: ReadWriteMany
Clone PVC to PVC yaml :
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: golden-fedora
namespace: default
spec:
accessModes:
- ReadWriteMany
storageClassName: vastdata-filesystem
resources:
requests:
storage: 30Gi
dataSource:
kind: PersistentVolumeClaim
name: fedoraNote - cloning a PVC does not create a snapshot.
Create from the PVC golden-pvc the VM VMFedora.
Note the PVC source name, spec, and SSH keys (optional).
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
creationTimestamp: 2018-07-04T15:03:08Z
generation: 1
labels:
kubevirt.io/os: linux
name: vmfedora
spec:
running: true
template:
metadata:
creationTimestamp: null
labels:
kubevirt.io/domain: vm1
spec:
domain:
cpu:
cores: 2
devices:
disks:
- disk:
bus: virtio
name: disk0
- cdrom:
bus: sata
readonly: true
name: cloudinitdisk
machine:
type: q35
resources:
requests:
memory: 1024M
volumes:
- name: disk0
persistentVolumeClaim:
claimName: golden-fedora
- cloudInitNoCloud:
userData: |
#cloud-config
hostname: vmfedora
ssh_pwauth: True
disable_root: false
ssh_authorized_keys:
- ssh-rsa <COPY YOUR PUBLIC KEY>
name: cloudinitdiskWe'll create a new VM called Fedora, SSH, and write something to it.
root@v55lg2:/opt/LLNL/vms# kubectl apply -f vm_fedora.yaml
virtualmachine.kubevirt.io/vmfedora created
root@v55lg2:/opt/LLNL/process# kubectl get vms
NAME AGE STATUS READY
vmfedora 52s Running True
root@v55lg2:/opt/LLNL/process# kubectl get vmis
NAME AGE PHASE IP NODENAME READY
vmfedora 56s Running 10.44.0.8 v55lg3 True
root@v55lg2:/opt/LLNL/process# kubectl get pods
NAME READY STATUS RESTARTS AGE
virt-launcher-vmfedora-z9779 2/2 Running 0 96s
root@v55lg2:/opt/LLNL/process# ssh 10.44.0.8
[root@vmfedora ~]# date > /home/`date +%d.%m_%H.%M.%S`.txt
[root@vmfedora ~]# cat /home/22.01_14.02.14.txt
Mon Jan 22 02:02:14 PM UTC 2024As explained at the beginning of the section, I'll tweak this VM a little bit, and to make a long story short, here's a set of commands that is very high-level:
### ssh to the vm (or using virtctl cmd)
### elbencho yum dependencies, make and deploy
yum install -y cronie cronie-anacron crontabs boost-devel gcc-c++ git libaio-devel make ncurses-devel numactl-devel rpm-build vim
git clone https://github.com/breuner/elbencho.git
cd elbencho
make rpm
yum install /root/elbencho/packaging/RPMS/x86_64/elbencho-3.0.5-1.x86_64.rpm -y
### Optional :
### Create a bash script to generate some I/O with the following elbencho command, and add it to crontab @reboot
/usr/bin/elbencho --write --mkdirs --threads 2 --dirs 3 --files 4 --size 1m --block 1m /home/elben_data --infloopOnce the VM is ready, we'll shut it down and create a snapshot. Let's name it snap-fedora-daily
snapshot yaml:
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: snap-fedora-daily
spec:
volumeSnapshotClassName: vastdata-snapshot
source:
persistentVolumeClaimName: golden-fedoraThe snapshot is ready to clone.
root@v55lg2:/opt/LLNL/process# kubectl get volumesnapshot
NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE
snap-fedora-daily true golden-fedora 0 vastdata-snapshot snapcontent-da56675d-785e-4220-ae09-48c9a1d86440 7s 8sNow, let's create two writable PVCs from the snapshot, named fedora-daily01 & fedora-daily02.
Please note that the clone-from-snapshot process does not erase the snapshot from the storage; we can reuse it.
PVC from snapshot yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: fedora-daily01
spec:
storageClassName: vastdata-filesystem
dataSource:
name: snap-fedora-daily
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
accessModes:
- ReadWriteMany
resources:
requests:
storage: 30GiMonitor global snapshot creation status from the cluster UI :
We have x3 PVCs: the imported data volume, the golden PVC, and the new daily PVC we created from the snapshots. In addition, if we examine the mounted folder for the VAST kubevirt images, the data is being filled as new PVCs are created.
root@v55lg2:/opt/LLNL/process# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
fedora Bound pvc-1d8ca312-f0f6-4427-b3de-ddde9f0e2699 30Gi ROX vastdata-filesystem 171m
fedora-daily01 Bound pvc-22907e3f-2298-40a7-bb5f-3f0b14b4241b 30Gi RWX vastdata-filesystem 10s
fedora-daily02 Bound pvc-52300027-56d6-415d-9c2b-7cf8cd4cde19 30Gi RWX vastdata-filesystem 7s
golden-fedora Bound pvc-d1cc5448-a8b8-4ea0-b3ea-59ac14dee5cb 30Gi RWX vastdata-filesystem 103m
#### Du -sh #######
root@v55lg2:/opt/LLNL/process# du -sh /mnt/v55/kubevirt/*
5.0G /mnt/v55/kubevirt/pvc-1d8ca312-f0f6-4427-b3de-ddde9f0e2699
15G /mnt/v55/kubevirt/pvc-22907e3f-2298-40a7-bb5f-3f0b14b4241b <-- PVC is ready when reaches 29G
11G /mnt/v55/kubevirt/pvc-52300027-56d6-415d-9c2b-7cf8cd4cde19 <-- PVC is ready when reaches 29G
29G /mnt/v55/kubevirt/pvc-d1cc5448-a8b8-4ea0-b3ea-59ac14dee5cbFinally, we'll create new VMs called daily-fedora01 && daily-fedora02, access them, and write more stuff to them.
root@v55lg2:/opt/LLNL/process# kubectl get vmis
NAME AGE PHASE IP NODENAME READY
daily-fedora01 18s Running 10.36.0.8 v55lg4 True
daily-fedora02 14s Running 10.44.0.9 v55lg3 True
vmfedora 50m Running 10.44.0.8 v55lg3 True
root@v55lg2:/opt/LLNL/process# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
virt-launcher-daily-fedora01-jsrwr 2/2 Running 0 26s 10.36.0.8 v55lg4 <none> 1/1
virt-launcher-daily-fedora02-wk9d5 2/2 Running 0 22s 10.44.0.9 v55lg3 <none> 1/1
virt-launcher-vmfedora-pbmwz 2/2 Running 0 50m 10.44.0.8 v55lg3 <none> 1/1
root@v55lg2:/opt/LLNL/process# ssh 10.44.0.9
root@daily-fedora02 ~]# date >> /home/22.01_14.02.14.txt
[root@daily-fedora02 ~]# cat /home/22.01_14.02.14.txt
Mon Jan 22 02:02:14 PM UTC 2024
Mon Jan 22 03:33:37 PM UTC 2024
[root@daily-fedora02 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs 459M 0 459M 0% /dev/shm
tmpfs 184M 3.0M 181M 2% /run
/dev/vda5 28G 27G 703M 98% /
tmpfs 459M 0 459M 0% /tmp
/dev/vda5 28G 27G 703M 98% /home
/dev/vda2 966M 53M 848M 6% /boot
/dev/vda3 100M 9.9M 90M 10% /boot/efi
tmpfs 92M 0 92M 0% /run/user/0
[root@daily-fedora02 ~]# ps aufx|grep elbencho
root 907 0.0 0.2 6216 2188 pts/0 S+ 15:34 0:00 \_ grep --color=auto elbencho
root 831 197 0.7 156028 7088 ? Sl 15:29 10:49 \_ /usr/bin/elbencho --write --mkdirs --threads 2 --dirs 3 --files 4 --size 1m --block 1m /home/elben_data --infloop
Nice utilization and Data flow distribution :
Stress Test
We'll measure how the system handles cloning x50 PVCs and attach the I/O-generating VMs to each one. Then, initiate a snapshot from one PVC and clone x30 new PVCs from it. Finally, attach x30 new VMs to the newly created PVCs.
Creating x50 PVC from the golden image at once, each PVC 30GB total of 1.5TB.
Data Flow | DNODES writes/read bandwidth | Cluster IOPS and Latency | Cluster Migrate Data Reduction Ratio |
|---|---|---|---|
|
|
|
|
Creating x50 PVC, roughly at 1.2GBps writes, all data is similar, takes roughly 28 minutes.
Every 2.0s: du -sh /mnt/v55/kubevirt/*|sort -nk1 v55lg2: Wed Jan 17 14:37:09 2024
5.1G /mnt/v55/kubevirt/pvc-44e28051-bb71-465f-860b-968b94e51f4b. <----- Last PVC ready with 29G provisioned
13G /mnt/v55/kubevirt/pvc-f0d01b22-9407-43d4-bfc6-d2c2f14fea61
16G /mnt/v55/kubevirt/pvc-39ef66cc-7be0-40b2-b24a-88e64a682626
20G /mnt/v55/kubevirt/pvc-4a13201a-cc18-48ec-8a7a-92bc166f2989
22G /mnt/v55/kubevirt/pvc-4ab3012e-5f66-4bd6-937e-f150af3d69df
24G /mnt/v55/kubevirt/pvc-b8595c12-d018-40e9-acda-7272e414f774
26G /mnt/v55/kubevirt/pvc-86733fe8-ed5c-440f-bd51-6087bfdec940
27G /mnt/v55/kubevirt/pvc-af7620f7-19f1-4961-a36d-059b18a18719
28G /mnt/v55/kubevirt/pvc-07911809-15c0-45a2-b4d6-23a8af13b150
28G /mnt/v55/kubevirt/pvc-603793aa-4177-4369-8a3c-627bafb713aa
29G /mnt/v55/kubevirt/pvc-0ca42e3f-5f74-491b-85b9-ff43e3da82d1
29G /mnt/v55/kubevirt/pvc-0cdd5162-c073-4e15-bb0e-f03dd02d69cf
...Attaching VM to each PVC.
VMs are running, each VM is attached to its own PVC.
root@v55lg2:~# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
virt-launcher-vmfedora01-h9hjw 2/2 Running 0 94m 10.36.0.32 v55lg4 <none> 1/1
virt-launcher-vmfedora02-s7f7h 2/2 Running 0 94m 10.44.0.8 v55lg3 <none> 1/1
virt-launcher-vmfedora03-htsw5 2/2 Running 0 94m 10.44.0.9 v55lg3 <none> 1/1
virt-launcher-vmfedora04-7kpx8 2/2 Running 0 94m 10.36.0.11 v55lg4 <none> 1/1
virt-launcher-vmfedora05-q8pnz 2/2 Running 0 94m 10.36.0.19 v55lg4 <none> 1/1
virt-launcher-vmfedora06-z4cpp 2/2 Running 0 94m 10.44.0.10 v55lg3 <none> 1/1
virt-launcher-vmfedora07-bgrfg 2/2 Running 0 94m 10.44.0.11 v55lg3 <none> 1/1
virt-launcher-vmfedora08-wl8vr 2/2 Running 0 94m 10.36.0.16 v55lg4 <none> 1/1
virt-launcher-vmfedora09-whdts 2/2 Running 0 94m 10.36.0.20 v55lg4 <none> 1/1
virt-launcher-vmfedora10-cfghk 2/2 Running 0 94m 10.44.0.23 v55lg3 <none> 1/1
virt-launcher-vmfedora11-c9g2r 2/2 Running 0 94m 10.44.0.12 v55lg3 <none> 1/1
virt-launcher-vmfedora12-pvtxx 2/2 Running 0 94m 10.44.0.13 v55lg3 <none> 1/1
virt-launcher-vmfedora13-ttckk 2/2 Running 0 94m 10.36.0.30 v55lg4 <none> 1/1
virt-launcher-vmfedora14-fktgb 2/2 Running 0 93m 10.36.0.21 v55lg4 <none> 1/1
virt-launcher-vmfedora15-rsmz2 2/2 Running 0 93m 10.44.0.14 v55lg3 <none> 1/1
virt-launcher-vmfedora16-hqf4n 2/2 Running 0 93m 10.44.0.15 v55lg3 <none> 1/1
virt-launcher-vmfedora17-4b527 2/2 Running 0 93m 10.36.0.22 v55lg4 <none> 1/1
virt-launcher-vmfedora18-5nrwd 2/2 Running 0 93m 10.36.0.25 v55lg4 <none> 1/1
virt-launcher-vmfedora19-v7h8d 2/2 Running 0 93m 10.44.0.17 v55lg3 <none> 1/1
virt-launcher-vmfedora20-phtlz 2/2 Running 0 93m 10.44.0.24 v55lg3 <none> 1/1
virt-launcher-vmfedora21-2lrbj 2/2 Running 0 93m 10.44.0.16 v55lg3 <none> 1/1
virt-launcher-vmfedora22-l5drd 2/2 Running 0 93m 10.36.0.17 v55lg4 <none> 1/1
virt-launcher-vmfedora23-45drd 2/2 Running 0 93m 10.36.0.26 v55lg4 <none> 1/1
virt-launcher-vmfedora24-pksk8 2/2 Running 0 93m 10.36.0.14 v55lg4 <none> 1/1
virt-launcher-vmfedora25-k4c7x 2/2 Running 0 93m 10.36.0.28 v55lg4 <none> 1/1
virt-launcher-vmfedora26-rvqcl 2/2 Running 0 93m 10.44.0.22 v55lg3 <none> 1/1
virt-launcher-vmfedora27-qfv6b 2/2 Running 0 93m 10.36.0.18 v55lg4 <none> 1/1
virt-launcher-vmfedora28-mdhdj 2/2 Running 0 93m 10.44.0.18 v55lg3 <none> 1/1
virt-launcher-vmfedora29-6mp2v 2/2 Running 0 93m 10.36.0.23 v55lg4 <none> 1/1
virt-launcher-vmfedora30-wcsgh 2/2 Running 0 93m 10.44.0.25 v55lg3 <none> 1/1
virt-launcher-vmfedora31-bjkmg 2/2 Running 0 93m 10.44.0.19 v55lg3 <none> 1/1
virt-launcher-vmfedora32-79dhq 2/2 Running 0 93m 10.36.0.13 v55lg4 <none> 1/1
virt-launcher-vmfedora33-gfblt 2/2 Running 0 92m 10.36.0.24 v55lg4 <none> 1/1
virt-launcher-vmfedora34-kgxc2 2/2 Running 0 92m 10.36.0.27 v55lg4 <none> 1/1
virt-launcher-vmfedora35-zrf5n 2/2 Running 0 92m 10.44.0.21 v55lg3 <none> 1/1
virt-launcher-vmfedora36-4zwb4 2/2 Running 0 92m 10.44.0.20 v55lg3 <none> 1/1
virt-launcher-vmfedora37-5g9x8 2/2 Running 0 92m 10.36.0.8 v55lg4 <none> 1/1
virt-launcher-vmfedora38-b4sgg 2/2 Running 0 92m 10.44.0.27 v55lg3 <none> 1/1
virt-launcher-vmfedora39-rlhxr 2/2 Running 0 92m 10.44.0.26 v55lg3 <none> 1/1
virt-launcher-vmfedora40-d5lk5 2/2 Running 0 92m 10.36.0.29 v55lg4 <none> 1/1
virt-launcher-vmfedora41-cxr4q 2/2 Running 0 92m 10.36.0.31 v55lg4 <none> 1/1
virt-launcher-vmfedora42-n7xlf 2/2 Running 0 92m 10.44.0.28 v55lg3 <none> 1/1
virt-launcher-vmfedora43-x84j7 2/2 Running 0 92m 10.44.0.30 v55lg3 <none> 1/1
virt-launcher-vmfedora44-j7g5k 2/2 Running 0 92m 10.36.0.12 v55lg4 <none> 1/1
virt-launcher-vmfedora45-skbxl 2/2 Running 0 92m 10.44.0.31 v55lg3 <none> 1/1
virt-launcher-vmfedora46-pg6g2 2/2 Running 0 92m 10.44.0.29 v55lg3 <none> 1/1
virt-launcher-vmfedora47-7plzq 2/2 Running 0 92m 10.36.0.10 v55lg4 <none> 1/1
virt-launcher-vmfedora48-rfshb 2/2 Running 0 92m 10.36.0.15 v55lg4 <none> 1/1
virt-launcher-vmfedora49-7hgrg 2/2 Running 0 92m 10.36.0.9 v55lg4 <none> 1/1
virt-launcher-vmfedora50-8vb5p 2/2 Running 0 92m 10.44.0.32 v55lg3 <none> 1/1Clone from snapshot
Taking a snapshot from the first PVC created: "pvc-fedora01", and creating x20 new PVC + VMs from it.
While snapshot creation is completed immediately, creating x20 PVCs has a similar rate of 1.3 GB/s, as it took 13 minutes to create 600GB of x20 30GB PVCs.
root@v55lg2:/opt/LLNL/snapshots# kubectl apply -f snap_fedora.yaml
volumesnapshot.snapshot.storage.k8s.io/snap-fedora01 created
root@v55lg2:/opt/LLNL/snapshots# kubectl get volumesnapshot
NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE
snap-fedora01 true pvc-fedora01 0 vastdata-snapshot snapcontent-2182a4
root@v55lg2:/opt/LLNL/vms# kubectl apply -f vm_fedora_fromsnap.yaml
virtualmachine.kubevirt.io/vmfedora-fromsnap created
root@v55lg2:/opt/LLNL/vms# kubectl get vms|grep snap
vmfedora-fromsnap 17s Running True
root@v55lg2:/opt/LLNL/vms# kubectl get vmis|grep snap
vmfedora-fromsnap 2m1s Running 10.44.0.33 v55lg3 True
root@v55lg2:/opt/LLNL/vms# ssh 10.44.0.33
Last login: Wed Jan 17 13:29:46 2024
[[root@vmfedora-fromsnap ~]# touch /home/snapwrite
[root@vmfedora-fromsnap ~]# ls -l /home
total 409600
drwxr-xr-x. 1 root root 144 Jan 17 13:27 elbencho
drwx------. 1 fedora fedora 72 Jan 17 12:44 fedora
-rw-r--r--. 1 root root 104857600 Jan 17 13:19 files100mb1
-rw-r--r--. 1 root root 104857600 Jan 17 13:19 files100mb2
-rw-r--r--. 1 root root 104857600 Jan 17 13:19 files100mb3
-rw-r--r--. 1 root root 104857600 Jan 17 13:19 files100mb4
drwxr-xr-x. 1 root root 0 Jan 17 12:45 oz-vm1
-rw-r--r--. 1 root root 0 Jan 17 17:49 snapwrite
Mass deletion of all PVC and snapshots
The best practice to remove PVC:
Shutdown associated VM
Delete the VM → this will also terminate its running pod
Delete the PVC
Massive deletion of x70 PVC, took less than 1 minute, with no queue detected on the cluster while checking the following metrics:
NVRamHashMapMetrics,name=new_token_mapper,block_used_space_percent__avg
TreeUnlinkerMetrics,unlinks_count__sum
TreeUnlinkerMetrics,unlinks_count__sum_squares
TreeUnlinkerMetrics,unlinks_count__max
TreeUnlinkerMetrics,unlinks_count__rate
TreeUnlinkerMetrics,unlinks_count__std
TreeUnlinkerMetrics,unlinks_count__avg
Thoughts and tl;dr
VM/PVC Similarity is very impressive - ratio of 30.1:1
2. Writable snapshots cannot be initiated without disabling the VAST Catalog.
3. Deletion with CSI - enable trash folder.
#2: Direct Clone a VM
Cloning a VM is basically the fastest way to clone, but snapshots and PVCs are automatically deleted from the storage.
Flow:
Once a VM has been created and is running from a golden image, we can clone it directly.
Clone the VM to a new VM. (Daily3 → Daily4)
The snapshot and PVC are automatically created during the process.
Functionality
This, in fact, can speed up and automate the cloning of VMs, since the underlying snapshot and the associated new PVC (restore) are created for you.
After the new VM was created, the snapshot is automatically deleted. This use case should not be used against ransomware attacks, for example, by saving immutable snapshots since they are removed.
Method
root@v55lg2:/opt/LLNL/clone# kubectl create -f clone.yaml
virtualmachineclone.clone.kubevirt.io/testclone created
root@v55lg2:/opt/LLNL/clone# cat clone.yaml
kind: VirtualMachineClone
apiVersion: "clone.kubevirt.io/v1alpha1"
metadata:
name: testclone
spec:
# source & target definitions
source:
apiGroup: kubevirt.io
kind: VirtualMachine
name: vmfedora01
target:
apiGroup: kubevirt.io
kind: VirtualMachine
name: vm-clone-target
# other identity stripping specs:
newMacAddresses:
interfaceName: "00-11-22"
newSMBiosSerial: "new-serial"
root@v55lg2:/opt/LLNL/clone# kubectl get volumesnapshot
NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE
vmsnapshot-fd042778-7f39-4a86-8a8b-47784c2b7dd5-volume-disk0 true fedora 0 vastdata-snapshot snapcontent-5b854b90-d157-4b84-bf02-74936adbb407 21s 22s
root@v55lg2:/opt/LLNL/clone# kubectl get virtualmachine
NAME AGE STATUS READY
fedora-clone-pvc01 5m25s Running True
vm-clone-target 91s Running True
vmfedora01 73m Running True
root@v55lg2:/opt/LLNL/clone# kubectl get volumesnapshot
No resources found in default namespace.
root@v55lg2:/opt/LLNL/clone# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
clone-of-defora-pvc Bound pvc-fbeba0fd-c1bc-42ad-8878-b125ecdb96df 20Gi RWX vastdata-filesystem 11m
fedora Bound pvc-6f82df62-f93a-4d12-8331-962da29282d8 20Gi RWX vastdata-filesystem 93m
restore-6af6ca21-6711-4024-9cb2-f4fe61f70663-disk0 Bound pvc-c0abba80-f8d2-40fc-84d0-55a94fff0b23 20Gi RWX vastdata-filesystem 2m9sAppendix A - Environment setup
Create a K8s cluster with at least x1 worker node. This setup was tested with version 1.28.2 and the containerized container runtime.
Install the vastcsi 2.3 driver according to the docs.
Ensure the VAST view for virtual machines is configured with no_root_squash on the k8s control plane host.
Validate hardware virtualization support:
VMware (for lab/testing) - enable "Expose hardware-assisted virtualization to the guest OS" in CPU options.
Bare Metal - BIOS
Linux OS - Install KVM and libvirt:
apt-get install qemu-kvm libvirt-binvalidate hardware support (don't forget BIOS conf in 2.b) to ensure that your host are capable of running virtualization workloads:
$ virt-host-validate qemu
QEMU: Checking for hardware virtualization : PASS
QEMU: Checking if device /dev/kvm exists : PASS
QEMU: Checking if device /dev/kvm is accessible : PASS
QEMU: Checking if device /dev/vhost-net exists : PASS
QEMU: Checking if device /dev/net/tun exists : PASS
...Appendix B - Deploy KubeVirt
Deploy kubevirt, according to this guide:
Deploy KubeVirt
Deploy the KubeVirt operator using the latest KubeVirt version.
An Operator is a method of packaging, deploying, and managing a Kubernetes application. A Kubernetes application is one that is deployed on Kubernetes and managed using the Kubernetes APIs and kubectl tooling. You can think of Operators as the runtime that manages this type of application on Kubernetes. If you want to learn more about Operators you can check the Kubernetes documentation
Here, we query GitHub's API to get the latest available release: (click on the text to automatically execute the commands on the console):
export KUBEVIRT_VERSION=$(curl -s https://api.github.com/repos/kubevirt/kubevirt/releases/latest | jq -r .tag_name)
echo $KUBEVIRT_VERSION
Run the following command to deploy the KubeVirt Operator:
kubectl create -f https://github.com/kubevirt/kubevirt/releases/download/${KUBEVIRT_VERSION}/kubevirt-operator.yaml
Now deploy KubeVirt by creating a Custom Resource that will trigger the 'operator' reaction and perform the deployment:
kubectl create -f https://github.com/kubevirt/kubevirt/releases/download/${KUBEVIRT_VERSION}/kubevirt-cr.yaml
Next, we need to configure KubeVirt to use software emulation for virtualization. This is necessary for the course environment but results in poor performance so avoid this step in production environments.
kubectl -n kubevirt patch kubevirt kubevirt --type=merge --patch '{"spec":{"configuration":{"developerConfiguration":{"useEmulation":true}}}}'
Install Virtctl
While we are waiting for the KubeVirt operator to start up all its Pods, we can take some time to download the client we will need to use in the next step.
virtctl is a client utility that helps interact with VM's (start/stop/console, etc):
wget -O virtctl https://github.com/kubevirt/kubevirt/releases/download/${KUBEVIRT_VERSION}/virtctl-${KUBEVIRT_VERSION}-linux-amd64
chmod +x virtctl
Wait for KubeVirt deployment to finalize
Let's check the deployment:
kubectl get pods -n kubevirt
Once it's ready, it will show something similar to:
controlplane $ kubectl get pods -n kubevirt
NAME READY STATUS RESTARTS AGE
virt-api-7fc57db6dd-g4s4w 1/1 Running 0 3m
virt-api-7fc57db6dd-zd95q 1/1 Running 0 3m
virt-controller-6849d45bcc-88zd4 1/1 Running 0 3m
virt-controller-6849d45bcc-cmfzk 1/1 Running 0 3m
virt-handler-fvsqw 1/1 Running 0 3m
virt-operator-5649f67475-gmphg 1/1 Running 0 4m
virt-operator-5649f67475-sw78k 1/1 Running 0 4m
As there are multiple deployments involved, the best way to determine whether the operator is fully installed is to check the operator's Custom Resource itself:
kubectl -n kubevirt get kubevirt
Once fully deployed, this will look like:
NAME AGE PHASE
kubevirt 3m Deployed
Now everything is ready to continue and launch a VM.
Deploy a Virtual Machine
The command below applies a YAML definition of a virtual machine into the current Kubernetes environment, defining the VM name, the resources required (disk, CPU, memory), etc. You can take a look at the vm.yaml file if you have interest in knowing more about a virtual machine definition:
kubectl apply -f https://kubevirt.io/labs/manifests/vm.yaml
We are creating a Virtual Machine in the same way as we would create any other Kubernetes resource thanks to the KubeVirt operator in our environment. Now we have a Virtual Machine as a Kubernetes resource.
After the vm resource has been created, you can manage the VMs with standard 'kubectl' commands:
kubectl get vms
kubectl get vms -o yaml testvm | grep -E 'running:.*|$'
Notice from the output that the VM is not running yet.
To start a VM, use virtctl with the start verb:
./virtctl start testvm
Again, check the VM status:
kubectl get vms
A VirtualMachine resource contains a VM's definition and status. An instance of a running VM has an additional associated resource, a VirtualMachineInstance.
Once the VM is running you can inspect its status:
kubectl get vmis
Once it is ready, the command above will print something like:
NAME AGE PHASE IP NODENAME
testvm 1m Running 10.32.0.11 controlplane
Access a VM (serial console & vnc)
Now that the VM is running you can access its serial console:
WARNING: in some OS and browser environments you may not be able to escape the serial console in this course.
NOTE: ^] means: press the "CTRL" and "]" keys to escape the console.
./virtctl console testvm
If you opened the serial console within the Killercoda course environment and you can't escape from it by pressing ^], you can click on the + at the top of the terminal window to start a new shell. You should be able to continue with the following steps in the shutdown and cleanup section.
In environments where VNC client access is available, the graphical console of a VM can be accessed with the virtctl vnc command.
Shutdown and cleanup
As with starting, stopping a VM also may be accomplished with the virtctl command:
./virtctl stop testvm
Finally, the VM can be deleted as any other Kubernetes resource using kubectl:
kubectl delete vms testvm5. Deploy CDI
Install CDI
Introduction to Containerized Data Importer
CDI is a utility designed to import Virtual Machine images for use with Kubevirt.
The process for importing disk images into KubeVirt involves a helper Custom Resource (CR) called a DataVolume (DV). DataVolumes handle the provisioning of PersistentVolumeClaims (PVCs) for KubeVirt. A custom controller watches for new DVs, and when one is discovered, starts an import process to create a new PVC into which it will import the desired content.
In this exercise we start by deploying the CDI operator. Then, we import a CirrOS disk image and use it to start a VM.
Wait for KubeVirt to deploy
The setup for this scenario includes installation of KubeVirt and the virtctl utility.
Before we can start, we need to wait for the KubeVirt initialization script to run. (a command prompt will appear once everything is ready).
Check Default Storage Class
Before we can install CDI, we have some prerequisites, namely a supported storage class and provisioner. For this example, we make use of the local-path StorageClass provided by k3s which provisions PVCs using node local storage. This is an option for proof of concept exercises like this one, but should not be used in production because it does not support accessing a volume across nodes.
kubectl get storageclass
Install the Containerized Data Importer
Next we determine the latest version of CDI and apply both the Operator and the CR that starts the deployment:
export VERSION=$(curl -Ls https://github.com/kubevirt/containerized-data-importer/releases/latest | grep -m 1 -o "v[0-9]\.[0-9]*\.[0-9]*")
echo $VERSION
Deploy the operator (and scale its replicas down to one due to the resource limitations of the environment):
kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-operator.yaml
kubectl -n cdi scale deployment/cdi-operator --replicas=1
Create CRD to trigger operator deployment of CDI:
kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-cr.yaml
Check status of CDI deployment. It may take some time before the cdi "PHASE" reads "Deployed"
kubectl get cdi -n cdi
To have kubectl do the checking for you and let you know when the operator finishes its deployment, use the wait command:
kubectl wait -n cdi --for=jsonpath='{.status.phase}'=Deployed cdi/cdi
At times, the environment this scenario runs in may be under heavy load; if the above command errors out, wait a moment and re-run.
Review the "cdi" pods that were added.
kubectl -n cdi get pods