Reclaiming disk space by safely resetting the containerd snapshotter storage without losing your Kubernetes configuration.

Critical Warning#

⚠️ Back up your system before proceeding. This procedure is performed at your own risk.

This tutorial involves stopping MicroK8s, removing container images and their filesystem layers, which will cause downtime and require Kubernetes to repull all images. Ensure you have backups of any critical data and configuration before starting.

Understanding containerd and OverlayFS#

Before diving into the cleanup procedure, it’s essential to understand what containerd does and how the OverlayFS snapshotter works.12

What is containerd?#

containerd is a high-level container runtime that manages the complete container lifecycle on your system. It handles:34

  • Pulling and storing container images from registries
  • Managing container creation, startup, and shutdown
  • Resource allocation and isolation using Linux namespaces and cgroups
  • Container monitoring and health checks

containerd operates as a daemon that exposes an API for higher-level tools like Docker, Kubernetes, and MicroK8s to interact with containers.25

The Runtime Architecture#

The container runtime architecture consists of multiple layers:62

  1. High-level runtime (containerd): Manages images, prepares container configurations, and coordinates the lifecycle
  2. containerd-shim: Acts as an intermediary process that persists after container creation, allowing containerd to be upgraded without affecting running containers78
  3. Low-level runtime (runc): Performs the actual container execution using Linux kernel primitives like namespaces and cgroups96

A Critical Point: Containers Can Run Without containerd#

One important characteristic of this architecture is that containers do not require containerd to remain running once they’ve been started. The containerd-shim process becomes the parent of the container process, and runc exits immediately after starting the container. This design allows administrators to upgrade or restart containerd without disrupting running containers.897

However, for management operations (creating, stopping, or inspecting containers), containerd must be running to handle these API requests.

Understanding the OverlayFS Snapshotter#

The directory /var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/ is where containerd stores container image layers and filesystem snapshots using the OverlayFS storage driver.101

OverlayFS is a union filesystem that layers multiple directories to present them as a single unified filesystem. It works by:111213

  • Lower layers: Read-only image layers containing the base operating system and application files
  • Upper layer: A writable layer where container modifications are stored
  • Merged view: The combined filesystem that the container actually sees

Each container image consists of multiple layers stacked on top of each other. When you pull an image, containerd unpacks these layers into the snapshots directory. The snapshotter creates filesystem snapshots for each layer, and OverlayFS efficiently combines them without duplicating data.14121

Why the directory grows large: Over time, as you deploy applications, upgrade images, and redeploy pods, old image layers accumulate. Even after removing pods, the underlying image layers may remain if garbage collection fails or if the disk becomes too full.1516

When to Clean the OverlayFS Directory#

Consider cleaning this directory when:

  • The /var/snap/microk8s/common/var/lib/containerd/ directory consumes excessive disk space1715
  • Kubernetes garbage collection is failing with errors like “Failed to garbage collect required amount of images”15
  • Standard cleanup commands (microk8s.ctr images prune or crictl rmi --prune) are insufficient1819
  • You experience filesystem corruption or snapshot inconsistencies

Prerequisites#

  • Root or sudo access to your MicroK8s node
  • A complete backup of your system
  • Understanding that all running pods will be terminated
  • Awareness that all container images will need to be repulled

Step-by-Step Cleanup Procedure#

Step 1: Stop MicroK8s#

First, gracefully stop all MicroK8s services:

microk8s stop

This command stops the kubelet, API server, and other Kubernetes components. Wait for the command to complete before proceeding.

Step 2: Kill All Running Container Tasks#

Even though MicroK8s is stopped, container processes may still be running due to the containerd-shim architecture. Forcefully terminate all container tasks:78

microk8s.ctr task ls | tail -n+2 | cut -f1 -d ' ' | xargs -r -n1 microk8s.ctr task kill -s SIGKILL

What this command does:

  • microk8s.ctr task ls: Lists all running container tasks
  • tail -n+2: Removes the header row
  • cut -f1 -d ' ': Extracts the first column (task ID)
  • xargs -r -n1 microk8s.ctr task kill -s SIGKILL: Sends SIGKILL signal to each task

Step 3: Navigate to the containerd Directory#

Change to the containerd data directory:

cd /var/snap/microk8s/common/var/lib/containerd/

Step 4: Rename the OverlayFS Directory#

Rather than immediately deleting the snapshotter directory, rename it as a safety measure:

mv io.containerd.snapshotter.v1.overlayfs io.containerd.snapshotter.v1.overlayfs.old

This approach allows you to recover the original data if something goes wrong. You can delete the .old directory later once you’ve verified everything works correctly.

Step 5: Delete All Containers#

Remove all container metadata from containerd:

microk8s.ctr containers ls | cut -f1 -d' ' | tail -n +2 | xargs -n1 -r microk8s.ctr containers delete

What this command does:

  • microk8s.ctr containers ls: Lists all containers (running or stopped)
  • cut -f1 -d' ': Extracts container IDs
  • tail -n +2: Skips the header
  • xargs -n1 -r microk8s.ctr containers delete: Deletes each container

Step 6: Delete All Images#

Remove all container image metadata:

microk8s.ctr images ls | cut -f1 -d' ' | tail -n +2 | xargs -n1 -r microk8s.ctr images rm

What this command does:

  • microk8s.ctr images ls: Lists all stored images
  • cut -f1 -d' ': Extracts image names
  • tail -n +2: Skips the header
  • xargs -n1 -r microk8s.ctr images rm: Removes each image reference

This step clears containerd’s image database, ensuring a clean slate for the new snapshotter directory.

Step 7: Recreate the OverlayFS Directory Structure#

Create a fresh snapshotter directory with the proper structure:

mkdir -p /var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots

The snapshots subdirectory is where containerd will store individual layer snapshots.201

Step 8: Start MicroK8s#

Restart MicroK8s to begin normal operations:

microk8s start

containerd will initialize with the clean snapshotter directory. Kubernetes will automatically begin pulling images and recreating pods according to your deployments, daemonsets, and statefulsets.

Step 9: Verify Operation#

Monitor the cluster to ensure pods are being recreated:

microk8s kubectl get pods --all-namespaces -w

Check that images are being pulled successfully:

microk8s.ctr images ls

Step 10: Remove the Old Directory (Optional)#

Once you’ve verified that MicroK8s is operating normally and all pods are running, you can safely remove the old snapshotter directory to reclaim disk space:

rm -rf /var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs.old

What Happens During This Process#

  1. Stopping MicroK8s: All Kubernetes components shut down gracefully, terminating pod connections and stopping the kubelet
  2. Killing tasks: Any remaining container processes are forcefully terminated87
  3. Renaming the directory: The existing OverlayFS snapshot data is preserved but made inaccessible to containerd
  4. Deleting containers and images: containerd’s internal database is cleared of all container and image references
  5. Recreating the directory: A fresh, empty snapshotter directory is created
  6. Starting MicroK8s: containerd starts with a clean slate, and Kubernetes begins reconciling the cluster state by pulling images and creating containers216

Expected Outcomes#

After completing this procedure:

  • Disk space recovered: The old snapshot directory can be removed, freeing significant disk space1715
  • All images repulled: Kubernetes will download fresh copies of all required container images
  • Pods recreated: All pods will be recreated from scratch according to your manifests
  • Clean filesystem state: No orphaned layers or corrupted snapshots remain

Prevention and Maintenance#

To avoid needing this drastic cleanup in the future:

  • Monitor disk usage regularly, especially the containerd directory1517
  • Configure image garbage collection properly by adjusting kubelet settings22
  • Use crictl rmi --prune periodically to remove unused images19
  • Implement microk8s.ctr content prune references as a scheduled task15
  • Set appropriate image pull policies to avoid unnecessary image accumulation
  • Monitor node events for garbage collection warnings15

Troubleshooting#

If MicroK8s fails to start after cleanup:

  1. Check containerd logs: journalctl -u snap.microk8s.daemon-containerd -n 100
  2. Verify directory permissions: ls -la /var/snap/microk8s/common/var/lib/containerd/
  3. Ensure the snapshots subdirectory exists

If pods fail to start:

  1. Check pod events: microk8s kubectl describe pod <pod-name>
  2. Verify image pull progress: microk8s.ctr tasks ls
  3. Check network connectivity for image registries

If you need to restore the old directory:

microk8s stop
rm -rf /var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs
mv /var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs.old io.containerd.snapshotter.v1.overlayfs
microk8s start

Conclusion#

Cleaning the containerd OverlayFS directory in MicroK8s is a powerful maintenance procedure that can reclaim significant disk space when standard garbage collection methods fail. By understanding the relationship between containerd, the containerd-shim, and the OverlayFS snapshotter, you can safely perform this operation while minimizing risks.

Remember that this is a disruptive operation that should only be performed when necessary, and always with a complete system backup. Regular maintenance and monitoring will help prevent the need for such drastic measures in most production environments. 23242526272829303132333435363738394041424344454647484950515253

⁂

  1. https://dev.to/napicella/what-is-a-containerd-snapshotters-3eo2 ↩︎ ↩︎ ↩︎ ↩︎

  2. https://iamachs.com/blog/docker/part-6-understanding-containerd-and-cri-o/ ↩︎ ↩︎ ↩︎

  3. https://containerd.io ↩︎

  4. https://github.com/containerd/containerd ↩︎

  5. https://spacelift.io/blog/containerd-vs-docker ↩︎

  6. https://zesty.co/finops-glossary/runc-in-kubernetes/ ↩︎ ↩︎ ↩︎

  7. https://container42.com ↩︎ ↩︎ ↩︎ ↩︎

  8. https://www.sobyte.net/post/2021-09/containerd-usage/ ↩︎ ↩︎ ↩︎ ↩︎

  9. https://devoriales.com/post/318/understanding-kubernetes-container-runtime-cri-containerd-and-runc-explained ↩︎ ↩︎

  10. https://stackoverflow.com/questions/71900937/is-it-possible-to-shrink-the-spaces-of-io-containerd-snapshotter-v1-overlayfs-fo ↩︎

  11. https://docs.docker.com/engine/storage/drivers/overlayfs-driver/ ↩︎

  12. https://ops.tips/notes/practical-look-into-overlayfs/ ↩︎ ↩︎

  13. https://jvns.ca/blog/2019/11/18/how-containers-work--overlayfs/ ↩︎

  14. https://k8studio.io/tutorials/container-architecture-namespaces-cgroups-overlayfs/ ↩︎

  15. https://pgmac.net.au/technology/2023/12/26/microk8s-garbage-collection.html ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎

  16. https://discuss.kubernetes.io/t/cleanup-unused-containerd-images/21827 ↩︎

  17. https://discuss.kubernetes.io/t/40gb-snapshot-folder/32547 ↩︎ ↩︎ ↩︎

  18. https://github.com/philippelt/microk8s_prune ↩︎

  19. https://stackoverflow.com/questions/72434877/how-to-make-microk8s-ctr-image-prune ↩︎ ↩︎

  20. https://github.com/containerd/containerd/discussions/10053 ↩︎

  21. https://www.upwind.io/glossary/container-runtimes-explained ↩︎

  22. https://github.com/ubuntu/microk8s/issues/482 ↩︎

  23. https://github.com/containerd/fuse-overlayfs-snapshotter ↩︎

  24. https://docs.keyfactor.com/ejbca/9.0/tutorial-clean-up-microk8s-cluster-and-redeploy ↩︎

  25. https://docs.docker.com/engine/storage/containerd/ ↩︎

  26. https://www.docker.com/blog/containerd-vs-docker/ ↩︎

  27. https://pkg.go.dev/github.com/containerd/containerd/snapshots/overlay ↩︎

  28. https://kubernetes.io/docs/setup/production-environment/container-runtimes/ ↩︎

  29. https://discuss.linuxcontainers.org/t/adding-overlayfs-to-run-docker-image-in-container/13858 ↩︎

  30. https://docs.cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/concepts/using-containerd ↩︎

  31. https://forums.docker.com/t/dockerd-uses-systemd-containerd-without-containerd-argument/124184 ↩︎

  32. https://raesene.github.io/blog/2018/08/05/Docker-Containers-Without-Docker/ ↩︎

  33. https://stackoverflow.com/questions/29599632/container-is-not-running ↩︎

  34. https://stackoverflow.com/questions/41645665/how-containerd-compares-to-runc ↩︎

  35. https://microk8s.io/docs/command-reference ↩︎

  36. https://dev.to/xavki/ultimate-guide-to-container-runtimes-from-docker-to-runc-and-beyond-3k4k ↩︎

  37. https://github.com/containerd/containerd/issues/2288 ↩︎

  38. https://docs.docker.com/engine/daemon/alternative-runtimes/ ↩︎

  39. https://github.com/containerd/containerd/issues/7076 ↩︎

  40. https://www.aquasec.com/cloud-native-academy/container-security/container-runtime/ ↩︎

  41. https://discuss.circleci.com/t/configuring-docker-daemon-for-using-containerd-image-store-instead-of-docker/51428 ↩︎

  42. https://artkai.io/blog/best-containerization-tools ↩︎

  43. https://github.com/containerd/containerd/issues/5708 ↩︎

  44. https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/7.2_release_notes/technology-preview-file_systems ↩︎

  45. https://microk8s.io/docs/troubleshooting ↩︎

  46. https://docs.docker.com/reference/cli/dockerd/ ↩︎

  47. https://docs.kernel.org/filesystems/overlayfs.html ↩︎

  48. https://microk8s.io/docs/build-microk8s-snap ↩︎

  49. https://stackoverflow.com/questions/67585798/what-process-is-the-parent-process-of-containerd-shim ↩︎

  50. https://forums.docker.com/t/need-help-with-stopping-containers-and-starting-docker-daemon/81507 ↩︎

  51. https://github.com/ubuntu/microk8s/issues/2740 ↩︎

  52. https://www.aquasec.com/blog/cve-2020-15257-containerd-shim-api-vulnerability/ ↩︎

  53. https://influentcoder.com/posts/overlayfs/ ↩︎