What to Do When Prometheus Runs Out of Space

If Prometheus stops working because it runs out of disk space, you need to increase the size of its persistent volume. To do this, complete these steps:

Note: In these commands, NAMESPACE_OF_INFRASTRUCTURE_COMPONENTS is the namespace where your infrastructure components such as the logging solution and the monitoring and alerting solution are installed.

1. Scale the number of Pods of the Prometheus Operator Deployment to 0:

   kubectl -n NAMESPACE_OF_INFRASTRUCTURE_COMPONENTS scale deployment prometheus-operator-kube-p-operator --replicas=0

2. Scale the number of Pods of the Prometheus StatefulSet to 0:

   kubectl -n NAMESPACE_OF_INFRASTRUCTURE_COMPONENTS scale statefulset prometheus-prometheus-operator-kube-p-prometheus --replicas=0

3. Increase the size of the persistent volume used by Prometheus:

   kubectl -n NAMESPACE_OF_INFRASTRUCTURE_COMPONENTS patch pvc prometheus-prometheus-operator-kube-p-prometheus-db-prometheus-prometheus-operator-kube-p-prometheus-0 --patch '{\"spec\": {\"resources\": {\"requests\": {\"storage\": \"NEW_SIZE_IN_GIB\"}}}}'

   In this command, NEW_SIZE_IN_GIB is a new size of the persistent volume in GiB. For example: 100Gi

4. Scale the number of Pods of the Prometheus Operator Deployment to 1:

   kubectl -n NAMESPACE_OF_INFRASTRUCTURE_COMPONENTS scale deployment prometheus-operator-kube-p-operator --replicas=1

As a result, the Prometheus Operator will start Prometheus, and the size of the persistent volume will be increased during the start of Prometheus.