My coworker and I needed to increase the amount of days that Prometheus keeps logs from 15 days to around 90 days. This meant that we wanted to increase our storage, or our persistent volume.
We use the kube-prometheus-stack helm chart for installing helm and have customized it to have persistent storage.
It turned out that all we had to do was the following to make it work:
1 Allow volume expansion to the default storage class
kubectl patch sc gp2 -p '{"allowVolumeExpansion": true}'2. Change the chart to be the size you want it to be. Then reapply the helm chart!
3. Delete the old prometheus pod. A new one will start and it will be back up to the right size you wanted!
I thought we would loose all our prometheus metrics but turns out we kept everything!
