We have a couple of pods that get pretty intense memory use. It’s possible there is a memory leak because it OOM’d a node due to a spike in usage. One of our temporary solutions while we investigate is to put a pod anti-affinity rule in the deployment. What this means is that we don’t like it when pod instances run on the same node. That way we spread them out.
As we may have more pods then nodes, we don’t want to make this a blocking requirement. Therefore we use the
affinity: # schedule pods on different nodes podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app.kubernetes.io/name operator: In values: - pod-name topologyKey: kubernetes.io/hostname
Here, the label on the pods are
app.kubernetes.io/name: pod-name This will now make it so the pods aren’t up on the same place.
You can see this with:
kubectl get pods -o wide
And it will show that pods that have the same name are scheduled on different nodes.