How can I unevenly distribute pods across Instance Groups
-
What I'd like to do is distribute 10% of my pods to one instance group with the other 90% on the other group.
I would like to experiment with using different AWS instance types (AMD, graviton, etc) but I only want to put a limited number of pods onto these instances. Ideally, I'd like to do it by service.
I've looked into PodTopologySpread but it appears to try to get roughly even distributions within a range. Alternatively, set the pod scheduler to prefer the different instance type and just limit how many nodes I can run but that doesn't work well if multiple services are in the instance group. The final option is to have two deployment objects with different node counts and but the same service selectors. That would work but seems like a work to tweak pod distributions.
How can I distribute a small number of pods to different instance types?
-
you are going the right way with PodTopologySpreadConstraints(PTSC for short) to have different instance of the same pod in an uneven distribution through all your k8s nodes. On that note, I would remind you that you can pair various PTSC rules to achieve your desired goal. Based on instance type labels as well as per availability zone.
But what is missing, and could be a solution to your problem, is to incrementally send a slice of your traffic to those pods in order to do a comparison in performance or whatever you want to compare.
You should look into what your ingress controller can manage. For instance, NGINX ingress-controller uses https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#canary and Traefik ingress-controller uses https://traefik.io/glossary/kubernetes-deployment-strategies-blue-green-canary/ to send a certain percentage of traffic to those "test" pod and once you have your metrics you can go on and complete the rollout.
