HorizontalPodAutoscaler scales up pods but then terminates them instantly
-
So I have a
HorizontalPodAutoscaler
set up for my backend (an fpm-server and an Nginx server for a Laravel application).The problem is that when the
HPA
is under load, it scales up the pods but it terminates them instantly, not even letting them get into the Running state.The metrics are good, the scale-up behavior is as expected the only problem is that the pods get terminated right after scaling.
What could be the problem?
Edit: The same
HPA
is used on the frontend and it's working as expected, the problem seems to be only on the backend.Edit 2: I have
Cluster Autoscaler
enabled, it does it's job, nodes get added when they are needed and then cleaned up so it's not an issue about available resources.deployment.yaml
apiVersion: apps/v1 kind: Deployment metadata: name: fpm-server labels: tier: backend layer: fpm spec: replicas: {{ .Values.replicaCount }} selector: matchLabels: tier: backend layer: fpm template: metadata: labels: tier: backend layer: fpm spec: {{- with .Values.imagePullSecrets }} imagePullSecrets: {{- toYaml . | nindent 8 }} {{- end }} containers: - name: fpm image: "{{ .Values.fpm.image.repository }}:{{ .Values.fpm.image.tag }}" ports: - name: http containerPort: 9000 protocol: TCP env: {{- range $name, $value := .Values.env }} - name: {{ $name }} value: "{{ $value }}" {{- end }} envFrom: - secretRef: name: backend-secrets resources: {{- toYaml .Values.resources | nindent 12 }}
hpa.yaml
apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: fpm-server-hpa labels: tier: backend layer: fpm spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: fpm-server minReplicas: {{ .Values.autoscaling.minReplicas }} maxReplicas: {{ .Values.autoscaling.maxReplicas }} metrics: {{- if .Values.autoscaling.targetCPUUtilizationPercentage }} - type: Resource resource: name: cpu targetAverageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }} {{- end }} {{- if .Values.autoscaling.targetMemoryUtilizationPercentage }} - type: Resource resource: name: memory targetAverageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }} {{- end }}
-
Seems that the problem was with the
replicas: {{ .Values.replicaCount }}
definition. It seems that, if you are using HPA,replicas
can't be used. I removed this line and the HPA started scaling.