HorizontalPodAutoscaler scales up pods but then terminates them instantly



  • So I have a HorizontalPodAutoscaler set up for my backend (an fpm-server and an Nginx server for a Laravel application).

    The problem is that when the HPA is under load, it scales up the pods but it terminates them instantly, not even letting them get into the Running state.

    The metrics are good, the scale-up behavior is as expected the only problem is that the pods get terminated right after scaling.

    What could be the problem?

    Edit: The same HPA is used on the frontend and it's working as expected, the problem seems to be only on the backend.

    Edit 2: I have Cluster Autoscaler enabled, it does it's job, nodes get added when they are needed and then cleaned up so it's not an issue about available resources.

    deployment.yaml

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: fpm-server
      labels:
        tier: backend
        layer: fpm
    spec:
      replicas: {{ .Values.replicaCount }}
      selector:
        matchLabels:
          tier: backend
          layer: fpm
      template:
        metadata:
          labels:
            tier: backend
            layer: fpm
        spec:
          {{- with .Values.imagePullSecrets }}
          imagePullSecrets:
            {{- toYaml . | nindent 8 }}
          {{- end }}           
          containers:
            - name: fpm
              image: "{{ .Values.fpm.image.repository }}:{{ .Values.fpm.image.tag }}"
              ports:
                - name: http
                  containerPort: 9000
                  protocol: TCP
              env:
              {{- range $name, $value := .Values.env }}
              - name: {{ $name }}
                value: "{{ $value }}" 
              {{- end }}             
              envFrom:
                - secretRef:
                    name: backend-secrets
              resources:
                {{- toYaml .Values.resources | nindent 12 }}
    

    hpa.yaml

    apiVersion: autoscaling/v2beta1
    kind: HorizontalPodAutoscaler
    metadata:
      name: fpm-server-hpa
      labels:
        tier: backend
        layer: fpm
    spec:
      scaleTargetRef:
        apiVersion: apps/v1
        kind: Deployment
        name: fpm-server
      minReplicas: {{ .Values.autoscaling.minReplicas }}
      maxReplicas: {{ .Values.autoscaling.maxReplicas }}
      metrics:
        {{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
        - type: Resource
          resource:
            name: cpu
            targetAverageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
        {{- end }}
        {{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
        - type: Resource
          resource:
            name: memory
            targetAverageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
        {{- end }}
    


  • Seems that the problem was with the replicas: {{ .Values.replicaCount }} definition. It seems that, if you are using HPA, replicas can't be used. I removed this line and the HPA started scaling.


Log in to reply
 


Suggested Topics