Load Balancing

Load Balancing

Configure load balancing for Mantis API and orchestration services.

Overview

Mantis uses different load balancing strategies for different components:

API Load Balancing (Mandible)

HTTP/HTTPS Load Balancing

Mandible instances are stateless and can be load balanced with any strategy.

NGINX Configuration:

upstream mandible {
    least_conn;
    server mandible-1:3000 weight=1;
    server mandible-2:3000 weight=1;
    server mandible-3:3000 weight=1;

    keepalive 32;
}

server {
    listen 443 ssl http2;
    server_name api.mantis.local;

    ssl_certificate /etc/nginx/ssl/server.crt;
    ssl_certificate_key /etc/nginx/ssl/server.key;

    location / {
        proxy_pass https://mandible;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Connection "";

        # Timeouts
        proxy_connect_timeout 60s;
        proxy_send_timeout 300s;
        proxy_read_timeout 300s;
    }

    # SSE endpoints need longer timeouts (streaming is served from /api/v1/sse/)
    location /api/v1/sse/ {
        proxy_pass https://mandible;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header Connection "";
        proxy_buffering off;
        proxy_cache off;

        # Extended timeout for SSE
        proxy_read_timeout 86400s;
    }
}

HAProxy Configuration:

frontend api_frontend
    bind *:443 ssl crt /etc/haproxy/certs/server.pem
    default_backend mandible_backend

backend mandible_backend
    balance leastconn
    option httpchk GET /api/v1/health

    server mandible-1 mandible-1:3000 check ssl verify none
    server mandible-2 mandible-2:3000 check ssl verify none
    server mandible-3 mandible-3:3000 check ssl verify none

Session Affinity

Mandible is stateless, so session affinity is not required. However, for SSE connections, sticky sessions can improve performance:

upstream mandible {
    ip_hash;  # Or use sticky cookie
    server mandible-1:3000;
    server mandible-2:3000;
}

Health Checks

Mandible exposes health endpoints:

Endpoint	Purpose	Success Response
/api/v1/health	Basic liveness	200 OK
/api/v1/health/ready	Full readiness	200 OK
/api/v1/health/live	Kubernetes liveness	200 OK

backend mandible_backend
    option httpchk GET /api/v1/health/ready
    http-check expect status 200

gRPC Load Balancing (Thorax)

Internal gRPC (Mandible → Thorax)

For Mandible to Thorax communication, use gRPC-aware load balancing:

Envoy Configuration:

static_resources:
  listeners:
    - name: grpc_listener
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 50051
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: grpc
                codec_type: AUTO
                route_config:
                  name: grpc_route
                  virtual_hosts:
                    - name: thorax
                      domains: ['*']
                      routes:
                        - match:
                            prefix: '/'
                          route:
                            cluster: thorax_cluster
                http_filters:
                  - name: envoy.filters.http.router

  clusters:
    - name: thorax_cluster
      connect_timeout: 5s
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      http2_protocol_options: {}
      load_assignment:
        cluster_name: thorax_cluster
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: thorax-1
                      port_value: 50051
              - endpoint:
                  address:
                    socket_address:
                      address: thorax-2
                      port_value: 50051
      health_checks:
        - timeout: 5s
          interval: 10s
          unhealthy_threshold: 2
          healthy_threshold: 1
          grpc_health_check: {}

External gRPC (Tarsus → Thorax)

Tarsus agents need direct connections to specific Thorax instances for target affinity. Do not load balance these connections.

Use Kubernetes headless service:

apiVersion: v1
kind: Service
metadata:
  name: thorax
spec:
  clusterIP: None # Headless - no load balancing
  selector:
    app: thorax
  ports:
    - port: 50051
      name: grpc

Kubernetes Load Balancing

Ingress for API

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mantis-api
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: 'HTTPS'
    nginx.ingress.kubernetes.io/proxy-body-size: '100m'
    nginx.ingress.kubernetes.io/proxy-read-timeout: '300'
spec:
  tls:
    - hosts:
        - api.mantis.local
      secretName: mantis-tls
  rules:
    - host: api.mantis.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: mandible
                port:
                  number: 3000

Service for Mandible

apiVersion: v1
kind: Service
metadata:
  name: mandible
spec:
  type: ClusterIP
  selector:
    app: mandible
  ports:
    - port: 3000
      targetPort: 3000
      name: https

Internal gRPC Service

apiVersion: v1
kind: Service
metadata:
  name: mandible-internal
spec:
  type: ClusterIP
  selector:
    app: mandible
  ports:
    - port: 50052
      targetPort: 50052
      name: grpc

Load Balancing Algorithms

For HTTP APIs (Mandible)

Algorithm	Use Case	Pros	Cons
Round Robin	Default	Simple, even distribution	Ignores server load
Least Connections	Varied request times	Balances actual load	More state tracking
IP Hash	SSE connections	Session affinity	Uneven distribution

Recommendation: Least Connections for most deployments.

For gRPC (Thorax)

Algorithm	Use Case	Pros	Cons
Round Robin	General	Simple	May interrupt streams
Pick First	Client-side	No infra needed	No HA
Weighted	Heterogeneous	Handles capacity diff	Complex config

Recommendation: Round Robin with health checks.

Connection Draining

Graceful Shutdown

Configure connection draining for zero-downtime deployments:

HAProxy:

backend mandible_backend
    option httpchk GET /api/v1/health/ready

    server mandible-1 mandible-1:3000 check ssl verify none
    server mandible-2 mandible-2:3000 check ssl verify none backup

    # Drain connections over 30 seconds
    default-server inter 3s fall 3 rise 2 slowstart 30s

Kubernetes:

spec:
  containers:
    - name: mandible
      lifecycle:
        preStop:
          exec:
            command: ['/bin/sh', '-c', 'sleep 15']
  terminationGracePeriodSeconds: 30

SSE Connection Handling

SSE connections require special handling during shutdown:

1. Stop accepting new connections
2. Send close event to existing SSE clients
3. Wait for clients to reconnect to other instances
4. Terminate remaining connections

TLS Configuration

TLS Termination at Load Balancer

server {
    listen 443 ssl http2;

    # Modern TLS configuration
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
    ssl_prefer_server_ciphers off;

    # Certificate
    ssl_certificate /etc/nginx/ssl/server.crt;
    ssl_certificate_key /etc/nginx/ssl/server.key;

    # OCSP Stapling
    ssl_stapling on;
    ssl_stapling_verify on;

    # Session caching
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;
    ssl_session_tickets off;
}

End-to-End TLS

For internal traffic encryption, configure backend TLS:

upstream mandible {
    server mandible-1:3000;
}

server {
    location / {
        proxy_pass https://mandible;
        proxy_ssl_verify on;
        proxy_ssl_trusted_certificate /etc/nginx/ssl/ca.crt;
    }
}

Monitoring

Load Balancer Metrics

Metric	Description	Alert Threshold
Active connections	Current connections	>1000/instance
Request rate	Requests per second	Capacity dependent
Error rate	5xx responses	>1%
Latency p99	99th percentile latency	>500ms
Backend health	Healthy backends	<2

Prometheus Metrics

# NGINX exporter
scrape_configs:
  - job_name: nginx
    static_configs:
      - targets: ['nginx-exporter:9113']

  # HAProxy exporter
  - job_name: haproxy
    static_configs:
      - targets: ['haproxy-exporter:9101']

Troubleshooting

Uneven Load Distribution

1. Check algorithm configuration
2. Verify health checks passing
3. Check for long-lived connections
4. Review connection limits

Connection Timeouts

1. Verify backend health
2. Check timeout configuration
3. Review network connectivity
4. Check TLS certificate validity

SSE Connection Drops

1. Increase proxy timeouts
2. Disable buffering for SSE paths
3. Check keep-alive configuration
4. Verify no connection limits hit

Next Steps

• Multi-Thorax Deployment - Deploy orchestrators
• Competing Consumers - RabbitMQ distribution
• Instance Affinity - Target binding