DevMate

Morning. I picked up a failed production deploy on acme/payments-api — commit 9f2c1d4, namespace payments-prod. Traffic is still on the previous revision, so customers aren't affected. Want the breakdown?

Build and manifest generation both passed. The rollout failed at verification: 3 of 4 pods were OOMKilled roughly 10 seconds after start and dropped into CrashLoopBackOff, so it never hit the minimum of 3 available replicas. The container limit is 256Mi and I measured a peak of 249Mi on the one pod that survived — that's 97% of the limit with no headroom.

The limit didn't change — the workload did. This commit adds the nightly batch reconciler, and SettlementReconciler.loadPendingBatches() pulls the full pending set into a list at startup. With SETTLEMENT_BATCH_SIZE at 5000 that's about 60Mi of extra heap before the service takes its first request. Every earlier deploy on this limit was fine because nothing allocated at boot.

Raise the memory limit and redeploy — it's a one-line manifest change and gives you ~50% headroom over the observed peak. identity-service already runs at 512Mi with a similar profile.

          resources:            requests:              cpu: 100m              memory: 384Mi            limits:              cpu: 500m              memory: 512Mi

Page the reconciler instead of loading everything at once — 500 rows at a time keeps the container under 300Mi even as settlement volume grows, so you're not re-tuning this limit next quarter. I'd ship the limit bump now and open a follow-up for the paging change. Want me to draft the updated manifest and a rollback command you can keep handy?