Why my Stripe test mode passed and production failed

The staging environment was green. Every test scenario passed: successful payment, declined card, 3D Secure challenge, partial refund, subscription upgrade. We had 47 integration tests covering the Stripe flow. All passed.

We pushed to production on a Tuesday morning. By Tuesday afternoon, 3 out of the first 8 real payments were stuck in a "processing" state. The customers were charged but our system didn't acknowledge it.

Here's what went wrong — three issues, all invisible in test mode.

Issue 1: Webhook signature verification with the wrong secret

We had two webhook endpoints: one for staging, one for production. Each gets its own signing secret from the Stripe dashboard. Our config looked like this:

// .env.production
STRIPE_WEBHOOK_SECRET = whsec_live_xxxxx

Except it didn't. During the staging-to-production migration, someone (me) copied the staging .env file and updated the API keys but missed the webhook secret. The production server was verifying webhook signatures against the staging secret.

const event = stripe.webhooks.constructEvent(
  body,
  signature,
  process.env.STRIPE_WEBHOOK_SECRET // ← staging secret in production
)

In test mode, this "worked" because we were testing against the staging endpoint with the staging secret. In production, Stripe signs with the production secret. Every webhook arrived, failed signature verification, and was silently dropped.

The silent part is what got us. Our error handling caught the StripeSignatureVerificationError and returned a 400 — which Stripe retried 3 times, then gave up. No alert on our side because the error was "expected" (we had a catch block that logged it at warn level, not error).

Fix: Separate env validation that checks the webhook secret matches the API key environment. And promote webhook signature failures to error level with an alert.

// Startup validation
if (
  process.env.STRIPE_SECRET_KEY?.startsWith('sk_live_') &&
  !process.env.STRIPE_WEBHOOK_SECRET?.startsWith('whsec_live_')
) {
  throw new Error('Production Stripe key with non-production webhook secret')
}

Issue 2: Idempotency keys that collided

Our checkout flow used idempotency keys to prevent double charges:

const session = await stripe.checkout.sessions.create(
  {
    // ...session config
  },
  {
    idempotencyKey: `checkout_${userId}_${eventId}`,
  }
)

In test mode, we created and deleted test data freely. The same user-event combination was reused across test runs. No problem — Stripe test mode resets idempotency keys after 24 hours.

In production, a real user tried to buy tickets for an event, got a network timeout, refreshed, and tried again within 30 seconds. The idempotency key was identical. Stripe returned the original session (correct behavior), but our frontend created a new UI state expecting a new session URL. The user saw a "session expired" page.

The deeper issue: our idempotency key didn't include a timestamp or attempt counter. Two legitimate attempts within the same minute got the same key.

Fix: Include a checkout attempt counter in the idempotency key:

const session = await stripe.checkout.sessions.create(
  {
    // ...session config
  },
  {
    idempotencyKey: `checkout_${userId}_${eventId}_${attemptNumber}`,
  }
)

Where attemptNumber is stored in Redis with a 5-minute TTL. Same user + same event within 5 minutes increments the counter.

Issue 3: Metadata size limit

Our checkout sessions included metadata about the purchase:

metadata: {
  userId: user.id,
  eventId: event.id,
  seats: JSON.stringify(selectedSeats), // ← the problem
  locale: currentLocale,
  referralCode: referralCode ?? '',
}

In test mode, we tested with 1–4 seats. selectedSeats was something like ["A1","A2","A3"] — 20 bytes. Fine.

In production, a group purchase selected 12 seats. The seats metadata value was 180 characters. Stripe's metadata value limit is 500 characters, so it fit. But the total metadata size exceeded Stripe's limit when combined with other fields and long UUIDs. Stripe rejected the request with a generic validation error.

Test mode didn't catch this because we never tested with more than 4 seats. The metadata limit isn't enforced differently in test mode — we just never hit it.

Fix: Don't store seat lists in metadata. Store a reference:

metadata: {
  userId: user.id,
  eventId: event.id,
  seatCount: String(selectedSeats.length),
  bookingRef: bookingId, // look up seats from our DB, not Stripe
}

Stripe metadata is for cross-referencing, not data storage.

What test mode doesn't test

After this incident, I made a list of things Stripe test mode can't catch:

1. Wrong webhook secret — test mode uses the test secret, which is correct for the test endpoint. You can't simulate a secret mismatch.
2. Real card network behavior — 3D Secure challenges in test mode are instant. In production, some banks add 10-second delays. Our timeout was 5 seconds.
3. Metadata edge cases — test data is usually minimal. Production data is messy, long, and Unicode-heavy.
4. Rate limits — Stripe test mode has generous rate limits. Production limits are lower for new accounts. Our ticket sale launch hit the limit on the first minute.
5. Currency edge cases — test mode accepts any currency. Production enforces your Stripe account's enabled currencies. We had a user with a browser locale that triggered an unsupported currency.

The checklist I now run before every Stripe go-live

1. Verify webhook secret matches the environment. Log the first 8 characters of both the API key and webhook secret at startup. If they're from different environments, fail loudly.
2. Test with realistic data volumes. If users can select 20 items, test with 20 items. If names can be 200 characters, test with 200 characters.
3. Test idempotency with rapid retries. Open two tabs, submit the same checkout simultaneously. Verify both tabs resolve correctly.
4. Set up a production webhook monitor. A Sentry alert for any webhook that returns non-200. Not just signature failures — any failure.
5. Do one real transaction. Buy your own product with a real card before opening to users. The €1 test charge catches more issues than 100 test-mode transactions.

I've written more about Stripe webhook failure modes — that post covers the 6 patterns that break after your first 1,000 events.

Shipping a Stripe integration to production and want a sanity check? Let's talk — I've shipped Stripe into 4 production systems and can review your integration before real money flows through it.