Payment Webhooks & Reconciliation

Webhooks push payment events to your system in real-time, eliminating the need to poll for status changes.

Common payment webhooks: - payment_intent.succeeded - payment_intent.payment_failed - invoice.paid - invoice.payment_failed - customer.subscription.updated - charge.dispute.created

Webhook delivery: - HTTP POST to your endpoint - JSON payload with event details - Retry on failure (with backoff) - Signature for authenticity verification

Why webhooks are essential: - Real-time notification - Asynchronous payment flows (3DS, bank transfers) - Server-to-server reliability - Complete event history

Without proper webhook handling, your system will have blind spots where payment states don't match reality.

Verify every webhook to prevent spoofed events from attacking your system.

Signature verification:

Stripe: - Uses HMAC-SHA256 with webhook secret - Signature in Stripe-Signature header - Timestamp prevents replay attacks - Use Stripe's SDK verification methods

MangoPay: - Uses webhook signature header - Verify using shared secret - Check timestamp freshness

Implementation rules: - NEVER skip signature verification - HTTPS only (reject HTTP) - Consider IP whitelisting (additional layer) - Reject invalid signatures immediately

Example (Stripe): 1. Extract signature from header 2. Compute expected signature using raw body + secret 3. Compare using constant-time comparison 4. Reject if mismatch

Always use the SDK's built-in verification—don't implement your own.

Webhooks may be delivered multiple times. Your handler must produce the same result regardless.

Why duplicates happen: - Your server was slow to respond (timeout → retry) - Network issues during acknowledgment - Payment provider retry logic - Manual retries during debugging

Idempotency implementation:

Event ID tracking: - Store processed event IDs - Check before processing - Skip if already processed - Use database unique constraint

State machine approach: - Define valid state transitions - Ignore events that don't advance state - subscription: active → canceled (valid) - subscription: canceled → active (check carefully)

Database transaction: - Wrap processing in transaction - Rollback on any failure - Prevents partial updates

Example flow: 1. Receive webhook 2. Verify signature 3. Check if event_id in processed_events 4. If exists, return 200 (acknowledge, don't reprocess) 5. If new, process and store event_id atomically

Production webhook handling requires robust architecture.

Acknowledge quickly: - Return 200 within 5-10 seconds - Providers retry on timeout - Queue heavy processing for async

Async processing pattern: 1. Receive webhook 2. Verify signature 3. Store raw event in database 4. Return 200 immediately 5. Background job processes event 6. Retry logic for processing failures

Error handling: - Return 4xx for invalid signatures (don't retry) - Return 5xx for server errors (do retry) - Never return 200 if you didn't process successfully

Dead letter queue: - Store events that fail processing - Alert on accumulation - Investigate and replay manually - Don't lose events

Monitoring: - Track webhook volume - Alert on delivery failures - Monitor processing lag - Dashboard for operational visibility

Reconciliation ensures your data matches the payment provider's records.

Why reconciliation matters: - Webhooks can be missed (outages, bugs) - Processing errors may occur - Prevents revenue leakage - Required for accurate accounting

Reconciliation levels:

Real-time (per transaction): - After webhook processing, verify via API - Catch immediate discrepancies - Expensive at scale

Batch (daily/weekly): - Pull transaction list from provider - Compare against internal records - Flag mismatches for investigation

Settlement (monthly): - Match bank deposits to expected payouts - Verify fee calculations - Generate audit reports

Implementation: 1. Pull transactions from payment provider API 2. Match against internal transaction table 3. Identify: missing, extra, amount mismatches 4. Generate exception report 5. Investigate and resolve discrepancies

Learn from common webhook and reconciliation problems.

Missed webhooks: - Cause: Server downtime, network issues - Solution: Daily reconciliation catches gaps - Prevention: Redundant webhook endpoints

Out-of-order events: - Cause: Async delivery, retries - Solution: Check timestamps, use state machine - Don't assume sequential delivery

Duplicate charges: - Cause: Retry logic bugs, race conditions - Solution: Idempotency keys on creation - Reconciliation catches after the fact

Refund mismatches: - Cause: Refund webhook missed - Solution: Pull refund list in reconciliation - Alert on unexpected balance changes

Fee discrepancies: - Cause: Rate changes, volume tier changes - Solution: Pull fee details from provider - Monthly fee reconciliation

Currency conversion: - Cause: Different conversion times - Solution: Use consistent rate source - Track conversion rate with transaction

Best practices: - Log everything (requests, responses, decisions) - Build reconciliation from day one - Alert on any discrepancy - Regular audits of payment data