Fern: a Neobank Backend, Load Tested
Fern is a small neobank backend I built to see what it takes to move money correctly: a double-entry ledger on Postgres, ACH through a real bank sandbox, and the failure modes a money system actually meets. It runs on AWS for a week with a demo UI that shows the trace of every request next to a phone.
- Live demo: waken.dev/apps/fern (live until 2026-11-19; after that a static archive that replays a recorded run)
- Source: github.com/wakenmeng/fern
Sandbox only. No real money, no real identities.
The five cases
Each case is an everyday flow, chosen for the failure it forces the system to handle.
| # | Case | Failure it demonstrates |
|---|---|---|
| S1 | Get paid: a payroll ACH credit lands | the bank’s webhook arrives twice |
| S2 | Pull $500 from an external bank (ACH debit) | the money is held until settlement and the return window |
| S3 | Send $50 to a friend | concurrent spends never overdraw |
| S4 | Withdraw to an external bank (ACH credit) | the bank accepts it, but the response is lost |
| S5 | The pulled $500 comes back as unauthorized (R10) | the balance goes negative, the account freezes, ops writes the loss off with two approvals |
The demo walks through them in order, one button at a time. The panel on the right is the real trace of the request you just made: every SQL statement, every bank, Plaid and Redis call, and the journal entries it posted.
Architecture
1
2
3
4
Browser ─► Cloudflare Worker (waken.dev/apps/fern) ─► Cloudflare Tunnel
└─► AWS EKS: FastAPI (Python 3.13) ─┬─► RDS Postgres 16, Multi-AZ (ledger)
├─► ElastiCache Redis 7 (rate limits)
└─► Column (bank, ACH) · Plaid (balances) · OpenAI (ops report)
Ledger. Every accounting effect is one journal with two postings, and each effect has exactly one legal shape: ach_debit.settle must debit bank and credit ach_debit_receivable, nothing else. Any effect that touches cash must point at a bank transaction of the same amount. Customer balances may only go negative through a return, never through a spend. Account rows are locked in id order, so two transfers between the same accounts can’t deadlock.
Idempotency everywhere. Every money-moving request carries an Idempotency-Key; every journal has a dedupe key. Replaying a request, or receiving the same bank webhook twice, posts nothing new. Posting a different effect under an existing key is an error worth alerting on.
The bank is the source of truth. When a call to the bank times out (S4), the money stays held and the transfer is unknown, not failed. Recovery asks the bank what happened and books only what the bank confirms. Reconciliation compares the ledger with the bank’s own records and opens an exception for anything that doesn’t match.
Integrations. Column’s sandbox for accounts and ACH, behind the same interface as an in-process simulator that follows Column’s documented behaviour (outgoing credits leave at initiation, pulled debits arrive at settlement, webhooks repeat and arrive out of order). Plaid’s sandbox checks the linked account’s balance before a pull. OpenAI writes the daily ops report, with a rules-based fallback and a daily budget.
Rate limits. A token bucket per space in Redis, as one Lua script. Redis holds no money state, so if it’s unreachable the limiter lets the request through and says so in the trace.
Infra. EKS on two Graviton t4g.medium nodes, RDS db.t4g.micro Multi-AZ, ElastiCache, provisioned with OpenTofu. Traffic comes in through a Cloudflare Tunnel, so there is no load balancer, no NAT gateway and no inbound port. The estimate for 72 hours was about $18.
Shutting down. At a fixed time the Worker stops forwarding to AWS and serves only static files. The demo page was recorded on the live system with ?record=1; in the archive it replays that recording, so the buttons still work without a backend.
Experiments
Load test method
The load generator runs as a Kubernetes Job inside the cluster, against a separate copy of the API with the in-process bank (the Column sandbox is not load tested). Every run starts from a fresh database.
- Mix: 40%
GET /me, 15%GET /transfers, 15%POST /p2p, 12%POST /withdrawals, 12%POST /pulls, 6% replays of an earlier request with the same idempotency key. - Think time: 0–1 s between requests.
- Step test: 5 stages of 60 s, more load at every stage. A stage is within target when p95 ≤ 500 ms with no errors.
- Money check after the run: every space is compared against a model of the requests it accepted. Books must balance, balances must match exactly, nothing may be negative, and no replay may have created a second transfer.
Throughput alone says nothing about a money system. The money check is the part that matters.
A deadlock first
The load test found a deadlock before it produced a throughput number. A request held its database connection for the whole response, and saving the request’s trace needed a second connection from the same pool. Under load, every connection belonged to a request waiting for another connection. The fix was a small separate pool for traces, and a test that fires 40 concurrent writes and fails on a timeout.
Run 1: production size
Against the same db.t4g.micro the demo uses, with two API replicas:
| Stage | Requests | req/s | p50 ms | p95 ms | p99 ms | Errors |
|---|---|---|---|---|---|---|
| 1 | 2,904 | 48 | 9.7 | 60 | 141 | 0 |
| 2 | 5,808 | 97 | 10.4 | 57 | 109 | 0 |
| 3 | 9,882 | 165 | 31.9 | 633 | 1,177 | 0 |
| 4 | 9,876 | 165 | 159 | 1,302 | 1,662 | 0 |
| 5 | 5,970 | 100 | 1,050 | 4,285 | 5,755 | 0 |
Capacity within target: about 97 req/s. Throughput levels off at 165 req/s and then falls as queries queue for CPU: RDS sat at 85–95% CPU from stage 3, while the API pods were not the limit. 35,133 requests, 13,856 transfers, 0 errors, 0 money errors.
Run 2: a larger database
The second run used its own RDS db.m7g.2xlarge (8 vCPU, Multi-AZ), 12 API pods on two c7g.4xlarge nodes, and a build that writes traces with group commit. It cost about $2.50 an hour and was torn down right after.
| Stage | Requests | req/s | p50 ms | p95 ms | p99 ms | Errors |
|---|---|---|---|---|---|---|
| 1 | 11,736 | 196 | 5.8 | 39 | 42 | 0 |
| 2 | 23,262 | 388 | 6.4 | 40 | 43 | 0 |
| 3 | 46,428 | 774 | 8.1 | 44 | 50 | 0 |
| 4 | 68,598 | 1,143 | 11.7 | 57 | 99 | 0 |
| 5 | 79,524 | 1,325 | 83.2 | 271 | 416 | 0 |
Every stage met the target. 1,325 req/s at p95 271 ms, 1,442 req/s at the best 10 seconds. 230,894 requests in 5 minutes, 90,222 transfers across 800 spaces, 0 errors, 0 money errors.
Per endpoint over the whole run:
| Endpoint | Requests | p50 ms | p95 ms | p99 ms |
|---|---|---|---|---|
GET /me | 92,355 | 9.4 | 109 | 190 |
POST /p2p | 34,843 | 31.0 | 207 | 335 |
GET /transfers | 34,534 | 7.9 | 93 | 171 |
POST /pulls | 27,706 | 39.8 | 189 | 344 |
POST /withdrawals | 27,673 | 52.2 | 300 | 451 |
| replay (same key) | 13,783 | 13.6 | 66 | 197 |
What the numbers do and don’t say
- RDS CPU peaked around 57% in run 2, so the database was no longer the limit. The latency rise at the last stage more likely comes from the API processes: 11 of the 12 pods landed on one node. I didn’t measure it directly.
- Between the two runs both the database class and the trace writes changed, so the 13× gain is not split between them.
- Both runs use the simulated bank. They measure Fern’s API, Postgres and Redis, not Column.
- The number I care about most is the same in both runs: zero money errors, including under 6% duplicate replays.
Takeaways
- Model every money movement as an effect with one legal shape, and make the database reject everything else. Most correctness came from that table of effects, not from careful handlers.
- Treat a timeout from the bank as “unknown”, hold the money, and ask. Never retry a transfer blindly, and never mark it failed on a guess.
- A load test for a money system needs a model to check the books against afterwards. Requests per second without that check is just a number.
- Load tests find concurrency bugs that unit tests don’t. The load test’s first useful result was a deadlock, not a throughput figure.