An auto-trader postmortem

So I’ve been meaning to write this one for a while, and now that the project is officially dead, it feels like the right time. This is the story of how I spent the better part of a year, and a chunk of my own money, building a bot to trade crypto on Polymarket, getting a model to 78% accuracy, and still losing.

TLDR:
Polymarket has a “Will BTC go up or down in the next 15mins”. That seems like something we can get data on, train a model and consistently win. Most of the data is publicly available, so what could go wrong. Well… xD

Quick disclaimer up front: none of this is financial advice. If anything this is anti-financial advice – meaning, please don’t do this at home.
I’m just a developer who got nerd-sniped by a market that reprices every 15 minutes. Absolutely don’t remortgage your house on the back of a blog post.

What we’re actually trading

Polymarket runs rolling 15-minute “Will BTC be up or down?” binaries. Every quarter hour a new market opens, and it resolves against the price move over the window. You buy “Up” shares or “Down” shares at some price between 0 and 1, and if you’re right each share pays out 1.

Two things make this look like free money:

1. A clean label every 15 minutes. That’s ~96 resolved outcomes a day. Thousands a month. For anyone who’s tried to train a model on sparse data, this is a buffet.
2. It’s genuinely predictable. Feed a model order-flow, some spot microstructure from Pyth and Binance, and where you are in the slot, and it’ll call direction at ~78-82% at high confidence. That number is real. It held up across months of walk-forward testing.

78% model accuracy. Free money, right? Heh, nope.

The boring math

Here’s the thing. On a binary priced at p, you pay p per share and win 1. Your break-even win rate is p, you need to be right at least as often as the price implies. That’s before fees. Polymarket takes a ~7% taker fee on the size.

Now, when your model is confident, the market has usually already priced the favourite at 0.80-0.85. It’s not stupid. So:

• You buy the favourite at 0.82.
• Break-even is ~82%, plus fee, call it ~84%.
• Your model gives you ~80%.
• You lose. Slowly, politely, but you lose.

That’s it. That’s the whole trap. 78% accuracy is a losing bet at the prices you’re actually offered. The only bands where the model clears its hurdle are the cheap ones (0.50-0.65). The thing is that these bands are rare, truly rare. I managed to get in a few and those were a real winner, but the market almost never hands you the favourite that cheap when the model is confident, because the market is also good at this. You’re fighting a reasonably efficient book with a 7% handicap strapped to your ankle.

The service setup

The architecture, quickly, because a couple of bits are worth stealing:

• A Python worker does everything that matters, ingest data, run the model, apply the gates, decide the trade.
• A tiny Node bridge is the only thing allowed to place orders. Polymarket’s deposit-wallet auth needs a signature type the Python client just can’t build, so the bridge owns order placement and nothing else.
• Postgres as the single source of truth. Every book snapshot, every tick, every resolved slot, every trade.
• Four trading rails running side by side: one paper rail that fires on everything with no gates (the honest view of the raw signal), a couple of “realistic” paper rails with simulated fills and fees, and the live rail with real money.

I set up two paper traders that ran 24/7 and a live trader on the infra above. The live trader was switched on/off when needed, and the paper traders were used as a control, theoretically.

8 months of iterations

This is the part where I show you the graveyard.

Gate tuning

Confidence floors, time-in-slot windows, flow-score gates, anti-whipsaw, loss-streak pauses. Each one, fitted to recent data, helped for about a week and then quietly decayed. Classic overfitting to a regime that won’t sit still. I eventually wrote myself a rule: never trust a change backtested on less than the full history. Short windows will happily sell you a pattern the long history laughs at.

Fractional Kelly sizing

Size each bet by the modelled edge. Sounds disciplined. The failure mode is nasty and non-obvious: Kelly bets more exactly when it thinks the edge is biggest, so on a marginally-negative edge it amplifies your losers. On a small, real, shrinking wallet that becomes a death spiral, big bets while the bankroll’s healthy, tiny recovery bets once it’s bled. Paper never felt this, because paper’s bankroll doesn’t actually shrink. Keep that thought.

“Let the model calibrate itself and be the gate.” Recalibrate the raw probability against realised outcomes on a rolling window, feed the honest number to Kelly, and let Kelly’s edge check refuse the bad trades on its own. Elegant on paper. It ran live. It didn’t save me, because it optimises a modelled edge and the real bottom line gets eaten by fees, sizing skew, and noise.

A LightGBM meta-labeler

A second model whose whole job is to predict when the first model is right. Kept it in shadow. First forward evaluation came back with an AUC of 0.36, for the non-ML folks, that’s worse than a coin flip. Retrained it, got it respectable in cross-validation, never convincingly better live. Shelved.

Regime detection. 

Per-regime confidence floors. This one genuinely helped, it filtered a couple of structurally awful market states. Did it flip the sign of the overall result? No.

You see the pattern. Every single one of these is decent engineering. Every single one is polishing the execution around a signal that can’t clear its costs.

A floor that tunes itself

By late August I had a sharp diagnosis. The profitable price band moves over time, a band that printed money all year had quietly gone red over the last two months, while a previously-losing band had turned green. Any fixed floor I set was stale the day I set it.

So I built the thing I was actually proud of: a fill-band gate that tunes itself. For each price band, track the realised, fee-adjusted P&L on a rolling window, run a proper 95% confidence-interval significance test, and skip only the bands that are statistically-significant losers. Neutral or thin-data bands? Let them through, blocking those is just fitting noise. As a band decays, its interval slides below zero and it gets blocked. When it recovers, it comes back. A floor that walks.

And the backtest was good. Replaying every historical live trade, walk-forward, no peeking: the blocked bands visibly migrated red-to-green over time, and the counterfactual added +$112 (+71%) to realised P&L. I was chuffed.

So, what happened when it went live? Nothing. Literally nothing. It blocked zero trades while the wallet bled 5%.

Here’s the punchline of the whole project, so pay attention because it’s the bit that actually taught me something. The gate can only fire on a band that’s a decisive, consistent loser, that’s the exact property that makes it trustworthy instead of noise-fitting. But live’s real failure mode isn’t a decisive loser. It’s a diffuse, marginal bleed: minus a dollar or two per trade, buried under ten dollars of per-trade variance. The significance test is designed not to fire on that. And you can’t buy your way out with volume, I checked, coarsening the bands to pool more samples, and the intervals still straddled zero. A tool that only catches the decisive loser cannot save you from the slow one. And a slow, marginal, negative edge is precisely what an efficient market leaves lying around for a taker paying 7%.

The problem with my paper traders

The realistic paper rail showed +$112 over a stretch where live did −$33. Same model, same window. I chased that gap for a while assuming it was execution, bad fills, slippage, something I could fix. It wasn’t. Matched slot-for-slot, the fills were nearly identical.

It was two quieter things:

1. Bankroll scale. Paper’s Kelly sized against a big, non-shrinking pretend bankroll. Live sized against the real, shrinking one. Paper’s dollar figures were inflated and its variance magically healed itself.
2. Optimistic fills. Paper filled about 10-15 cents cheaper than the real order book, which shoves the apparent profitable band downwards and makes cheap longshots look like heroes.

If there’s one line to take from this whole post, it’s this: a paper rail is only honest if it sizes against your real wallet and fills at real prices. Otherwise it’s not a research tool, it’s a machine for telling you you’re a genius while you lose money.

The lessons, as best I can summarise

1. Accuracy is not edge. 78% on a 0.82 favourite with a 7% fee is a losing bet. Work out your fee-adjusted break-even win rate at the price you’ll pay, first, before you write a single line of model code.
2. The signal was the constraint, not the tooling. Kelly, calibration, meta-labeling, the self-tuning gate, I spent months perfecting the 20% and never once fixed the 80%.
3. Fees are the whole game on short-horizon binaries. 7% a pop on a market that reprices every 15 minutes quietly ate about a third of gross. Brutal.
4. Beware the safety tool that can’t fire when you need it. A conservative gate is trustworthy because it won’t act on a marginal bleed, which means it can’t protect you from one.
5. Small real money teaches faster than large paper money. The Kelly spiral, the fill gap, the fee drag, none of it showed up in paper. All of it showed up in the first fifty quid of real losses.

The scoreboard

• Live wallet: about $490 at the start of the final experiment → $366 at shutdown. Down over the life of the thing.
• Live trades actually placed: 5,270.
• Model accuracy: ~78-82% the whole way through. Never the problem. Never enough.

Live trades breakdown

What I kept: polytrader

Here is the thing nobody tells you: the hard part of a Polymarket bot is not the model, it is talking to Polymarket at all. Their deposit-wallet API needs an ERC-1271 signature the Python SDK simply cannot produce, so every order has to go through a little Node sidecar. Getting that plumbing right cost me more hours than the model ever did.

So before I burned it all down, I ripped the plumbing out into a clean, standalone package: polytrader. No strategy, no model, no opinions. Just the boring, reliable connectivity: market data, orders, balance, health, and treasury, behind a small async API.

from polytrader import PolyTrader, PolyTraderConfig

async with PolyTrader(PolyTraderConfig(bridge_url="http://localhost:3000")) as pt:
    if (await pt.verify_conn_health()).ok:
        res = await pt.bet_on("btc-updown-15m-...", "UP", amount_usd=5)
        print(res.ok, res.filled_shares, res.avg_price)

The scars are baked in as features. Orders never auto-retry, because the one time a network blip makes you double-fill a bet is the time it costs you real money. Sub-$1 buys get rejected before they waste a round trip. Deposits and withdrawals are honest about what they can and cannot do. It ships with 113 tests and 93% coverage, it is MIT licensed, and the docs (README, AGENTS.md, CLAUDE.md) tell you and your AI assistant exactly how to wire it in.

Take it, use it. Issues and pull requests are welcome. If you build something with it, I would love to hear about it.

So, was it worth it?

The bot? No. The lesson? Genuinely, yes. I now understand, in my bones and not just my spreadsheet, why “high accuracy” and “profitable” are completely different animals once fees and an efficient counterparty enter the room.

That’s it. If you’re about to build one of these: do the break-even math on a napkin first. It’ll cost you five minutes instead of eight months.