What to Do When Your AI Coding Agent Can't Read a Web Page

I built fetch-anything to be stubborn. It’s a skill my coding agent loads when it needs a web page and the built-in WebFetch tool comes back empty: when a page refuses to be read, don’t give up, climb to a more aggressive tool and try again. Four rungs deep if that’s what it takes. No excuses.

It worked. It also spent months doing something I hadn’t designed it to do. I had built a test harness instead of a tool and learned I should be checking what my skills are really doing more often.

The Web Fetch Fallback Ladder I Built

The ordering rule was start with the tool that burns the fewest tokens and takes the least time, and only climb when a rung fails, which the skill decides by pattern matching the response for the usual tells.

Four rungs, in this order:

1. defuddle strips a page down to its readable article and hands back markdown. No browser, nearly free, and it works on more of the web than you would guess.
2. crawl4ai drives a real stealth browser, so it renders JavaScript and survives the bot checks that flatten the first rung.
3. playwright-cli, a full scriptable browser, for when a page needs something clicked or waited on before the content exists.
4. browser-act, the heavy artillery: anti-detection browser, captcha solving, proxies, persistent sessions, and a hand-the-user-the-wheel mode for the pages that beat everything else.

Underneath these sat a small set of per-domain handlers, hardcoded knowledge for sites where the standard approach is wrong. Reddit is the obvious one, where you fetch old.reddit.com instead of even fighting the new site and move on.

When WebFetch Gives Up

When I’m scraping on purpose, I’ve patience for a page that fights back. Fighting back is the job I signed up for when I chose the work. I’m already thinking about headers and rate limits and rendering, and a hard page is just the lay of the land.

The case this skill exists for is different. I need information from one page. The page decides I’m a robot and shows me nothing. And now I’m not doing my work anymore, I’m doing scraping work, which I didn’t sign up for.

Before the skill, my move was the Obsidian Web Clipper. Open the page in a browser, clip it, drop the file into the project, tell the agent to read the file. It works every single time. It’s also an easy way to break your flow.

The other thing is I had accumulated a pile of half-working tools, each one covering some slice of the problem, none of them the answer. I wanted one skill to use. If that one place couldn’t get the page, fine, I’d go back to clipping it by hand. But I should not have to remember which of four things to try.

Get the content off the page. Hand it back as markdown, main content only. Not HTML, because a page of raw markup landing in a context window is how a cheap task turns into an expensive one, and the whole point was to spend less attention on this, not more.

I Went Looking for the Numbers

Writing this post, I decided I should prove it. My agent keeps a session log for every conversation, and the ladder runs as ordinary commands inside those logs. So the evidence was sitting on my own disk. I wrote a scanner, pointed it at 187 sessions, and waited for a nice table of success rates per rung.

In that same month, the full ladder ran 29 times against 416 plain fetches that never needed it. The ones that did need something extra stopped at crawl4ai.

browser-act, rung four, was invoked zero times. Not rarely. Zero. The only commands in the entire corpus that contain the string browser-act are the searches I ran while writing this post.

Go back and look at what that rung was called in the shipped source:

## Step 5 — Rung 5: browser-act (UNTESTED — last resort before debug-research)

I labeled it untested, in its own heading, and shipped it anyway. Then it sat at the bottom of the ladder for two months and never ran. It didn’t need a success rate. There was nothing to compute.

The New Ladder: Defuddle, Crawl4AI, and Firecrawl

Four rungs became three. Here’s what happened to each one:

“I deleted a rung” is the wrong summary: the stubbornness didn’t go anywhere. Rule one of the skill still says a fetch is never impossible, and “this page can’t be scraped” is still defined as a bug rather than a fact. Playwright and browser-act are still in the skill, in case I need to debug what happened.

They used to run by rote. The ladder reached them because the page was still failing, which isn’t a reason, it’s just an ordinal. Now they sit in a separate debug step that only starts after the ladder is exhausted, and that step diagnoses the page first and picks a tool to match. Same tools, same refusal to give up. The difference is that guessing got confined to the place where guessing is the actual job.

Playwright earned its demotion on the same criterion. It’s a good tool for building and testing sites, and I still use it for exactly that, but for getting text off a page it’s clunky, and clunky is the specific thing this skill exists to eliminate. It also has two faces. Driven through its MCP server, a browser window pops up in my face, and I’m watching a machine use a computer instead of doing my own work. Maybe there’s a setting for that. I’m tired of dicking with settings. Driven through the CLI it runs in the background, gets what it came for, and never asks for my attention. A tool that interrupts me has to earn the interruption, and a fetch I only needed because I was busy with something else can never earn it.

The ladder got shorter because the answer was known. And Firecrawl may not get hit for days or months, but it will be there waiting as the final rung and is almost guaranteed to work, or so they say.

Firecrawl at the Bottom

Running firecrawl config printed Status: ✓ Authenticated and then, on the very next line, API Key: Not set. Both statements were rendered with total confidence. If you hit the same thing, the key lives in FIRECRAWL_API_KEY, and if you’re on fish, the command you want is set -Ux FIRECRAWL_API_KEY fc-..., with the -U, because the session-only version evaporates the moment you close the terminal and you’ll be back here in ten minutes wondering why.

That was the hard part of setup. The new rung 3 is Firecrawl, which is a hosted scraping API that handles the rendering, the rotation, and the blocking on somebody else’s machine and hands back markdown. It costs money per call, which is precisely why it belongs at the bottom.

So it’s worth being specific about the actual exposure. The free tier is a thousand pages a month, and what matters about that number is less its size than what it tells you about how the service expects to be used. A thousand is nothing if you’re crawling. But this skill never crawls anything. It fetches the one page I asked for, and only after two free rungs have already failed on that exact page, which is a small fraction of an already small number. To get near a thousand of those in a month I’d have to be using the tool for something other than what I built it for.

So rung 3 isn’t really “the paid one.” It’s the rung that would start costing money if my habits changed a lot, and until then is just the thing that makes the failures stop. Sign up, put the key in your environment, forget it exists.

I drive it through the CLI rather than its MCP server, and that’s deliberate. The install one-liner and firecrawl init push Firecrawl’s own agent skills, and optionally its MCP server, into every editor they can detect on your machine. I didn’t want a vendor writing into my tooling to solve a problem that npm install -g firecrawl-cli solves.

The other thing I kept, untouched, is the per-domain handlers. Those were never part of the experiment. A handler isn’t a guess about which tool might work, it’s a thing I learned about a specific site and wrote down, like knowing that old.reddit.com still serves clean HTML while the modern site fights you. Reddit is the only one so far. There will be more, because some sources come up over and over, and I’m happy to build a one-off for anything I’m going to need a hundred times.

Your Fallback Chain Is an Experiment

None of this was planned. The rewrite happened just after midnight on August 1st, inside a session in another project, which has nothing to do with scraping. I needed a page, watched the thing grind, and finally noticed I’d been watching it grind the same way for months. Forty minutes later it was three rungs and I went back to what I’d been doing.

The generalizable part isn’t about scraping at all. Any time you write “try A, and if that fails try B, and if that fails try C,” you have built two things whether you meant to or not. You built a fallback chain, and you built a running experiment that records which option actually resolves things. The chain is the part you designed. The experiment is the part that accumulates an answer while you get on with your life.

I didn’t go back to see what was happening for months, and when I finally did it was by accident, and even then I only bothered to gather the evidence properly because I was writing this post.

But once the experiment has returned a result, continuing to run it on every call is stupid. You already paid for that certainty. Re-buying it on each use costs time and tokens and, in my case, the exact attention the tool was supposed to protect.

So go read your own logs. Not for the tool that wins, that part you probably already know in your gut. Read them for the option at the bottom that has never once been the answer. Mine had a rung that never ran a single time in two months, and I had labeled it UNTESTED myself, in its own heading, where I’d see it every time I opened the file.

I built the thing to be stubborn. It turned out stubbornness is a fine quality in a debugger and a bad one in a plan.