What to Do When Your AI Coding Agent Can't Read a Web Page

I built fetch-anything to be stubborn. It is a skill my coding agent loads when it needs a web page and the built-in WebFetch tool comes back empty: when a page refuses to be read, do not give up, climb to a more aggressive tool and try again. Four rungs deep if that’s what it takes. No excuses.

It worked. It also spent months doing something I hadn’t designed it to do. I had built a test harness instead of a tool and learned I should be checking what my skills are really doing more often.

The Web Fetch Fallback Ladder I Built

The ordering rule was start with the tool that burns the fewest tokens and takes the least time, and only climb when a rung fails, which the skill decides by pattern matching the response for the usual tells.

Four rungs, in this order:

1. defuddle strips a page down to its readable article and hands back markdown. No browser, nearly free, and it works on more of the web than you would guess.
2. crawl4ai drives a real stealth browser, so it renders JavaScript and survives the bot checks that flatten the first rung.
3. playwright-cli, a full scriptable browser, for when a page needs something clicked or waited on before the content exists.
4. browser-act, the heavy artillery: anti-detection browser, captcha solving, proxies, persistent sessions, and a hand-the-user-the-wheel mode for the pages that beat everything else.

Underneath these sat a small set of per-domain handlers, hardcoded knowledge for sites where the standard approach is wrong. Reddit is the obvious one, where you fetch old.reddit.com instead of even fighting the new site and move on.

When WebFetch Gives Up

When I am scraping on purpose, I have patience for a page that fights back. Fighting back is the job I signed up for when I chose the work. I am already thinking about headers and rate limits and rendering, and a hard page is just the lay of the land.

The case this skill exists for is different. I need information from one page. The page decides I am a robot and shows me nothing. And now I am not doing my work anymore, I am doing scraping work, which I did not sign up for.

Before the skill, my move was the Obsidian Web Clipper. Open the page in a browser, clip it, drop the file into the project, tell the agent to read the file. It works every single time. It also is an easy way to break your flow.

The other thing is I had accumulated a pile of half-working tools, each one covering some slice of the problem, none of them the answer. I wanted one skill to use. If that one place could not get the page, fine, I would go back to clipping it by hand. But I should not have to remember which of four things to try.

Get the content off the page. Hand it back as markdown, main content only. Not HTML, because a page of raw markup landing in a context window is how a cheap task turns into an expensive one, and the whole point was to spend less attention on this, not more.

I Went Looking for the Numbers

Writing this post, I decided I should prove it. My agent keeps a session log for every conversation, and the ladder runs as ordinary commands inside those logs. So the evidence was sitting on my own disk. I wrote a scanner, pointed it at 187 sessions, and waited for a nice table of success rates per rung.

In that same month, the full ladder ran 29 times against 416 plain fetches that never needed it. The ones that did need something extra stopped at crawl4ai.

browser-act, rung four, was invoked zero times. Not rarely. Zero. The only commands in the entire corpus that contain the string browser-act are the searches I ran while writing this post.

Go back and look at what that rung was called in the shipped source:

## Step 5 — Rung 5: browser-act (UNTESTED — last resort before debug-research)

I labeled it untested, in its own heading, and shipped it anyway. Then it sat at the bottom of the ladder for two months and never ran. It did not need a success rate. There was nothing to compute.

The New Ladder: Defuddle, Crawl4AI, and Firecrawl

Four rungs became three. Here is what happened to each one:

“I deleted a rung” is the wrong summary: the stubbornness did not go anywhere. Rule one of the skill still says a fetch is never impossible, and “this page can’t be scraped” is still defined as a bug rather than a fact. Playwright and browser-act are still in the skill, in case I need to debug what happened.

They used to run by rote. The ladder reached them because the page was still failing, which is not a reason, it is just an ordinal. Now they sit in a separate debug step that only starts after the ladder is exhausted, and that step diagnoses the page first and picks a tool to match. Same tools, same refusal to give up. The difference is that guessing got confined to the place where guessing is the actual job.

Playwright earned its demotion on the same criterion. It is a good tool for building and testing sites, and I still use it for exactly that, but for getting text off a page it is clunky, and clunky is the specific thing this skill exists to eliminate. It also has two faces. Driven through its MCP server, a browser window pops up in my face, and I am watching a machine use a computer instead of doing my own work. Maybe there is a setting for that. I am tired of dicking with settings. Driven through the CLI it runs in the background, gets what it came for, and never asks for my attention. A tool that interrupts me has to earn the interruption, and a fetch I only needed because I was busy with something else can never earn it.

The ladder got shorter because the answer was known. And Firecrawl may not get hit for days or months, but it will be there waiting as the final rung and is almost guaranteed to work, or so they say.

Firecrawl at the Bottom

Running firecrawl config printed Status: ✓ Authenticated and then, on the very next line, API Key: Not set. Both statements were rendered with total confidence. If you hit the same thing, the key lives in FIRECRAWL_API_KEY, and if you are on fish, the command you want is set -Ux FIRECRAWL_API_KEY fc-..., with the -U, because the session-only version evaporates the moment you close the terminal and you will be back here in ten minutes wondering why.

That was the hard part of setup. The new rung 3 is Firecrawl, which is a hosted scraping API that handles the rendering, the rotation, and the blocking on somebody else’s machine and hands back markdown. It costs money per call, which is precisely why it belongs at the bottom.

So it is worth being specific about the actual exposure. The free tier is a thousand pages a month, and the interesting thing about that number is not its size, it is what it tells you about how the service expects to be used. A thousand is nothing if you are crawling. But this skill never crawls anything. It fetches the one page I asked for, and only after two free rungs have already failed on that exact page, which is a small fraction of an already small number. To get near a thousand of those in a month I would have to be using the tool for something other than what I built it for.

So the honest description of rung 3 is not “the paid one.” It is the rung that would start costing money if my habits changed a lot, and until then is just the thing that makes the failures stop. Sign up, put the key in your environment, forget it exists.

I drive it through the CLI rather than its MCP server, and that is deliberate. The install one-liner and firecrawl init push Firecrawl’s own agent skills, and optionally its MCP server, into every editor they can detect on your machine. I did not want a vendor writing into my tooling to solve a problem that npm install -g firecrawl-cli solves.

The other thing I kept, untouched, is the per-domain handlers. Those were never part of the experiment. A handler is not a guess about which tool might work, it is a thing I learned about a specific site and wrote down, like knowing that old.reddit.com still serves clean HTML while the modern site fights you. Reddit is the only one so far. There will be more, because some sources come up over and over, and I am happy to build a one-off for anything I am going to need a hundred times.

Your Fallback Chain Is an Experiment

None of this was planned. The rewrite happened just after midnight on August 1st, inside a session in another project, which has nothing to do with scraping. I needed a page, watched the thing grind, and finally noticed I had been watching it grind the same way for months. Forty minutes later it was three rungs and I went back to what I had been doing.

The generalizable part is not about scraping at all. Any time you write “try A, and if that fails try B, and if that fails try C,” you have built two things whether you meant to or not. You built a fallback chain, and you built a running experiment that records which option actually resolves things. The chain is the part you designed. The experiment is the part that accumulates an answer while you get on with your life.

I did not go back to see what was happening for months, and when I finally did it was by accident, and even then I only bothered to gather the evidence properly because I was writing this post.

But once the experiment has returned a result, continuing to run it on every call stupid. You already paid for that certainty. Re-buying it on each use costs time and tokens and, in my case, the exact attention the tool was supposed to protect.

So go read your own logs. Not for the tool that wins, that part you probably already know in your gut. Read them for the option at the bottom that has never once been the answer. Mine had a rung that never ran a single time in two months, and I had labeled it UNTESTED myself, in its own heading, where I would see it every time I opened the file.

I built the thing to be stubborn. It turned out stubbornness is a fine quality in a debugger and a bad one in a plan.