anubis only stops scrapers that scrap a very large number of the web UI webpages. it does not stop scrapers that act as a fediverse instance and get the data access directly through the API.

the reason why we use it is because scraping the web UI is expensive and slows down the service for everyone else. meanwhile the API access is efficient and therefore not a problem to us.

It’s not designed to stop anybody. It’s goal is to slow down automated scrapers.

For a scraper, solving the Anubis challenge is a one-time, amortized-to-zero cost since the cookie can be cached and reused.

Wait there no request limit per token? The fuck?

Collectively we are wasting an impressive amount of time waiting for access to websites; time we didn’t spend before the AI era. As a human, time is precious and finite to me, whereas to a robot it is not.

If time not matter for robot why all the llm company scrape at site break speed?

The irony is that Anubis’s goal is to stop AI but it was incredibly easy for AI to circumvent it and yet the cost to humans and an open web remains.

Either he not understand POW or never read. Goal is not to stop but to make cost enough to make not worth.

This is just sad. Instead of understanding WHY a site would employ Anubis they just break it without a care or thought.

It’s a cat and mouse game.
The more popular something is, the more likely tools will be made to circumvent it.

In this case the author is advertising their tool as a method to circumvent Anubis:

https://github.com/fzakaria/anubis-fetch