HurrbeeBot

HurrbeeBot is the web crawler for Hurrbee, an independent search engine. If you found it in your server logs, this page tells you what it was doing and how to stop it.

HurrbeeBot/1.0 (+https://hurrbee.com/bot)

What it's for

Hurrbee is building its own index of the web rather than relying entirely on someone else's. HurrbeeBot fetches pages so we can understand what they're about and rank them for our users.

We don't crawl the whole web, and we're not trying to. HurrbeeBot visits three kinds of page: ones our users chose to open from search results, ones listed in the Curlie human-edited directory, and ones site owners submit to us directly. If nobody has shown any interest in a page, we leave it alone.

Our crawl commitment

These are promises, not aspirations. If HurrbeeBot breaks one, tell us and we'll fix it.

At most one request every 5 seconds to any single host — and usually far less. Where your robots.txt sets a longer Crawl-delay, we use yours.
One connection at a time. We never open parallel connections to the same host.
A few hundred pages per host per day at most, and for the overwhelming majority of sites, a handful or fewer.
Conditional requests. We send ETag and If-Modified-Since on every re-fetch, so a page that hasn't changed costs you almost nothing.
We back off. A 429 or 503 stops us crawling that host for at least 24 hours.
We store text, not archives. Title, description, and extracted text — enough to index a page, not a copy of your site.

How to block it

Add this to your robots.txt:

User-agent: HurrbeeBot
Disallow: /

We re-read robots.txt every 24 hours, so a block takes effect within a day. You can also block individual paths the usual way — we follow the longest matching rule, per RFC 9309.

If your server returns a 5xx for robots.txt, we treat that as "don't crawl" rather than "no rules", on the basis that a struggling server is the last thing that needs a crawler.

If something's wrong

A real person reads bot@hurrbee.com. If HurrbeeBot is hitting you too hard, ignoring your rules, or you'd like your site removed from our index, mail us and we'll sort it out. You don't need to wait for a robots.txt cycle — we can block a host immediately.

Submitting a site

If you'd rather we did crawl you, submit your site and we'll add it to the queue. Site owners who verify ownership get priority and can see their domain's Trust Authority — the score our own ranking uses.

With content from Curlie.org — the largest human-edited directory of the web. Contribute by submitting a website or becoming an editor.