HurrbeeBot

HurrbeeBot is the web crawler for Hurrbee, an independent search engine. If you found it in your server logs, this page tells you what it was doing and how to stop it.

HurrbeeBot/1.0 (+https://hurrbee.com/bot)

What it's for

Hurrbee is building its own index of the web rather than relying entirely on someone else's. HurrbeeBot fetches pages so we can understand what they're about and rank them for our users.

We don't crawl the whole web, and we're not trying to. HurrbeeBot visits pages people have shown interest in: results our users chose to open, pages they asked HurrbeeBot to read, and links a few steps out from those while someone is researching a topic. It also reads sites listed in the Curlie human-edited directory, the sitemaps of sites we already know, and sites their owners submit to us directly. If nobody has shown any interest in a page, we leave it alone.

Our crawl commitment

These are promises, not aspirations. If HurrbeeBot breaks one, tell us and we'll fix it.

At most one request every 5 seconds to any single host — and usually far less. Where your robots.txt sets a longer Crawl-delay, we use yours.
One connection at a time. We never open parallel connections to the same host.
A few hundred pages per host per day at most, and for the overwhelming majority of sites, a handful or fewer.
Conditional requests. We send ETag and If-Modified-Since on every re-fetch, so a page that hasn't changed costs you almost nothing.
We back off. A 429 or 503 stops us crawling that host for at least 24 hours.
We store text, not archives. Title, description, and extracted text — enough to index a page, not a copy of your site.
We index pages only. Files — PDFs, images, video, archives, data downloads — and feeds are noted once and left alone.
We read at most 2 MB of any page. If a page runs longer, we index the start of it and stop.
We follow your canonical tag within your own site. If it points to another site, we keep the page at the address we fetched.

How to block it

Add this to your robots.txt:

User-agent: HurrbeeBot
Disallow: /

We re-read robots.txt every 24 hours, so a block takes effect within a day. Pages that ask crawlers not to read them rank slightly lower, because Hurrbee can't check what they say. You can also block individual paths the usual way — we follow the longest matching rule, per RFC 9309.

If your server returns a 5xx for robots.txt, we treat that as "don't crawl" rather than "no rules", on the basis that a struggling server is the last thing that needs a crawler.

If something's wrong

A real person reads bot@hurrbee.com. If HurrbeeBot is hitting you too hard, ignoring your rules, or you'd like your site removed from our index, mail us and we'll sort it out. You don't need to wait for a robots.txt cycle — we can block a host immediately.

Submitting a site

If you'd rather we did crawl you, submit your site and we'll add it to the queue. Site owners who verify ownership get priority and can see their domain's Trust Authority — the score our own ranking uses.

With content from Curlie.org — the largest human-edited directory of the web. Contribute by submitting a website or becoming an editor.