Responsible crawling

Vunk maintains a verified catalog of the iGaming industry. To keep company facts fresh, a small crawler periodically checks companies’ own public pages and official registries. This page is our public commitment to how it behaves — and how to stop it.

Companies, not people: The crawler collects and validates facts about companies from their own public pages and official registries. It does not scrape personal contacts from social media or build a people database.
Identifiable: Every request is sent with a stable, identifiable User-Agent that links back to this page, so any site owner can recognise and contact us.
robots.txt is law: Before fetching any page we read the site’s robots.txt and obey it. A disallowed path is never fetched; if robots.txt is temporarily unavailable, we defer rather than guess.
Light footprint: We crawl a small, curated set of pages at a low rate, cache robots.txt per host, cap response sizes, and honour Crawl-delay. We are never a load problem.
Provenance & freshness: Anything we record carries its source, a confidence score, and a last-verified timestamp. A changed page becomes an editorial signal for review — it is never published as fact without grounding.
Easy to remove: If you do not want your site crawled, disallow our User-Agent in robots.txt or email us. We honour removal requests for sources we control.

Identifying & controlling the crawler

Our User-Agent

Requests are sent with this exact User-Agent. You can match it in your logs or in robots.txt:

VunkCrawler/1.0 (+https://bevunk.com)

Block or limit us in robots.txt

We obey the Robots Exclusion Protocol, including Allow, Disallow, * wildcards, a trailing $ anchor, and Crawl-delay. To keep us off your whole site:

User-agent: vunkcrawler
Disallow: /

# …or just slow us down / fence off a section:
User-agent: vunkcrawler
Crawl-delay: 30
Disallow: /private

Load budget

HTTPS only; redirects are not followed and private/internal address ranges are blocked.
Responses are capped (128 KB by default) and time out quickly (5 s).
robots.txt is fetched once per host and cached, not re-fetched per page.
We crawl a small curated set at a low, bounded concurrency and request rate.

Source-removal & complaints

To request removal of a source we control, report a problem, or ask a question about the crawler, contact [email protected]. This is a role mailbox, not a personal contact.