What it reads
It reads websites in Belgium, a group at a time, each for one of these reasons:
- a place lists it on OpenStreetMap: its homepage, and up to four of its pages that look like opening hours, prices or contact;
- it's a town's own site, its tourism office's or its centre traders': up to 60 pages, chosen from what its homepage links to and what its sitemap lists;
- one of those pages links to it: up to 8 pages;
- many of the sites it already reads link to it: up to 30 pages.
It never reads social networks, review or booking sites, marketplaces or news sites, and no "about us" pages. It keeps the main text of the pages, without menus, headers, footers or email addresses, each part with the address of the page it came from.
On every page it reads, it notes the links and what they say, and which site they lead to, but not what's behind them. That is how it finds the sites a town's own pages point to. It notes no links marked nofollow, and none of a page that asks not to be indexed.
What it keeps
Only what a site may be used for. A place's site that is no longer the place's (a domain for sale, someone else's pages at the place's old address, a login page) or that is mostly about gambling: nothing of it is kept, only that it was found so, and when. Any other site is kept only if it isn't for sale, a login page, an adult site or mostly about gambling. Nothing about gambling is used from any site.
It keeps a copy of each page it reads, without email addresses, so it can read the page again without asking your site again, and so everything Inka says can be traced to where it came from. It keeps no copy of a page that asks not to be indexed or archived (noindex, noarchive), or that is reserved against text and data mining. The copies stay on its own server: the latest of each page, and older ones only while an index made from them is in use, never more than 90 days after it last saw the page.
How it's used
What it keeps goes into Baken, the index Inka searches. From a place's pages, BakenBot also reads its opening hours, and notes when the website listed for a place is no longer the place's own. Those two facts, with the address of the page and the date it was read, may also be used by Mapr, a map service, in its place details. Mapr is given nothing from any other site.
None of it is used to train models, sold, or published as a dataset.
How
It reads your robots.txt first and follows it. If robots.txt can't be read, it leaves the site alone. It respects noindex and noarchive, in a page's robots meta tag or an X-Robots-Tag header, and text-and-data-mining reservations (TDMRep), which it treats like a refusal. It reads many sites at once, but asks each for one page at a time, at least 10 seconds apart, or longer if your robots.txt sets a Crawl-delay; robots.txt and tdmrep.json count as pages too. If your site answers 429 or 503, it leaves it alone for as long as Retry-After says, or an hour. On later visits it asks only for what changed, when your site supports that (ETag or Last-Modified).
It identifies itself as BakenBot/0.1 (+https://baken.dev/bakenbot), and reads from 213.239.220.146 and 2a01:4f8:a0:70c4::2, whose reverse DNS is bakenbot.baken.dev.
Opting out
Add this to your robots.txt:
User-agent: BakenBot Disallow: /
If your robots.txt has no group for BakenBot, a group for InkaIndex, its name before, counts instead.
On its next visit, BakenBot stops reading your site and deletes what it kept of it: the text, the copies of its pages, and the links it noted on them. It leaves Baken's index, and what Mapr is given, the next time they are rebuilt. To keep only some pages out, disallow those paths instead. A text-and-data-mining reservation (TDMRep) keeps it out too.
To have something removed, write to bakenbot@baken.dev: it's removed within 7 days, from the index too.
BakenBot is run by SeeMe AI BV, Leschot 28, 2440 Geel, Belgium.