ThinwedgeBot
ThinwedgeBot/1.0 (+https://thinwedge.com/bot)
ThinwedgeBot collects public marketplace listings and public pricing pages so Thinwedge can find incumbents whose pricing leaves an opening. If it has turned up in your logs, this page is what it was doing and how to make it stop.
What it reads
- Public marketplace listing pages, including their review text and ratings.
- Public pricing pages, to record tier names, prices and usage limits over time.
What it will not do
- It sends no cookies, no authorisation headers and no other credentials, so it cannot reach anything behind a login. Requests carrying credential headers are rejected before they are sent, not filtered afterwards.
- It treats 403 and 429 as final answers. There is no retry and no back-off loop; the source is halted.
- It stops reading a response after 5 MB.
- It does not submit forms, and it does not execute your JavaScript.
How to block it
ThinwedgeBot reads robots.txt before it reads anything else and honours both the rules that match its name and Crawl-delay. Add this and it will stop within a day, which is how long a cached robots.txt is held before it is fetched again:
User-agent: ThinwedgeBot Disallow: /
To have anything already collected from your site removed, write to bot@thinwedge.com from an address at that domain.