Crawler
About bankerbuyerlist-bot
bankerbuyerlist-bot is the web crawler behind BankerBuyerList. It reads the public websites of private equity firms, their portfolio companies and other acquirers so bankers and business brokers can find out who buys businesses like the ones they sell. This page explains how it identifies itself, what it respects and how to keep it off your site.
How it identifies itself
Every request carries this user agent string:
bankerbuyerlist-bot/1.0 (+https://bankerbuyerlist.com/bot)The URL in the string points back to this page so an operator reading server logs can find out who is visiting and how to reach us.
What it respects
- robots.txt. It fetches robots.txt before any other page on a domain and obeys Disallow rules for bankerbuyerlist-bot and for
*. A page that robots.txt forbids is skipped and recorded as skipped, not fetched. - Crawl-delay. When robots.txt sets a Crawl-delay it is honoured. Otherwise the crawler waits at least 2 seconds between requests to the same domain.
- Public pages only. It never signs in, submits forms, or fetches anything behind authentication.
- Failures back off. On errors or network trouble it retries with increasing delays rather than hammering a site.
- No LinkedIn. It does not crawl LinkedIn or any social network.
How often it visits
Pages are revisited on a schedule by page type rather than continuously. The defaults are:
- Firm websites
- about every 30 days
- Portfolio company websites
- about every 60 days
- News and transaction pages
- about every 7 days
A typical site therefore sees a handful of requests per visit, a few times a month at most.
How to opt out
Add this to your robots.txt and the crawler will stop visiting:
User-agent: bankerbuyerlist-bot
Disallow: /Or email jack@bizvalwriters.com with the domain and we will exclude it from future crawls.
No tracking
The crawler sets no cookies, runs no analytics and sends no tracking identifiers. It reads what your site publishes and nothing else. Where a page only renders with JavaScript it may use a headless browser to load it, with the same user agent and the same limits.
Questions about the data itself, including corrections, are answered on the FAQ.