PhilosophicalProgressBot

PhilosophicalProgressBot is the crawler behind Philosophical Progress, a free, non-commercial index of new philosophy papers as they appear on journal sites, preprint archives, blogs and philosophers' own homepages. For each paper, the page shows title, authors, abstract, and topics, and links to where it was found. Full texts are never shown, although shorter ones are stored, if available, for topic classification and deduplication.

Philosophical Progress is a personal project developed and maintained by Wolfgang Schwarz.

What PhilosophicalProgressBot fetches

The crawler monitors a fixed list of pages, which it fetches at most twice a day, following new links to papers they might point to. The crawler also visits pages returned by web searches when it looks for a philosopher's homepage.

Pages are fetched with a headless Chrome browser, so that scripts and stylesheets load as in a browser.

How PhilosophicalProgressBot behaves

• It obeys^* robots.txt, following the rules for PhilosophicalProgressBot or, if there are none, those for *. It keeps a copy of a site's robots.txt for up to a day.
• It honours Crawl-delay, and otherwise waits at least three seconds between page requests to the same host.

^* There's one technical exception. Personal homepages sometimes link to papers on drive.google.com. Here, robots.txt allows fetching the web viewer, which embeds the paper, but it does not directly whitelist the paper's URL as embedded in the viewer. The crawler nonetheless fetches the file directly rather than loading the viewer and downloading the paper through its JS interface (which is possible but cumbersome).

How to recognise PhilosophicalProgressBot

Page and paper requests carry this user agent:

PhilosophicalProgressBot/1.0 (+https://philosophicalprogress.org/bot)

RSS and Atom feeds are fetched with this one:

philosophicalprogress/1.0 (+https://www.philosophicalprogress.org/bot) urllib/3.12

All requests come from the addresses listed in /bot/ips.json.

How to opt out

To keep the crawler off your site, add this to your robots.txt:

User-agent: PhilosophicalProgressBot
Disallow: /

It takes effect within a day.

For any other requests, questions, or concerns, please write to philosophicalprogress@umsu.de.