[NTLK] ProjectNewtonberg

[Grant Hutchinson]

> On Feb 26, 2026, at 7:05 PM, J Caffiney <caffiney at gmail.com> wrote:
> 
> Excellent. Thank you. It's a shame the archive didn't save the books themselves.

Agreed. Back in the day, a lot of sites would block indexing of download and image directories using the robots.txt file in order to mitigate huge bandwidth usage caused by search engine spiders and scrapers. That’s what we used to do on our visual content-heavy sites in the 90s. The Internet Archive has always obeyed those robot.txt directives ... for good or bad, I suppose.

g.