Readit Newsosdotsystem commented on Amazon's AI crawler is making my Git server unstable xeiaso.net/notes/2025/ama... · Posted by u/subset
I've heard that too. Your posting inspired me to check. I see daily accesses from claudebot and gtpbot to /robots.txt on two sites. Also from something called amazonbot, which may be another AI bot?
Me i dropped a bunch of dont allows for different bots, as many as i could find.