Can be described as Astroneer-like setting, Teardown voxel physics, in a Valheim-like online multiplayer survival game.
Game isn't really announced yet but I've shown some videos of the tech: https://x.com/Alientrap/status/1909316208563732866 (On Youtube: https://www.youtube.com/watch?v=ZWISaUmvit4 ) https://x.com/Alientrap/status/1918024969939808654
Their fetcher (not crawler) has user agent Perplexity-User. Since the fetching is user-requested, it ignores robots.txt . In the article, it discusses how blocking the “Perplexity-User” user agent doesn’t actually work, and how perplexity uses an anonymous user agent to avoid being blocked.