ENTRY
[ESC]Top line: neocities gives you access to make your own website and hosts it. They also provide a robust robots.txt to block AI crawlers.
Bottom line: AI crawlers don’t give a fuck about robots.txt, I’ve found out that Anubis exists as some sort of against-crawlers measure.
If I understand correctly, Anubis is a (?) reverse-proxy..??? And I need an apache/nginx server to use it?
Question is that whether it’s overcomplicating a problem and whether it’s possible to connect neocities website to this service
6 replies
Log in to read the replies and join the conversation