Today MJ12bot made 2570 request to my blog but it got trapped by garbage so 2487 answers involved random word salad from Moby Dick."Majestic is a UK based specialist search engine used by hundreds of thousands of businesses in 13 languages and over 60 countries to paint a map of the Internet independent of the consumer based search engines. Majestic also powers other legitimate technologies that help to understand the continually changing fabric of the web." No thanks."MJ12bot adheres to the robots.txt standard. If you want the bot to prevent website from being crawled then add the following text to your robots.txt:" User-agent: MJ12bot
Disallow: /
I guess my robots.txt doesn't match that exactly:# curl https://alexschroeder.ch/robots.txt
User-agent: Wibybot
User-agent: Xobaque
User-agent: search.marginalia.nu
Allow: /view/
User-agent: *
Disallow: /
DisallowAITraining: /
I still feel that this should block most bots. #ButlerianJihad
