Das Thema AI-Crawler, Bots und Scraper macht auch vor dem Fediverse nicht halt.
-
Das Thema AI-Crawler, Bots und Scraper macht auch vor dem Fediverse nicht halt. Nicht nur, dass hier unerwünscht Daten von Usern "abgesaugt" werden, diese Abfragen erzeugen auch immer mehr Last auf den Servern - viel unnötiger Traffic, der da entsteht und Ressourcen kostet.
Vieles lässt sich durch die robots.txt aussperren, aber leider hält sich nicht jeder Bot an diese. Natürlich ist auch von hartnäckigen Bots und Spidern ein Ausschluss in der .htaccess oder Apache-Config möglich. Auch mit Fail2Ban lässt sich da eine Menge bauen.
Allerdings ist es auch empfehlenswerter, diese IPs gleich kernelnah, also z.B. über nftables, zu sperren. Leider sind es nicht nur ein paar IPs, leider kann man nicht alle IPs dieser Bots sperren, weil auch immer wieder neue IPs hinzukommen. Es gibt aber bestimmte IPs und Adressbereiche, die von diesen Bots genutzt werden. Die Infos habe ich bei den "Anbietern" selbst gefunden, mit Ausnahme von SEMrush, der wirklich sehr aggressiv crawlen kann und gerne mal die robots.txt völlig ignoriert:
openai.com/gptbot.json
openai.com/chatgpt-user.json
openai.com/searchbot.json
docs.claude.com/en/api/ip-addr…
developers.google.com/static/s…
bing.com/toolbox/bingbot.json
search.developer.apple.com/app…
perplexity.com/perplexitybot.j…
perplexity.com/perplexity-user…
duckduckgo.com/duckduckgo-help…
index.commoncrawl.org/ccbot.js…
ipinfo.io/AS209366Ja, das sind natürlich nicht alle Bots, Crawler und Co! Das ist mir bewusst. Aber ein großer Teil Traffic verschwindet damit schon!
Da ich auf meinen Friendica-Instanzen aber auch keinen Besuch von Google und Co. haben möchte, habe ich heute mal einen Rundumschlag gemacht und diese IPs von oben genannten Seiten in nftables reingeworfen. Ergebnis: Es ist bedeutend ruhiger geworden! Und dabei sind es bei weitem natürlich nicht alle Bots und Crawler, aber doch schon einige!
Ich weiß nicht, ob jemand Interesse an den IPs hat, ich stelle diese hier einfach mal zur Verfügung. Quelle sind wie bereits erwähnt oben stehende Seiten. Nutzung natürlich auf eigene Gefahr.
IPv4-Adressen:
132.196.86.0/24,172.182.202.0/25,172.182.204.0/24,172.182.207.0/25,172.182.214.0/24,172.182.215.0/24,20.125.66.80/28,20.171.206.0/24,20.171.207.0/24,4.227.36.0/25,52.230.152.0/24,74.7.175.128/25,74.7.227.0/25,74.7.227.128/25,74.7.228.0/25,74.7.230.0/25,74.7.241.0/25,74.7.241.128/25,74.7.242.0/25,74.7.243.128/25,74.7.244.0/25,104.210.139.192/28,104.210.139.224/28,13.65.138.112/28,13.65.138.96/28,13.67.46.240/28,13.67.72.16/28,13.70.107.160/28,13.71.2.208/28,13.76.115.224/28,13.76.115.240/28,13.76.116.80/28,13.76.223.48/28,13.76.32.208/28,13.79.43.0/28,13.83.167.128/28,13.83.237.176/28,132.196.82.48/28,135.119.134.128/28,135.119.134.192/28,135.220.73.208/28,135.220.73.240/28,135.237.131.208/28,135.237.133.48/28,137.135.183.96/28,137.135.190.240/28,137.135.191.176/28,137.135.191.32/28,138.91.30.48/28,138.91.46.96/28,168.63.252.240/28,172.178.140.144/28,172.178.141.112/28,172.178.141.128/28,172.183.143.224/28,172.183.222.128/28,172.196.40.208/28,172.202.102.112/28,172.204.16.64/28,172.204.27.16/28,172.212.159.64/28,172.213.11.144/28,172.213.12.112/28,172.213.21.112/28,172.213.21.144/28,172.213.21.16/28,172.215.218.96/28,191.233.1.112/28,191.233.1.128/28,191.233.1.224/28,191.233.194.32/28,191.233.196.112/28,191.233.199.160/28,191.233.2.0/28,191.234.167.128/28,191.235.66.16/28,191.235.98.144/28,191.235.99.80/28,191.237.249.64/28,191.239.245.16/28,20.0.53.96/28,20.102.212.144/28,20.117.22.224/28,20.125.112.224/28,20.125.144.144/28,20.161.75.208/28,20.168.7.192/28,20.168.7.240/28,20.169.72.112/28,20.169.72.96/28,20.169.73.176/28,20.169.73.32/28,20.169.73.64/28,20.169.78.112/28,20.169.78.128/28,20.169.78.144/28,20.169.78.160/28,20.169.78.176/28,20.169.78.192/28,20.169.78.208/28,20.169.78.48/28,20.169.78.64/28,20.169.78.80/28,20.169.78.96/28,20.169.86.224/28,20.169.86.240/28,20.169.87.112/28,20.172.29.32/28,20.193.233.240/28,20.193.50.32/28,20.194.0.208/28,20.194.1.0/28,20.194.157.176/28,20.198.67.96/28,20.203.245.32/28,20.204.24.240/28,20.206.107.192/28,20.210.154.128/28,20.210.174.208/28,20.210.211.192/28,20.215.187.208/28,20.215.188.192/28,20.215.214.16/28,20.215.219.128/28,20.215.219.160/28,20.215.219.208/28,20.215.220.112/28,20.215.220.128/28,20.215.220.144/28,20.215.220.160/28,20.215.220.176/28,20.215.220.192/28,20.215.220.208/28,20.215.220.64/28,20.215.220.80/28,20.215.220.96/28,20.226.32.80/28,20.227.140.32/28,20.228.106.176/28,20.235.75.208/28,20.235.87.224/28,20.249.63.208/28,20.27.94.128/28,20.45.178.144/28,20.55.229.144/28,20.63.221.64/28,20.90.7.144/28,20.97.189.96/28,23.102.140.144/28,23.102.141.32/28,23.97.109.224/28,23.98.142.176/28,23.98.179.16/28,23.98.186.176/28,23.98.186.192/28,23.98.186.64/28,23.98.186.96/28,4.151.119.48/28,4.151.241.240/28,4.151.71.176/28,4.189.118.208/28,4.189.119.48/28,4.196.118.112/28,4.196.198.80/28,4.197.115.112/28,4.197.19.176/28,4.197.22.112/28,4.197.64.0/28,4.197.64.16/28,4.197.64.48/28,4.197.64.64/28,4.205.128.176/28,40.116.73.208/28,40.122.235.112/28,40.67.183.160/28,40.67.183.176/28,40.75.14.224/28,40.81.134.128/28,40.81.134.144/28,40.81.234.144/28,40.84.181.32/28,40.84.221.208/28,40.84.221.224/28,51.107.70.192/28,51.8.155.48/28,51.8.155.64/28,51.8.155.80/28,52.148.129.32/28,52.154.22.48/28,52.156.77.144/28,52.159.227.32/28,52.159.249.96/28,52.165.212.16/28,52.165.212.32/28,52.165.212.48/28,52.172.129.160/28,52.172.251.112/28,52.173.123.0/28,52.173.219.112/28,52.173.219.96/28,52.173.221.16/28,52.173.221.176/28,52.173.221.208/28,52.173.234.16/28,52.173.234.80/28,52.173.235.80/28,52.176.139.176/28,52.187.246.128/28,52.190.137.144/28,52.190.137.16/28,52.190.139.48/28,52.190.142.64/28,52.190.190.16/28,52.225.75.208/28,52.230.163.32/28,52.230.164.176/28,52.231.30.48/28,52.231.34.176/28,52.231.39.144/28,52.231.39.192/28,52.231.49.48/28,52.231.50.64/28,52.236.94.144/28,52.242.132.224/28,52.242.132.240/28,52.242.245.208/28,52.252.113.240/28,52.255.109.112/28,52.255.109.128/28,52.255.109.144/28,52.255.109.80/28,52.255.109.96/28,52.255.111.0/28,52.255.111.112/28,52.255.111.16/28,52.255.111.32/28,52.255.111.48/28,52.255.111.80/28,57.151.131.224/28,57.154.174.112/28,57.154.175.0/28,57.154.187.32/28,68.154.28.96/28,68.218.30.112/28,68.220.57.64/28,68.221.67.160/28,68.221.67.192/28,68.221.67.224/28,68.221.67.240/28,68.221.75.16/28,74.226.253.160/28,74.249.86.176/28,74.7.35.112/28,74.7.35.48/28,74.7.36.64/28,74.7.36.80/28,74.7.36.96/28,9.160.163.224/28,9.160.164.128/28,104.210.140.128/28,135.234.64.0/24,172.182.193.224/28,172.182.193.80/28,172.182.194.144/28,172.182.194.32/28,172.182.195.48/28,172.182.209.208/28,172.182.211.192/28,172.182.213.192/28,172.182.224.0/28,172.203.190.128/28,20.14.99.96/28,20.168.18.32/28,20.169.6.224/28,20.169.7.48/28,20.169.77.0/25,20.171.123.64/28,20.171.53.224/28,20.25.151.224/28,20.42.10.176/28,4.227.36.0/25,40.67.175.0/25,40.90.214.16/28,51.8.102.0/24,74.7.175.128/25,74.7.228.0/25,74.7.228.128/25,74.7.229.0/25,74.7.229.128/25,74.7.230.0/25,74.7.241.128/25,74.7.242.128/25,74.7.243.0/25,74.7.244.0/25,160.79.104.0/21,192.178.4.0/27,192.178.4.128/27,192.178.4.160/27,192.178.4.192/27,192.178.4.224/27,192.178.4.32/27,192.178.4.64/27,192.178.4.96/27,192.178.5.0/27,192.178.6.0/27,192.178.6.128/27,192.178.6.160/27,192.178.6.192/27,192.178.6.224/27,192.178.6.32/27,192.178.6.64/27,192.178.6.96/27,192.178.7.0/27,192.178.7.128/27,192.178.7.160/27,192.178.7.192/27,192.178.7.224/27,192.178.7.32/27,192.178.7.64/27,192.178.7.96/27,34.100.182.96/28,34.101.50.144/28,34.118.254.0/28,34.118.66.0/28,34.126.178.96/28,34.146.150.144/28,34.147.110.144/28,34.151.74.144/28,34.152.50.64/28,34.154.114.144/28,34.155.98.32/28,34.165.18.176/28,34.175.160.64/28,34.176.130.16/28,34.22.85.0/27,34.64.82.64/28,34.65.242.112/28,34.80.50.80/28,34.88.194.0/28,34.89.10.80/28,34.89.198.80/28,34.96.162.48/28,35.247.243.240/28,66.249.64.0/27,66.249.64.128/27,66.249.64.160/27,66.249.64.192/27,66.249.64.224/27,66.249.64.32/27,66.249.64.64/27,66.249.64.96/27,66.249.65.0/27,66.249.65.128/27,66.249.65.160/27,66.249.65.192/27,66.249.65.224/27,66.249.65.32/27,66.249.65.64/27,66.249.65.96/27,66.249.66.0/27,66.249.66.128/27,66.249.66.160/27,66.249.66.192/27,66.249.66.224/27,66.249.66.32/27,66.249.66.64/27,66.249.66.96/27,66.249.67.0/27,66.249.67.32/27,66.249.67.64/27,66.249.68.0/27,66.249.68.128/27,66.249.68.160/27,66.249.68.192/27,66.249.68.32/27,66.249.68.64/27,66.249.68.96/27,66.249.69.0/27,66.249.69.128/27,66.249.69.160/27,66.249.69.192/27,66.249.69.224/27,66.249.69.32/27,66.249.69.64/27,66.249.69.96/27,66.249.70.0/27,66.249.70.128/27,66.249.70.160/27,66.249.70.192/27,66.249.70.224/27,66.249.70.32/27,66.249.70.64/27,66.249.70.96/27,66.249.71.0/27,66.249.71.128/27,66.249.71.160/27,66.249.71.192/27,66.249.71.224/27,66.249.71.32/27,66.249.71.64/27,66.249.71.96/27,66.249.72.0/27,66.249.72.128/27,66.249.72.160/27,66.249.72.192/27,66.249.72.224/27,66.249.72.32/27,66.249.72.64/27,66.249.73.0/27,66.249.73.128/27,66.249.73.160/27,66.249.73.192/27,66.249.73.224/27,66.249.73.32/27,66.249.73.64/27,66.249.73.96/27,66.249.74.0/27,66.249.74.128/27,66.249.74.160/27,66.249.74.192/27,66.249.74.224/27,66.249.74.32/27,66.249.74.64/27,66.249.74.96/27,66.249.75.0/27,66.249.75.128/27,66.249.75.160/27,66.249.75.192/27,66.249.75.224/27,66.249.75.32/27,66.249.75.64/27,66.249.75.96/27,66.249.76.0/27,66.249.76.128/27,66.249.76.160/27,66.249.76.192/27,66.249.76.224/27,66.249.76.32/27,66.249.76.64/27,66.249.76.96/27,66.249.77.0/27,66.249.77.128/27,66.249.77.160/27,66.249.77.192/27,66.249.77.224/27,66.249.77.32/27,66.249.77.64/27,66.249.77.96/27,66.249.78.0/27,66.249.78.128/27,66.249.78.160/27,66.249.78.32/27,66.249.78.64/27,66.249.78.96/27,66.249.79.0/27,66.249.79.128/27,66.249.79.160/27,66.249.79.192/27,66.249.79.224/27,66.249.79.32/27,66.249.79.64/27,157.55.39.0/24,207.46.13.0/24,40.77.167.0/24,13.66.139.0/24,13.66.144.0/24,52.167.144.0/24,13.67.10.16/28,13.69.66.240/28,13.71.172.224/28,139.217.52.0/28,191.233.204.224/28,20.36.108.32/28,20.43.120.16/28,40.79.131.208/28,40.79.186.176/28,52.231.148.0/28,20.79.107.240/28,51.105.67.0/28,20.125.163.80/28,40.77.188.0/22,65.55.210.0/24,199.30.24.0/23,40.77.202.0/24,40.77.139.0/25,20.74.197.0/28,20.15.133.160/27,40.77.177.0/24,40.77.178.0/23,17.241.208.160/27,17.241.193.160/27,17.241.200.160/27,17.22.237.0/24,17.22.245.0/24,17.22.253.0/24,17.241.75.0/24,17.241.219.0/24,17.241.227.0/24,17.246.15.0/24,17.246.19.0/24,17.246.23.0/24,107.20.236.150/32,3.224.62.45/32,18.210.92.235/32,3.222.232.239/32,3.211.124.183/32,3.231.139.107/32,18.97.1.228/30,18.97.9.96/29,44.208.221.197/32,34.193.163.52/32,18.97.21.0/30,18.97.43.80/29,130.107.228.224/32,104.43.54.127/32,104.43.55.116/32,104.43.55.117/32,104.43.55.166/32,104.43.55.167/32,108.141.83.74/32,13.89.106.77/32,172.169.17.165/32,191.233.3.197/32,191.233.3.202/32,191.234.216.178/32,191.234.216.4/32,191.235.201.214/32,191.235.202.38/32,191.235.202.48/32,20.113.14.159/32,20.113.3.121/32,20.12.141.99/32,20.185.79.15/32,20.185.79.47/32,20.191.44.119/32,20.191.44.16/32,20.191.44.22/32,20.191.44.234/32,20.191.45.212/32,20.193.12.126/32,20.193.24.10/32,20.193.24.251/32,20.193.25.197/32,20.193.27.215/32,20.193.45.113/32,20.195.108.47/32,20.197.209.11/32,20.197.209.27/32,20.201.15.208/32,20.204.240.172/32,20.204.241.148/32,20.204.242.101/32,20.204.242.19/32,20.204.243.55/32,20.204.246.254/32,20.204.246.81/32,20.207.107.181/32,20.207.72.11/32,20.207.72.110/32,20.207.72.113/32,20.207.72.21/32,20.207.97.190/32,20.207.99.197/32,20.219.43.246/32,20.219.45.190/32,20.219.45.67/32,20.226.133.105/32,20.232.184.230/32,20.3.1.178/32,20.40.133.240/32,20.43.150.85/32,20.43.150.93/32,20.43.172.120/32,20.44.222.1/32,20.49.136.28/32,20.50.168.91/32,20.50.48.159/32,20.50.48.192/32,20.50.49.0/32,20.50.49.237/32,20.50.49.25/32,20.50.49.40/32,20.50.49.55/32,20.50.50.118/32,20.50.50.121/32,20.50.50.123/32,20.50.50.130/32,20.50.50.134/32,20.50.50.145/32,20.50.50.146/32,20.50.50.163/32,20.50.50.46/32,20.53.134.160/32,20.53.78.106/32,20.53.78.123/32,20.53.78.138/32,20.53.78.144/32,20.53.78.236/32,20.53.91.2/32,20.53.92.211/32,20.56.197.58/32,20.56.197.63/32,20.61.34.40/32,20.62.224.44/32,20.71.12.143/32,20.72.242.93/32,20.73.132.240/32,20.73.202.147/32,20.75.144.152/32,20.79.226.26/32,20.79.238.198/32,20.79.239.66/32,20.80.129.80/32,20.93.28.24/32,20.99.255.235/32,4.156.136.79/32,4.182.131.108/32,4.195.133.120/32,4.209.224.56/32,4.213.46.14/32,4.228.76.163/32,40.114.182.153/32,40.114.182.172/32,40.114.182.45/32,40.114.183.196/32,40.114.183.251/32,40.114.183.88/32,40.119.232.146/32,40.119.232.215/32,40.119.232.218/32,40.119.232.251/32,40.119.232.50/32,40.64.105.247/32,40.64.106.11/32,40.76.162.191/32,40.76.162.208/32,40.76.162.247/32,40.76.163.23/32,40.76.163.7/32,40.76.173.151/32,40.80.242.63/32,40.81.250.205/32,40.88.21.235/32,40.89.243.175/32,51.104.146.225/32,51.104.146.235/32,51.104.160.167/32,51.104.160.177/32,51.104.161.32/32,51.104.162.149/32,51.104.163.250/32,51.104.164.109/32,51.104.164.147/32,51.104.164.189/32,51.104.164.215/32,51.104.166.111/32,51.104.167.104/32,51.104.167.110/32,51.104.167.19/32,51.104.167.52/32,51.104.167.54/32,51.104.167.61/32,51.104.167.71/32,51.104.167.87/32,51.104.167.88/32,51.104.167.95/32,51.104.167.96/32,51.104.180.26/32,51.104.180.47/32,51.104.180.53/32,51.107.40.209/32,51.116.131.221/32,51.120.48.122/32,51.138.90.161/32,51.138.90.206/32,51.138.90.233/32,51.8.253.152/32,51.8.71.117/32,52.142.24.149/32,52.142.26.175/32,52.143.241.111/32,52.143.242.6/32,52.143.243.117/32,52.143.244.81/32,52.143.247.235/32,52.143.95.162/32,52.143.95.204/32,52.146.58.236/32,52.146.59.12/32,52.146.59.154/32,52.146.59.156/32,52.146.63.80/32,52.148.161.87/32,52.148.165.38/32,52.149.25.43/32,52.149.28.18/32,52.149.28.83/32,52.149.30.45/32,52.149.56.151/32,52.149.58.139/32,52.149.58.173/32,52.149.58.27/32,52.149.58.69/32,52.149.60.38/32,52.149.61.51/32,52.154.169.200/32,52.154.169.50/32,52.154.170.113/32,52.154.170.117/32,52.154.170.122/32,52.154.170.209/32,52.154.170.229/32,52.154.170.243/32,52.154.170.26/32,52.154.170.28/32,52.154.170.88/32,52.154.170.96/32,52.154.171.0/32,52.154.171.150/32,52.154.171.196/32,52.154.171.205/32,52.154.171.235/32,52.154.171.250/32,52.154.171.44/32,52.154.171.70/32,52.154.171.87/32,52.154.172.2/32,52.154.60.82/32,52.190.37.160/32,52.224.16.221/32,52.224.16.229/32,52.224.19.152/32,52.224.20.174/32,52.224.20.181/32,52.224.20.186/32,52.224.20.190/32,52.224.20.193/32,52.224.20.203/32,52.224.20.204/32,52.224.20.223/32,52.224.20.227/32,52.224.20.249/32,52.224.21.19/32,52.224.21.20/32,52.224.21.23/32,52.224.21.27/32,52.224.21.4/32,52.224.21.49/32,52.224.21.51/32,52.224.21.53/32,52.224.21.55/32,52.224.21.61/32,52.242.224.168/32,57.152.72.128/32,20.8.252.26/32,20.61.142.192/32,20.54.224.39/32,20.13.44.19/32,172.199.55.212/32,132.164.209.198/32,20.166.171.150/32,4.207.220.92/32,68.219.152.220/32,40.127.154.196/32,172.168.115.250/32,172.169.28.184/32,4.150.142.218/32,128.203.132.152/32,64.236.118.43/32,48.217.212.89/32,52.186.37.211/32,52.188.89.106/32,135.234.221.112/32,74.179.232.116/32,4.201.220.8/32,4.201.197.203/32,4.201.206.133/32,4.201.125.59/32,4.201.141.71/32,172.168.53.53/32,172.168.254.119/32,172.168.227.120/32,172.169.60.134/32,4.249.216.104/32,172.169.177.131/32,48.217.23.236/32,4.156.154.107/32,48.217.129.210/32,20.232.51.46/32,4.255.35.121/32,20.253.59.76/32,52.250.46.221/32,172.193.245.229/32,20.112.58.44/32,20.253.96.199/32,40.82.218.203/32,4.237.244.80/32,20.175.232.228/32,4.172.49.103/32,20.250.51.113/32,4.226.40.135/32,20.170.75.54/32,4.182.10.198/32,20.101.17.173/32,20.71.69.210/32,20.216.200.223/32,20.40.147.172/32,20.100.136.36/32,20.100.140.155/32,20.77.146.108/32,20.49.129.236/32,13.86.35.212/32,20.118.11.251/32,18.97.9.168/29,18.97.14.80/29,18.97.14.88/30,98.85.178.216/32,85.208.96.0/24,85.208.97.0/24,85.208.99.0/24,185.170.167.0/24,185.191.171.0/24
IPv6-Adressen:
2607:6bc0::/48,2001:4860:4801:10::/64,2001:4860:4801:12::/64,2001:4860:4801:13::/64,2001:4860:4801:14::/64,2001:4860:4801:15::/64,2001:4860:4801:16::/64,2001:4860:4801:17::/64,2001:4860:4801:18::/64,2001:4860:4801:19::/64,2001:4860:4801:1a::/64,2001:4860:4801:1b::/64,2001:4860:4801:1c::/64,2001:4860:4801:1d::/64,2001:4860:4801:1e::/64,2001:4860:4801:1f::/64,2001:4860:4801:20::/64,2001:4860:4801:21::/64,2001:4860:4801:22::/64,2001:4860:4801:23::/64,2001:4860:4801:24::/64,2001:4860:4801:25::/64,2001:4860:4801:26::/64,2001:4860:4801:27::/64,2001:4860:4801:28::/64,2001:4860:4801:29::/64,2001:4860:4801:2::/64,2001:4860:4801:2a::/64,2001:4860:4801:2b::/64,2001:4860:4801:2c::/64,2001:4860:4801:2d::/64,2001:4860:4801:2e::/64,2001:4860:4801:2f::/64,2001:4860:4801:30::/64,2001:4860:4801:31::/64,2001:4860:4801:32::/64,2001:4860:4801:33::/64,2001:4860:4801:34::/64,2001:4860:4801:35::/64,2001:4860:4801:36::/64,2001:4860:4801:37::/64,2001:4860:4801:38::/64,2001:4860:4801:39::/64,2001:4860:4801:3a::/64,2001:4860:4801:3b::/64,2001:4860:4801:3c::/64,2001:4860:4801:3d::/64,2001:4860:4801:3e::/64,2001:4860:4801:3f::/64,2001:4860:4801:40::/64,2001:4860:4801:41::/64,2001:4860:4801:42::/64,2001:4860:4801:44::/64,2001:4860:4801:45::/64,2001:4860:4801:46::/64,2001:4860:4801:47::/64,2001:4860:4801:48::/64,2001:4860:4801:49::/64,2001:4860:4801:4a::/64,2001:4860:4801:4b::/64,2001:4860:4801:4c::/64,2001:4860:4801:4d::/64,2001:4860:4801:4e::/64,2001:4860:4801:50::/64,2001:4860:4801:51::/64,2001:4860:4801:52::/64,2001:4860:4801:53::/64,2001:4860:4801:54::/64,2001:4860:4801:55::/64,2001:4860:4801:56::/64,2001:4860:4801:57::/64,2001:4860:4801:58::/64,2001:4860:4801:59::/64,2001:4860:4801:60::/64,2001:4860:4801:61::/64,2001:4860:4801:62::/64,2001:4860:4801:63::/64,2001:4860:4801:64::/64,2001:4860:4801:65::/64,2001:4860:4801:66::/64,2001:4860:4801:67::/64,2001:4860:4801:68::/64,2001:4860:4801:69::/64,2001:4860:4801:6a::/64,2001:4860:4801:6b::/64,2001:4860:4801:6c::/64,2001:4860:4801:6d::/64,2001:4860:4801:6e::/64,2001:4860:4801:6f::/64,2001:4860:4801:70::/64,2001:4860:4801:71::/64,2001:4860:4801:72::/64,2001:4860:4801:73::/64,2001:4860:4801:74::/64,2001:4860:4801:75::/64,2001:4860:4801:76::/64,2001:4860:4801:77::/64,2001:4860:4801:78::/64,2001:4860:4801:79::/64,2001:4860:4801:7a::/64,2001:4860:4801:7b::/64,2001:4860:4801:7c::/64,2001:4860:4801:7d::/64,2001:4860:4801:80::/64,2001:4860:4801:81::/64,2001:4860:4801:82::/64,2001:4860:4801:83::/64,2001:4860:4801:84::/64,2001:4860:4801:85::/64,2001:4860:4801:86::/64,2001:4860:4801:87::/64,2001:4860:4801:88::/64,2001:4860:4801:90::/64,2001:4860:4801:91::/64,2001:4860:4801:92::/64,2001:4860:4801:93::/64,2001:4860:4801:94::/64,2001:4860:4801:95::/64,2001:4860:4801:96::/64,2001:4860:4801:97::/64,2001:4860:4801:a0::/64,2001:4860:4801:a1::/64,2001:4860:4801:a2::/64,2001:4860:4801:a3::/64,2001:4860:4801:a4::/64,2001:4860:4801:a5::/64,2001:4860:4801:a6::/64,2001:4860:4801:a7::/64,2001:4860:4801:a8::/64,2001:4860:4801:a9::/64,2001:4860:4801:aa::/64,2001:4860:4801
:/64,2001:4860:4801:ac::/64,2001:4860:4801:ad::/64,2001:4860:4801:ae::/64,2001:4860:4801:b0::/64,2001:4860:4801:b1::/64,2001:4860:4801:b2::/64,2001:4860:4801:b3::/64,2001:4860:4801:b4::/64,2001:4860:4801:b5::/64,2001:4860:4801:b6::/64,2001:4860:4801:c::/64,2001:4860:4801:f::/64,2600:1f28:365:80b0::/60Dazu empfehle ich noch eine weitere Liste von blocklist.de.
#Friendica, #Server, #Bot, #Crawler, #Scraper, #SEMrush, #nftables
www.blocklist.de -- Fail2Ban-Reporting Service (we sent Reports from Attacks on Postfix, SSH, Apache-Attacks, Spambots, irc-Bots, Reg-Bots, DDos and more) from Fail2Ban via X-ARF. -
R relay@relay.mycrowd.ca shared this topic
-
Das Thema AI-Crawler, Bots und Scraper macht auch vor dem Fediverse nicht halt. Nicht nur, dass hier unerwünscht Daten von Usern "abgesaugt" werden, diese Abfragen erzeugen auch immer mehr Last auf den Servern - viel unnötiger Traffic, der da entsteht und Ressourcen kostet.
Vieles lässt sich durch die robots.txt aussperren, aber leider hält sich nicht jeder Bot an diese. Natürlich ist auch von hartnäckigen Bots und Spidern ein Ausschluss in der .htaccess oder Apache-Config möglich. Auch mit Fail2Ban lässt sich da eine Menge bauen.
Allerdings ist es auch empfehlenswerter, diese IPs gleich kernelnah, also z.B. über nftables, zu sperren. Leider sind es nicht nur ein paar IPs, leider kann man nicht alle IPs dieser Bots sperren, weil auch immer wieder neue IPs hinzukommen. Es gibt aber bestimmte IPs und Adressbereiche, die von diesen Bots genutzt werden. Die Infos habe ich bei den "Anbietern" selbst gefunden, mit Ausnahme von SEMrush, der wirklich sehr aggressiv crawlen kann und gerne mal die robots.txt völlig ignoriert:
openai.com/gptbot.json
openai.com/chatgpt-user.json
openai.com/searchbot.json
docs.claude.com/en/api/ip-addr…
developers.google.com/static/s…
bing.com/toolbox/bingbot.json
search.developer.apple.com/app…
perplexity.com/perplexitybot.j…
perplexity.com/perplexity-user…
duckduckgo.com/duckduckgo-help…
index.commoncrawl.org/ccbot.js…
ipinfo.io/AS209366Ja, das sind natürlich nicht alle Bots, Crawler und Co! Das ist mir bewusst. Aber ein großer Teil Traffic verschwindet damit schon!
Da ich auf meinen Friendica-Instanzen aber auch keinen Besuch von Google und Co. haben möchte, habe ich heute mal einen Rundumschlag gemacht und diese IPs von oben genannten Seiten in nftables reingeworfen. Ergebnis: Es ist bedeutend ruhiger geworden! Und dabei sind es bei weitem natürlich nicht alle Bots und Crawler, aber doch schon einige!
Ich weiß nicht, ob jemand Interesse an den IPs hat, ich stelle diese hier einfach mal zur Verfügung. Quelle sind wie bereits erwähnt oben stehende Seiten. Nutzung natürlich auf eigene Gefahr.
IPv4-Adressen:
132.196.86.0/24,172.182.202.0/25,172.182.204.0/24,172.182.207.0/25,172.182.214.0/24,172.182.215.0/24,20.125.66.80/28,20.171.206.0/24,20.171.207.0/24,4.227.36.0/25,52.230.152.0/24,74.7.175.128/25,74.7.227.0/25,74.7.227.128/25,74.7.228.0/25,74.7.230.0/25,74.7.241.0/25,74.7.241.128/25,74.7.242.0/25,74.7.243.128/25,74.7.244.0/25,104.210.139.192/28,104.210.139.224/28,13.65.138.112/28,13.65.138.96/28,13.67.46.240/28,13.67.72.16/28,13.70.107.160/28,13.71.2.208/28,13.76.115.224/28,13.76.115.240/28,13.76.116.80/28,13.76.223.48/28,13.76.32.208/28,13.79.43.0/28,13.83.167.128/28,13.83.237.176/28,132.196.82.48/28,135.119.134.128/28,135.119.134.192/28,135.220.73.208/28,135.220.73.240/28,135.237.131.208/28,135.237.133.48/28,137.135.183.96/28,137.135.190.240/28,137.135.191.176/28,137.135.191.32/28,138.91.30.48/28,138.91.46.96/28,168.63.252.240/28,172.178.140.144/28,172.178.141.112/28,172.178.141.128/28,172.183.143.224/28,172.183.222.128/28,172.196.40.208/28,172.202.102.112/28,172.204.16.64/28,172.204.27.16/28,172.212.159.64/28,172.213.11.144/28,172.213.12.112/28,172.213.21.112/28,172.213.21.144/28,172.213.21.16/28,172.215.218.96/28,191.233.1.112/28,191.233.1.128/28,191.233.1.224/28,191.233.194.32/28,191.233.196.112/28,191.233.199.160/28,191.233.2.0/28,191.234.167.128/28,191.235.66.16/28,191.235.98.144/28,191.235.99.80/28,191.237.249.64/28,191.239.245.16/28,20.0.53.96/28,20.102.212.144/28,20.117.22.224/28,20.125.112.224/28,20.125.144.144/28,20.161.75.208/28,20.168.7.192/28,20.168.7.240/28,20.169.72.112/28,20.169.72.96/28,20.169.73.176/28,20.169.73.32/28,20.169.73.64/28,20.169.78.112/28,20.169.78.128/28,20.169.78.144/28,20.169.78.160/28,20.169.78.176/28,20.169.78.192/28,20.169.78.208/28,20.169.78.48/28,20.169.78.64/28,20.169.78.80/28,20.169.78.96/28,20.169.86.224/28,20.169.86.240/28,20.169.87.112/28,20.172.29.32/28,20.193.233.240/28,20.193.50.32/28,20.194.0.208/28,20.194.1.0/28,20.194.157.176/28,20.198.67.96/28,20.203.245.32/28,20.204.24.240/28,20.206.107.192/28,20.210.154.128/28,20.210.174.208/28,20.210.211.192/28,20.215.187.208/28,20.215.188.192/28,20.215.214.16/28,20.215.219.128/28,20.215.219.160/28,20.215.219.208/28,20.215.220.112/28,20.215.220.128/28,20.215.220.144/28,20.215.220.160/28,20.215.220.176/28,20.215.220.192/28,20.215.220.208/28,20.215.220.64/28,20.215.220.80/28,20.215.220.96/28,20.226.32.80/28,20.227.140.32/28,20.228.106.176/28,20.235.75.208/28,20.235.87.224/28,20.249.63.208/28,20.27.94.128/28,20.45.178.144/28,20.55.229.144/28,20.63.221.64/28,20.90.7.144/28,20.97.189.96/28,23.102.140.144/28,23.102.141.32/28,23.97.109.224/28,23.98.142.176/28,23.98.179.16/28,23.98.186.176/28,23.98.186.192/28,23.98.186.64/28,23.98.186.96/28,4.151.119.48/28,4.151.241.240/28,4.151.71.176/28,4.189.118.208/28,4.189.119.48/28,4.196.118.112/28,4.196.198.80/28,4.197.115.112/28,4.197.19.176/28,4.197.22.112/28,4.197.64.0/28,4.197.64.16/28,4.197.64.48/28,4.197.64.64/28,4.205.128.176/28,40.116.73.208/28,40.122.235.112/28,40.67.183.160/28,40.67.183.176/28,40.75.14.224/28,40.81.134.128/28,40.81.134.144/28,40.81.234.144/28,40.84.181.32/28,40.84.221.208/28,40.84.221.224/28,51.107.70.192/28,51.8.155.48/28,51.8.155.64/28,51.8.155.80/28,52.148.129.32/28,52.154.22.48/28,52.156.77.144/28,52.159.227.32/28,52.159.249.96/28,52.165.212.16/28,52.165.212.32/28,52.165.212.48/28,52.172.129.160/28,52.172.251.112/28,52.173.123.0/28,52.173.219.112/28,52.173.219.96/28,52.173.221.16/28,52.173.221.176/28,52.173.221.208/28,52.173.234.16/28,52.173.234.80/28,52.173.235.80/28,52.176.139.176/28,52.187.246.128/28,52.190.137.144/28,52.190.137.16/28,52.190.139.48/28,52.190.142.64/28,52.190.190.16/28,52.225.75.208/28,52.230.163.32/28,52.230.164.176/28,52.231.30.48/28,52.231.34.176/28,52.231.39.144/28,52.231.39.192/28,52.231.49.48/28,52.231.50.64/28,52.236.94.144/28,52.242.132.224/28,52.242.132.240/28,52.242.245.208/28,52.252.113.240/28,52.255.109.112/28,52.255.109.128/28,52.255.109.144/28,52.255.109.80/28,52.255.109.96/28,52.255.111.0/28,52.255.111.112/28,52.255.111.16/28,52.255.111.32/28,52.255.111.48/28,52.255.111.80/28,57.151.131.224/28,57.154.174.112/28,57.154.175.0/28,57.154.187.32/28,68.154.28.96/28,68.218.30.112/28,68.220.57.64/28,68.221.67.160/28,68.221.67.192/28,68.221.67.224/28,68.221.67.240/28,68.221.75.16/28,74.226.253.160/28,74.249.86.176/28,74.7.35.112/28,74.7.35.48/28,74.7.36.64/28,74.7.36.80/28,74.7.36.96/28,9.160.163.224/28,9.160.164.128/28,104.210.140.128/28,135.234.64.0/24,172.182.193.224/28,172.182.193.80/28,172.182.194.144/28,172.182.194.32/28,172.182.195.48/28,172.182.209.208/28,172.182.211.192/28,172.182.213.192/28,172.182.224.0/28,172.203.190.128/28,20.14.99.96/28,20.168.18.32/28,20.169.6.224/28,20.169.7.48/28,20.169.77.0/25,20.171.123.64/28,20.171.53.224/28,20.25.151.224/28,20.42.10.176/28,4.227.36.0/25,40.67.175.0/25,40.90.214.16/28,51.8.102.0/24,74.7.175.128/25,74.7.228.0/25,74.7.228.128/25,74.7.229.0/25,74.7.229.128/25,74.7.230.0/25,74.7.241.128/25,74.7.242.128/25,74.7.243.0/25,74.7.244.0/25,160.79.104.0/21,192.178.4.0/27,192.178.4.128/27,192.178.4.160/27,192.178.4.192/27,192.178.4.224/27,192.178.4.32/27,192.178.4.64/27,192.178.4.96/27,192.178.5.0/27,192.178.6.0/27,192.178.6.128/27,192.178.6.160/27,192.178.6.192/27,192.178.6.224/27,192.178.6.32/27,192.178.6.64/27,192.178.6.96/27,192.178.7.0/27,192.178.7.128/27,192.178.7.160/27,192.178.7.192/27,192.178.7.224/27,192.178.7.32/27,192.178.7.64/27,192.178.7.96/27,34.100.182.96/28,34.101.50.144/28,34.118.254.0/28,34.118.66.0/28,34.126.178.96/28,34.146.150.144/28,34.147.110.144/28,34.151.74.144/28,34.152.50.64/28,34.154.114.144/28,34.155.98.32/28,34.165.18.176/28,34.175.160.64/28,34.176.130.16/28,34.22.85.0/27,34.64.82.64/28,34.65.242.112/28,34.80.50.80/28,34.88.194.0/28,34.89.10.80/28,34.89.198.80/28,34.96.162.48/28,35.247.243.240/28,66.249.64.0/27,66.249.64.128/27,66.249.64.160/27,66.249.64.192/27,66.249.64.224/27,66.249.64.32/27,66.249.64.64/27,66.249.64.96/27,66.249.65.0/27,66.249.65.128/27,66.249.65.160/27,66.249.65.192/27,66.249.65.224/27,66.249.65.32/27,66.249.65.64/27,66.249.65.96/27,66.249.66.0/27,66.249.66.128/27,66.249.66.160/27,66.249.66.192/27,66.249.66.224/27,66.249.66.32/27,66.249.66.64/27,66.249.66.96/27,66.249.67.0/27,66.249.67.32/27,66.249.67.64/27,66.249.68.0/27,66.249.68.128/27,66.249.68.160/27,66.249.68.192/27,66.249.68.32/27,66.249.68.64/27,66.249.68.96/27,66.249.69.0/27,66.249.69.128/27,66.249.69.160/27,66.249.69.192/27,66.249.69.224/27,66.249.69.32/27,66.249.69.64/27,66.249.69.96/27,66.249.70.0/27,66.249.70.128/27,66.249.70.160/27,66.249.70.192/27,66.249.70.224/27,66.249.70.32/27,66.249.70.64/27,66.249.70.96/27,66.249.71.0/27,66.249.71.128/27,66.249.71.160/27,66.249.71.192/27,66.249.71.224/27,66.249.71.32/27,66.249.71.64/27,66.249.71.96/27,66.249.72.0/27,66.249.72.128/27,66.249.72.160/27,66.249.72.192/27,66.249.72.224/27,66.249.72.32/27,66.249.72.64/27,66.249.73.0/27,66.249.73.128/27,66.249.73.160/27,66.249.73.192/27,66.249.73.224/27,66.249.73.32/27,66.249.73.64/27,66.249.73.96/27,66.249.74.0/27,66.249.74.128/27,66.249.74.160/27,66.249.74.192/27,66.249.74.224/27,66.249.74.32/27,66.249.74.64/27,66.249.74.96/27,66.249.75.0/27,66.249.75.128/27,66.249.75.160/27,66.249.75.192/27,66.249.75.224/27,66.249.75.32/27,66.249.75.64/27,66.249.75.96/27,66.249.76.0/27,66.249.76.128/27,66.249.76.160/27,66.249.76.192/27,66.249.76.224/27,66.249.76.32/27,66.249.76.64/27,66.249.76.96/27,66.249.77.0/27,66.249.77.128/27,66.249.77.160/27,66.249.77.192/27,66.249.77.224/27,66.249.77.32/27,66.249.77.64/27,66.249.77.96/27,66.249.78.0/27,66.249.78.128/27,66.249.78.160/27,66.249.78.32/27,66.249.78.64/27,66.249.78.96/27,66.249.79.0/27,66.249.79.128/27,66.249.79.160/27,66.249.79.192/27,66.249.79.224/27,66.249.79.32/27,66.249.79.64/27,157.55.39.0/24,207.46.13.0/24,40.77.167.0/24,13.66.139.0/24,13.66.144.0/24,52.167.144.0/24,13.67.10.16/28,13.69.66.240/28,13.71.172.224/28,139.217.52.0/28,191.233.204.224/28,20.36.108.32/28,20.43.120.16/28,40.79.131.208/28,40.79.186.176/28,52.231.148.0/28,20.79.107.240/28,51.105.67.0/28,20.125.163.80/28,40.77.188.0/22,65.55.210.0/24,199.30.24.0/23,40.77.202.0/24,40.77.139.0/25,20.74.197.0/28,20.15.133.160/27,40.77.177.0/24,40.77.178.0/23,17.241.208.160/27,17.241.193.160/27,17.241.200.160/27,17.22.237.0/24,17.22.245.0/24,17.22.253.0/24,17.241.75.0/24,17.241.219.0/24,17.241.227.0/24,17.246.15.0/24,17.246.19.0/24,17.246.23.0/24,107.20.236.150/32,3.224.62.45/32,18.210.92.235/32,3.222.232.239/32,3.211.124.183/32,3.231.139.107/32,18.97.1.228/30,18.97.9.96/29,44.208.221.197/32,34.193.163.52/32,18.97.21.0/30,18.97.43.80/29,130.107.228.224/32,104.43.54.127/32,104.43.55.116/32,104.43.55.117/32,104.43.55.166/32,104.43.55.167/32,108.141.83.74/32,13.89.106.77/32,172.169.17.165/32,191.233.3.197/32,191.233.3.202/32,191.234.216.178/32,191.234.216.4/32,191.235.201.214/32,191.235.202.38/32,191.235.202.48/32,20.113.14.159/32,20.113.3.121/32,20.12.141.99/32,20.185.79.15/32,20.185.79.47/32,20.191.44.119/32,20.191.44.16/32,20.191.44.22/32,20.191.44.234/32,20.191.45.212/32,20.193.12.126/32,20.193.24.10/32,20.193.24.251/32,20.193.25.197/32,20.193.27.215/32,20.193.45.113/32,20.195.108.47/32,20.197.209.11/32,20.197.209.27/32,20.201.15.208/32,20.204.240.172/32,20.204.241.148/32,20.204.242.101/32,20.204.242.19/32,20.204.243.55/32,20.204.246.254/32,20.204.246.81/32,20.207.107.181/32,20.207.72.11/32,20.207.72.110/32,20.207.72.113/32,20.207.72.21/32,20.207.97.190/32,20.207.99.197/32,20.219.43.246/32,20.219.45.190/32,20.219.45.67/32,20.226.133.105/32,20.232.184.230/32,20.3.1.178/32,20.40.133.240/32,20.43.150.85/32,20.43.150.93/32,20.43.172.120/32,20.44.222.1/32,20.49.136.28/32,20.50.168.91/32,20.50.48.159/32,20.50.48.192/32,20.50.49.0/32,20.50.49.237/32,20.50.49.25/32,20.50.49.40/32,20.50.49.55/32,20.50.50.118/32,20.50.50.121/32,20.50.50.123/32,20.50.50.130/32,20.50.50.134/32,20.50.50.145/32,20.50.50.146/32,20.50.50.163/32,20.50.50.46/32,20.53.134.160/32,20.53.78.106/32,20.53.78.123/32,20.53.78.138/32,20.53.78.144/32,20.53.78.236/32,20.53.91.2/32,20.53.92.211/32,20.56.197.58/32,20.56.197.63/32,20.61.34.40/32,20.62.224.44/32,20.71.12.143/32,20.72.242.93/32,20.73.132.240/32,20.73.202.147/32,20.75.144.152/32,20.79.226.26/32,20.79.238.198/32,20.79.239.66/32,20.80.129.80/32,20.93.28.24/32,20.99.255.235/32,4.156.136.79/32,4.182.131.108/32,4.195.133.120/32,4.209.224.56/32,4.213.46.14/32,4.228.76.163/32,40.114.182.153/32,40.114.182.172/32,40.114.182.45/32,40.114.183.196/32,40.114.183.251/32,40.114.183.88/32,40.119.232.146/32,40.119.232.215/32,40.119.232.218/32,40.119.232.251/32,40.119.232.50/32,40.64.105.247/32,40.64.106.11/32,40.76.162.191/32,40.76.162.208/32,40.76.162.247/32,40.76.163.23/32,40.76.163.7/32,40.76.173.151/32,40.80.242.63/32,40.81.250.205/32,40.88.21.235/32,40.89.243.175/32,51.104.146.225/32,51.104.146.235/32,51.104.160.167/32,51.104.160.177/32,51.104.161.32/32,51.104.162.149/32,51.104.163.250/32,51.104.164.109/32,51.104.164.147/32,51.104.164.189/32,51.104.164.215/32,51.104.166.111/32,51.104.167.104/32,51.104.167.110/32,51.104.167.19/32,51.104.167.52/32,51.104.167.54/32,51.104.167.61/32,51.104.167.71/32,51.104.167.87/32,51.104.167.88/32,51.104.167.95/32,51.104.167.96/32,51.104.180.26/32,51.104.180.47/32,51.104.180.53/32,51.107.40.209/32,51.116.131.221/32,51.120.48.122/32,51.138.90.161/32,51.138.90.206/32,51.138.90.233/32,51.8.253.152/32,51.8.71.117/32,52.142.24.149/32,52.142.26.175/32,52.143.241.111/32,52.143.242.6/32,52.143.243.117/32,52.143.244.81/32,52.143.247.235/32,52.143.95.162/32,52.143.95.204/32,52.146.58.236/32,52.146.59.12/32,52.146.59.154/32,52.146.59.156/32,52.146.63.80/32,52.148.161.87/32,52.148.165.38/32,52.149.25.43/32,52.149.28.18/32,52.149.28.83/32,52.149.30.45/32,52.149.56.151/32,52.149.58.139/32,52.149.58.173/32,52.149.58.27/32,52.149.58.69/32,52.149.60.38/32,52.149.61.51/32,52.154.169.200/32,52.154.169.50/32,52.154.170.113/32,52.154.170.117/32,52.154.170.122/32,52.154.170.209/32,52.154.170.229/32,52.154.170.243/32,52.154.170.26/32,52.154.170.28/32,52.154.170.88/32,52.154.170.96/32,52.154.171.0/32,52.154.171.150/32,52.154.171.196/32,52.154.171.205/32,52.154.171.235/32,52.154.171.250/32,52.154.171.44/32,52.154.171.70/32,52.154.171.87/32,52.154.172.2/32,52.154.60.82/32,52.190.37.160/32,52.224.16.221/32,52.224.16.229/32,52.224.19.152/32,52.224.20.174/32,52.224.20.181/32,52.224.20.186/32,52.224.20.190/32,52.224.20.193/32,52.224.20.203/32,52.224.20.204/32,52.224.20.223/32,52.224.20.227/32,52.224.20.249/32,52.224.21.19/32,52.224.21.20/32,52.224.21.23/32,52.224.21.27/32,52.224.21.4/32,52.224.21.49/32,52.224.21.51/32,52.224.21.53/32,52.224.21.55/32,52.224.21.61/32,52.242.224.168/32,57.152.72.128/32,20.8.252.26/32,20.61.142.192/32,20.54.224.39/32,20.13.44.19/32,172.199.55.212/32,132.164.209.198/32,20.166.171.150/32,4.207.220.92/32,68.219.152.220/32,40.127.154.196/32,172.168.115.250/32,172.169.28.184/32,4.150.142.218/32,128.203.132.152/32,64.236.118.43/32,48.217.212.89/32,52.186.37.211/32,52.188.89.106/32,135.234.221.112/32,74.179.232.116/32,4.201.220.8/32,4.201.197.203/32,4.201.206.133/32,4.201.125.59/32,4.201.141.71/32,172.168.53.53/32,172.168.254.119/32,172.168.227.120/32,172.169.60.134/32,4.249.216.104/32,172.169.177.131/32,48.217.23.236/32,4.156.154.107/32,48.217.129.210/32,20.232.51.46/32,4.255.35.121/32,20.253.59.76/32,52.250.46.221/32,172.193.245.229/32,20.112.58.44/32,20.253.96.199/32,40.82.218.203/32,4.237.244.80/32,20.175.232.228/32,4.172.49.103/32,20.250.51.113/32,4.226.40.135/32,20.170.75.54/32,4.182.10.198/32,20.101.17.173/32,20.71.69.210/32,20.216.200.223/32,20.40.147.172/32,20.100.136.36/32,20.100.140.155/32,20.77.146.108/32,20.49.129.236/32,13.86.35.212/32,20.118.11.251/32,18.97.9.168/29,18.97.14.80/29,18.97.14.88/30,98.85.178.216/32,85.208.96.0/24,85.208.97.0/24,85.208.99.0/24,185.170.167.0/24,185.191.171.0/24
IPv6-Adressen:
2607:6bc0::/48,2001:4860:4801:10::/64,2001:4860:4801:12::/64,2001:4860:4801:13::/64,2001:4860:4801:14::/64,2001:4860:4801:15::/64,2001:4860:4801:16::/64,2001:4860:4801:17::/64,2001:4860:4801:18::/64,2001:4860:4801:19::/64,2001:4860:4801:1a::/64,2001:4860:4801:1b::/64,2001:4860:4801:1c::/64,2001:4860:4801:1d::/64,2001:4860:4801:1e::/64,2001:4860:4801:1f::/64,2001:4860:4801:20::/64,2001:4860:4801:21::/64,2001:4860:4801:22::/64,2001:4860:4801:23::/64,2001:4860:4801:24::/64,2001:4860:4801:25::/64,2001:4860:4801:26::/64,2001:4860:4801:27::/64,2001:4860:4801:28::/64,2001:4860:4801:29::/64,2001:4860:4801:2::/64,2001:4860:4801:2a::/64,2001:4860:4801:2b::/64,2001:4860:4801:2c::/64,2001:4860:4801:2d::/64,2001:4860:4801:2e::/64,2001:4860:4801:2f::/64,2001:4860:4801:30::/64,2001:4860:4801:31::/64,2001:4860:4801:32::/64,2001:4860:4801:33::/64,2001:4860:4801:34::/64,2001:4860:4801:35::/64,2001:4860:4801:36::/64,2001:4860:4801:37::/64,2001:4860:4801:38::/64,2001:4860:4801:39::/64,2001:4860:4801:3a::/64,2001:4860:4801:3b::/64,2001:4860:4801:3c::/64,2001:4860:4801:3d::/64,2001:4860:4801:3e::/64,2001:4860:4801:3f::/64,2001:4860:4801:40::/64,2001:4860:4801:41::/64,2001:4860:4801:42::/64,2001:4860:4801:44::/64,2001:4860:4801:45::/64,2001:4860:4801:46::/64,2001:4860:4801:47::/64,2001:4860:4801:48::/64,2001:4860:4801:49::/64,2001:4860:4801:4a::/64,2001:4860:4801:4b::/64,2001:4860:4801:4c::/64,2001:4860:4801:4d::/64,2001:4860:4801:4e::/64,2001:4860:4801:50::/64,2001:4860:4801:51::/64,2001:4860:4801:52::/64,2001:4860:4801:53::/64,2001:4860:4801:54::/64,2001:4860:4801:55::/64,2001:4860:4801:56::/64,2001:4860:4801:57::/64,2001:4860:4801:58::/64,2001:4860:4801:59::/64,2001:4860:4801:60::/64,2001:4860:4801:61::/64,2001:4860:4801:62::/64,2001:4860:4801:63::/64,2001:4860:4801:64::/64,2001:4860:4801:65::/64,2001:4860:4801:66::/64,2001:4860:4801:67::/64,2001:4860:4801:68::/64,2001:4860:4801:69::/64,2001:4860:4801:6a::/64,2001:4860:4801:6b::/64,2001:4860:4801:6c::/64,2001:4860:4801:6d::/64,2001:4860:4801:6e::/64,2001:4860:4801:6f::/64,2001:4860:4801:70::/64,2001:4860:4801:71::/64,2001:4860:4801:72::/64,2001:4860:4801:73::/64,2001:4860:4801:74::/64,2001:4860:4801:75::/64,2001:4860:4801:76::/64,2001:4860:4801:77::/64,2001:4860:4801:78::/64,2001:4860:4801:79::/64,2001:4860:4801:7a::/64,2001:4860:4801:7b::/64,2001:4860:4801:7c::/64,2001:4860:4801:7d::/64,2001:4860:4801:80::/64,2001:4860:4801:81::/64,2001:4860:4801:82::/64,2001:4860:4801:83::/64,2001:4860:4801:84::/64,2001:4860:4801:85::/64,2001:4860:4801:86::/64,2001:4860:4801:87::/64,2001:4860:4801:88::/64,2001:4860:4801:90::/64,2001:4860:4801:91::/64,2001:4860:4801:92::/64,2001:4860:4801:93::/64,2001:4860:4801:94::/64,2001:4860:4801:95::/64,2001:4860:4801:96::/64,2001:4860:4801:97::/64,2001:4860:4801:a0::/64,2001:4860:4801:a1::/64,2001:4860:4801:a2::/64,2001:4860:4801:a3::/64,2001:4860:4801:a4::/64,2001:4860:4801:a5::/64,2001:4860:4801:a6::/64,2001:4860:4801:a7::/64,2001:4860:4801:a8::/64,2001:4860:4801:a9::/64,2001:4860:4801:aa::/64,2001:4860:4801
:/64,2001:4860:4801:ac::/64,2001:4860:4801:ad::/64,2001:4860:4801:ae::/64,2001:4860:4801:b0::/64,2001:4860:4801:b1::/64,2001:4860:4801:b2::/64,2001:4860:4801:b3::/64,2001:4860:4801:b4::/64,2001:4860:4801:b5::/64,2001:4860:4801:b6::/64,2001:4860:4801:c::/64,2001:4860:4801:f::/64,2600:1f28:365:80b0::/60Dazu empfehle ich noch eine weitere Liste von blocklist.de.
#Friendica, #Server, #Bot, #Crawler, #Scraper, #SEMrush, #nftables
www.blocklist.de -- Fail2Ban-Reporting Service (we sent Reports from Attacks on Postfix, SSH, Apache-Attacks, Spambots, irc-Bots, Reg-Bots, DDos and more) from Fail2Ban via X-ARF.@jools Könnte es d Sinn machen ggf. ganze Bereiche zu blocken? Z.B.: 172.183.* und 172.213.*
Könnte natürlich mehr blocken als nötig. Aber es könnte ja sehr gut möglich sein, das unter diesen IP-Teilen noch mehr Bots kommen. Also quasi Prävention.
Das würde die Liste auch deutlich verschlanken.
