A dear friend uses Feedly to monitor my site. He complained that he was getting 403s Banned, and asked why. Well, I have found that Feedly usually only takes my RSS feed, but sometimes, not often, it scrapes me mercilessly. Once I see a bot start scraping, I ban it. I moved him over to the more well behaved Feedburner by Google.
Here are the Feedly user agents:
Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)
The latter, FeedlyBot, runs off WZComm, and had previously scraped me, so I banned it. WZComm also runs the surdotly bot, which is also banned. The former, Feedly/1.0, runs off Level 3, and seems well behaved.
# AWS 22.214.171.124 – 126.96.36.199 188.8.131.52/11
deny from 184.108.40.206/13 220.127.116.11/16 18.104.22.168/17 22.214.171.124/18 126.96.36.199/19 188.8.131.52/20 184.108.40.206/21
# Pinterest 220.127.116.11/24 18.104.22.168/24
deny from 22.214.171.124/23 126.96.36.199/22 188.8.131.52/15 184.108.40.206/14 220.127.116.11/12
Delays in htaccess Implementation
tanyadokterkeluarga.blogspot is a persistent referrer spammer. They use a huge amount of Ip addresses that do not repeat the third octet. It has similar strategies to kosmetik-freaks.blogspot, in fact sharing identical IP ranges. They are sister referrer spammers. Both are not banned by the HTTP_REFERER in htaccess. If you kill one you kill the other, a nice double prize. As with the sister, this spammer runs out of Indonesia.
These are the referrers:
There are some scrapers and there are others that are ridiculous. I just got scraped hard by 18.104.22.168, 209-133-216-182.static.hvvc.us, with 105 server entries and 7 unique user agent names. Excessive, to say the least.
Here are the UA’s used:
Mozilla/5.0 (BlackBerry; U; BlackBerry 9900; en) AppleWebKit/534.11+ (KHTML, like Gecko) Version/22.214.171.1246 Mobile Safari/534.11+
Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20160721-2308 +http://www.exif-search.com)
Mozilla/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0
Mozilla/5.0 (Windows; U; Windows NT 6.1; rv:2.2) Gecko/20110201
Opera/12.02 (Android 4.1; Linux; Opera Mobi/ADR-1111101157; U; en-US) Presto/2.9.201 Version/12.02
126.96.36.199 – 188.8.131.52 184.108.40.206/19
NOC4Hosts, HIvelocity Network
I have sent an email to their ISP, email@example.com.
This kosmetik-freaks.blogspot is a referrer spammer that has been harassing me for quite a long time. I have tried to ban them with an HTTP_REFERER ban but this does not work. My ISP, Site5, will not help me. They are predominantly out of Indonesia. They are pret220.127.116.11
too sophisticated to evade my detection for so long.
The sister referrer spammer is tanyadokterkeluarga.blogspot, which uses the identical method and largely shares the same IP ranges. When you kill one you kill the other. Almost all these UAs are mobile devices, leading me to believe these are mobile customers that have downloaded the same spam app.
kwpublisher.com is a long-time referrer spammer that I would like to remove. I have tried to ban them with an HTTP_REFERER ban but this does not work. My ISP, Site5, will not help me. This guy seems to have a similar method to kosmetik-freaks.blogspot. They seem to be out of Pakistan mostly, but have gone to Indonesia and China. I am now tracking them closely.
Conclusion: Tracked down the code hotlinking to my site. Complained to their domain names provider. Them they disappeared. Goodbye.
18.104.22.168 x 4 22.214.171.124 – 126.96.36.199 Pakistan Tel