Tag: user agent

Nikto Web Server Scan: View from the Access Log

Playing, I am, with the Nikto web server scanning package. I scanned my own site, just for fun. While it does take some time, it did finish. I wondered how it would look from my site’s raw access log viewpoint. In summary, Nikto is not stealthy at all. It is also easily detected and banned mid-scan, as it takes a long time to complete.

Essentially you start a Terminal, and type “nikto -h “. There are lots of options, such as output to a log. The Nikto output highlights web site vulnerabilities and cross references these with a database of known hacks. Using this tool you can highlight the site’s weaknesses and then strengthen your site from hackers.

Managing a Large Apache htaccess: Tips

After a long while your htaccess might get a tad long. My favourite htaccess checker only processes files up to 5,000 lines. Often this is due to lots of comments, which I encourage. Let us cover some ways you can shorten your htaccess:

Combine your user agents/referrers
If you have multiple user agent or referrers that have similar names, combine them into a single statement,
RewriteCond %{HTTP_USER_AGENT} ^.*Blackboard\ Safeassign [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*BlackWidow [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*Black(board\ Safeassign|Widow) [OR]

RewriteCond %{HTTP_REFERER} ^http://.*pizza\-imperia [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*pizza\-tycoon [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*pizza(\-imperia|\-tycoon) [NC,OR]

User Agents I Could not Ban with htaccess

These user agents, or bots, somehow fool and subvert my .htaccess user agent rules and continue to scrape my site. I’ve looked at my htaccess user agent rule many times and don’t know why. The next step is to ban their IP.

AhrefsBot is a large content scraper that hits my site hard, reads robots.txt but ignores it, fools my htaccess, bot is “Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)”