My htaccess file is getting large as I continually ban more bad bots of the world. As it gets larger there are bound to be more mistakes. One of the mistakes can occur in “deny from” lines, which account for the vast majority of lines in the htaccess. If you add any alpha characters to the ip addresses in “deny from” lines, the Apache server will do all host lookups and try to not return IP addresses. This means that some spammers’ ip addresses will be hidden behind bogus host names. For accuracy it is best for the Apache server to return their IP addresses. Using IPs you can then do host and search lookups, find them and ban them.
I have created 2 shell scripts that will highlight any “deny from” line that has alpha characters. Both these scripts run in terminal on Unix. I use Ubuntu. If you have Windows you figure out what to do.
Filter htaccess for Only deny From Lines
To check my “deny from” lines I first strip out all my comments and other lines from my htaccess file, leaving me with a file containing only the “deny from” lines. This is the purpose of denycheck.sh. The output file from denycheck.sh contains only your “deny from” lines. you can review the output files with an editor to ensure that the script has been completed properly.
strin=$1
strout=${strin/\.htaccess/denycheck}
grep deny $1 | grep from | grep -v "#" > tempfile
echo "Deny from statements checked..."
sed -e 's/deny from//g' tempfile | grep '[a-z,A-Z,#,-]'
echo "end output for errant deny From statements"
mv tempfile ${strout/\new/out}
Comments: filters for lines with “deny”
filters for lines with “from”
excludes lines with “#”, all comments
highligh deny from lines that contain alpha characters or “-“, as this is a syntax errors
save as denycheck.sh
to execute: ./denycheck.sh .htaccess-2016-aug-20-input-new
A file denycheck-2016-aug-20-input-out will be created
Display Deny From lines with Errant Alpha Characters or hyphen
The sed line in the script will display in terminal only those “deny from” lines that have alpha characters and would send your Apache server to return host names only. The sed command strips out the “deny from” literal and then removes all numeric only lines. This should leave you with only those lines that have alpha characters, which would be the lines you want to review and correct. Output is displayed on terminal. If you see alpha characters in your deny from line you can then use the IP address to search and correct your error.
Terminal command:
sed -e ‘s/deny from//g’ htaccess-checked-file-output-2016-aug-07.txt
sed -e ‘s/deny from//g’ $1 | grep ‘[a-z,A-Z]’
Comments:
-replace “deny from’ with null, highlight a alpha, non-numeric
Use this script so that your “deny from” statements only contain IP addresses and your Apache server will return accurate IP addresses in your raw access log.
Remove Comments from htaccess: Remove comments to allow long htaccess files to be processed by the htaccess checker, and to make it easier for your ISP tech staff to review. All those comments are great for you but not for the tech staff.
strin=$1
grep "^[^#;]" $1 > ${strin/\new/out}
to execute: ./nocomments.sh .htaccess-2016-aug-05-new
output: a file .htaccess-2016-aug-05-out will be created. Use an htaccess checker to ensure nothing has changed
Always check your htaccess for syntax errors before you upload it to your server. I use htaccess check. Otherwise an error will result in a Bad Gateway error and make your site unreachable to all. Check your htaccess every time you change it.
And try not to ban yourself…too often.
