Managing a Large Apache htaccess: Tips

After a long while your htaccess might get a tad long. My favourite htaccess checker only processes files up to 5,000 lines. Often this is due to lots of comments, which I encourage. Let us cover some ways you can shorten your htaccess:

Combine your user agents/referrers
If you have multiple user agent or referrers that have similar names, combine them into a single statement,
from:
RewriteCond %{HTTP_USER_AGENT} ^.*Blackboard\ Safeassign [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*BlackWidow [OR]
to:
RewriteCond %{HTTP_USER_AGENT} ^.*Black(board\ Safeassign|Widow) [OR]

From:
RewriteCond %{HTTP_REFERER} ^http://.*pizza\-imperia [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*pizza\-tycoon [NC,OR]
To:
RewriteCond %{HTTP_REFERER} ^http://.*pizza(\-imperia|\-tycoon) [NC,OR]

Comment your IP ranges
Spammers flock together into groups, and like a bad rash, they often return. Add comments to your htaccess and put like ip ranges together, on the same deny line. This saves a lot of space. The next time they visit you already have their IP range so just add them to the appropriate line. This saves time and space.

If IP ranges are related combine on a single line
Let’s say that you have 3 ip ranges to ban for Chinanet Guangdong. Instead of listing one IP range per line, combine them into one line separated by a space.

Go from:
# CHINANET Guangdong 14.144.0.0 – 14.159.255.255
deny from 14.146.34.0/24
deny from 14.150.64.0/24
deny from 14.150.245.0/24
to:
# CHINANET Guangdong 14.144.0.0 – 14.159.255.255
deny from 14.146.34.0/24 14.150.64.0/24 14.150.245.0/24

Combine IP ranges into larger ranges
If you have 2 contiguous IP ranges you can consider combining them, using CIDR notation, for a shorter, more concise IP range. If there are a large number of IP bans at the .0/24 level you can simplify greatly by going down to a smaller CIDR number. This also means that if you have exceptions it will be more difficult to get around.

Move from:
# Digital Ocean
deny from 162.247.72.0/24 162.247.73.0/24 162.247.74.0/24 162.247.75.0/24
to:
# Digital Ocean
deny from 162.247.72.0/22

The deny range is exactly the same, but now you only use one statement, simpler, easier to read and easier to process.

Remove Comments
If something goes wrong with your site and you ask your ISP for help, it is best to run through a script that removes all comments. After comment removal by htaccess size drops by 45%. Check the no comments version through the htaccess checker. The script creates a new copy of your htaccess without comments, is fast and easy to run.

Reading an unfamiliar htaccess without comments is simpler for tech support and they can treat all your Deny From statements as one large block.

As the htaccess is read for each server request, having no comments might make it a smidge faster, but probably not by much.

Keep the original htaccess with comments as comments make it so much easier to manage.

Test only a portion of your htaccess through the checker
So you have hit the 5k mark on the checker. If you are like me your htaccess does not markedly change from day to day. Divide your file in half and check each separately through the htaccess checker. Remove comment and then check the no comments version. The no comments version should be markedly smaller

Always backup your htaccess versions
You never know when someone will point out that there was a problem, which you trace back to last Tuesday. You should have a working htaccess version from last Tuesday that you can install temporarily while you beaver away to find the issue.

Use an editor that can compare multiple files
In Ubuntu gedit has a plugin called Meld, which allows you to compare two files and backout or redo any differences between the files. This has been a life saver when you are troubleshooting a newer version.

I am sure there are lots more hints, tips and tricks. If you have any please write in and share them.