I try to keep a close eye on my web logs. They tell me things like when I'm under attack, when spammers are going to strike, and what topics are popular. (Although this blog is a
writing exercise for me, I always appreciate feedback and conversations.)
Proactive Defense
As someone in the computer security field, I am occasionally a target for noob hackers. They think it would be a coup to compromise any site run by a security professional. I don't try to kid myself: I don't keep anything valuable on this web site. (It's hosted at GoDaddy and not on my own servers. So how sensitive could it be?) If/when this site is compromised, it probably won't be more than a defacement, and certainly won't compromise any of my sensitive information.
Having said that, this site does have a
custom intrusion prevention system. Since the only significant threat (that is within my control) comes from the CGI interface, my tools watch for potential attacks. If it sees something that looks like a real threat, it is blocked before it ever reaches the actual server-side scripts. A more serious threat results in a ban -- either the network address or other attributes from the attacker are blocked. Not only is the attack prevented, future attacks are also blocked.
(Other threats, such as remote logins or server patches, are outside of my control. This site is hosted at GoDaddy, after all. I'm trusting them to properly administer the system.)
My attack-detect-and-block scripts current ban about a dozen clients per month. That's actually a fairly average/low volume. Also, most of the bans come from manual attacks, not automated scanners. (There are ways to tell.) If this site were a bank, government web site, or major online presence (e.g., Google, Facebook, or Twitter), then I'd expect many more attacks per month.
Right now, I leave banned entries for a few months. (I manually clear them out every now and then.)
Obnoxious Users
A couple of regular users have been blocked from accessing this site. Two people repeatedly posted offensive messages to the comment form. (Swearing with off-topic comments never get past the moderator.) A third person decided to mirror my web site weekly. But a few other people have been blocked for RSS abuse. For a blog that is updated 1-2 times per week, there is no reason to request an RSS update every 2 minutes. (Google doesn't even rescan this site that often!)
Frankly, I don't know any RSS feeds to need to be polled more often than every 15 minutes. Regular blogs rarely get new content less than hourly. Some sites, like Google News,
USA Today, and
Oh No They Didn't, might update more often, but even these high-volume feeds don't update every minute. The only things that update more often are real-time feeds or microblogs like Twitter. But there is no real reason to refresh an RSS feed more than once every few hours for something other than a real-time/micro blog.
So what's the harm in refreshing more often? Bandwidth, cost, and server load. If just one guy refreshes the RSS link every minute, then there is no serious damage. But with just a few dozen people, the requests begin to add up. The result is a higher server load, less available bandwidth, and fewer available connections at any given time. There is also the issue of cost. While most hosting providers give you a huge bandwidth allocation, nearly all still track bandwidth usage. If you go over bandwidth, the hosting site will either block your site (out of bandwidth) or charge you extra to cover the hosting costs. Either way, rapid refreshes does have a negative impact.
With RSS, there is no method for informing users that they are refreshing too often. The only options are blocking or permitting. Personally, I don't mind blocking people who automatically refresh every few minutes. They aren't paying attention to the feed results (no users watch their RSS streams 24-hours a day) and they are probably not even aware of the abuse they are causing. If the user isn't conscious enough to configure their RSS reader to refresh at a reasonable rate, then this blog is probably too technical for them anyway.
Blocking Subnets
While I have no problem blocking individuals, it really takes a lot for me to consider blocking access from an entire subnet. Currently, I am blocking two Class-B subnets and one Class-C: two in China and one in Russia.
The two Class-B subnets are both in China and affiliated with comment-form spam. I noticed that scans from 60.177.0.0/16 were always followed by spam-comments posted by a human at 218.240.0.0/16. I also noticed that these subnets have never been used by any user who "just visits" my site. Thus, blocking these subnets ended up stopping all of the comment-form spam and does not appear to impact any real humans.
The Russian subnet 95.142.46.0/24 appears to only be associated with referrer spam. They post a bunch of referrer links to everything from scam sites to porn. Some of the sites also host malware. The attack assumes that the victim will be reviewing their logs and click on a link in order to see who is talking about them. This subnet began generating more traffic to my site than Google. (And anyone who reviews web logs knows that Google is usually the highest volume visitor.) Blocking them cut web traffic to my site in half.
Undo
The biggest problem isn't determining who to block. Rather, the problem is determining when to unblock. If I block forever, then eventually the network address will be reassigned and some innocent user will be assigned these blocked addresses. They will end up being unfairly blocked from this site. However, unblocking too soon results in more of the abuse that initially lead to the block. I'm still working on a good way to determine when it is safe to unblock an offensive client.
