Secure Computing: Sec-C

I use FreeImage for image analysis because it supports loading dozens of different image file formats (and because I like the FIPL licensing). However, FreeImage does not support every format that I need. For example, it has problems with PNG files that use something other than 8-bit data. It also has problems with 16-bit PPM/PGM files and it does not support lossless JPEG compression. So for these formats (and a few others), I use my own rendering systems.

However, I seem to be coming across more odd file formats that FreeImage simply does not recognize. In these cases, I usually implement my own rendering system. For example, I recently encountered the Digital Imaging and Communications in Medicine (DICOM) file format. Wow... I used to think that JPEG was a horrible file format because it is inconsistent and cannot store true color. And TIFF is bad because you jump all over the file. And PDF is bad due to the complexity (objects reference objects reference objects...). But then comes along DICOM, and suddenly I have a newfound respect for JPEG.

Is there a doctor in the house?

Let's face it: hospitals have problems. Many hospitals and doctors offices still use paper records and digital records are not necessarily compatible between vendors. If you switch hospitals, then you either need to bring over your medical records or you need to start with no records on file.

So it should come as no surprise that the image file format used for CT scans, MR scans, PET, and other imaging systems uses a format that, on the surface appears complete, yet actually isn't.

Thinking Positively

With JPEG, PNG, and other file formats, the image usually comprises most of the file. However, many DICOM files store much more meta information that images. Each DICOM file includes information about the patient (name, age, gender), doctor (name, address, hospital, referring physician, etc.), and the image itself (date, time, sequence number, equipment, scanning duration, relative slice position, and much more). For the medical community, this is outstanding! You will never have a case of the image being separated from the relevant medical information.

Although DICOM is capable of storing multiple images in one file, this appears to be uncommon. Instead, each picture in a series is stored in an independent DICOM file. Each file stores one image plus all of the relevant meta data. I view this as a positive thing -- it makes it very easy to send one specific image to a coworker for a second opinion. (Since a DICOM can be a multi-megabyte file, why send a Gig of images when you can send a meg?)

Unlike PNG or JPEG, DICOM has literally thousands of pre-defined meta fields. This is beneficial since it guarantees that everyone will operate with compatible meta data. The list of meta data fields reads like a committee trying to guess about every possible need -- and covering it! For example, there is no generic "comment" field. But there are fields for Patient Comments, Image Comments, Frame Comments, Study Comments, Comments on Radiation Dose, Comments on Schedule Procedure Steps, and much more.

While DICOM overflows with detailed meta data, it totally lacks flexibility. For example, let's say that a new scanner appears on the market tomorrow. They want to be compatible, so they want to use DICOM to record their medical images. The doctor, patient, and related information is still relevant. However, there is no place to store the new, scanner-specific parameters. At this point, the vendor has three options: (1) use vendor-specific fields (incompatible with other vendors), (2) submit new fields to the Medical Imaging and Technology Alliance (committee that updates the standard almost annually), or (3) don't record the data (not useful).

Moreover, every meta data field is defined by a 4-byte numeric type. Thus, if you don't already know what 0018:9090 means, then you are out of luck -- I have yet to find a central repository that details each standard tag and type of data that goes with it.


Most DICOM seem to store 12-bit or 16-bit grayscale images. I applied a coloring algorithm.

Unnecessary Procedures

The problem with DICOM really doesn't come from the plethora of meta data. Rather, it comes from the file format itself. The basic DICOM record consists of tag-type-length-data blocks. However, they are a little more complicated.

Tags. The tags actually consist of two fields. The first two bytes define the class of data element. For example 0x0010 defines patient information, 0x0028 describes the image file format, and 0x300A seems to be for radiation therapy. The next two bytes describe the specific meta data. For example, 0028:0010 and 0028:0011 are the image width and height (called "Rows" and "Columns"), 0010:21C0 is the patient's pregnancy status, and 300A:0018 is for radiation dose reference point coordinates.

Type. Called the "Value Representation" (or VR), this two-byte field defines the type of data. Most data types are pretty standard: short string, unsigned integer, date, float, etc. The only real odd-balls are the "other" fields -- other byte (OB), other word (OW), and other float (OF). These define arbitrary data lists.

Length. This is the amount of data following the tag. Unlike JPEG or TIFF, this is actually the number of bytes not including the tag-vr-length. (Excellent! Zero actually means no data! With JPEG, no data is a length of 2.)

This seems simple enough to parse, right? Except that there are a few exceptions... For example:

• All length are 16-bits (2-bytes) long. Except for records with a VR of OB, OW, OF, sequences (SQ), unlimited text (UT), and unknown (UN). These exceptions use 4-byte lengths.
  
  
• Tag 0002:0010 defines the transfer encoding. One encoding method specifies no VR field. In this case, there may be no VR in records following this record and all lengths for undefined VR fields are 4-bytes long. Also, if you don't know the default VR for the particular record, then you cannot parse the data. (If you don't know that 0008:0092 should be a string, then you won't be able to parse the referring physician's address.)
  
  
• Assuming you can parse the lengths, some records have a length of 0xffffffff. This means that the length is unspecified and the subsequent records are nested under this record. (Just keep reading the next records.) Eventually there will be an FFFE:E0DD tag with length zero. This tag marks the end of the nested data sequence. Oh, and these undefined length records can be nested...
  
  
• Most tags only store one piece of data, but some have complex data formats. For example, text fields may use "^", "\", or "/" to separate strings. (Doctor names use "^" as in "REIL^TODD^D.^^M.D.", image type uses "\", and addresses use "/".)
  
  
• All values are stored in little endian format. Unless Tag 0002:0010 defines big endian. In that case, 16-bit image data should be processed in big endian but all other values remain in little endian.

Frankly, the data fields should be consistent. Either always specify the type of data or require the type to be known. Making the data type optional is just going to lead to parsing problems. While I'm not a big fan of hard-coded tag numbers (0028:0031 is hard-coded as the zoom factor), they could at least embed the type of data in the tag. For example, there are 27 different data types. They could reserve 5 bits in the Tag for specifying the data type, or 1 bit for indicating 2 or 4 bytes for the length.

I should also point out that you must be able to parse the entire file. Image data is almost always found at the very end. So if you cannot parse the file then you cannot find the start of the image data.

Wrong Prescriptions

So you finally found the image data itself (record 7FE0:0010). The image data is either stored in one big block (OB or OW), or in multiple fragments (frames).

Image data itself is even more complicated. For example, 0028:0100 defines the number of bits allocated per pixel data record (either 8 or 16). (I don't know why they use this when OB/OW also define the data size. But I like using 0028:0100 better.) 0028:00101 defines the number of bits stored, and 0028:0102 defines the position of the most significant bit (MSB). So, for example, they may allocate 16 bits per sample, use 12 bits, and have the MSB at position 14. This means bits 3-14 contain the data, and bits 0, 1, 2, and 15 are unused. (Unless there are overlays... yet more complexity.)

And what format is the data in? This is defined by Tag 0002:0010 (Transfer Syntax UID). The value is a sequence of numbers (that looks a lot like an SNMP MIB entry). For example, 1.2.840.10008.1.2 defines an implicit VR (VR field is removed) and the image is a bitmap in little endian. 1.2.840.10008.1.2.2 defines a bitmap in big endian. 1.2.840.10008.1.2.4.50 through 1.2.840.10008.1.2.4.93 are different JPEG formats (lossy and lossless, JPEG and JPEG-2000). 1.2.840.10008.1.2.5 uses a bitmap with run-length encoding while 1.2.840.10008.1.2.1.99 uses zlib compression. Exiftool identifies over 250 different possible values. (I currently support all JPEGs and bitmaps, but nothing else. I'll be adding RLE this weekend.)

Of course, this isn't even including the 3D image information... There's an entirely different section for storing voxel data.

Feeling Better

Consistency is always a good thing. With image formats, consistency simplifies parsing, reduces potential implementation problems, and increases compatibility. While DICOM includes an amazing number of pre-defined data fields, it lacks generic expansion and the hard-coded tag identifiers restrict the total number of possible tags (there is a finite amount of expansion). DICOM's meta fields are great for today, but could be problematic in the future.

The inconsistent file format options and overly complex 0002:0010 tag means that a parser must understand hundreds of very different variations before being able to render an arbitrary DICOM file. While I can understand the need for lossy, lossless, and 3D data storage, couldn't they decided on a single lossless method? I mean, all lossless will look the same anyway since they are lossless. So why provide a dozen different storage methods? (Alright, DICOM dates back to 1985 so some formats may be old residues. Yet none have been obsoleted. And this doesn't explain the dozens of system-specific formats like 1.2.840.10008.5.1.4.1.1.2, which is a CT Image Storage format, or 1.2.840.10008.5.1.4.1.1.1.2, which is Digital Mammography X-Ray Image Storage for presentations as opposed to 1.2.840.10008.5.1.4.1.1.1.2.1 which is the same thing for processing.)

While DICOM is very detailed, it lacks consistency in file structure, meta data format, and image format. All of the problems with the medical community have clearly been passed into their image file format. And with all the implementation complexity, medical software is bound to be more expensive. Perhaps the medical community should consult a computer scientist before creating any future image formats. I don't prescribe medicines and they should not design file formats.

I recently watched a short video titled "Patently Absurd". (Thanks to Shawn Merdinger for the pointer.) This 30-minute video discusses the harm caused by software patents and a recent Supreme Court case (Bilski v. Kappos) which will be decided soon.

The video itself isn't going to win an Oscar. (As video production and acting goes, it ranks up there with Star Wars Episode I, the Phantom Menace.) However, this documentary is about the message and not the filmography. And the message is clear: the patent system is totally screwed up.

Patent Problems

Patents were initially intended to give credit and protections to someone who develops a novel technical advancement. For example, if you developed a new farm tool that help cut costs and improve harvesting, then you should be rewarded for that good idea. The reward comes in the form of licensing fees for other people to use your idea.

Patentable ideas should be novel and distinct; someone with the same basic knowledge and skill set should not be able to trivially re-develop the same good idea. For example, if a chair with 4 legs exists, then you can't patent another 4-legged chair (not novel or distinct). And expanding to a 5-legged chair is a trivial extension, so it shouldn't be patented either.

Unfortunately, that isn't how patents are being awarded or used. Today, patents are given out to anyone who files a distinct idea. The novel aspect is no longer a requirement. For example, patent 6,368,227 is for a way to swing at a playground, and patent 6,004,596 is for a peanut butter and jelly sandwich. Neither of these are novel or non-trivial. For example, the sandwich one is for making sealed peanut butter and jelly pockets. But... Pop-Tarts are sealed jelly pockets. So isn't the inclusion of peanut butter a trivial addition?

Today, patents are used to retain a monopoly. I have an idea and I'm going to tell everyone so I can stop them from using my idea. Monopoly, extortion... call it what you want. I was once advised by a patent attorney: don't file a patent on anything you cannot afford to protect. In other words, unless you have enough money to sue everyone and their cousin for using the idea, it isn't worth filing a patent.

Big companies like HP, IBM, and Microsoft have a whole slew of patents that they use as trading cards -- each company is violating someone else's patents, so they agree to no sue each other. For example, if Nvidia ever decides to sue IBM over some patent claim, IBM will likely pull out 100 IBM patents that Nvidia is violating (with the sheer number of patents, it is hard not to violate something...) and force Nvidia to concede. In the worst case, someone will likely determine that the patents are either too vague or too trivial to be enforced. For example, Rambus sued Nvidia over some alleged patent infringement. The USPTO decided that none of the claims were an infringement. Rambus withdrew some of their allegations, and the rest are still pending.

Patenting Software

There used to be a rule that you could not patent something found in nature. And math was considered a natural system. Since programs are nothing more than applied math, you couldn't get a software patent.

However, that isn't the case anymore. Anyone can submit a software patent as a "method" for accomplishing a task. If you have ever used Photoshop then you have seen that startup window that lists dozens of patents that "protect" the software. Photoshop Elements 4.0 lists over 50 patents. (Good luck trying to look them up since you can't cut-and-paste the patent numbers and the "About" window scrolls them off the screen before you can write them down.)

For full disclosure, I have one software patent (7,296,084) and seven others pending at the USPTO (most have been pending for 7 years). However, I don't actually "own" them -- all of the rights were transferred to a corporate entity. (Most companies pay their employees a bounty for patentable ideas. So I was compensated for these.)

Today, software patents are causing a chilling effect. Software developers fear distributing code because they might infringe on some patent written in legalese. And while a patent lawsuit may have no basis, a lone software developer will likely go bankrupt defending himself.

Personally, I don't spend any time looking up patents. My rational:

1. If you know about a patent and infringe on it, then you can lose triple damages. So not knowing is cheaper.
   
2. I usually create my own code for complex systems. If I end up recreating someone else's patented system, then it clearly fails the "novel" test since someone else with the same basic knowledge and skill (me) reached the same conclusion.
   
3. If it is complex enough, then it is distinct enough to not infringe.
   
4. Most software patents are vague to the point that you cannot implement a solution based on the patent's information. (Most software patents are almost as bad as academic research papers. They leave out critical steps and details needed to validate or implement the system.) Software patents usually contain very little useful information.
   
5. Most software patents were written by patent attorneys. Their legalese is worse than contract lawyer legalese. Unless you are a patent attorney and technical, you probably won't understand the patent.

Frankly, I'm more concerned with software licensing than patents. (I believe that GPL is evil since it dictates distribution requirements.)

Finally, I don't like how patents require public disclosure. If I don't want someone else to copy my work, then why would I release details of my work publicly? Instead, I use security-by-obscurity, and the knowledge that anyone copying my work will violate copyright laws and constantly be playing catch-up to me. If someone tries to recreate my software, then they will likely be at least two major revisions behind the current development cycle. As long as they play catch-up, I will always be the leader.

In contrast to patents, I am a huge fan of copyright law. And plagiarism should be a death-penalty offense.

The Library of Congress recently announced an ambitious archiving project. Every posting on twitter will be collected and archived forever. I still haven't decided whether this is a good thing or not.

The Benefits

There are some seriously positive benefits for archiving this information. For example, people use twitter to post their immediate thoughts and activities. This isn't just a snapshot of American life, this is a snapshot of the world! If we ask what life was like 100, 200, or even 500 years ago, we need to estimate based on the available materials. Few people wrote down their daily actions and thoughts. Writing was a time consuming hobby. And it was expensive -- paper wasn't free.

With Twitter, people constantly post the most menial things. But those little details are the most enlightening. In 500 years, the question won't be "what was life like back in 2010?" There are detailed histories from regular people all over the world. They talk about daily life, common activities, desires, and even opinions about political and environmental topics.

Twitter also permits researchers to follow trends in vocabulary, observe the impact of technologies, and even watch how news and events propagate through society. Twitter is an information goldmine.

The Downside

While there is a definite research benefit, it comes at a price of privacy. We have become a world that never forgets. Frankly, not everything done in public needs to be recorded forever. For example, a quiet conversation between friends -- even if made in public -- may still have an expectation of privacy. And just because I step outside my house does not mean someone should record my every step. Although the Internet operates with a very public profile, not every public comment needs to be recorded as if the paparazzi found everyone important. Many people post things to Twitter that they would never have written if they thought it would be archived for posterity.

Today, there are people who find that items posted to public forums have very serious consequences. Like the woman who was denied a teaching certificate because of a MySpace picture. Students have been expelled over Facebook comments, and two people were arrested for directing protesters via Twitter.

Imagine what people will find if they look more closely at archived tweets... Old-school politicians only needed to worry about items recorded in the media. But we now have an entire generation that is living online. That innocent tweet you made when you were 12 could be the reason you lose your run for governor when you turned 34. And that snide remark posted when you were 14 could be why you were passed on a promotion when you turned 28, were laid off when you turned 29, and can't find a new job.

Roach Motel

The Library of Congress is supposed to have a copy of every book that has been publish in the United States. Their American Memory Project includes detailed interviews of everyday people; recording current thoughts, fears, and daily life for posterity. And the National Digital Preservation Project has been archiving the Web since December of 2000.

However, there is a big difference between these projects and Twitter. When a book is published, there is an expectation that people will read it. There is an expectation that it will be archived. When a web page is published, it is published for the world; most web masters hope that Google will index it. Cached or archived web pages are the norm. In fact, the Internet Archive's Wayback Machine has been mirroring sites since 1996. And while the American Memory Project records detailed snapshots of real people, the people are interviewed for the purpose of being archived.

But what about Twitter? These people weren't interviewed. And while it is a public forum, there hasn't been an expectation that someone would archive and datamine all tweets forever. Moreover, Twitter is used by people world-wide, and not just in the United States. Did international people have any expectation for privacy -- even on public tweets?

Where does it end?

Besides posting text, many people tweet pictures. Will those also be archived?

I've been periodically downloading pictures from Twitpic. I've probably archived close to 100,000 pictures. I'm doing it because I want unmodified samples from digital cameras (great for photo ballistics). I've also noticed certain trends. For example, I know what type of pictures to expect depending on the time of day. I can actually associate some camera models with certain regional and social economic classes, I know who is the target customer for an iPhone, and I know the most common thing people photograph: their food. It is really weird... it doesn't have to be a great presentation at an expensive restaurant. It is usually just "here's what I'm about to eat for lunch".

Besides food, people also post semi- and fully-exposed self photographs. While it is stupid to take those kinds of pictures, and really stupid to post them in a public forum, should their stupidity be archived forever? (Not in my archives. I deleted all amateur porn that I download from Twitpic. I just need camera samples and I prefer ones that are workplace safe. Besides, most of these people are really not as attractive as they think they are.)

I recently had a request to create an API for an application that I wrote a few years ago. Basically, one of my associates wanted to create his own functions for use with the application. Rather than hacking the code, it was clearly a better idea to create plugin modules. Since I use a modular programming style, converting the modules to plugins really wasn't much of a conceptual difference or coding change. The big issue was the linker...

Basic Plugins

Plugins are actually shared object libraries (.so files under Linux). They are loaded using dlopen() and functions are found using dlsym(). For simplicity, I made every plugin have the same function names. "Init()" performs any one-time preprocessing, Destroy() closes it down, and Run() performs the actual action. (With these plugins, they all take the same parameters and return the same format data.) The basic structure for managing them looks like:

struct plugin

  {

  void *handle; // pointer returned by dlopen()

  int (*Init)(...); // the init function returns an integer

  void (*Destroy)(...);

  float (*Run)(float p1, int p2);

  struct plugin *next; // linked list of plugins

  };

...

  struct plugin *P;

  // allocate P

  P->handle = dlopen("/path/plugin1.so",RTLD_LAZY);

  if (!P) { ... } // handle the error

  else

    {

    P->Init = (int(*)(...))dlsym(P->handle,"Init");

    P->Destroy = (void(*)(...))dlsym(P->handle,"Destroy");

    P->Run = (float(*)(float p1, int p2))dlsym(P->handle,"Run");

    }

  // check for errors (if the function pointer is null)

  P->Init(); // run the init function

The problem really isn't creating the plugin. The problem is compiling it. There are plenty of tutorials online, but they seem to glance over the different parameters or working examples. And many of the examples that I stumbled across didn't work.

Naming

Compiling and linking the plugins were not as simple as running gcc or g++. The big caveats concern naming and linking.

The dlsym() function takes a handle to the shared library and the name of the function to load. This isn't a problem when using gcc, but g++ mangles names. The actual function name in the library is a combination of the name and data types (this is how C++ distinguishes between different prototypes). So your function called "Run" is likely named something like _Z3Runfi.

There are two ways to resolve the naming issues. First, you can run 'nm' on the shared library to list the function names. The alternative is to export the function name as a "C" name (not C++).

extern "C" int Run(float p1, int p2);

or

extern "C" {

int Run(float p1, int p2);

}

There are other options for defining C (or "flat") naming, but these options seemed the simplest to me. You don't need to define everything this way -- just the prototypes that you plan to link.

Linking

Compiling the plugins is not too complicated:

g++ -O2 -Wall -fPIC -shared -Wl,-E -o file.so file.c

This works for both Linux and Mac. The -fPIC option creates position independent code, -shared makes a shared library, and -Wl,-E passes the -E option to the linker; ld -E says to export dynamic symbols for use with dlsym().

If your plugins are standalone, then there is nothing fancy for the main program beyond linking with "g++ -ldl". libdl.a provides dlopen, dlsym, and related dynamic functions.

For me, compiling is a little more difficult since my plugins needed to use functions provided by the main program. This means that the main program must export symbols:

g++ -O2 -Wall -fPIC -Wl,-E -o main main.c -ldl

Normally I use 'strip' to shrink my programs. However, strip will remove exported symbols. If you want to strip the code, use 'strip -x' to leave global exports.

Of course, my code needs to support both Linux and Mac. Here's the equivalent options for the Mac:

g++ -O2 -Wall -fPIC -arch ppc -arch i386 -bundle -flat_namespace -undefined suppress file.so file.c

g++ -O2 -Wall -fPIC -arch ppc -arch i386 -Wl,-exported_symbols_list,exportlist.txt -o main main.c

Unlike Linux, the Mac linker needs a list of functions to export (the exportlist.txt file). Also, I use "-arch ppc -arch i386" to create a universal binary. If you only need to support one platform, then you can leave this out.

Getting Really Complicated

I never seems to be content with the simple solution. If I'm moving functions into dynamic libraries, then I'm going to move lots of functions into libraries. And since my program has a configuration file, I want the file to specify where the library is located.

Suddenly things get complicated. You see, if I just link to the shared library, then the location becomes static; it must be located in the shared library path, but that isn't controlled by my configuration file. Also, the linked executable wants to load all required functions when it starts, but I need the program to read its configuration file first. The program needs to run with unresolved symbols.

Thankfully, Valdis Kletnieks gave me a great hint toward the solution. Here's how to compile the main program:

g++ -O2 -Wall -fPIC -Wl,-E -Wl,--unresolved-symbols=ignore-in-object-files main main.c -ldl

This tells the linker to ignore functions that cannot be resolved when the program starts. If your program calls an unlinked function, then it will abort (or crash). So the first thing main.c does is process the configuration file and use dlopen() to load the correct shared library. I don't even need to use dlsym() -- the linker is smart enough to link as-needed as long as the functions are available.

Doing the same thing on the Mac is a little more complicated:

g++ -O2 -Wall -fPIC -arch ppc -arch i386 -Wl,-exported_symbols_list,exportlist.txt \

-Wl,-undefined,dynamic_lookup -o main main.c

With Linux, static libraries are immediately linked during compilation, shared libraries are loaded when the program starts, but unresolved functions will only be resolved when you call dlopen().

With the Mac, I found that fewer shared libraries are loaded (D'oh!). You may need to use dlopen() to load additional dynamic libraries (dylib).

More to come...

Fortunately, I have not needed to support Windows systems. As I understand it, dlopen/dlsym exist but with different names, and the compiler options are different.

I try to keep a close eye on my web logs. They tell me things like when I'm under attack, when spammers are going to strike, and what topics are popular. (Although this blog is a writing exercise for me, I always appreciate feedback and conversations.)

Proactive Defense

As someone in the computer security field, I am occasionally a target for noob hackers. They think it would be a coup to compromise any site run by a security professional. I don't try to kid myself: I don't keep anything valuable on this web site. (It's hosted at GoDaddy and not on my own servers. So how sensitive could it be?) If/when this site is compromised, it probably won't be more than a defacement, and certainly won't compromise any of my sensitive information.

Having said that, this site does have a custom intrusion prevention system. Since the only significant threat (that is within my control) comes from the CGI interface, my tools watch for potential attacks. If it sees something that looks like a real threat, it is blocked before it ever reaches the actual server-side scripts. A more serious threat results in a ban -- either the network address or other attributes from the attacker are blocked. Not only is the attack prevented, future attacks are also blocked.

(Other threats, such as remote logins or server patches, are outside of my control. This site is hosted at GoDaddy, after all. I'm trusting them to properly administer the system.)

My attack-detect-and-block scripts current ban about a dozen clients per month. That's actually a fairly average/low volume. Also, most of the bans come from manual attacks, not automated scanners. (There are ways to tell.) If this site were a bank, government web site, or major online presence (e.g., Google, Facebook, or Twitter), then I'd expect many more attacks per month.

Right now, I leave banned entries for a few months. (I manually clear them out every now and then.)

Obnoxious Users

A couple of regular users have been blocked from accessing this site. Two people repeatedly posted offensive messages to the comment form. (Swearing with off-topic comments never get past the moderator.) A third person decided to mirror my web site weekly. But a few other people have been blocked for RSS abuse. For a blog that is updated 1-2 times per week, there is no reason to request an RSS update every 2 minutes. (Google doesn't even rescan this site that often!)

Frankly, I don't know any RSS feeds to need to be polled more often than every 15 minutes. Regular blogs rarely get new content less than hourly. Some sites, like Google News, USA Today, and Oh No They Didn't, might update more often, but even these high-volume feeds don't update every minute. The only things that update more often are real-time feeds or microblogs like Twitter. But there is no real reason to refresh an RSS feed more than once every few hours for something other than a real-time/micro blog.

So what's the harm in refreshing more often? Bandwidth, cost, and server load. If just one guy refreshes the RSS link every minute, then there is no serious damage. But with just a few dozen people, the requests begin to add up. The result is a higher server load, less available bandwidth, and fewer available connections at any given time. There is also the issue of cost. While most hosting providers give you a huge bandwidth allocation, nearly all still track bandwidth usage. If you go over bandwidth, the hosting site will either block your site (out of bandwidth) or charge you extra to cover the hosting costs. Either way, rapid refreshes does have a negative impact.

With RSS, there is no method for informing users that they are refreshing too often. The only options are blocking or permitting. Personally, I don't mind blocking people who automatically refresh every few minutes. They aren't paying attention to the feed results (no users watch their RSS streams 24-hours a day) and they are probably not even aware of the abuse they are causing. If the user isn't conscious enough to configure their RSS reader to refresh at a reasonable rate, then this blog is probably too technical for them anyway.

Blocking Subnets

While I have no problem blocking individuals, it really takes a lot for me to consider blocking access from an entire subnet. Currently, I am blocking two Class-B subnets and one Class-C: two in China and one in Russia.

The two Class-B subnets are both in China and affiliated with comment-form spam. I noticed that scans from 60.177.0.0/16 were always followed by spam-comments posted by a human at 218.240.0.0/16. I also noticed that these subnets have never been used by any user who "just visits" my site. Thus, blocking these subnets ended up stopping all of the comment-form spam and does not appear to impact any real humans.

The Russian subnet 95.142.46.0/24 appears to only be associated with referrer spam. They post a bunch of referrer links to everything from scam sites to porn. Some of the sites also host malware. The attack assumes that the victim will be reviewing their logs and click on a link in order to see who is talking about them. This subnet began generating more traffic to my site than Google. (And anyone who reviews web logs knows that Google is usually the highest volume visitor.) Blocking them cut web traffic to my site in half.

Undo

The biggest problem isn't determining who to block. Rather, the problem is determining when to unblock. If I block forever, then eventually the network address will be reassigned and some innocent user will be assigned these blocked addresses. They will end up being unfairly blocked from this site. However, unblocking too soon results in more of the abuse that initially lead to the block. I'm still working on a good way to determine when it is safe to unblock an offensive client.