The Hacker Factor Blog

I recently came across an odd problem. One of the shell scripts that I wrote, and which works fine under RedHat, Fedora, and Ubuntu's Dapper Drake, was failing under Ubuntu Edge Eft and later versions. The problem turned out to be the shell itself!

My script was written to use the Bourne Shell (/bin/sh), but had been using the Bourne-Again Shell (bash). This is because, under most Linux distributions, /bin/sh is a symbolic link to /bin/bash.

However, there was a fairly recent and subtle change that I did not notice. Under the Debian Linux, /bin/sh is required to be a POSIX shell, but not the Bourne shell. (Slight difference in functionality.) Bash is POSIX compliant, but contains many additional features that are not required by POSIX. For example, Bash supports command-line editing. As a result, Bash has bloated in size; it consumes over 650 Kb. This bloat is costly since every start-up script, system script, and user terminal needs to run an instance of the shell.

An associate of mine, who is a Debian developer, educated me. Debian replaced the default /bin/sh; they switched from Bash to a lighter version of the POSIX shell called the Debian Almquist Shell (/bin/dash). (Dash is a variant of ash, a POSIX compliant shell that is a variant of the Bourne shell.) Although Debian made this change a few years ago, Ubuntu -- based on Debian -- did not do the change until Edge Eft. The reason I did not notice is that all of my Linux systems either run RedHat or Dapper Drake.

While both Bash and Dash are POSIX compliant, Bash has more additional features. The switch to the minimal Dash shell ended up breaking lots of scripts (including mine) that used Bash-specific features. For example, Dash does not have strong regular expression support for variables. Conventions like "${!MODE_*}" and "${MODEVAR//_/\/}" work fine under Bash but not under Dash. In addition, compiled programs that use system() and popen() calls to run shell scripts may see a problem. These function calls run "/bin/sh" so the code may be sent to Bash or Dash, depending on the host operating system.

There are a couple of mitigation options if you find yourself in this situation, with a shell script that stops working on newer Ubuntu systems:

• You can replace the default shell with Bash:
  sudo ln -sf /bin/bash /bin/sh
  This shouldn't break anything, but will slow down boot times. It will also make your system non-standard so keep an eye out for any future patches or updates that attempt to re-link /bin/sh to dash.
  
  
• Change your script. If your script needs Bash, it should use "#!/bin/bash" instead of "#!/bin/sh". (I'm changing all of my scripts so they no longer depend on "/bin/sh" -- they will explicitly state the shell they need.)

I hope that the shell becomes more standardized someday. As a programmer, I don't want to worry about which shell is being executed when I run "/bin/sh". Right now, every Linux and BSD distribution seems to use a different shell for "/bin/sh". When you use /bin/sh, you may end up running bash, dash, ash, jsh, ksh, psh, or some other variation of the Bourne shell.

The Internet Storm Center (ISC) is continuing their month of security tips. Today's tip concerns E-mail (PGP, Attachments, etc), IM, IRC. These protocols are widely used, but they are not necessarily secure. The question posed by handler John Bambenek was this: "how can those who use [these services] do so safely?"

Here's my reply:

There is always a distinction between "internal" and "external" services. Every big company that I have worked for has had one or more internal IRC/Jabber servers. Employees may be anywhere in the world, but real-time communications like IRC can really improve collaboration.

As an example, one of my current projects uses an internal IRC channel to communicate. For very complicated issues or detailed explanations, we pick up the phone. But it is wonderful to be able to say, "the SQL query is not working for me. Can you paste your code?" and poof it appears in the IRC window.

When I run my own multi-person projects, I always setup an internal IRC network. This provides a logging history, a message-board, and a quick way to communicate with people.

Even when people are in the same building, IM/IRC/Jabber can be more efficient. Rather than wandering over to someone's cubicle and seeing that they are not there, you can just type into the channel: "Hey Marc, are you there?" or "marc: ping". If you get a reply then you can wander over and not waste time.

While the greatest risks to a company come from internal sources (people), internal resources (systems) are usually very safe. (At least, the risk is no worse than the rest of the internal network.)

However, while I trust internal IM/IRC/Jabber, there are things that I would never do on a external IM/IRC/Jabber system -- even in a private channel. Basically, there is no authentication (okay, Jabber does but IRC does not by itself) and no guarantee that someone (the admin or sniffer) is not recording a private conversation. Also, IRC exposes your IP address, making you a target for an attack. There are some registration systems for IRC (so you can register your nick), but you know nothing about the people running the registration system. These are not really issues for internal systems, but external systems pose very high risks.

Email is a different issue -- it is trivial to do impersonations. Even with internal environments, I always double-check the email headers when the contents appear unusual. For example, if I am asked to do something outside of my work scope, I'll validate that it is real. Fortunately, email is usually associated with a continuing concept. If the email's contents are not related to a known topic, then you can quickly identify it as being suspicious. I do know a few non-techies who take email at face value, but most of the people I work with question things when the content seems out of scope.

When it comes to email, I treat internal just like external. No HTML, show headers, and never blindly open any attachments. I trust external IRC more than internal email. In my opinion, corporate email only serves one primary purpose: it is a paper trail and a CYA. Everything else, including collaboration and basic workplace messages, is a secondary purpose.

As with any technology, nothing is inherently evil. (Well, maybe the guillotine...) IM, IRC, and Email do have legitimate workplace uses, and can be used safely. However, internal usage is not the same as external and external use poses much greater risks.

In my "Point-of-Sale Vulnerabilities" paper, I criticized the payment card industry for doing too little too late. For example, why did it take the industry decades to decided that default passwords should be changed? The computer industry has widely known for decades, but the PCI DSS did not require this until 2004.

However, now it is time to criticize the Transportation Security Administration (TSA) for their lack of common sense. According to a report at USA Today, "TSA ran its background check after someone started working at an airport. It would order ID cards revoked for those found to be problematic." (WHAT?!?) And better yet, now TSA is complaining that the new rule (background checks before IDs) is taking too long so they cannot hire people fast enough.

So let's make sure we get this straight... Before 1-Oct-2007, a known terrorist could get a job doing airport security and would have full access to restricted areas -- until the security clearance comes back and says "denied". Then, and only then, would TSA take away the badge and restrict access. This is like giving a new and unknown user your root password, and then taking it away if you find out that they are not qualified to be an administrator on the system.

In computers, removing privileged access from an unprivileged user does not remove all of the risks. You still risk backdoors, trojans, and worse; you can never be certain that the person is out of the system. Now, consider TSA: if it takes a background check one month to find the person unqualified, and the person knows they will be fired, then they still have a month to do damage. Moreover, they know they can come back a month later at a different airport using a fake name and still be certain that they will have a month of access.

TSA has historically suffered from a high turnover rate. I cannot help but wonder if this is because unqualified people were being kicked out after being denied clearance. TSA should release metrics about the percent of employees denied clearance after being given access to restricted locations. I'd also be curious if some airports have a higher denial rate. For example, Los Angeles, Chicago, New York, and Washington DC have airports that have been targeted by terrorists. Do they have a higher percentage of people denied clearance?

And then there are the issues with stolen TSA badges and uniforms. According to this report:

"If you have a badge and a uniform," [terrorist prevention expert Saul] Wilen explains, "you are invincible in terms of the system. Not only can you get in and get around, you can get known and become a regular that becomes more and more recognized, so the next time you are less liable to have to go through the system's security, and the next time even less."

This means that the TSA decision to not grant access until after issuing a badge and a uniform effectively gave people considered national security risks direct access to airports. These same people could easily steal badges and uniforms, and could do this over and over using fictitious names.

The current complaint from TSA is the amount of time it takes to complete a clearance check. But how long does it take to get clearance? According to TAOnline, it usually takes 1 to 3 months. To put things in context, that's about how long it takes to get a passport. And considering that passports are required when traveling through an airport (international travel), why should it not be required for someone working at an airport? Yes: the process is slow. No: it is not unexpected.

For an organization with "Security" literally as their middle name, they sure appear to know nothing about security. They have released documents that exposed the names of DHS employees, employed poor web practices that look like phishing sites, lost badges, lost uniforms, failed to identify 75% of bombs in a test, and gave restricted access to people without clearance. This should not be happening in a post-9/11 environment.

With all of these problems at TSA, it is no wonder airports are no more secure than before 9/11. It is also no wonder airlines are overreacting to things like t-shirts, praying (jews and muslims), and babies saying, "Bye bye plane".

Perhaps it is time for TSA to take a more proactive step. Security is a process and the current process is not working. Perhaps it is time for people to step down. (I'd blame Kip Hawley, but he was appointed by the White House.)

By the way, has anyone else noticed that the people responsible for managing security at TSA (Kip Hawley and Stephanie Rowe) used to be consultants?


Thanks to John Bambenek for the blog title.

Today's media is far more interested in sensationalism than covering important topics. Unfortunately, there is no limit to the number of media whores who are more than eager to provide baseless metrics in order to gain attention.

Before I get much further, I should temper my rant... While I have been interviewed by members of the media before, I know that some of them seriously twist what they are told and are only interested in sensationalistic tangents. Many of these myopic reports hyperfocus on asides and information taken out of context. However, people who post unsubstantiated metrics are also responsible for sensationalizing minutia.

Recently Slashdot featured a news report that sensationalized a single finding from a paper published by a company called Commtouch.

Just to make sure I am saying this correctly: Slashdot featured a news report on a paper, rather than the paper itself. And the news report focused on one unsubstantiated metric in the entire 13-page paper. Much of the paper is interesting, but not scientific. This particular metric is mentioned in a summary side-bar bullet on page 1, and in one sentence on page 10.

The news article from 16-Oct-2007 has the sensational title "Spam reaches all-time high of 95% of all email" and states that Commtouch announced that "Global spam levels reached an all-time high of 95% of all emails at its peak during the quarter." It is true that the paper says this on page 10, but there are many aspects that make it un-newsworthy.

95% Fake

"News" implies that the information is "new". This is not the case with the reported claim that 95% of email is spam.

• 27-July-2006, "More than 95% of e-mail is 'junk'". This BBC report says, "More than 95% of e-mail is junk, be it spam, error messages or viruses, report mail monitoring firms." This is attributed to e-mail security firm Return Path.
  
  
• 27-Sept-2007, "95% of email is now spam and it’s still rising". Network monitoring, messaging, and managed file transfer systems developer Ipswitch says that 95% of email is now spam.
  
  
• 24-May-2007, "People are less bothered by spam; E-mails continue, but filters lessen annoyance factor" (and here). According to this news report, "Various studies show that 80 to 95 percent of all e-mail is now spam."
  
  
• 24-Aug-2007, "Spam fighters hit criminals' weak spot". Greenfield's IT department measures 96 (ninety-six) percent of email is spam.

So, here are four news reports that show the same findings, as much as a year earlier. This is not new, not newsworthy, and not even reported well in this latest instantiation.

Comparing Spam with Spam

The second issue is about the Commtouch report. They don't describe their collection methods, don't identify any biases in their data, and don't identify what they measure. These are the same issues that I wrote about in Laptop Losses and Phishing Fruit Salad. In effect, these numbers are bogus since Commtouch does not identify what they are measuring or how they are measuring it.

On the other hand, I can compare spam volumes received in various archives. For example:
2007-01 180 (180 spam emails received in January 2007)
2007-02 215
2007-03 273
2007-04 193
2007-05 271
2007-06 345
2007-07 237
2007-08 158
2007-09 135
2007-10 198
According to this single email account, received spam volume dropped in Q3. This archive only includes one email account that was harvested by spammers in 2003 and 2004. This email address has never been used to send email, never enrolled in any services (beyond the ISP where the account was harvested), and is not listed on any web page. However, this account may not be representative of spam in general.

As a comparison, I can look at the news.admin.net-abuse.sightings (NANAS) archive that I've been mirroring.
2007-01 50,722
2007-02 61,632
2007-03 75,996
2007-04 61,962
2007-05 59,698
2007-06 66,051
2007-07 93,766
2007-08 98,667
2007-09 98,723
NANAS is a collection of spam samples submitted by people all over the Internet. The archive only represents emails that people submitted, and people do not always submit everything that they have. In particular, people may not submit multiple samples of the same kind of spam. However, the total volume of email in NANAS is usually representative of the total volume of spam. In this case, NANAS volume variance for Q1 matches my sample account. NANAS slightly differs in Q2, and shows a significant increase in email during Q3.

What does this indicate? First, my individual account is probably not representative of spam in general. In fact, the graphs provided in the Commtouch report do not even match the variations reported in NANAS. This means that they are all measuring different things. Considering that the graph of NANAS's volume looks similar to graphs put out by Kaspersky Lab, Spamcop, and other anti-spam vendors, NANAS is likely very representative of world-wide spam volume (and Commtouch is not representative of spam in general).

More importantly, spam has been increasingly targeted toward specific markets. People in America rarely receive Chinese spam, and people in Germany (emails ending with ".de") don't receive phish for Bank of America. Since my sample account has virtually no demographics associated with it, it is likely to see fewer spam emails.

Comparing Spam with Email

All of these metrics measure total spam volume, but not spam in relation to desirable emails. NANAS and my sample account both have "100% spam". My personal email account is more like 99% real email (and I don't use a spam filter), but I have seen a slight increase in spam this month (from 2 a day to 5 a day).

When someone says that "95% of email is spam", we need to ask "why?" Is spam increasing, or is email use decreasing? The 'younger' generation is leaning toward IM and text messaging and moving away from email. If fewer people use email, then the total amount of desirable email will drop. If the spam volume stays constant, then it will increase in percentage.

The metrics by Commtouch do not say how many emails were desirable and how many were spam; they only give a relative percentage. Moreover, Commtouch does not identify how they determine whether an email is spam or not. Do they base this on some filtering software? If so, then which software, how accurate is the software, how large is their sample, and how was the sample collected?

Although different companies measure different things, spam has always been between 75% and 95% of total email (depending on the metrics you follow). However, spam volume has definitely increased in Q3 -- and more than just a little. (NANAS, Kaspersky, and Spamcop all show similar increases). So if spam volume increased by around 30%, but total percent of email just hit 95%, then why did the total percent of spam not increase more? Was there a large increase in desirable email to help balance out the metrics? Did total spam volume need to increase 30% in order to cross the 95%-total boundary? Or are spam filters keeping the actual numbers much lower?

The report by Commtouch and associated news coverage are sensationalist, but they are not new, not newsworthy, and not even complete enough to tell what they are reporting on. 95% spam? How about "95% bunk".

I rarely reboot my computer. Uptimes of three or more months is not uncommon, and I've had one computer stay up for more than two years (including through a UPS battery change). However, I needed to reboot my Ubuntu system.

This computer had been up for over four months. However, following a software upgrade (apt-get update ; apt-get upgrade), the mouse stopped working and Gnome stopped too. (Probably had to do with an upgrade to Gnome and my bad decision not to reboot.) However, after rebooting, the X-Windows server refused to start.

I checked the xorg.conf file, and it had not changed. It was complaining that the nVidia drivers would not load. My video card is a nVidia GeForce4 MX 4000 AGP 8x. This uses the "nvidia-glx-legacy" driver. I uninstalled and reinstalled the nVidia driver, thinking that the problem was due to some change that happened during the upgrade. However, that did not fix the problem.

Then I made a big mistake that cost me a few hours this evening. I went to Google and looked up terms like "nvidia ubuntu fails" and "nvidia geforce4 mx 4000 dapper". My idea was to find other people who had this same problem, and use whatever solution worked for them.

Yes: there are plenty of people who report the exact same problem. Following an upgrade, the nVidia driver stops working. The solutions ranged from reinstalling the nvidia driver (did not work) to recompiling the kernel (you've got to be kidding me). A few things people swore to did not work for me:

• Changing xorg.conf to use Driver "nv" instead of Driver "nvidia" did get the video working, but I lost GLX support. Since my favorite game requires GLX, this is not a viable solution.
  
  One person posted a recommendation to change the driver to "legacy" or "glx", yet there is no video driver on my system with these names. (He was probably just making this up.)
  
  If I had thought about it, I would not have even touched my xorg.conf. It was reporting the problem, but it was never changed by the upgrade so it wouldn't be the problem.
  
  
• A few people recommended downloading older drivers from nVidia. I tried this, but I could never get them to install. (There are a ton of dependencies for just getting past the compiling phase. I got past that, but it would not load the new drivers.)
  
  Again, if I had l thought about it, I would have realized that Ubuntu provides great package support. I should never need to download and compile drivers that previously existed. I mean, there are drivers for some really old hardware, and if old hardware is supported then the "barely old" GeForce4 should be supported.

One of the problems with open source is that the people who provide solutions frequently try to solve problems that they have never experienced. And people who do solve the problem frequently forget to include all of the necessary steps. Many of the tips I came across said things like "you will need to install some additional packages to make this work." Great -- which packages? Moreover, some solutions are not reproducable, or are far more complicated than necessary.

I finally found the problem by simply looking at the error in /var/log/Xorg.0.log. It said that the "nvidia" driver could not be found. So I went looking for it:
find /lib -name nvidia* | sort
There should be two sets of drivers per kernel. One is found in /lib/modules/version and the other is in /lib/linux-restricted-modules/version. I noticed that some of the installed kernels did not have restricted modules.

Now I know the problem: even though apt-get upgrade installed a new kernel, it did not bring in the restricted modules. When I rebooted, the new kernel was used, but the proprietary nVidia drivers were not installed so X-Windows could not run.

The solution: install the restricted drivers that go with the kernel.
apt-get install linux-restricted-modules-`uname -r`
Then reboot and everything will be happy.

The instructions for installing the drivers for nVidia and ATI video cards are in my book, Hacking Ubuntu. If the publisher asks me to update the book for Hardy Heron (the next long-term support version), then I'll be sure to add in a note about restricted drivers and kernel upgrades.