The Hacker Factor Blog

Google Abuse

Dr. Neal Krawetz — Sun, 13 May 2012 10:47:13 -0700

I watch my web logs closely. I'm basically looking for possible attacks, unexpected errors, and other forms of abuse. If the error is on the server side, I want to fix it as fast as possible. If the error is from the user, then I want to understand and address the problem.

The Joys of HTML

One of the most common problems that I have been seeing at the FotoForensics site concerns the upload-by-URL option. Users are supposed to paste in a URL to an image, and the site will retrieve and analyze the picture.

Unfortunately, many users don't understand the difference between a URL to a picture and a URL to a web page that contains a picture. I'm mostly seeing this problem with URLs from other hosting sites (e.g., flickr or imgur) and from Facebook. These web pages contain one big picture (likely the one the user wanted to analyze) and a bunch of smaller pictures in the header, along the margin, and at the bottom.

I finally came up with a solution. If the user uploads HTML, then the site now displays a small notice about right-clicking on the image to find the URL to the image. I thought it was kind of wordy, but I've already seen users upload HTML, pause, and then resend the URL to the big picture on the web page. So this looks like a fast fix via a better error message.

After I get a little more infrastructure implemented on the back-end of the service, I'll release another solution to this problem which will be much more convenient. Hopefully it will be released by the end of this month.

Slammed By Google

One of the reasons I looked into the "uploading HTML" issue is that I saw a huge increase in uploads making this mistake. Basically, I had one user who was uploading tons of pictures from imgur, but was specifying the URL to the HTML page and not to the images.

Even after I put up the message telling users how to find the image URL, I still had this one user uploading imgur HTML. The uploads happened every few seconds. So it looks like a bot. And then I noticed the IP address and user-agent: Google.

66.249.72.152 - - [13/May/2012:06:25:17 -0500] "GET /upload-url.php?url=http%3A%2F%2Fimgur.com%2F3iCJ8 HTTP/1.1" 200 1594 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.72.152 - - [13/May/2012:06:25:38 -0500] "GET /upload-url.php?url=http%3A%2F%2Fimgur.com%2F32yOZ HTTP/1.1" 200 1594 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.72.152 - - [13/May/2012:06:25:45 -0500] "GET /upload-url.php?url=http%3A%2F%2Fimgur.com%2FUamV7 HTTP/1.1" 200 1594 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.72.152 - - [13/May/2012:06:25:59 -0500] "GET /upload-url.php?url=http%3A%2F%2Fimgur.com%2F32GkV HTTP/1.1" 200 1594 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
...

66.249.72.xx is Google and the user agent is Googlebot. Anyone who runs a web service knows that Google's indexing system will attempt to submit crap to any form it finds. They want to find local search engines on blogs and web sites and index the results. This way, if someone searches for "hypnophonic", then Google can direct them to whatever results page my own site's search would find. If Googlebot finds the search form on your blog, then it will submit all sorts of random words in order to find possible results. And it will do it for days.

In this case, Google seems to know that FotoForensics analyzes images. And it knows that the non-Google service imgur contains images. So Google was trying to submit every image at imgur to FotoForensics for analysis. I'm not kidding.

This uploading by Google was failing because they were submitting URLs to HTML web pages and not to the actual images. It started on Sat May 12 22:57:04 2012. I stopped it on Sun May 13 11:15:17 2012. (All times are CDT.) During those 12 hours, Googlebot submitted 1,061 URLs. My logs for them look like:

...

[Sat May 12 23:17:36 2012] [client 66.249.72.152] POST url: http://imgur.com/5C4Ul

[Sat May 12 23:18:04 2012] [client 66.249.72.152] POST url: http://imgur.com/w3Cwo

[Sat May 12 23:18:31 2012] [client 66.249.72.152] POST url: http://imgur.com/0KByw

[Sat May 12 23:18:59 2012] [client 66.249.72.152] POST url: http://imgur.com/XwYEC

[Sat May 12 23:19:27 2012] [client 66.249.72.152] POST url: http://imgur.com/gDnDG

[Sat May 12 23:19:55 2012] [client 66.249.72.152] POST url: http://imgur.com/DiBKf

[Sat May 12 23:20:17 2012] [client 66.249.72.152] POST url: http://imgur.com/EYmG1

[Sat May 12 23:20:18 2012] [client 66.249.72.152] POST url: http://imgur.com/sMleL

[Sat May 12 23:20:22 2012] [client 66.249.72.152] POST url: http://imgur.com/zUYB6

[Sat May 12 23:20:50 2012] [client 66.249.72.152] POST url: http://imgur.com/Xh3oI

[Sat May 12 23:21:18 2012] [client 66.249.72.152] POST url: http://imgur.com/JrmZb

[Sat May 12 23:21:45 2012] [client 66.249.72.152] POST url: http://imgur.com/7aHsS

[Sat May 12 23:22:13 2012] [client 66.249.72.152] POST url: http://imgur.com/vMcRj

[Sat May 12 23:22:41 2012] [client 66.249.72.152] POST url: http://imgur.com/yXNh1

[Sat May 12 23:23:09 2012] [client 66.249.72.152] POST url: http://imgur.com/RF4Lj

[Sat May 12 23:23:36 2012] [client 66.249.72.152] POST url: http://imgur.com/MJwDN

[Sat May 12 23:24:04 2012] [client 66.249.72.152] POST url: http://imgur.com/xF00H

[Sat May 12 23:24:32 2012] [client 66.249.72.152] POST url: http://imgur.com/wAXDI

[Sat May 12 23:25:00 2012] [client 66.249.72.152] POST url: http://imgur.com/B8bon

[Sat May 12 23:25:27 2012] [client 66.249.72.152] POST url: http://imgur.com/SOVBZ

...

The pictures follow a theme, so it looks like Googlebot was doing what web bots do well: traverse a set of links.

Do No Evil

The biggest problem with blocking Google is that I want them to index and link to the site. I just don't want them to submit crap to my service, or try to submit all of the content from some other service to my service. (I view this as a service abuse.) Moreover, there is no reliable method for telling Google to not post content to a web form.

However, I did find a solution. I'm blocking them in my .htaccess file:

SetEnvIf User-Agent ".*Googlebot.*" search_bot=1
SetEnvIf Request_URI "^/upload.*" has_upload=1

RewriteEngine on

RewriteCond %{env:search_bot} 1
RewriteCond %{env:has_upload} 1
RewriteRule $ - [R=404,L]

This sets the environment variable "search_bot" to 1 if the user-agent contains the string "Googlebot". And if the URL begins with "/upload" (where all of my submission forms go), then set the flag "has_upload". Finally, I have a rewrite rule: if it is a search bot and it has upload, then block them with a 404 error. This request will never even get to my back-end server script. Googlebot no longer abuses my site and it no longer uses my service to abuse imgur.

They can still index the site. They can still index links to results that other people post to other sites. But Googlebot can no longer post new content to my site.

In the first hour that this was implemented, Googlebot received 38 "404" errors. I wonder how long it will take before Googlebot realizes that all of its uploads will fail...

Truthfully, I don't think this abuse is coming from the main Google search engine. When the main Googlebot finds a form, it will slam it with requests. Thousands upon thousands of them. I really suspect that this is some Google employee trying out a side-project. This explains the low request volume (well, low for Google) and the fact that every request is coming from the same network address. (The main Googlebot uses a variety of network addresses.) Some Google employee probably thought it would be fun to submit every imgur picture through my FotoForensics site. Nevermind that they were submitting bad image URLs, nevermind that they are ignoring return codes and ignoring results, and nevermind that they are abusing imgur by harvesting their content and sending it to another web service.

I wonder how "do no evil" compares against "just be stupid"?

A Different Request

Dr. Neal Krawetz — Fri, 04 May 2012 17:52:24 -0700

The FotoForensics web site has been far more successful that I originally expected. Not knowing any better, we allocated enough resources to managed at least 250,000 pictures over a two year period. I got that from the original errorlevelanalysis.com web site -- which processed about 250,000 pictures in two years. Although the first month had relatively little traffic (5038 unique pictures), it has been steadily ramping up. April shot up to 16,421 unique pictures analyzed -- including one day with 2,545 unique uploads. At the current rate, we expect to processed over 100,000 unique pictures by October or November (far faster than the original estimate). For this reason, we are taking the server down for a few hours this Sunday and increasing the storage capacity.

In my previous update I discussed the "porn problem". Since then, we've designed a multi-stage filtering solution and it is partially implemented; porn dropped from 5% in March to 2% in April. We are getting much better at stopping pornographic uploads.

When people get blocked for uploading porn, we don't just give them 404 errors. Instead, we show them a simple page that says to read the FAQ. The FAQ has a red highlight telling them to read the section about being banned for uploading porn. About 70% of the people who are banned read the FAQ. The remaining people didn't return to the site so they never saw the ban.

The FAQ also includes an email address where people can protest being banned. I really didn't expect anyone to write in. But so far, I've had two people contact me. The first person claimed that he didn't upload porn. After I reminded him about when and what he uploaded, he changed his story to claim that it was an accident. (If you saw the series of pictures he uploaded, you wouldn't believe it was an accident.) So he is still banned.

The second person wrote in last night. She was irate at being blocked. Her email subject was "A different request"; she requested being banned forever. Between the venomous attacks and harsh wording, she brings up a number of points that I've discussed with other people but have not made public. So I'm reprinting my reply here, with her letter included in bold. Since I do believe in privacy, I have redacted personally identifiable information (name, age, etc.) in square brackets [redacted] and given her name the anonymous letter "B".

ps. You will notice that I bounce between "me" and "we". I do have a few people helping me with the site. I also corrected a few of my spelling/grammar errors.

A different request

Dear [B],

Thank you for taking the time to detail your concerns about FotoForensics. My responses to each of your issues are included below.

On Thu, May 03, 2012 at 09:49:14PM [redacted timezone], [B] wrote:
Dear fotoforensics.com nerd(s),

I do not want you to reinstate my abilities to use your crappy website, nor do I want any uploaded pictures removed. I am requesting something else, and I'll get to that right after I elaborate as to why I'm a little pissed off.

1. No one reads the god damn terms of service... ever. I don't read them so often I can't even remember if your site had any prior to using it or not.

The site has always had the terms of service in the FAQ. According to the web logs, the FAQ is the second most popular page on the site, after the tutorial.

You might want to look at the terms of service from other picture hosting web sites. Such as Google's Picasa (http://picasa.google.com/policy.html), yfrog (http://yfrog.com/page/tos), and Twitpic (http://twitpic.com/terms.do). You don't need to read the whole text -- just search for the word "pornography". They all say the same basic thing: pornography is not permitted, pornography will be removed, and you may have your account blocked. Why would you assume that my site would be any different? Had you uploaded the same genitalia picture to Picasa, yfrog, Twitpic, Flickr, Photobucket, Zooomer, or most other image sharing sites, you would have had your account blocked as soon as they noticed.

Some sites, like Picasa, appear to use Google's powerful image search engine and pagerank system to flag potential pornography. Other sites seem to review pictures that suddenly become popular. (Post the link to a porn web site and traffic will increase dramatically.) And still other services have a room full of people who review content, or leave it up to visitors to flag content as inappropriate.

As a research site, we're actively developing a multi-stage porn filter system. The first two stages are already functional. (This is how your pictures were caught.) The next stage will be coming online this month. The final filter is already built and currently being tested; when deployed, this will permit real-time automated detection. And while these filters are currently being designed for the pornography problem, they are really generic and could easily be modified for other tasks.

2. I don't imagine too many people begged you to host these images. It's probably safe to assume that the majority of your web traffic comes from online daters who want to see if their new girlfriend is enlarging her tits or what not. Therefore the hosting aspect is simply bullshit. I just can't imagine the sharing factor in this unless you're trying to encourage people to use your site in combination with facebook/twitter to ridicule people who photoshop their pictures. If so, that's pretty pathetic and probably not very well received considering how facebook is already taking shit for being a popular tool for high school bullying. And since the news tells me that high school bullying directly causes suicide and psycho rampages... well, all I can say is shame on you, shame, shame.

You mention a number of points here.

Although nobody begged me, I was asked politely by a handful of people to fill the hole that errorlevelanalysis.com created when it went offline. For this reason, I chose to start fotoforensics.com as a service for the world-wide community.

The main themes of the uploaded pictures are not sexual in nature. Rather, they fall into three main categories: testing, evaluation, and debunking. Since the purposes of this site are (1) research, and (2) give people an opportunity to try out some real photo forensic methods, I think it has been a smashing success. Remember: this is not a generic file sharing site like yfrog or Twitpic; this is a research-oriented web service.

I do not tell people what to upload or how to use the pictures that they upload. (Although I do specify what not to upload.) And due to legal liability, I cannot even provide any automated interpretation of the results. The site applies a test and leaves the interpretation to the viewer. The tutorial and FAQ are designed to help people understand the results and how to use the site.

While the majority of pictures are not shared, a few have become very popular. For example, the most popular picture in the last week is of a tweet. While I don't read Arabic and I don't know the details behind this picture or the controversy, I do know that it is very popular and they even put a site-by-side comparison on a web site: http://mz-mz.net/38282/

As of this morning, the site is hosting 30,727 unique images. (That averages to about 10,000 a month, but the average is skewed since the average daily upload rate is still increasing.) Of the 30,727 images, 1,070 have been blocked -- either due to pornographic content or by request. That means that about 97% of the images are non-sexual in nature.

Your final point in issue #2 concerns the distribution of images. As stated in the FAQ, we do not publicly provide the URLs to images that have been uploaded. So far, we only publicly mention links that other people have already posted publicly.

3. The fact that you even have admins sitting around viewing each picture is very creepy in itself. Since I've already determined that no one wants cloud storage from you, for anything constructive anyway, why not just let people view the picture comparisons and let that be the end of it? But hey, it's your website, you want to ban people go right ahead. There are plenty of sites just like this that do exactly what I'm talking about. I'm just saying if you'd be less of a dick you might get more web traffic and then you wouldn't have to panhandle for donations.

Before I even agreed to run this service, before I coded the first line of the server, I was discussing my concerns related to hosting this site with attorneys and law enforcement officers. The main problem is due to the US legal system: some types of pictures are illegal in the United States. The main issues are related to child pornography and harassment.

To make it clear how strict (and screwed up) US laws are related to child porn: there have been court cases related to child porn where the judge, jury, and legal counsels on both sides never saw the pictures. Just showing them the pictures means that they would be in possession and distributing prohibited content. Imagine a court case where nobody can see the evidence -- that's a child porn lawsuit.

With regards to reviewing content, this goes back to the legal issue. Since it is "my server" and the law will hold me liable for anything on my server, shouldn't I know what is on my server? Myself and my fellow moderators have better things to do than ogle over every picture uploaded to the site. We rely on various filter techniques to reduce the amount of time we spend on this task (right now, one person can review the filter results for a day of uploads in under two minutes -- and I'm spending hours building filters that will further reduce that time). I am confident that FotoForensics does not host pornography or child pornography.

Finally, although there is a donation button, I don't think anyone has clicked on it yet. The site has received no donations. And since the donation button is buried in the FAQ, I don't view it as panhandling. (Not like Wikipedia, which ran a huge banner on every page.)

4. What is wrong with nudity anyway? Are you mormon? Are you Mitt Romney? I did not upload "pornography", I uploaded a nude picture of my [20+] year old SELF. Nudity is not always pornography... in fact we were all born completely bare-ass naked! It's almost like it's a... I don't know... natural thing, maybe? Yeah, wrap your offended little brain around that. And if you must know, the reason I chose that picture is because I dramatically lightened it, as well as cropped it.And rest assured I had no intentions what-so-fucking-ever to distribute any links to that page, I just wanted to test out the tool with a picture that I knew was photoshopped. So just to get things straight, the only people who can see the uploaded photo are site admins, the person who uploaded it, and anyone they share the link with. You say it's a PUBLIC WEB SERVICE but... only if you have the link, right? Not really PUBLIC in a comprehensive sense of the word, now is it? If you had a home page with something like "latest 100 uploads from everyone" then I could see a point in restricting content.

For site moderation, we have a written policy used by the moderators concerning acceptable content. Basically, if the picture could be found on the cover of a supermarket magazine that doesn't have a black plastic wrapper (for example, Cosmo, Vanity Fair, the Victoria's Secret catalog, or even Playboy -- that is, Playboy in the US and not Playboy in Brazil since Brazil permits full nudity on the cover), then it is permitted. But if it couldn't be found on the cover, then it is pornography. Two of your pictures clearly fall into this definition for pornography.

Personally, I don't care about pornography. But it was made abundantly clear to me before I started building the server that pornography is like a "gateway drug" to child porn. If people are permitted to upload porn, then it will attract the pedophiles. If we don't permit porn, then the pedophiles will stay away. I can honestly say that, out of 30,000 pictures over three months, a total of 7 pictures -- uploaded by two people -- were classified as child pornography and had to be deleted immediately. (The deletion shows our intent to not be in possession. And I hope that "Mr. Chicago" and the "Poland Pervert" both get arrested.)

You mentioned that the site says it is public yet the content is generally not publicly accessible. If the content was publicly viewable, would it have prevented you from uploading photos of yourself? Or, having admitted to not reading the FAQ, would you be upset about having hundreds of creepy stalkers who saw your photos?

Finally, of all the things you could have photographed, modified, and uploaded for analysis, could you really not have come up with anything better than a topless reflection and a picture of you fingering yourself? I think this says a lot more about you than me. For my own tests, I used a picture of my bookshelf; removing my clothing never even crossed my mind.

5. You have created something that is obviously going to be used with bad intentions most of the time. Yet you act like that will not be tolerated. And that's sort of like giving someone a crackpipe then saying 'you will not dare smoke crack out of it, just keep it as a nice paperweight'.

I fully disagree here. 97% of the time, it appears to be used with good intentions. 3% of the time, it is used by people who don't care that the site contains public content. And a very tiny fraction (2 out of approximately 48,000 people who have uploaded content) have uploaded photos with "bad intentions". (And if you want the full statistics, that is 48,000 uploads of 30,000 unique pictures and viewed by at least 73,500 unique visitors in three months.)

Now then, my request. I request not to be banned for 90 days after my last visit, but to be banned FOREVER. I don't want to forget about this in the future and then accidentally use your site. I certainly don't want that to happen and then I go around telling other people to use it. I would also like to request that you suck my ass and get the fuck over yourself.

Ignoring the hostile tone of your email, you do have valid concerns. Hopefully I have addressed those concerns and explained why the site is run this way.

Per your request we will not remove the ban. But as we state in the FAQ, the ban will be removed after 90 days of inactivity. This prevents someone with a dynamic configuration from inheriting a ban that was caused by someone else.

With your permission, I would like to include your remarks and my responses on my blog this evening (http://www.hackerfactor.com/blog/). I will remove any directly-identifying information such as your name, email, age, and network address. Without your permission, I will summarize your remarks and include my responses. Without a reply, I will assume that you grant permission. And I promise to not publicly post the links to your pictures.

Neal Krawetz, Ph.D.
Hacker Factor Solutions
http://www.hackerfactor.com/ and http://fotoforensics.com/

A Little More Information

I didn't hear back from her.

Her original email did have a small addendum. She concluded it with this message:

Thanks!
[redacted network address]
( ^ my IP addy, and being the nerd that I know you are, I'm sure you can find
out my blood type with that... so it's all you need to fucking know)

It's funny that she mentioned blood type. One of my side research projects is really panning out. I gave a rough presentation about this at a little technical meet-up group I attended last year. Given a photo of a person, you can sometimes tell other things about them. In some rare cases, you can even determine their blood type. (It's scary-accurate but still working on the confidence interval.)

Put A Bird On It

Dr. Neal Krawetz — Sat, 28 Apr 2012 11:53:00 -0700

Perhaps our lawmakers have always been this slimy and we've only begun to notice because of rapid communication systems like the Internet. Or perhaps they have sunk to a new low.

On Thursday (April 26, 2012), the House of Representatives voted on the Cyber Intelligence Sharing and Protection Act of 2011 (H.R.3523 aka CISPA). This passing is the first step in becoming a law. It still needs to pass the Senate and be signed by the President. And the President has already vowed to veto it.

Nitty Gritty

There are many things that trouble me about this bill. The 18 page PDF is fairly straight forward, but has many unmentioned implications.

For starters, Section 2(a)(2) makes it easier for government agencies to share information. My concern is with information oversight. Many government agencies are not permitted to collect or hold information related to US citizens. (It's fine to spy on others, just not on our own people.) And there are some legitimate reasons to collect information about specific US citizens. For example, if some guy is part of an active criminal investigation then federal law enforcement can gather information about him. Also, if information is publicly available, then it can be collected.

However, a few branches of the government are permitted to collect limited information about US citizens. For example, the Transportation Security Administration (TSA) can (and does) collect information about US citizens. This permits them to compare names against the no-fly list, request your papers at the security checkpoint, and employ full-body scanners on US citizens.

Under this bill, it would be easier for TSA to pass anything they collect to other government agencies. They just have to claim that they have a reasonable belief that it relates to a threat.

Section 2(b) focuses on cybersecurity. In particular, 2(b)(1)(a) states that a cybersecurity provider may share cybersecurity information with the government. Section 2(h) defines a "cybersecurity provider" as any non-government entity that provides any kind of computer security. This includes your ISP (they provide firewalls and look for network attacks; the larger ones also give away anti-virus software for free), email systems (e.g., Gmail and Yahoo! employ spam filters for your security), and other online services. If there's a password involved, then that's cybersecurity!

Section 2(b)(4) (PDF page 8 for anyone following along) states that any cybersecurity provider -- or any person with access to the provider's information -- that shares information with the Federal government is immune from prosecution. This immunity is just like the kind proposed by SOPA; SOPA granted immunity from all lawsuits. This is actually more lenient that the previous draft of H.R.3523 (PDF, see page 5), which did not grant immunity from civil prosecution. (Perhaps this is why Facebook proposed changes their use policy days before the House vote, making it easier for Facebook to not be in violation if they hand over anything to the Government. And Facebook is a CISPA supporter.)

Section 2(h) defines terms. Like a "protected entity" is anyone who does business with the cybersecurity provider. It doesn't say that the provider must provide the "protected entity" with any protections. Google could offer protections to some users and still claim (under this definition) that any Google users -- even those who do not receive protections -- are protected entities. This ties in well with Section 2(b)(1)(B), which describes a "Self-Protected Entity". So my FotoForensics web site -- which offers no user accounts to web clients -- is still a cybersecurity provider because it is self-protected; I use SSH to remotely manage the system and SSH is cybersecurity!

Section 2(h)(2) brings back the horrors of SOPA. It mentions "unauthorized access" or "misappropriate" private or government information. This revisits copyright infringement. It places file sharing and blogging (sources of potential copyright infringement) on par with cyber-terrorism. If your web site contains infringing material ("unauthorized access" or "misappropriation"), then it is considered "cyber threat information".

Section 2(h)(3) defines "cyber threat intelligence". This is anything that could potentially be used to evaluate a cyber threat. The actual text of the bill says, "directly pertaining to a vulnerability of, or threat to, a system or network of a government or private entity". Watch those conjunctions! (... or ... or ... or ...) This section grants any service provider permission to read your emails just in case you make a threat against anything. Simply saying "Screw that!" in an email about a cyber security bill could be considered a threat -- since you have voiced an opinion that could jeopardize the actions of a government entity.

Hush and Rush

CISPA is clearly a bad cyber security bill. As Rep. Jared Polis (D-Colo.) said, "This bill in its current form ... is an unprecedented, sweeping piece of legislation that would waive every single privacy law ever enacted in the name of cybersecurity."

Amazingly, that isn't what bothers me the most. The thing that really bothers me is how this bill was handled.

Multiple copies. If you search for the text of H.R.3523, you will find many different versions. The variants on the House web site range from 8 pages to 16 pages. The GOP web site has an 11-page version. The official document that the House voted on is 18 pages, and as far as I can tell, it was only available after they voted on it. Each of these versions have different text and different nuances.
Rush to vote. They were supposed to consider CISPA on Friday, April 27. But at the last minute they moved the consideration vote up to Thursday, April 26 and voted on it.
Partisan Politics. The bill passed with a vote of 248-168. But the 248-168 is a little misleading. These numbers are almost completely down party lines! The Republican votes were 206-28, and the Democrats were 42-140 (non-votes were 7 Republicans and 8 Democrats). Fortunately, the 248-168 split isn't enough to override a Presidential veto.
One more thing. There was a last-minute modification that is currently not found in the PDF. This modification was proposed by the bill's sponsor, Mike Rogers (R-MI), and directed the clerk to 'Insert "deny access to or" before "degrade" in each place it appears.' On the surface, this does not seem major and passed unanimously. Basically, the bill now describes a cyber threat as something that "deny access to, degrade, disrupt, or destroy". Now doing multiple bad login attempts which would block a Yahoo! account is considered a cyber threat and on par with cyber terrorism.
Other amendments. A bunch of other amendments were added to H.R.3523 but are not found in the PDF. For example, H.AMDT.1022 added an amendment that limits the purpose of information shared with the Government. The list now includes "(1) cybersecurity; (2) investigation and prosecution of cybersecurity crimes; (3) protection of individuals from the danger of death or physical injury; (4) protection of minors from physical or psychological harm; and (5) protection of the national security of the United States." Now the bill does not just cover cyber security, it also covers death threats, things that may cause mental anguish, and anything inappropriate for children. This "limit" grossly expands this bill's reach. For example, a web site that caters to adult entertainment, even with a password and validation requirement prior to seeing content, could be considered a cyber threat since the content is not appropriate for children. (And the TV show South Park would certainly be classified as a cyber threat.)

In contrast, amendment H.AMDT.1023 forbids the government from using library records, firearms sales records, and tax returns that are passed in by cybersecurity providers. So just to be clear, your gun purchase and tax records showing donations to groups that support terrorism are private and cannot be used to detect a threat, even if someone willingly supplies the information to the government. However your grocery store receipt can be collected and used by the government if someone passes it to them. (Is it just me, or is this really backwards? I'm only good with the library restriction.)

H.AMDT.1031 was an amendment that authorized "the Secretary to intercept and deploy countermeasure with regard to system traffic for cybersecurity purposes in effect identification of cybersecurity risks to federal systems." This would have been like the Great Firewall of China. Fortunately, this amendment was withdrawn.

As far as I can tell, the only good amendment is H.AMDT.1027. It states that "those who choose not to participate in the voluntary program authorized by this bill are not subject to new liabilities." So you can decline to participate in the voluntary part of this bill.

The Soft Sell

This bill and associated amendments really strike me more as job justification, catering to special interests, and an attempt to expand government reach than any actual attempt at stemming cyber threats. Moreover, the bipartisan rift clearly shows that this is more of a power play along party lines than an actual attempt to pass a useful law. For example, Benjamin Quayle's (R-AZ) amendment that added in protection of children seems more like an afterthought (it is an amendment after all). And when the President vetos the bill, the Republicans can claim that Obama likes to show pornography to children since he was against a bill that protected minors.

I am becoming convinced that the Republican party has evolved into a bully that does not know how to negotiate or make a soft sell. Rather than forcing through legislation in an attempt to be relevant, perhaps they should learn from other fields. In the pharmaceutical community, they learned to sugar coat medicines and lower dosages when there are undesirable side effects. And when something does not sell in the hobby community, they put a bird on it because items with birds always sell.

Update 2012-04-28: Microsoft has dropped support of CISPA. They cited privacy issues, telling CNET that any new law must allow "us to honor the privacy and security promises we make to our customers" and protect "consumer privacy."

Gradient Map Test

Dr. Neal Krawetz — Tue, 24 Apr 2012 09:38:41 -0700

My site admin regularly watches the logs and lets me know when there is a sudden increase in unexpected activity. Recently there has been an increase in people searching for something called a "Gradient Map Test".

A new photo analysis algorithm? I'm intrigued!

The First Clue

Most of the requests were coming from a site called the Clues Forum. As far as I can tell, this site focuses on conspiracies. In some threads they debunk hoaxes, in others threads they promote them.

Frankly, I'm not interested in getting involved with another conspiracy. And this blog entry is not about conspiracies around the 9/11 attack.

What got my attention was an animated GIF that used something called a "Gradient Map Test" to highlight alleged modifications to the image.

This GIF looks impressive. The posting claims that it shows firemen added to the photo. I wanted to know how the algorithm works, so I did a literature search. Unfortunately, I found nothing about a "gradient map test" -- nobody else seems to be using this, or at least, not calling it by this name. Usually I can find something about an algorithm.

And then I noticed one more thing in the Clues Forum posting: someone attributed the algorithm to me. WHAT?!? This isn't one of mine!

Caution: Steep Gradient

When someone talks about a gradient related to images, they are usually talking about color differentials. How much are two colors similar or different? Gradients make great edge detection algorithms and can also be used for things like topology conversions.

A "gradient map" can refer to the array of gradients across an image. However, that isn't how the animated GIF is using it. If they used a vector based gradient mapping, then the blues in the middle of the people's clothing should be muted since blue next to blue has virtually no gradient slope.

Maybe they are using a larger gradient measurement? In the animated GIF, the black box next to the person on the left is a very different color from the black square to the right of the people (in the white box). Yet in the original picture, these black squares are the same shades of black. Both squares are large black objects with similar coloring and similar textures. So regardless of the algorithm, I would have expected them to look similar.

Original picture

Animate GIF frame 13

A gradient map can also refer to a colorization. Basically, you convert the image to a grayscale with 256 possible values. Then you map these 256 values to 256 colorizations. For example, I can convert the original image to a grayscale and apply a gradient mapping based on a sinusoidal curve.

×=

With a gradient map colorization, the people stand out because they are the darkest items in the picture and are basically the only thing that maps to that color.

Gradient map colorizations are commonly used on weather maps and medical imaging to highlight measurement differences, in astronomy for converting non-visible light to a viewable spectrum, and on Ghost Hunters to find cold spots. It really can be used in any field that needs to visualize or colorize non-RGB data.

The problem is, the animated GIF did not apply a gradient map colorization. We know this because the people did not change color. If the GIF was showing a gradient map colorization, then the people (skin and clothing) should have been recolored based on the gradient map coloring.

So we don't know what they are doing in the animated GIF. We only know that they have thrown out a wizbang term and gave a picture. They never said how they created it, and they gave it a false attribution (the algorithm did not come from me).

Using my own tools, both public and proprietary, I cannot detect anything suggesting that the people were added. I can detect a few things in the original picture, but none suggest a hoax:

A color profile was applied. It operated in the XYZ color space.
A minor amount of color scaling was applied in the RGB color space.
There is a minor amount of edge sharpening, similar to what I would expect from an Adobe application.
The error level analysis does include a little rainbowing, but not a lot. A lot of rainbowing strongly suggests an Adobe product. A little rainbowing could come from Adobe, Gimp, or a half-dozen other graphic programs.

In contrast, I do not detect splicing. I do not detect blending. I do not detect selective color corrections. I do not detect lighting abnormalities, missing shadows, or inconsistent focal points. Frankly, I'm not seeing anything abnormal or suspicious in this image.

That's Odd

The picture is attributed to a FEMA photographer named Kurt Sonnenfeld. By himself, he is an interesting character: he was given exclusive access to photograph the World Trade Center area after the 9/11 attack, but he refuses to show over some of his 9/11 footage. And he is wanted for murder in Colorado concerning the death of his wife. There are a half dozen conspiracies that involve him.

The Clues Forum and many other conspiracy-oriented forums have repeatedly attacked his 9/11 photos, as well as those by other photographers, as being faked. For this reason, I am hesitant to point out the one abnormality that I did detect. I want to emphasize that this abnormality does not imply malicious intent and does not support the conspiracy around this photo allegedly having people added.

The oddity comes from the meta data. The source image that I looked at is hosted on Flickr. Flickr permits you to download the "original" image. Now for clarity, "original" refers to the picture that was uploaded to Flickr, and not necessarily the camera-original (straight from the camera) photo.

According to the EXIF meta data, which should represent the camera's data, the picture was taken on 2002-08-21 16:12:39. The problem is, that date is almost a year after the picture was supposed to have been taken. (And about 2 months after he was released from jail after being detained relating to his wife's murder investigation.)

The EXIF data also identifies the camera as an Olympus E-10. But this is inconsistent with the photo. Specifically, the Olympus E-10 uses big-endian byte ordering for the EXIF data, but this file uses little-endian. The E-10 specifies three quantization tables, but this file only has two tables. And there are other differences in the meta data. This file fails a camera ballistics test. We can be certain that this is not a camera-original image.

So what happened to this picture? We know it is not camera original. But that is consistent with the application of a color profile and minor color corrections. Since I know of a half-dozen ways that the meta data could have been altered without malicious intent, I'm not willing to conclude that this denotes a hoax. Moreover, the Flickr account is not attributed to Kurt Sonnenfeld. So I cannot even rule out modifications made by the person who uploaded the photo to Flickr.

As far as I can tell, people in the Clues Forum want there to be a conspiracy so badly, that they will provide a false graphical analysis. Moreover, they have attributed their mysterious algorithm to me in order to give it some kind of credibility.

Finally, I want to make this very clear: I am not interested in conspiracies around 9/11, Kurt Sonnenfeld, empty vaults, missing black boxes, or the death of his wife. I only want to point out that I can find no basis for this "Gradient Map Test" that the Clues Forum attributed to me.

Jail Bait

Dr. Neal Krawetz — Fri, 13 Apr 2012 19:15:58 -0700

Earlier today the news was abuzz with an announcement. The FBI announced the arrest of Higinio O. Ochoa III. He was charged with digital trespassing into government computer systems and releasing personal details related to law enforcement officers.

Of course, every news story focused on the fact that there was a picture of a woman's breasts. The headlines all played on the pun. For example, CBS wrote, "Suspected Anonymous Hacker Busted By FBI -- Thanks To A Racy Photo. (Get it? Busted? It's funny because it's her bust.) And Gizmodo declared, "Body of Evidence: These Breasts Nailed a Hacker For the FBI". (You see, it's her body and it's evidence, and saying "nailed" gives a sexual undertone.) Personally, I like my own title: Jail Bait (nothing but trouble).

Look Closely

The thing that really got my attention was the fact that Ochoa was caught through digital photo forensics. According to reports, "someone" posted some information to PasteHtml. The posting was credited to w0rmer @cabincr3w. At the bottom of the posting was a picture of a woman's torso and some text:

We Are ALL Anonymous

We NEVER Forgive.

We NEVER Forget.
<3 @Anonw0rmer

And that's where he screwed up. (It's as if he never read my blog! I'm shocked!)

Cyber-stalking

According to the Criminal Complaint, someone using the nick "wormer" had repeatedly compromised systems and posted harassing messages directed at law enforcement officers. In effect, he was just begging to be caught.

The FBI had a suspicion that wormer was Ochoa due to a previous online posting that was from an account attributed to a user named "wormer" and signed using Ochoa's name. However, they couldn't be certain that this was the same person. Perhaps it was another user using the same name...

The picture is what sealed the link. Specifically, the picture includes meta data and GPS information. The GPS information identifies the location as 37° 51' 25.20" S, 145° 15' 1.20" E. That's a suburb of Melbourne, Australia. The FBI linked a Facebook account for Ochoa as having a friend in Australia named Kylie Gardner. Moreover, Ochoa is apparently "in a relationship" with Kylie. (The complaint cites sources and emphasizes that all of this was publicly available information. No subpoena required for this part.)

The complaint also states that they traced network addresses to Ochoa.

As far as I can tell, there is really only one thing missing from the criminal complaint. The report links the account for wormer to a person named Ochoa (who lives in Texas), and a network address to a location near where Ochoa resides. It links Ochoa in Texas to Gardner in Melbourne, and it links a photo of a woman used by wormer to an address in the Melbourne suburbs. However, the report does not state that Kylie Gardner is the woman in the photos (plural -- the complaint lists multiple images of this woman and used by wormer), nor does it identify Gardner as residing at the GPS location.

Although the linkage between Ochoa/w0rmer is very strong, it is incomplete. But, it is enough for the FBI to claim "probable cause" and to make an arrest. They have charged Ochoa with at least four counts of unauthorized access to a protected computer. With the arrest, I suspect that they will get subpoenas to search his home and computer systems. And if those systems contain any of the compromised data or links between Ochoa and the wormer accounts used to post the compromised data, then he will be going away for a very long time.

Thanks to Xenon for forwarding me the news of the arrest and JG for the blog title and links. Jack: Hope your eyes feel better.

Slip Me Some Skin

Dr. Neal Krawetz — Fri, 06 Apr 2012 19:08:00 -0700

Normally when I write about politics, I focus on the United States (because that's where I'm located). But on occasion I mention other countries -- particularly when they work on laws that I wish this country would consider. (Like truth in advertising laws regarding digitally modified pictures.) However, this time the UK has proposed an amazingly ignorant law.

Let me preface this with: I'm not a barrister or solicitor (or whatever they call an attorney in the UK), this is not legal advice, and I'm certainly not familiar with the UK legal system (well, anything other than how to break away from their empire after they imposed the Stamp Act of 1765). Worse: I learned of this topic on Slashdot, so I don't even know how serious it really is.

It appears that the UK has proposed a bill that demands ISPs and device makers block porn. The bill, called the Online Safety Bill (PDF) would force service providers to restrict access to pornographic content. ISPs, mobile phone services, and other providers would need to maintain an opt-in list of people who want pornography and only permit them to see pornographic images. (As an aside: if this were in the USA, we would have given the bill a snappier acronym like the "Protect Lasses from Adult Content and Erotica using Network Technologies Act" commonly called PLACENTA.)

Porn? What? Where!

While I am not an Internet Service Provider or Internet Access Service, I do run an online photo analysis service that has a zero-tolerance policy concerning pornographic content. So any technology that would make this law possible definitely has my interest. I mean, I don't want to be known as the guy who runs the porn web site. (Other sites may not mind that reputation, but I'm not them.)

Lately I've been doing a lot of literature searching for anti-porn filtering technologies. And frankly, there's no really good solutions right now. Here's a summary of the basic options that comprise the current state of the art:

Filter by Domain
Some web sites are explicitly known to host adult content. If you see a picture from any of those domains, then assume it is prohibited and filter it. The problem is, if you are not careful then you can end up filtering a lot of content that isn't porn.

There are plenty of domain blacklists out there. URL Blacklist has their "bigblacklist" of potentially offending (and potentially safe) domains. While the bigblacklist is one of the better lists out there, it has some issues. For example, it includes sites like Flickr and Photobucket in their adult-related categories. While some Flickr and Photobucket users do upload adult content, that certainly isn't the majority of either site. And frankly, other photo-sharing sites like Zooomr, Facebook, and Google's Picasa also host some adult images, yet they are not classified as adult-oriented domains.

The worst thing about filtering by domain is that you are likely to filter pictures that are not adult in nature. If the law requires filtering pictures, then filtering by domain fails; it does not filter based on pictures, and misses pictures that are not at those domains.

To give you a real example of this, India passed a law requiring search engines to filter pornographic content. However, this law has had little impact because there are just too many ways around the filters. The filtering errors (Type I, Type II, and Type III) grossly outnumber the actual matches. (For you non-statistical people, a Type I error is a false-positive and a Type II error is a false-negative. A Type-III error is when you get the correct result for the wrong reason, such as getting a non-porn picture because you went to the unfiltered domain for Twitpic -- as if Twitpic didn't host porn.)

Filter by Keywords
Another common solution filters by keywords. The hope is that filenames or some part of the URL will be descriptive and permit filtering. However, there are plenty of false positives (e.g., "Onanie Bomb Meets the Sex Pistols.jpg" really is workplace safe, even though it contains the word "sex"). Also, this approach does nothing to filter pictures with numerical filenames.

Filter by Image Content
In the image analysis world, there have been interesting efforts to identify pornographic content. For example, there are face detection algorithms that look for specific features: eyes, nose, mouth, etc. If they find likely features in the correct relative orientation (two eyes above one nose above one mouth) then they determine that it is a face.

I read one paper about a decade ago that used an eye-filter classifier to identify boobs. (Well, they kind of look like eyes: big, round, dark center...) Since then, people have tried to train classifiers to identify various body parts. And while many of these systems work well in specific circumstances, most seem to fail in the general cases. For example, it is difficult to tell a close-up of a bent elbow or a butt-crack from chest cleavage, and I have yet to see a system that can distinguish man-boobs from their female counterparts.

Filter by Colour
Another very common approach tries to identify skin tones. This seems to be the most popular analysis approach.

Basically, skin usually comes in shades of tan. You don't typically see blue skin except on a Smurf, in the movie Avatar, or on a corpse. (Unless it is spring break and there's ample body paint.) Same goes for green, yellow, red, and anything else not tan, brown, or beige.

All of these algorithms use the same basic approach: define colour range that denotes "likely skin" and the look for a percentage of match. Some algorithms also try to determine if the patches make up the shape of a person. If there is too much "skin", then assume it is porn.

However, these algorithms are very primitive and inaccurate; they have lots of limitations. Simply using a percentage approach, a close-up of an arm will easily pass the threshold (while not being pornographic), while a nudist in the background will likely be missed (low percentage of the picture). These algorithms also have trouble with women in bikinis (large percentage of skin) and flesh-coloured shirts (false-positive on colours) -- these are not the same as nudes and will likely be classified incorrectly. (I read one paper that found a red firetruck classified as a nude because the lighting put it at the edge of the skin colourspace.)

Algorithms usually define skin-tone based on a fixed range in the RGB, HSV, or YUV colourspaces. Each approach has their own limitations. Basically, there are a huge number of false-positives and false-negatives, and they are very dependent on the lighting and tone. A picture that works well with one algorithm will probably not work well with another algorithm. And using a majority rule to combine multiple algorithms typically does not work well since the majority perform poorly.

However, the biggest problem has to do with skin colouring. People with extreme colouring (albinos, very pale, or very dark skin) usually fall outside of these colour ranges. This created a huge problem for Hewlett-Packard a few years ago, when they had a camera that identified faces using skin-colour and then focused on the faces. They were criticized for having racist cameras since the devices would not focus on black people. Technically, the skin colour fell outside of the colour swath, so it failed to match. (In HP's case, they used an algorithm that was optimized for people with typical Asian skin tones because that is where the research originated.) As I understand it, very pale people had the same focus problem, but dark-skinned people had it much worse.

Making Flesh Crawl

To give you an example of how these skin detection algorithms work, here are a few pictures that use the YUV and HSV colourspace definitions from "Detection of Pornographic Digital Images" (J. Marcial-Basilio, et al., International Journal of Computers, Issue 2, Volume 5, 2011) and the normalized RGB colourspace from "Simple Face-detection Algorithm Based on Minimum Facial Features" (Y. Chen and Y. Lin, The 33rd Annual Conference of the IEEE Industrial Electronics Society (IECON), Nov. 5-8, 2007, Taipei, Taiwan). Each picture includes (in order): the original image, normalized RGB skin tones, HSV skin tones, and YUV skin tones. Notice how none match the exact same things.

(As an aside: due to criticism for continually using pictures of distractingly beautiful women, I've decided to use an alternate set of photos for the examples.)

Pregnant in Flesh coloured Shirt (Source: Zoe Strauss's Photostream)

For the flesh-colored shirt, the RGB filter did a good job identifying actual skin. The HSV filter did a better job identifying flesh tones. In contrast, the YUV filter matched the shirt and not the face or arm.

Beer Belly (Source: BigBeerBelly1's Photostream)

In the beer belly picture, RGB was very noisy and missed nearly all of the people in the background. HSV got the background people, but also marked the woman's yellow top as skin. YUV performed much like HSV, but matched fewer people in the background.

"Best. Wedding. Ever." (Source: Reddit)

In the wedding photo (they seriously know how to party...), RGB matched the most skin, but still failed on everyone except the bride. HSV did a great job matching the ground, trees, and shirts (non-skin), but missed most of the actual skin. And YUV, well, it missed every person. (I actually suspect that this YUV filter is the same algorithm used by the HP cameras.)

While these examples show YUV performing the worst, I have plenty of other examples where YUV performs better than the other options. It all depends on the picture's content, lighting, and colouring.

Smut and Innuendo

I looked up British Humour on Wikipedia and immediately saw that it was based on smut, innuendo, and satire. "Benny Hill" and "Monty Python" were listed as examples. (I wonder how many Monty Python movies have no nudity...) So perhaps I am reading too much into this proposed bill. Maybe it is British humour -- and I missed the joke. Or maybe some lawmaker really does want to curtail this aspect of British culture?

In any case, it appears that this bill misses the target. It requires network service providers to perform complex content filtering. This is ignoring the fact that access service providers just pass bits between computers and may not have access to the content due to cryptography between the client and server. This bill also requires filtering technologies that do not currently exist with any significant degree of accuracy.

If the UK lawmakers seriously wanted to address this issue, they would (1) require hosting providers, and not service providers, to filter the content that they host, (2) put in a clause that limits filtering efforts to the currently available technologies, and (3) leave the type of filtering up to the people running the filter, rather than explicitly specifying image-based filtering in Section 1.

Fool Me Once

Dr. Neal Krawetz — Sun, 01 Apr 2012 15:40:00 -0700

In my previous blog entry, I wrote about using photo forensics to identify fake pictures of lottery tickets. However, one forgery looked good enough to deserved a closer inspection. This particular picture was posted on Reddit. (Thanks to Than Tibbetts for the pointer.)

In the Reddit message, the guy claimed to have a real winning lottery ticket and even took multiple photos -- at the request of other Reddit readers -- to prove that it is real.

The initial photo shows the winning numbers on line D.

Requested to write "Villa22" next to the ticket -- to prove a current photo. But the winning ticket it only partially in the picture.

Requested an uncropped ticket next to "Villa22".

The initial problem mentioned by the Reddit crowd is that this was a ticket for Massachusetts, but the three winners were in Illinois, Kansas, and Maryland. However, when this posting first came out, the media had only begun to declare one winner. Later, the media reports were updated to say "at least one winner" and then three winners. So at the time that this message was posted, we couldn't be certain that there were not four or more winners... Even though Massachusetts was not mentioned as one of the winning states, we could not be sure that the media was accurate.

A few Reddit readers also noticed that the numbers were not monotonically increasing. However, I'm not a lottery expert and I don't know if states can choose to sort the numbers on the ticket.

So rather than depending on speculation about number order and winning states, I rely on photo forensics to identify forgeries.

Basic Tests

I have a small set of basic image analysis tests that I run on every image. I call these my "90% tests" because they catch about 90% of forgeries. And while there is a steep learning curve, if it is a fake, you'll know it within seconds or minutes. For the forgeries that they do not catch, they will usually show something odd -- not enough to call it a fake, but enough to take a deeper look.

With the first picture, it easily passed most tests. However, I was bothered by the image quality. This is not a camera-original picture, it lacks major Photoshop artifacts, and yet the ELA was high across the entire image. Whatever was applied to this picture was applied across the entire image.

In contrast, the second picture has a low ELA and significant rainbowing (the red/blue splotches typically associated with Adobe products), while the third picture has a high ELA and no major rainbowing artifacts.

Why are the first and third pictures different from the second? Shouldn't they all have similar ELA results? And why is the ELA from the first and third so high? There's something odd going on here...

PCA

One of my favorite tests is the Principal Component Analysis. PCA identifies an orthogonal coordinate system based on the variance of the data space. (Uh, can you explain that in English?)

All of the RGB colors in the picture can be represented as XYZ coordinates. This yields an RGB cloud. For this picture, the cloud is spear-shaped since it doesn't have much color.

The widest path through the cloud defines the first principal component (PC1; in red). PC2 is the second widest path (at a right angle to the first path; in green), and PC3 is whatever is left over (the shortest path relative to PC1 and PC2; in blue).

There's a bunch of different things that can be done with PCA. One of my common tests is to measure the distance from each color to the PC1 line. This shows JPEG resave artifacts and can be used to identify splicing or editing based on image qualities. However, with this picture nothing stands out as unusual; the picture has lots of large chunks indicating multiple JPEG resaves, but no obvious splicing or drawing.

The JPEG quantization tables claim that this was saved at 90%, but this is much less than 90%. The difference could be explained by the fact that it is not camera original. So far, this picture could be real...

Cross-Product

One of the tests that I usually do not need to use checks the PCA cross-product. The cross-product identifies a vector normal to two other vectors. It is usually used to identify the angle between two vectors or the orientation between vectors.

In this case, I have two vectors: the PC1 line and a vector from the center of the color cloud to the specific color. Adjacent-but-different colors in the picture should form a sweeping arc between the cross-product vectors. So if the text is gray on white with a faint pink background (as seen in the first photo), then whatever color results from the cross-product, all text should show the same cross-product result.

Colored based on cross-product with PC1.

All of the lottery numbers appear as a turquoise on a faint purple background.

And there's the problem... Don't see it? Mouse over the image to see a blend between the cross-product image and the original. The thing that immediately stands out: the "04 02" is the wrong color. The cross-product lacks the turquoise. The blended image shows the "04 02" as much darker than all other numbers. So we know that at least two of the "winning" numbers are fake.

A little more subtle is the shape of Massachusetts in the center of the ticket (the watermark). The corner of the state under the "38" (line D) has the wrong shape. In fact, it appears to have almost the same shape as the "38" found in line E. So we have two winning numbers that are modified, and a third that is likely modified.

Whoops

However, the big screw-up happened when the guy released his third picture. PCA (PC1 measured from the line) shows that all five of the winning numbers are significantly different from all other lottery numbers; they were modified. The only original number was the bonus "23".

PCA identifies that all five of the winning numbers in line D are forgeries.

Now we know why the second photo only showed the bonus number and why the second photo's ELA was lower. Since the final "23" is real, he didn't need to edit the second picture.

One More Thing

Whenever I show investigators how to do photo forensics, I always emphasize the need to validate findings with multiple algorithms. A single algorithm can trigger false-positive or false-negative results. If something really is tampered with, then it should show up with multiple analysis algorithms. And if something is really real, then it should pass everything. In this case, the first picture fails the PCA cross-product test (and a few other tests, but this blog entry is long enough). And the third picture fails the PCA line test (and PCA cross-product test and a few other tests).

ELA -- by itself -- identifies nothing abnormal with any of these pictures. That would be a false-negative when asking "is it fake?" But combined with (1) the image quality and ballistics, (2) the results from the PCA line test, and (3) the results from the other two pictures, we have a contradiction -- the ELA seems too high. And when tested with PCA's cross-product, we have a high-confidence result regarding the manipulation.

It is one thing to say "the ticket is fake because the state is wrong". It is another thing to be able to show the actual modifications. In this case, we can prove that the picture was modified despite the odd results from ELA.

We Have A Winner!

Dr. Neal Krawetz — Sat, 31 Mar 2012 12:13:44 -0700

Last night's Mega Millions Jackpot of $650 MILLON dollars matched three lottery tickets. Each ticket is worth over $215 million dollars. (Well, before taxes and expenses like hiring an accountant.) The tickets were reportedly sold in Illinois, Kansas, and Maryland.

Over at FotoForensics, it wasn't more than 15 minutes after the drawing that we began to see an influx of fake lottery tickets being uploaded for analysis. Some fakes were much better than others...

Right Numbers, Wrong State

One of the first ones claimed to be from Louisiana and was posted over at 4chan. (4chan? Why am I not surprised...) Error Level Analysis (ELA) immediately identified this fake. (As an aside, I don't mind re-posting the image and direct FotoForensics link since the source originated from a public URL.)

For Louisiana, ELA clearly identifies that the "4" in "04" and "46", and both "23" values were altered.

Similarly, the two-ticket picture of New York lottery tickets that was tweeted by @MelissaStetten shows up as fake under ELA; the "4" in "04", "3" in "38", "6" in "46" and both "23" values were modified to look like a winner.

More than one winner

There are other ways besides ELA to tell that these were fakes. For example, we can check the text alignment. The New York picture is rotated about 22.6 degrees clockwise (measuring a line across the fives in the "C" line). By straightening the picture and adding a grid, we can see that all of the digits fall into straight lines. Well, all except the modified line. In the modification, the "38" and "46" are too low, and the "23" appears a little skewed.

Using the copy-move detector, we can further identify that all three "23" values are identical (cloned -- from line C), and the "04" was copied from line B and not from line A.

Really Winning

Of course, what most people forget (or don't really care about when fabricating pictures like this) is that the winning number isn't tracked by the numbers printed digits on the lottery ticket. It's tracked by the barcode. The encoded number is mapped in the lottery's database to the winning ticket. Even if you manage to change the printed numbers to look like the winning sequence, the barcode will be used to identify the forgery and validate the winner. (But if you know the barcode, then you could -- theoretically -- create a second winning ticket.)

Update: Not every fake lottery ticket shows up under ELA. An example is covered in the next blog entry: Fool Me Once.

Thanks to Than Tibbetts for the pointer to the original tweet.

It's not all about flattening tummies

Dr. Neal Krawetz — Thu, 29 Mar 2012 12:58:52 -0700

We seem to be hearing more and more about photo manipulation in the news. Is it because people are becoming conscious of it, tools are becoming more available (probably not), or that people are starting to realize how manipulative the media really has become. (I'll vote for the latter.)

Dead Cat Bounce

The Shelby 1000 (a car for you non-autophiles) is getting some serious media coverage this week. It seems that someone photoshopped the press release to show the frontend off the ground. (Implying a lot of torque, or as the Honda forums suggest, it could indicate bad motor mounts.)

What really surprises me is that this photo manipulation was identified by someone at USA Today! (I typically associate them with manipulation and not detection.)

When caught releasing doctored photos, John Luft, CEO of Carroll Shelby's Las Vegas-based aftermarket outfit, stated that the pictures were edited tongue-in-cheek and released accidentally to the media. He continued on, saying that the photos were never intended to be misleading.

Yeah, right -- an accident and never intended to be misleading? All of their photos have been digitally manipulated. For example, the picture of Shelby's 2012 Shelby Mustang GT 1000 coupe seems innocent enough; the car is sitting in front of a yellow building. Yet, according to ExifTool, this picture was resaved at least 18 times over a month and a half.

History Action: saved, saved, saved, saved, saved, saved, saved, converted, derived, saved, saved, saved, converted, derived, saved, saved, saved, saved, saved, saved, saved, converted, derived, saved

History Instance ID: xmp.iid:5DC5EBD225206811A84DDF80EC75E9A6, xmp.iid:5EC5EBD225206811A84DDF80EC75E9A6, xmp.iid:BA44176C26206811A84DDF80EC75E9A6, xmp.iid:BB44176C26206811A84DDF80EC75E9A6, xmp.iid:8F223F0E2B206811A84DDF80EC75E9A6, xmp.iid:90223F0E2B206811A84DDF80EC75E9A6, xmp.iid:887F1A922B206811A84DDF80EC75E9A6, xmp.iid:897F1A922B206811A84DDF80EC75E9A6, xmp.iid:8A7F1A922B206811A84DDF80EC75E9A6, xmp.iid:A3908EE7155CE1118387C874E4F410D9, xmp.iid:A4908EE7155CE1118387C874E4F410D9, xmp.iid:A6908EE7155CE1118387C874E4F410D9, xmp.iid:A7908EE7155CE1118387C874E4F410D9, xmp.iid:31E0C2720872E11199D88B59CD45EE3E, xmp.iid:32E0C2720872E11199D88B59CD45EE3E, xmp.iid:33E0C2720872E11199D88B59CD45EE3E, xmp.iid:D7D478623474E111B678BF5FBD5033F3, xmp.iid:D8D478623474E111B678BF5FBD5033F3

History When: 2012:02:12 14:46:44-08:00, 2012:02:12 14:46:44-08:00, 2012:02:12 14:47:02-08:00, 2012:02:12 14:47:02-08:00, 2012:02:12 15:23:44-08:00, 2012:02:12 15:23:44-08:00, 2012:02:12 15:23:53-08:00, 2012:02:12 15:23:53-08:00, 2012:02:12 15:24:22-08:00, 2012:02:20 15:14:51-08:00, 2012:02:20 15:14:51-08:00, 2012:02:21 07:13:06-08:00, 2012:02:21 07:19:29-08:00, 2012:03:20 07:09:34-07:00, 2012:03:20 07:11:12-07:00, 2012:03:20 07:12:09-07:00, 2012:03:22 09:02:09-07:00, 2012:03:22 09:02:09-07:00

History Software Agent: Adobe Photoshop CS5 Macintosh, Adobe Photoshop CS5 Macintosh, Adobe Photoshop CS5 Macintosh, Adobe Photoshop CS5 Macintosh, Adobe Photoshop CS5 Macintosh, Adobe Photoshop CS5 Macintosh, Adobe Photoshop CS5 Macintosh, Adobe Photoshop CS5 Macintosh, Adobe Photoshop CS5 Macintosh, Adobe Photoshop CS5.1 Windows, Adobe Photoshop CS5.1 Windows, Adobe Photoshop CS5.1 Windows, Adobe Photoshop CS5.1 Windows, Adobe Photoshop CS5.1 Windows, Adobe Photoshop CS5.1 Windows, Adobe Photoshop CS5.1 Windows, Adobe Photoshop CS5.1 Windows, Adobe Photoshop CS5.1 Windows

Even this seemingly simple picture of a car in front of a building was touched up to the extreme. (By the way, the meta data says it was taken with a Nikon D2X. But the image is 1419x1296 -- a resolution not supported by the camera. So at minimum there was cropping. A deeper analysis detects at least two different color modifications -- one in the YUV/YIQ color space and the second in the XYZ color space, a conversion in the subsampling, and some blemish removals on the yellow wall.) So to say that a modified photo was "accidentally" released seems like an outrageous claim.

Frankly, Shelby would have been better off saying that it was a tricked out car bouncing. Instead, they claimed that the manipulation was a joke that was not intended to be released.

And keep in mind: Shelby isn't the only automobile manufacturer who doctors photos. They all do it. It's the norm in their industry. And even extreme modifications are commonplace. Otherwise, Ford would have us believe that they have a convertible that can drive really fast without messing up the woman's hair. (And I don't know what to say about Honda's speeding car with tree leaves following it, except "very artistic".) Shelby's press release wasn't an accident. Auto companies have so many people involved in advertisements that they don't let anything out the door without multiple checks and authoritative approval.

For Shelby, caught with a manipulated photo is bad. Caught lying about it is worse. Perhaps this is why their stock is doing the dead-cat-bounce.

If Looks Could Kill

I had a friend who was told by a receptionist to not bother applying for a particular job because he'd never get it. Even though the person telling him this wasn't the decision maker, she had seen enough applicants go through the process that she could already tell the outcome. By the same means, law enforcement officers are trained to evaluate situations. They're not supposed to detain people who do not appear to have been in the wrong. At least, that's how the system is supposed to work.

I've heard lots of different -- and conflicting -- stories about the Trayvon Martin shooting. One version says that the shooter (George Zimmerman) fired in self defense. Another says that he shot an innocent person. The police claim that Zimmerman's actions were justified. The media claims otherwise. Frankly, I'd rather see a trial in a court of law than performed by the media. It isn't like the family doesn't have the option to sue the shooter in civil court. But instead, they have taken to the media rather than the judicial system. And the amount of media bias is astounding.

Now keep in mind, I'm not making a decision about who was the victim, whether the shooting was justified, or even about the accuracy of the information provided by the news reports. I'm only focusing on how the "facts" are being reported by the media.

People have begun to notice how the media has been presenting this case. Most reports show two photos side-by-side. One picture shows Martin (the dead guy) smiling and fresh faced. Next to him is Zimmerman (the shooter) looking dour.

These photos actually remind me of the 2007 campaign photos that were enhanced to show a nice looking Obama countered by a shriveled Clinton. (ELA shows that the picture was enhanced since Clinton is not at the same error level as Obama.)

In this case, Martin's picture is an old photo, showing him younger than 17 years old. They don't show the picture of him with (alleged) gold teeth (seen in a photo from his now-closed online account). And early reports didn't mention his drug-related suspension from high school a month before the shooting, or his other charges including vandalism, carrying burglary tools, and being in possession of women's jewelry (that he claimed did not belong to him).

The Zimmerman's photo is reportedly from a 2005/2006 mug shot for domestic violence, speeding, and "tussling with a police office". (Seriously? "Tussling"? Was that the official charge? Gotta find the humor in media reports.)

Ironically, Martin's family says that pulling up Trayvon's past (from as recent as a month before his death) is a "smear campaign" initiated by the police. While nobody mentions that doing the same thing to Zimmerman (mention his arrests more than 5 years ago) is a smear campaign against him. From my viewpoint, it appears that neither person was an angel. But the media seems to be trying to heavily sway a bias.

As I See It

Both of these stories from this week's news drive home the importance of truth in photography and balanced news reporting. Pictures sway opinions. When combined with biased news reports, they can polarize an entire population.

It is clear that the media is not fair and reports are not balanced. It is also clear that photo manipulation -- whether to make a vehicle look more desirable or a person more innocent/guilty -- is commonplace. It is for this reason that we need to watch closely for signs of manipulation. Not just manipulation to make pictures look nicer, but manipulation designed to sway our opinions.

Thanks to FourAndSix for the news stories, and Woody for the blog title.

You Want My Facebook Password?

Dr. Neal Krawetz — Tue, 20 Mar 2012 19:04:59 -0700

According to the Wall Street Journal, job seekers are being asked during the interview to provide their Facebook passwords.

Wow. The number one rule that every security evangelist promotes is "Don't Share Passwords". This includes giving your Facebook password to a potential employer. Moreover, Facebook is a personal social network with public and private content. What kind of employer wants to judge your worthiness based on private, and not public, content? Does that private message your mom wrote on your wall really reflect your skill set?

Pick Your Battles Carefully

I always say that people shouldn't fight TSA at the airport -- unless you don't mind missing your flight. Instead, file a complaint the instant you get past the security checkpoint. Similarly, if you really want the job, then don't fight the interviewer on this point -- there are plenty of other options available.

So, what should you do if the job interviewer asks you for this information? My first through is to ask yourself: do you really want to work for a company that is this intrusive? But then again, I have a job. Other people are desperate for work and in these desperate times, we do desperate things. However, you can still ask them -- with curiosity and not defensiveness -- why they require this information, who will have access to your password, who will see your private information, and how often will they be checking your account? Emphasize that you take security very seriously and sharing passwords with anyone does not seem like a good idea.

You might also want to mention that sharing your password with anyone is a direct violation of the Facebook Terms of Service. (While their ToS changes regularly, this issue is currently listed under Section 4, "Registration and Account Security", item #8: "You will not share your password, (or in the case of developers, your secret key), let anyone else access your account, or do anything else that might jeopardize the security of your account.") Are they seriously asking you to breach a service contract in order to successfully complete the interview? Let them know that, just as you won't breach any contract with their company, you will not breach a contract with Facebook.

The WSJ article said that people who feel uncomfortable with providing their passwords may be asked to login at the human resources desk or to "friend" the HR person so they can monitor your non-public content. Even in these cases, ask them if you can use your own laptop to login because you are concerned about cookie theft and the security of your account. I'm also fine with accepting HR as a "friend" since you can place friends in categories and you can put their HR person in a restricted category that can only see your public information.

An Ounce of Prevention

If you suspect ahead of time that you might be asked this question, then you can create a new Facebook account before going to the interview. Give it a minimal configuration and a unique (and friendly) password, like "AllWorkNoPlay". Let their HR person read into this whatever they want. Inform them that you rarely use Facebook.

Then again, your existing profile may include your name, pictures, and more. So even a semi-competent HR minion will see through this ploy. Since you're giving them your login, be sure to minimize the damage: don't use that password anywhere else and disconnect your links to Twitter, Gmail, and other services. The last thing you want is for HR to gain access to more than Facebook.

And since you suspect that you'll be giving them access, clean the place up. Remove all swear words, comments that insult anyone, and pictures of you drinking beer or partying.

Oh Those Games We Play

Of course, my brain immediately went to the lottery scenario. I could only wish that someone would ask me for my Facebook information! This is like winning the lottery! You cannot lose here! Some of the options that just popped into my head:

No job? Sue!
Regardless of what they give as a reason for not hiring you, file a complaint that you were dismissed after the interviewer took offense at something in your Facebook account. What was there? Your religious preference? Political orientation? Close ties to family or friends? Taste in music? Whatever it was, that is a bias that is unrelated to your ability to perform the job. Sue them.

Smear it!
I don't care whether you got the job or not. When you get home, anonymously post your Facebook password to one of 4chan's random boards. Let other people into the account -- other people who you know will deface, impersonate, post insults, and worse. Then, blame the company for leaking your password! Be sure to point out that the compromise didn't happen until after you gave them your password during the interview.

If they ask how it was leaked, well, you don't know and you don't have to know. Maybe they didn't have an antivirus or their antivirus was ineffective (accuse them of having malware). Maybe they told someone else who leaked it. Heck, maybe the interviewer leaked it just for fun. You don't care. You only know that your online profile and reputation has been tarnished -- and you are going to sue... right after you report this in a YouTube rant and to CNN!

Of course, no company will ever let this sit. (Particularly if they are a publicly traded company.) They will want to shut you up as fast as possible. "Please don't post to YouTube, give us a moment to investigate this." Apply pressure: don't wait for more than a day because this is your reputation. Meanwhile, contact Facebook and ask them to suspend the account.

The employer won't want the negative press, they won't want a lawsuit, and they will know that they were in the wrong to require your password. Expect a big payout just to keep you quiet. (And if you got the job and then lose it because of this? Get a second payout for wrongful termination.)

Change is Good

But let's say you want to take the easy-out. You don't want to get rich quick through a legal complaint; you just want a job. And you feel that you must show them your account. When you get home, change your password. Nobody except you should know your password.

Or Perish

Dr. Neal Krawetz — Fri, 16 Mar 2012 13:45:21 -0700

One of my coworkers is trying to convince me to submit some of my research to academic journals. Although I reference other people's published papers, I have only made my work public through conferences, blog entries, and white papers; not through scholarly journals.

It isn't that I don't think I could get them published. Or even that I don't want to publish my work. Rather, during my six years of graduate school, I became bitter and disillusioned by the publication process. Here are my six reasons for not submitting for publication...

1. Speed to Publication

The publication process is not speedy. Papers may take over a year just to be reviewed. After the review (assuming no revisions needed), the paper's publication may still be delayed by a year or more as it enters the queue of items needing to be published.

In the computer security world, speed means everything. A month is the difference between a few compromised systems and widespread havoc. What if a paper describing a new way viruses spread isn't published for two years? By the time it comes out, the vulnerability will likely be patched (or made obsolete by updates) and the novel approach becomes old news.

In contrast, conferences generally take six months to a year from submission to presentation. I can submit a talk at a security conference and present the material to my peers long before it would ever be printed in a journal. For an even faster disclosure, I have the option to blog, post to a widely-read forum, or otherwise just put it out there.

2. Peer Review

Some people associate journals with a thorough peer review process. However, I have seen far too many papers that are either incomplete or inaccurate. The incompleteness is usually due to missing steps, unspecified assumptions, or an assumed common background which may not exist.

For example, I was recently reading a paper on saliency. In computer vision, a saliency algorithm identifies the areas that initially attract the eye. This is the visually important area. As Craig Reynolds wrote in LinkedIn's Image Processing Interest Group:

[A]n iconic example of saliency is a photo of a field of green plants with yellow flowers, and one red flower. The saliency map will have a peak at the red flower. It finds areas of the image which differ in hue, brightness, dominant frequency, etc. However ... Imagine a photo of a dozen smiling faces and one frowning face. A human observer would zero in on the frowning face while saliency presumably wouldn't notice it.

The paper I was reading is "Real-time Visual Saliency by Division of Gaussians" (I. Katramados, T.P. Breckon), In Proc. International Conference on Image Processing, IEEE, pp. 1741-1744, 2011. This paper describes a very interesting variation to saliency mapping that is supposed to be wickedly fast. And the paper is well written... until you get to Step 2 on page 2. This step defines a second Gaussian pyramid 'D', but never explains what 'D' is, or how 'D' differs from 'U' ('U' is defined in Step 1 as a Gaussian pyramid). Step 3 references a 'MiR' matrix, but never defines this term. I would hope that, if I could find out what MiR means, I could properly evaluate this paper. But there isn't even a reference to this term (and Google pulls up nothing relevant). In effect, this paper describes the speed of a very fast algorithm, but lacks details needed to understand how the algorithm works.

In contrast, I found the paper "Frequency-tuned Salient Region Detection" to be very informative (R. Achanta, S. Hemami, F. Estrada and S. Süsstrunk, "Frequency-tuned Salient Region Detection", IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2009). This paper is easy to read and the diagrams really explain the algorithm well. (I implemented it in 10 minutes and, with a little tweaking, got it working with even better results than they claim.) If it wasn't for one typographical error in their third figure, I would say that this paper is perfect. (The 'if' condition should compare each S_k to 2×S_µ, and the scalar 2 should be adjusted based on the number of graph segments.)

So the peer review process failed to identify missing steps, undefined variables, and typographical errors in these papers. And this is just from papers I was reading one night; I see this kind of mistake in nearly every paper. On occasion, I review papers for IEEE. I end up rejecting most of them due to readability, missing steps, errors, and false premises. (I still shake my head about a paper on a vulnerability that started off with the assumption that the system was fully compromised by another exploit from the same attacker. Uh, if the system is already fully compromised, then exploiting another vulnerability doesn't gain you anything.)

3. Readability

I'm dismayed by the number of papers that are nothing more than technical mumbo jumbo. I described most of these problems in a previous blog entry. I really do believe that the number of mathematical symbols is inversely proportional to the value of the content.

On occasion I see this leak through to source code. If I see a program with variable names like "omega" and "phi", then this tells me that the programmer doesn't know what the hell they are doing. They are just copying an algorithm that someone else wrote. If you really understand the algorithm, then you will use intelligent variable names. Rather than writing "lambda=7", you might write "AdjustmentRate=7". Frankly, I'd rather see "L=7" than "lambda=7" since at least this tells me that the variable's name is just a shortcut for something, and not the spelled out shortcut to something that someone wrote on a whiteboard. (Writing out "lambda" is like writing "Ell=7". Spelling out the phonetic pronunciation of the single-letter variable doesn't make it any more readable.)

4. References

Back in grad school, I had a teacher who assigned a few books and a ton of papers as light reading. Everyone in the class quickly noticed that every single reference was limited to a small group of authors (which included the professor). It isn't that there were not other authors in this field or that some of those other papers were not as good or better than the ones supplied. This was strictly friends citing friends -- no outsiders allowed.

As it turns out, the problem of cyclical citations is very wide-spread. This is similar to Google's issue with cyclical links in order to raise your page-rank. A small circle of friends repeatedly citing your paper doesn't mean you have a good paper. But unlike Google, the journals don't seem to be actively combating this problem.

On occasion, I have also seen papers that list fake sources, unavailable references, or that ignore the main paper on the topic in order to claim some partial credit for the discovery. But the thing that offends me the most is that most academic papers won't cite non-academic sources. Conference proceedings, presentations, whitepapers, and online resources are usually ignored. (I'm not saying that everyone should cite Wikipedia, but some Wikipedia entries are well written and make good references. It just depends on the Wikipedia article and the current state of the entry.)

This is different from conferences, whitepapers, blogs, etc. -- these types of disclosures have no problem citing references and/or linking to authoritative sources beyond academic journals. And if you miss a big one or attempt to plagiarize work, your peers will let you know.

5. Cost

The price for some journal articles is just staggering. While some papers cost $10 or $20 for people who want to read it, others cost $100 or more. And that's just for the papers. What if you want the whole journal? Now you're talking about the $100 to $2000 range. And a subscription? Prices can range from a few hundred to tens of thousands of dollars per year. (Is the journal Adverse Reactions Titles (Section 38 EMBASE) really worth $31,298 for 12 issues?)

Conferences may not be cheap. Transportation, food, lodging, and the admission fee could cost for a few thousands dollars. But you still get more out of it than one academic journal offers. And that's assuming that you attend in person. Many conferences offer streaming videos, CDs of papers/slides, and more. And most of the expensive conferences have more talks than any single journal has articles.

The best thing about conferences is that it is a true peer-review system. You are speaking to your peers. If you are bad or wrong, they will let you know it. And if your content is good, they will explicitly tell you how good it is.

The blogosphere is another good medium. Content is usually free and peer feedback is immediate (and sometimes brutal).

But what if you want to make money? I mean, it took time and effort to write that whitepaper. Well, self-published ebooks are a good option. If the paper is really good and useful, I'll pay a few dollars for it.

While academic journals claim to be about making information available, they are price limiting. Only the richest can afford this information. I might be persuaded that they are worth the cost if the entries are complete, accurate, and readable, but they usually are not.

6. Variations

At most universities, the quantity of publications is a key tenure metric. So rather than describing everything in one paper, they will spread it over a dozen papers. What you end up with are dozens of very similar papers that differ in the minor details. In order to get the full concept, you must track down all of the variants and find out what each one contains.

Academic papers also have a sense of one-upmanship. Every paper with a minor increase in performance, accuracy, or precision is a 'scholarly topic'. What you end up with are tons of papers with minor variations or special circumstances that claim to be better than other papers.

Ironically, most journals are open to minor variations but closed to completely radical concepts. In the February issue of Wired magazine is a story about a brain researcher who discovered how to erase memories. The concept seems sound and the experiments were reported to be repeatable, provable, and definitive. (The brain creates proteins to recall memories. If you stop the proteins from forming, then you forget the memory.) Yet the researcher couldn’t get published anywhere. This is because things that run contrary to the norm are undesirable to journals. (He should have refocused his paper topic on 'forgetting faster' or 'more accurately forgetting' since one-upmanship is publishable.)

Perhaps the journals are unwilling to risk being proven wrong later. Or maybe they don't want to contradict things that they previously published. Or maybe some of the reviewers had a personal stake in not being proven wrong. ("Contrary research" and "peer review" means you are proving your peers wrong. And if you are trying to get tenure and your work is shown to be wrong, then this impacts your livelihood. Following this mentality, it is better to have a nice house and a steady income than permit correct and radical work to be published.)

The Up Side

There is an up-side to academic publications. These journals are usually the first place people go when doing research. (Well, today they go to Google first and the stacks second...) The content is considered authoritative and the authors are considered experts.

Perhaps I'm just hyper-focusing on the negatives. Are there other benefits from scholarly journals? Are my views too biased?

Keystone Kops

Dr. Neal Krawetz — Mon, 12 Mar 2012 17:40:08 -0700

Ever since Maricopa County Sheriff Joe Arpaio publicly declared that Obama's birth certificate was a forgery, I have been receiving a steady flow of requests to evaluate this claim. Personally, I've been trying to stay out of it since it really isn't new. Arpaio's entire claim is based on the fact that the PDF document contains "layers" and he interprets that as fraud.

I debunked this specific claim of fraud nearly a year ago. The layers are actually due to a simple optimized setting when the PDF was created. Since then, major media outlets have duplicated the same layer-effect by enabling the optimization, and there are even YouTube videos that show how to do it yourself. To reiterate: it is not fraud; it is a configuration setting for one type of optimized PDF. The results are repeatable and conclusive. And most importantly: this claim of fraud was debunked nearly a year ago.

Behind the Badge

Normally I would just disregard this as another Birther conspiracy -- one that the extremists want to believe so strongly that they will disregard all evidence to the contrary. Kind of like how they said Obama was a Muslim who follows a radical Christian Pastor (except that Muslims don't follow Christian Pastors).

However, Joe Arpaio isn't just your run-of-the-mill conspiracy nut. He's a sheriff. He's a law enforcement officer. He has taken an oath to protect and to serve. So when he calls for a criminal investigation, one would assume that he has real facts to back up his claim.

Over at the East Valley Tribune and Politico are detailed write-ups of Sheriff Arpaio's claim. The facts listed by these media outlets are backed up by the press conference that Arpaio held; he really did say these things.

Frankly, I am just amazed that someone who is supposed to uphold the law would conduct an investigation this way. According to Arpaio, both he and Mike Zullo -- head of the "Cold Case Posse" -- relied on research conducted by unnamed 'volunteers'. He said many things that are provably inaccurate. For example, he said that the layers in the PDF were an indication of tampering, even though there are plenty of videos and write-ups that show step-by-step how to enable the optimization and see the same type of layers yourself. One would think that, after six months of investigation, they would have actually seen that it is trivial to duplicate by enabling the optimization. I cannot help by think that the unnamed volunteers are actually people like Ron Polland -- a person who repeatedly lied about his credentials and used pseudo-science to support his baseless claims of fraud and tampering.

More stunning is this quote from the East Valley Tribune:

Zullo also said the fact the Honolulu newspaper had a birth announcement at the time proves nothing, saying the newspaper accepted such messages without requiring proof the person actually had been born in the state.

So... Zullo is implying that the conspiracy to elect Obama dates back over 51 years. I mean, Zullo is implying that the Honolulu newspaper regularly announces non-Hawaiian births for foreigners. This would make sense if the paper had a few million birth announcements for 1961. Otherwise, why would the paper randomly select one alleged foreign birth out of everyone born in the world and forget to mention Kenya? And what are the odds that the random selection became President nearly a half-century later? (Hey, perhaps this is related to the Bay of Pigs invasion that happened a few months before Obama was born...)

I have some serious concerns here. If this is the type of logic that Zullo and Arpaio use for investigations, then all of their previous work should be suspect.

Then again

Then again, this isn't the first time Arpaio has gotten in trouble for shoddy investigations. For example, his office failed to investigate the sexual assault of a 14-year-old. It was later found that his office had failed to investigate over 400 cases of sexual abuse, including crimes against children. According to the Sheriff's office, there was a backlog and it was "cleared up after the problem was brought to Arpaio's attention." I cannot help but wonder how they closed out these cases since they claim to have accumulated 400 cases in 3 years and resolved the backlog in less than 2 years. That's a rate of closing more than 2 cases per day (assuming new reports were coming in at the same rate). How thorough could these investigations have really been?

Sheriff Arpaio has also been sued by the Justice Department for civil rights violations, investigated for deaths and injuries of people in custody, and found to violate the constitutional rights of inmates. (I was surprised to find an entire Wikipedia page devoted to controversies from his Sheriff's office.)

Not Alone, Yet So Alone

Today, most people seem to view the Birthers as nuts who fear a black democratic president -- even though he has already been in office for over 3 years, the world has not ended, and we are much better off compared to how Bush left us.

Most of the notable Birthers have gone quiet on the topic. Donald Trump and Roy Blunt have not said much since Obama produced his birth certificate, Newt Gingrich has mainly limited his comments to political gaffs, and even idiot ~~savant~~ Sarah Palin isn't in the news much anymore.

While researching this story, I only came upon one interesting birther update. Last month Orly Taitz managed to bring her complaint -- that Obama is not eligible to be President -- to a court of law. The defense (Obama's legal team) decided that the claims were so baseless that they didn't even show up to court. That's right, Taitz was allowed to make her claims and present her evidence without any counter argument or challenge from the defense. And she still lost. The judge found that Taitz offered no credible evidence and had no case.

So I have to wonder: if the courts believe Obama qualifies for the Presidency, all Birther arguments about a fake birth certificate are easily explained with rational arguments and provable demonstrations, and even the State of Hawaii has consistently said that Obama is a US citizen, why is it that Sheriff Arpaio could not uncover any of this in his "six month investigation"? I can only conclude that Arpaio either did not conduct an investigation (fraud) or was so grossly incompetent that all investigation performed by his office should be reviewed for criminal ineptitude. (Arpaio: I hope you like pink underwear.)

Use Cases

Dr. Neal Krawetz — Sat, 03 Mar 2012 15:23:26 -0700

One of my big concerns regarding any kind of public forensic analysis site is that it would be misused. I worried that people would upload pictures and -- regardless of the results -- claim that a picture was real or fake (or whatever supports their viewpoint). For this reason, I created a basic tutorial and FAQ to help people understand the results. Then again, I was also concerned that people would ignore these documents.

Well, I am no longer worried. I can now honestly say that the tutorial is being used. I've been watching the web logs and the vast majority of new visitors either go the the tutorial and then upload pictures, or upload a picture and then go to the tutorial.

The Naked Truth

We've been working on a few behind-the-scenes items. For example, we're getting better at rapidly banning perverts who upload porn. As the FAQ states, FotoForensics is not for your personal porn archive. Right now, porn comprises about 3% of the uploaded images but comes from far less than 1% of people who upload images. However, without content filtering the perverts would run rampant over the entire site.

We have a simple policy concerning pornography: block and ban. The content is blocked and the person is banned. And we might even choose to humiliate perverts who complain about being banned without telling the whole truth about why they were banned. Keep in mind: we are only censoring pornography. So if you see someone complain about the site banning users, you should ask them: what kind of perverted content did they upload? (And if you're curious or bi-curious about Ricardo, he's an ass-man.)

So far, the most disturbing source of pictures has been from the Corrections Corporation of America. CCA runs many of the US prisons. I sent them a notification through their site's contact form, but haven't heard a reply. The disturbing part is that I don't know if it is an inmate or a guard. Then again, a major airline appears to employ a pervert with child-related fantasies and that company didn't reply either.

From Abuse To Use

The vast majority of the people who use FotoForensics are not uploading porn. (97% of the pictures and over 99% of the site visitors are basically workplace safe. And 3% of the pictures have already been blocked and removed.)

Today I decided to search and see who was using FotoForensics. I was thrilled to see that Google has already indexed plenty of people using the system -- and using it correctly! (I guess the tutorial works!)

For example, over at Fstoppers, someone used FotoForensics to identify that pictures of the new Canon EOS 5D Mark III were actually modified. And they are correct -- the pictures are not camera-original. This particular ELA result is typically found with artificial sharpening. Someone wanted the picture to look sharper than the original photo appeared. However, this isn't too grievous of a manipulation and does not appear to have been malicious.

Red Light District

But my favorite use of FotoForensics so far comes from the UFO community. I like these people because many of them really do want to investigate and not just blindly accept things as fact.

Over at 2012hoax.org is a posting about a Big Red Object New Mexico. This is a continuation of the original posting found at Above Top Secret.

The photo shows the night sky, a bright white object (the moon) and a bright red object near it (the unidentified object). According to the posting, "ALL HE DID WAS EMAIL THE IMAGE FROM HIS PHONE TO HIS COMPUTER THEN POSTED IT ONTO THE WEBSITE" (it was written in ALL CAPS). Here's the picture:

There are so many problems with this picture that it makes a great analysis example.

Meta data. The first claim is that this came directly from his cellphone camera. However, it is missing camera meta data; it is not camera-original. Moreover, it has an embedded JPEG comment: "CREATOR: gd-jpeg v1.0 (using IJG JPEG v62), quality = 80". So now we know that it used a software image library to last save it at 80% quality.
Quality. Just looking at the picture, you can see grainy 8x8 blocky artifacts. This comes from multiple resaves.
ELA. The error level analysis is mostly black, supporting the observed artifacts from multiple resaves. The picture uses 16x16 chrominance subsampling, so I've drawn in the 16x16 grid.

Notice how the red dot impacts the four 16x16 grids that contain it; it is at a much higher quality than the rest of the image. If this were simply an issue of the red dot being bright, then the white/moon should also have a higher error level potential. If this red dot were original, it would look similar to the white light's faded ELA result.
LG. The luminance gradient shows us something else.

The white light has a very wide gradient -- like a real light. In fact, the spurious noise seen far from the white light actually seem to form a loose ring around the white light. In contrast, the gradient from the red object is strictly limited to the four 16x16 squares; there is no ringing and it cuts off sharply. If it were really that bright, it would radiate out further than 16 pixels from the center.
PCA. The Principal Component Analysis is sensitive to the image quality and JPEG artifacts. (I've applied a colorization just to make the dark picture easier to see.)

The overall background is a solid purple color -- it is such a low quality that it appears uniformly colored. The white moon has very large artifacts that align with the 16x16 grid. So this is a very low-quality picture. However, the red light has very fine details. Relative to the rest of the picture, this is a high-quality region.

The people at 2012hoax noticed a few other things as well. For example, the photographer described the camera settings (focal length, exposure time, etc.). However, this picture's quality is inconsistent with those settings.

The net result? The red light is a hoax.

The Real World

Dr. Neal Krawetz — Sun, 26 Feb 2012 19:29:46 -0700

This has been a crazy month, so I haven't had much time to cover all of the topics that have been on my mind.

Earlier this month, the Washington Post featured a cover photo that used high dynamic range (HDR) imaging. HDR is a type of pseudo-color that gives pictures a wider color range. The use of the picture caused some controversy since photo-journalism is not supposed to manipulate the image. This controversy peaked when Sean Elliot, the president of the National Press Photographers Association, declared that "HDR is not appropriate for documentary photojournalism."

The net result? Press photographers should not use high dynamic range (HDR) imaging.

Color My World

Elliot's ruling makes sense from a manipulation standpoint. The application of false-color can alter the meaning of the image. For example, are the false color images released by BP (after the gulf oil spill) an artistic rendering, or an attempt to make the brown water look blue and the brown sand look white?

Of course, not everyone agrees with Elliot. For example, John Omvik at Unified Color strongly disagreed. He points out that photos captured by the camera don't always look like what our eyes see. HDR can make the picture appear more like how the photographer saw it. As he wrote: "Using HDR software and processing tools is the only method a photographer has to deliver precisely what he or she witnessed at the time of an image capture."

I'll take a moment to point out that John Omvik is the V.P. of Marketing at Unified Color Technologies, and UCT creates HDR software. So it in his company's best interest to try to promote this technology.

The one big thing that caught my attention came from Omvik's description of HDR's potential. He wrote, "When properly used, HDR does the most accurate job of reconstructing the dynamic range of the original scene at the time the photo was taken."

And that's the problem: "when properly used". There is no license or certification for identifying HDR competency. And there's no deterministic method to identify when a picture's color has been over-corrected.

There's also the little problem about color -- we all see things differently. While the artist (or for people who feel slighted, "the photographer") may try to match the blue in the sky, he may not notice that he's blown out the blues in the water.

A Vision to Behold

Personally, I experience this color issue daily. I wear glasses and glasses refract light. I've noticed that I have a choice concerning the way I see the world. I can either wear my glasses and see the world with crisp edges but muted colors, or I can remove my classes and see the world as a big blur with vibrant colors. (If you're nearsighted, try looking at something bright and colorful with and without glasses. Which looks more colorful to you?) So if I adjust the photo to match what I saw, then you'll either see something blurry but colorful, or you'll see something crisp but with colors that don't pop out.

With a camera, there are many components that impact the colors in the captured photo. There is the lens, color filter array, sensor, JPEG quality, and JPEG color loss. All of these add up to colors that approximate the scene but do not duplicate it. However, this approach is deterministic and not an arbitrary skew based on the photographer's recollection of how bright elements appeared.

Consistent View

Personally, I think Elliot's opinion is consistent with industry practices. For example, the National Press Photographer's Association (NPPA) has a published Code of Ethics. It states that "Editing should maintain the integrity of the photographic images' content and context." It's about the photo and not the photographer. Or more specifically, it isn't "as the photographer saw it" -- it is "as the camera recorded it".

I previously wrote how major photo suppliers, such as Reuters, AP, and Getty Images, all have requirements concerning photo accuracy. Getty goes so far as specifying specific cameras and permitted camera functions; nothing else is approved. And for clarity, Getty and the other media outlets have not approved HDR for photojournalism.

Awesome First Week

Dr. Neal Krawetz — Sun, 12 Feb 2012 12:49:51 -0700

The FotoForensics site has been up for a week. I'm thrilled -- it's been years since I needed to create a new, hardened server and I half expected auto-scanning bots to compromise the site through some unpatched vulnerability. I've already recorded a few dozen automated attacks; the first happened about 4 hours after I got the web server installed. (For a comparison, the Google and Microsoft search engines indexed the site faster than all of the attack-bots -- and I didn't even have links on the site when Google found it!)

I can honestly say that installing a "more secure" system is easier today than it was a few years ago. Just four years ago, there were dozens of services and auto-update systems that posed risks and needed to be disabled. Today, well, most of the system was secure right out of the box, and there are some great instructions for quickly securing the rest. I decided to go with an Ubuntu Lucid Lynx (10.04 LTS) since it is the current version with long-term support. The install is the basic server, not the full desktop, and I added in whatever minimal additional tools I need. (Thanks to Bill and Chris for the installation!) I was provided with a basic install and one remote login account. I hardened it from there, installed the web server, scripts, and detection/prevention elements.

I consider this exercise to be a good refresher and learning experience. I've been so engrossed in computer forensics that I haven't had time to keep up my sysadmin skills. (It's like falling off a bike. You never really forget how, but you can always use the reminder.) And as a learning exercise, I certainly got some great experience.

Lacking Sense

The first big problem to overcome came from Websense. They provide a web filtering system that blocks hostile sites. And a huge number of big companies use their filtering software. The problem is, Websense blocked my site. According to their online "try-before-you-buy" ACE Insight system, my server was classified as "Suspicious content was found or the reputation of the server is low." Their security categories said that the site contained "Potentially Damaging Content".

However, their scanner also reported that there were no viruses detected, no links, and they really didn't know anything about the site.

When I clicked on their scanner, I watched the system for their accesses.

204.15.64.200 - - [09/Feb/2012:11:43:31 -0600] "GET / HTTP/1.0" 200 1021 "-" "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2; SLCC1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 2.0.50727)"

They only went to "/", and that only returned HTML. No Java, no JavaScript, and nothing active. So how can plain HTML be hostile? Even the text in the HTML was passive ("Submit a picture for Error Level Analysis") -- it isn't like it said "boobs" or "exploits". Websense claimed that there were no links, but the site did have hyperlinks to other pages on the same site. Yet, they didn't even scan the rest of the site; they didn't download the CSS, any images, or even visit the other pages. Is Websense really afraid of plain HTML?

I wrote to an acquaintance at Websense and asked why they had blocked me. It turns out, the ASN assigned to the hosting subnet also hosted some porn and warez sites. As I understand it, those sites get shutdown if they are reported and found to be conducting illegal activities. However, because they are/were at the same upstream provider, Websense decided that my server lives in a bad neighborhood and blocked it by default. This explains why they blocked access, but not why they claimed that the site contained potentially damaging content -- even after they "scanned" it and found nothing harmful.

After sending my query, Websense reclassified FotoForensics as "Computer Security", noted that there is "No threat detected at this time", and removed the block. However, it is still listed as a "Potential Security Risk" because it is new (low reputation).

Considering that Websense describes themselves as "a global leader in unified Web, data, and email content security, delivers the best security for modern threats", one would think that they would actually check for modern threats before claiming that a site poses a risk. Blocking access to a site and classifying it as a source for "potentially damaging content" without checking first seems along the same lines as slander, libel, and security snake-oil.

Are they real?

The second issue I needed to contend with was porn. As stated in the FotoForensics' FAQ, the site administrators may review content. I've been keeping watch on the site to make sure it isn't compromised, everything stays up and running, and the content doesn't violate US laws concerning pornography and obscenity.

Even though the site has only been up for a week (and less than a week since the formal announcement), I'm already seeing a few patterns. For example, 99.9% of the uploaded pictures seem to be sent once and then not visited again. But the few pictures that are revisited by other people seem to be very popular.

Most of the uploaded pictures seem to be associated with controversy. Basically, someone somewhere said that it looked fake. The second biggest category are pictures from the media. It seems that many people don't trust what they are shown. (People are skeptical! Awesome! This reestablishes my belief that people are not as stupid as the media wants us to believe.)

Unfortunately, about 5% of the uploads right now are porn. And I'm not taking about the good kind. (Girl-on-girl with spatulas and duct tape is just wrong.) In the first 24 hours, I had to add an enhancement to the admin script. I had a means to quickly view, block, or delete content, but I also needed a way to ban users. You see, regular people seem to just upload one or two pictures at a time. But if there are a bunch of pictures uploaded in a short amount of time, then it is usually the creepy porn. These perverts do not seem interested in photo forensics; they seem interested in uploading their collections. So, I ban them. Right now, about a dozen users are banned. That isn't much, but it cut off 5% of the uploaded content. (Like the 90-10 rule: a small group of people cause the most problems.) I also updated the FAQ just for the pervs: this site is not for your personal porn collection.

On the positive side, a few people are giving the ELA system a serious workout. Some have even submitted controlled analysis experiments. AWESOME! I'm eager to hear their thoughts if they are interested in sharing.

What's next?

Based on the content I've seen and a little feedback I've received, I'm currently updating the FAQ and training page. In particular, I'm adding descriptions about how to get better results. (I'm seeing a good number of pictures with multiple resaves and scaling, and those deter ELA analysis. For better results, find a picture that is closer to the original and not a resave of a resave.)

I promised myself that I wouldn't update or expand the system for at least a while -- just to give it time to see if the current system works. However, I do have dreams of it doing more than just ELA. So I guess that's one question I'll throw out there: what else would the users of this system like to see? My dream list includes:

Comments. Something like Disqus or Intense Debate, where people could comment on uploaded pictures. However, this requires a moderator and I don't have that kind of time. (It would need volunteer moderators.)
Forum. Perhaps the site could use a forum where people comment on pictures, assist with analysis, or discuss anything related to computer forensics. (Again, would require assistance since I really hate spammers and have no time to moderate.)
More analyzers. ELA is cute, but other analyzers would be even more entertaining. I'm actually thinking that people could submit their own analyzers for inclusion on the system. (I'd need to document the API and write up requirements -- like must run in real-time and not consume 100% CPU or 100% memory.) Of course, there would need to be a training page for each analyzer that says what it does, how it works, and the limitations... I am also thinking that I might add ExifTool to the analysis process -- many photo-sharing sites already use ExifTool for meta data. But if there are no volunteers, then I'll just make it do what I want (which is probably not what other people want).
Detectors. Right now, the site doesn't generate any interpretations about the data. Even things like "outside photo" vs "night shot" or "in a room" would be awesome. Other detectors, like a people counter (e.g., "there are three people") or identifying known objects (e.g., "there is a car") would be very cool. (Can anyone reliably tell if the picture contains pets and distinguish dogs from cats?) Automatically identifying real vs graphics and edits would be a great bonus. This would be along the same lines as the analyzers: if other people with existing (real or experimental) detectors want to help, I'm open to the idea, but if people expect it from me, then you probably shouldn't hold your breath.

So having said that, FotoForensics has a lot of potential to be more than an error level analyzer. What else would people like to see and would people be willing to contribute?

I still have serious concerns about distributing code that bad guys can use for offline analysis. If they had the tools, they would attempt to exploit existing analysis method -- either to hide their identities or frame other people. So even with an API and contributions, I don't see this becoming an open source project.