Secure Computing: Sec-C

In my previous blog posting, I mentioned how some people really do "get it" when it comes to digital manipulation and photo fakery. However, others like "photographer" Nicholas Routzen and BP's Marc Morrison still don't understand why representing modified photos as if they were "real" is nothing other than fraud.

BP was heavily criticized in the media for releasing edited photos. In fact on 22-July-2010, White House Press Secretary Robert Gibbs even commented that it was sheer stupidity:

"I think it's genuinely on the stupidity part of the transparency scale," Gibbs said this afternoon at the White House daily briefing. "I mean, if you want to show a picture of what the room looks like, just take a picture."

Upon the discovery of BP's digital manipulation, BP decided to come clean. Sort of. It was actually more of a "throw the photographer under the bus" than an actual correction:

BP cast the blame entirely on a hired photographer and claimed to have no part in the decision to alter the photos. "One of BP's contract photographers used Photoshop to edit images posted on the bp.com Gulf of Mexico Response web site," the company said, adding, "[W]e've instructed the photographer who created the images to refrain from cutting-and-pasting in the future and to adhere to standard photo journalistic best practices."

Too bad this isn't an isolated incident... and it still has not stopped.

As part of their corrections, BP created a special Flickr set where they show the before and after photos of the three pictures that America Blog and Gizmodo identified as modified. However, BP is only showing the three outed photos.

Standard is Better than Better

I really like that phrase, "Standard Photo Journalistic Best Practices". There is no such standard. As I detailed last year, different organizations have different rules about acceptable manipulation. However, there are some generalizations that can be made.

For Photographers

In general, if the photo is supposed to represent something real then the person providing the photo to the media should abide by these guidelines (a combination of rules from Reuters, Associated Press, Getty Images, and other photo providers including China's Xinhua news agency):

• No splicing, no drawing. Whether it is for removal or enhancement does not matter. You never splice images and you never alter content. If Billy blinked during the photo, then the photo must have Billy's eyes closed -- don't draw in eyes or splice them from a different photo.
  
• Minor cropping. A little cropping from an edge ("little" as in "up to 5% from an edge") is acceptable if it does not remove a subject from the image. Major cropping, such as Morrison's removal of the entire upper half of the photo, chairs, and two people (before and after) is not permitted.
  
• Minor dust and speck removal. Minor dust and speck removal from a non-critical section of the photo is generally permitted. However, this really depends on the photo provider. Some providers (like the AP) are more critical than others. In general, if there is a tiny speck of dust on the lens that ends up looking like a distant UFO in the sky, then you can remove it. But if there are lots of specks, then you must suffer with a dirty picture (next time, clean your lens!). And if the speck is located on the subject matter (like Hillary Clinton's shoulder), then don't touch it!
  
• Minor color enhancements. Color corrections that do not alter the subject are permitted. As the AP described:
  

  Minor adjustments in PhotoShop are acceptable. These include cropping, dodging and burning, conversion into grayscale, and normal toning and color adjustments that should be limited to those minimally necessary for clear and accurate reproduction (analogous to the burning and dodging often used in darkroom processing of images) and that restore the authentic nature of the photograph. Changes in density, contrast, color and saturation levels that substantially alter the original scene are not acceptable. Backgrounds should not be digitally blurred or eliminated by burning down or by aggressive toning.

  
  In general, any acceptable, minor color enhancements should be equally applied over the entire image and not isolated to a specific region. Highly focused or region-specific color alterations are no different than drawing. ("Select all" or "select none", but if you touch the selection tool or magic wand then you are drawing.)
  
• After effects. Blur, sharpen, smudge, liquify, rotate, blend, and other enhancements are not permitted. Basically, if it is not found in the real picture then this is considered "drawing".
  
• Acceptable drawing. There are only two situations where drawing, such as blurring or using a black censor box, are permitted: (1) to protect anonymity, and (2) to prevent advertising. A blurred face, blurred logo on a hat or shirt, or black box over a license plate is acceptable. However, the blurring/drawing must be blatant and obvious.

For Media Outlets

The photographers who provide the photos to the media must abide by much stricter rules than the media outlets. In contrast, outlets are permitted to perform manipulations that match their medium and format. These include:

• Scaling and cropping. While the photographer must provide the whole picture, the media is permitted to crop it to show a specific subject matter. For example, the photo may show President Obama surrounded by thirty people, but USA Today may crop it to just show his head.
  
• Color adjustment. The media, particularly printed media, are permitted to color correct an image. The photographer's picture may be too dark or not print well for a magazine. The media outlet can, and usually does, color correct the image. In this case, the pictures may actually be color corrected in specific regions or specific color bands, and not applied uniformly across the image.
  
• Acceptable drawing. The media may also choose to blur faces, censor logos, or annotate features with arrows or circles that highlight specific items. However, these modifications must be blatant. The subtle removal of a logo, when discovered, can lead to sharp criticism (or worse).

BP used to take photos and use them in their advertising campaigns; anything goes in advertisements. However, that role has changed. Since the Gulf disaster, BP has been providing photos that document recovery and cleanup efforts to the mass media. As someone who provides photos to the media, BP is expected to adhere to the higher standard. BP should not be making modifications reserved for media outlets.

BP: Best Practices

Unfortunately, BP seems to be making up their "Standard Photo Journalistic Best Practices" as they go. While I have not seen any splicing in the last few days, some of their photographers are still taking liberties with the crop tool and recoloring. Here are a few examples from BP's Flickr feed. (Click on the photo to see the full picture.)

Creative Cropping


This photo by Marc Morrison is dated 26-July-2010 but was last modified on 27-July-2010. The full picture is 3981x1496. The problem is, the Canon EOS-1Ds Mark II does not take photos at these dimensions. The closest it gets is 4992x3328. This means that Marc cropped nearly 20% from the horizontal and over 55% from the vertical. So what did Marc not want us to see?

A few years ago I was told a story about a photo from China. It appeared to show a government vehicle with people standing around it cheering. But the uncropped photo showed the crowd throwing stones; the people were not cheering, they were yelling. Creative cropping can alter the meaning of a picture. For this reason, "Standard Photo Journalistic Best Practices" requires the photographer to submit the whole picture and not something with creative cropping. For all we know, there could be a dead whale on the right, and that gray structure in the top-left could actually be pollution filling the sky. If the picture has too much sky, then BP needs to let the media outlets decide what to crop.

BP's True Colors
Here's a very colorful photo by BP:



This photo by Harrison McClary is dated 26-July-2010 and last modified a day later. The image itself measures 3600x2400. That is close to a native resolution for the Canon EOS-1D Mark III, which can take pictures at 3888x2592 (cropped or scaled 7% horizontal and 7% vertical). However, McClary over-applied the color correction. We can see this in the color histogram (graphing HSV).



There are two things that really stand out as abnormal: (1) the clusters of blue and yellow at the top shows a blown-out color space, and (2) the wide color blobs are too wide, too tall, and too blended for a natural picture. This is not a typical color space for a Canon EOS-1D Mark III.

For a comparison, consider this sample photo from the same model camera (and not provided by BP):




Notice how the unmodified photo does not blow out colors at the extreme intensities, and has less-blended color bands. This is very typical for a digital camera, including cameras made by Canon, Olympus, Nikon, Ricoh, and other manufacturers.

So why would BP's Harrison McClary over-correct the color space? Perhaps he is inexperienced with cameras. Or maybe he really wanted that brown water to look blue. By blowing out the color spectrum, he has given the image a "clean" look -- the sand is white, the sky and water are blue, the tractor does not look dirty, and even the brown grasses look green.

Here's another example from Harrison McClary:




Again, the blue and green are blown out (blobs at the upper intensities). Also, notice how the orange spike actually curves with intensity (vertical). That's why they call it a "color curve adjustment".

Of course, McClary isn't the only one tweaking colors. BP's Robert Seale also did some color corrections.




Notice how Robert's dark red, blue, and green all lean toward the left at the top? While he didn't blow out the color range, he did adjust the sky, grass, and maroon stripe on the bookmobile (the RV in the background-right that says "Vermilion Parish Public Library").

Seeing Red

Dear British Petroleum,

If you want to us to believe that the pictures are real, then please release real pictures. Don't crop out stuff you don't want us to see. Don't make the sky and water look bluer. And most importantly, don't think that we won't notice.

Having been caught splicing images, BP promised to adhere to "Standard Photo Journalistic Best Practices". However, this is clearly not the case. While BP claimed that the modifications were limited to one photographer, the actual problem is more systemic. BP's photographers may no longer be splicing, but they are still striving to literally show that the grass is always greener. This isn't a problem with BP's photographers; this is a problem with BP.

This week really gave me a thrill. Readers, models, and even large companies have taken steps against digital photo manipulation in the media.

The first big congrats goes to Domino's Pizza. They recently announced a promise to use real photos of real pizzas in their advertisements. No more cardboard, glue, and partially-cooked food that looks "better" when photographed.

Our Photo Promise
Here at Domino's, we don't think our inspired Domino's pizza needs the "extra" things typically done to food at photo shoots to look mouth watering. Our pizza is good enough to stand on its own. That's why we're making the following promises about how we photograph our pizzas from this day forward. Did we just buck the food photography trend? Oh yes we did.

1. We will only photograph real, honest-to-goodness pizzas.
That means fresh from our own ovens, with exactly the same ingredients we deliver to your doorstep. Nothing else added.

2. Our employees will make the pizza we shoot.
Not an art director or model maker or food stylist. A Domino's employee trained to make pizzas the only way they know how: by hand.

3. We will not artificially manipulate the food we shoot.
No tweezers, no steam guns, no model knives cutting perfect perforations in the cheese. The only thing that will touch the pizzas we shoot is the pizza-maker's hands and a standard Domino's pizza cutter.

Russell Weiner, Chief Marketing Officer

Bravo! I've looked at some of the pizza photos on their web site and I must say: no detectable manipulation (beyond scaling and cropping, which does not modify the look of the food). Moreover, the food actually looks good! (Good enough for me to now have a pizza craving.)

Pizza Photo by Makena B. from Houston, TX

Worth the Wait

Not to be outdone, plus-size model (and super hottie) Crystal Renn just went on the record saying that she is offended by some photoshopping done to her picture. As she said in her Today Show interview this morning, "When I first saw the photos, I would have to say I was absolutely shocked." The photographer turned this well-known size-10 into much thinner version. (But at least he didn't give her noodle arms, right Ralphie?)

The photographer, Nicholas Routzen, has this reply:

I want to reiterate that I feel Crystal looks amazing in both images and the minimal retouching that I did do - it's nothing you wouldn't see in any magazine today. There is nothing hidden about this.

This tells me three things: (1) he sways to peer pressure (everyone else is doing it...), (2) he does not listen to the models that he shoots (Renn has been a strong voice against the unhealthy, unrealistic anorexic female shape that most of the fashion companies strive for), and (3) he photoshops his pictures. It makes me want to take a much closer look and see if he also does splicing, smoothing, and other common forms of deceptive manipulation.

However, I would not recommend browsing Routzen's blog. Some of his photos could easily pass for child pornography. (Full frontal nudity of a minor.)

Feeling Pumped

But I am saving my largest applause for America Blog and Gizmodo. These people have been looking at the media photos released by British Petroleum (BP).

It isn't enough that BP's runaway deep-sea oil well poisoned the Gulf of Mexico, after they lied to the United States by claiming that they knew how to handle any deep-sea accidents. Or when they repeatedly underestimated the amount of oil and would not assist scientists in creating an accurate estimate (we still don't know how much oil was leaked). Or that they only provided low resolution video feeds to the public while they had high resolution footage available. Or that they tried to stop the media from documenting the disaster. No... they also have to doctor pictures. (Is anyone really surprised?)

One photo has the title "Aerials over Gulf of Mexico". With a name like "aerials", one would think it would be taken from the air...

The problem is, the view out the window has been photoshopped. I noticed many things in this picture, but the people on Reddit just shredded the photo. Some of the findings:

• The display clearly says that the door is open, ramp is open, rotor brake is on, and parking break is on. There is no way this helicopter is in the air.
  
  
• The radar shows something to the far left, but nothing in front of him. Thus, no boats.
  
• There is a light that says APU GEN ON. This is the alternate power unit. It provider power until the engines are started.
  
• The pilot is holding a pre-flight checklist. (Ironic that his fingers are crossed.)
  
• Neither pilot is holding the flight stick!
  
• There is a waterbottle resting in the handhold above the guy on the right. The water in the bottle is smooth and flat -- no vibrations at all.
  
• The pilot on the left is wearing glasses. The glasses are reflecting some type of straight-line object. This is likely a runway or edge for the helipad.
  
• The outside water goes from clear blue to smokey. You can clearly see the waves in the blue and smokey areas, but the waves are fuzzy/blended where the two meet.
  
• The water is also blurry around the pilot on the left and near the top of the right window.
  
• The edge of the boats on the left are precisely in the fuzzy section.
  
• The first boat in the right window has a very visible shadow. So all boats should have shadows. However, none of the boats in the left window have shadows.
  
• The boat with shadow indicates that the sun in in front of the helicopter. However, the entire copter is in shadow and so is the tower structure (top left).

This isn't even the entire list. It is suffice to say that this is not an "aerial" photo and it has been grossly modified.

Another photo shows people in front of some monitors. The problem is, the image shown in some of the monitors was changed. Technically, content from three screens was replicated into the three off-line screens. Oh, and the picture has an internal timestamp indicating that it was created in 2001 (2001-03-06 15:16:50.25) and not 2010 (EXIF data modified time 2010-07-19 18:54:04.25). In either case, the timestamps do not match the "HIVE at Houston Command Center 16 July 2010" as BP captioned the picture.

Modified

Allegedly Unmodified

The final picture (so far) shows people in a meeting room. However, the splicing of the content on the screen was done very poorly.

Here's a closeup of some of the splicing:

Frankly, I'm not sure what is more offensive -- the fact that the picture was modified, or the quality of the modification. In either case, this should be a firing offense.

Of course, I began to do what everyone else is probably doing -- poring over bp.com and looking for more doctored photos. That's when I noticed something. All of the modified photos appear to have something in common. The meta data and associated credits identify the photographer as "Marc Morrison".

Hello, Marc

According to his bio, Marc has been a photographer for 26 years and works for BP. A significant number of photos released by BP were taken by Marc.

Marc prefers Canon cameras like the EOS-1Ds Mark II or EOS 5D. While these cameras usually take very good photos, Marc's pictures always have a large mount of sensor noise and discoloration. (I can actually pick out Marc's photos on BP's site just by looking for the sensor noise and grainy coloring. Not every picture has had content modifications, but all look grainy and noisy.)

When it comes to manipulation, Marc seems to rely on overlaying and blending. He primarily targets flat surfaces like monitors or windows. His non-grainy photos appear to have color enhancements to make bright colors pop -- look for things that are red or yellow (his favorite bright colors). I have not seen him advance to people splicing, reflections, or lighting. He also appears to be fond of image cropping; I have yet to see any of his photos that are anywhere near close to a native camera resolution size. Oh, and Marc likes to use something called Photoshelter. (Since I have no experience with it, I can't tell if it is a program for editing or only web creations and annotations... In either case, many of his photos were modified by it.)

Two photos by Morrison. The left is "AP Photo/BP, Marc Morrison". The right is "EPA/Marc Morrison/BP Handout". Both show the same room and same people but at slightly different times. Some monitors are the same, some are different. Is either unmodified?

Now, for clarity, there appears to be many photographers named "Marc Morrison". One lives in Steamboat Springs, Colorado -- I really don't think it is him. Another lives in Houston, Texas. The Houston guy seems to take some celebrity photos as well as plenty of oil rig and related industrial photos. However, I haven't seen anything that says the guy in Houston works for BP. (This Marc could be a different Marc.)

In any case, many of the photos provided by BP's Marc Morrison were credited as "AP Photos/BP, Marc Morrison" and "Marc Morrison - AP". (Example: Washington Post, look at the slide show.) However, I cannot find any of Marc's photos at AP's web site. I wonder if they already booted him for altering images...

(Thanks to the 11 people who sent me links to this BP story. Keep 'em coming!)

The hardest parts of forensic analysis isn't the tools; it's the training. Anyone can buy rubber gloves, swabs for collecting blood samples, and plastic evidence bags. But if you are not trained to properly collect, handle, and evaluate evidence, then the tools and methods are meaningless.

The learning curve is the hardest part. To address this, I've been working on documentation and worksheets for digital image analysis and photo forensics. While there is still a steep learning curve, the investigator can review the worksheets as a checklist for common things to evaluate. The associated documentation provides details regarding the checklist items, in case the investigator needs to review how a particular system works.

An Eye For Details

While luminance gradient and error level analysis draw pretty pictures, the most important tool is basic observation. It is one thing to see the big and obvious signs of manipulation. It is something else to remember all of the fine details.

The folks at Photoshop Disasters recently posted a couple of amazingly bad shopped pictures that clearly illustrate the power of observation for detecting image modifications.

The first picture comes from an ad campaign for fingernail polish. The picture is supposed to show a model and some nail polish. The magical stars that go from her elbow to the picture frame are just artistic. However, it is the fine details that make this such an obvious disaster... Just using your eyes, what stands out as abnormal and not intentionally artistic? Give yourself a minute to look over it, then scroll down and see how many things you noticed.



If you only saw the disconnected leg, then give yourself one point. (If you didn't notice the leg, then go back and try again. As Thall commented at PsD, "That women could birth a horse or two with those hips!") Other oddities include:

• Her waist is out of proportion.
  
• There is a black triangle between her torso and floating leg. The artist forgot to cut out this area.
  
• Her left forearm (photo right) is significantly longer than her right forearm.
  
• She has two thumbs on her lower hand. (One thumb could be her foot showing through a strap in the shoe, but it is actually blended into the hand.) Oh, and don't mind the "S" on her finger; other photos show it as a tattoo or something.
  
• Her neck is showing a tendon, indicating that her head is turned. However, her head is looking straight and is not centered on the neck. Yes, they cut off her head and pasted on a different head.
  
• She is missing a clavicle (shoulder bone) -- one is there, but the other was erased.
  
• Her boobs are different sizes, and not in a natural way (unless the left one is half as long and deflated...).
  
• The shadow under her head indicates a bright light to the upper right. But the floating leg isn't casting the same shadow onto the other leg. And the shadow from the sleeve onto her straight arm has a shadow going the other direction. Inconsistent lighting means splices.
  
• Of course, all of these are issues with the woman. Over at PsD, ZaphodQB noticed that the reflection of the black polish does not meet the bottom of the black bottle.

This isn't the full list. What else do you see? No wonder their product is called "Oops!"

The Perfect Model

I'm always looking for good sample images that demonstrate specific points. Ideally, I want one picture that only demonstrates one thing, then another that demonstrates the same thing with more complexity, and finally an example that brings everything together.

From the Oops! example, we know to look for different classes of manipulation. These attributes become our checklist:

• Limbs: Are all of them accounted for? Are all connected? Are they the right proportions?
  
• Reflections: Do items line up properly?
  
• Shadows and lighting: Are they consistent?

Now we can apply this to a new set of pictures.

At Photoshop Disasters, they featured a picture from the French fashion house, Louis Vuitton. However, the web page at Fashion Gone Rogue contains many pictures from the "Louis Vuitton Fall 2010 Campaign" (also available at Fashionologie). It is an homage to digital distortions.

Starting at the top is the banner for Fashion Gone Rogue. Her upper arms are very different lengths. It is also faint (better seen with luminance gradient), but it looks like there is a strap or something going across her shoulder and down her cleavage. (This could be where the artist stopped altering the skin.)

Mirror Mirror On The Wall

The various photos from Louis Vuitton have been equally mangled. Let's use our new checklist...

The picture claims to show three women in a dressing room. Each has different color hair: red, blonde, and brunette.

Limbs
Every person has two arms? Check! Extra fingers? Nope. Legs and feet? Uh... the brunette on the right has an ankle but is missing toes.

Reflections
The right-most mirror (behind the toe-less brunette) is not reflecting anyone in the room. The blonde has her hand up in the room but her hand is down in the mirror. That same mirror also shows a light bulb in the reflection, but the bulb does not exist in the room.

The second mirror from the right shows bulbs but they don't align with the bulbs in the room.

The mirror on the far left shows red's head from the back. However, red's head is not turned to show her back to that mirror. And the mirror's reflection shows the lamp on the wrong side. The reflection does not match the room.

Lights and Shadows
When an item sits next to a illuminated light, it is made brighter. And when items are facing away from the light, they are in shadow. Complex lighting, such as floods, reflectors, and bright ambient lighting, can mitigate shadows.

However, those mirrors have a lot of bright lights. The women should have brightly lit backs. But this isn't what we're seeing. The brunette has bright reflections off her chest but not her back. The blonde has a bright clavicle but an under-lit neck. The pile of junk in the back has a brown fabric thing above the handbag; it is lying next to a light bulb and not lit up.

This isn't a comprehensive list and there are other oddities that are not in our checklist. For example, the blonde's dress seems to have a layering issue with red's chair. The dress fabric suddenly becomes semi-transparent and you can see the chair through it.

Frankly, I kind of doubt that these three women even posed together for this picture.

Some of the pictures in this series are much worse than others...


Dear Louis: While fabrics may be diaphanous, people are not. And while models may be vamps, they are not vampires. Please fix the left mirrors. You know, the ones with the time-delay reflections that show the brunette in two alternate positions and don't reflect the blonde.

Dress For Success

While I can criticize these ads for pasting in people, changing reflections, and digitally altering lighting, I have to give Vuitton one piece of credit:

Beyond expected color enhancements (applied to the entire picture) and spicing blends (expected from a composite image), I have not detected any modifications to the clothing. Well done. Unlike Ralph Lauren and Victoria's Secret, Vuitton's pictures do not appear to be a product bait-and-switch.

My friend Bob G. recently told me about a column on page 12 of National Geographic Magazine's June 2010 issue. The column titled "Getting Real" is a rant about how a photographer supplied the magazine with a digitally altered photo. (A shorter version of the article is available online under the title "Your Shot Digital Manipulation".)

Basically, "photographer" William Lascelles submitted a photo to National Geographic and claimed that it was real. The magazine asked Lascelles to verify the photo, and he submitted a second fake photo. National Geographic then printed the image, only to learn that Lascelles lied to them twice; they were duped into printing a fake photo.

Whenever I post blog entries about photo manipulation, there is always someone who asks "what's the harm?" and "what about artistic freedom?" In this case, the harm directly impacts National Geographic's reputation. The photo contest is restricted to real photos for a reason: the magazine strives to use photos based on reality and not fake or doctored or modified situations that are fictitious.

In response to being duped, National Geographic published a full column identifying the fraud by William Lascelles and repeatedly naming him. The printed column also names another fake, by Dobrev and published in December 2009. This is more than just posting a small correction. This is a full outing.

Told You So

Like National Geographic, Smithsonian Magazine holds an annual photo contest. In March 2009, I wrote to the Smithsonian and informed them that some of their finalists used digital modifications that appeared to be outside the permitted amount of allowed manipulations. The Smithsonian responded a few days later. They had investigated the images and queried the photographers. In the end, three of the five modified photos were disqualified from the contest. The remaining two were determined to have an acceptable amount of digital modification.

I was thrilled with the Smithsonian's reply. They listened to the concern, investigated the situation, and took appropriate action.

I sent a similar letter to National Geographic on Tue, 24 Nov 2009 21:36:10 -0700. I had noticed that at least one of their contest submissions was digitally modified. Here's the reply they sent me:

Subject: Re: Digital Manipulation in the Photo Contest
To: "Dr. Neal Krawetz"
Sender: [redacted]@ngs.org
Date: Tue, 8 Dec 2009 13:48:19 -0500

Dear Dr. Neal Krawetz:

Thank you for contacting the National Geographic Society.

Your comments regarding photos submitted to the 2009 International Photo Contest are very much appreciated. While we provide information on photo manipulation and what is and is not accepted in the contest rules <http://ngm.nationalgeographic.com/photo-contest/manipulation>, all the photos in the contest are submitted by the individual photographers. We do not go through each one of them and remove them, even if we feel they have been manipulated. We simply do not have the time or staff to do that. You can view the winning photos on our website at http://ngm.nationalgeographic.com/photo-contest/past-winners

Best wishes,

[Name redacted]
National Geographic Society

That's right: National Geographic does not review each of the images that they accept for publication on their web pages, and they do not remove images that they know or believe are digitally modified. As seen with William Lascelles, their "verification" consists of asking the photographer -- who has already lied and won't mind lying again. I can understand that they don't have a big budget or staff for image analysis. However, this is their reputation. One would think that they would be interested in protecting it.

In the fraud exposure column, National Geographic wrote that they have changed their policy: "Now we're looking more closely at all Your Shot pictures." It's about time!

Frankly, if you are going to hold a photo contest and require original photos (not digitally altered), then you should take the time to verify every finalist. I'm not saying that you should police every submission. Rather, attempt to evaluate every finalist or semifinalist -- probably a dozen or so pictures. You usually don't even need special tools; a critical eye is usually good enough. If you don't have the time or resources to validate potential winners, then perhaps you shouldn't hold the contest.

And I'm sure that someone is going to ask "Where can they get my tools?" It isn't so much the tools as the training. If you are not trained to spot photo manipulation, then the best tools in the world won't help you. And as my friend Cynthia Baron has repeatedly demonstrated to me: people with the right training and experience can do this without any specialized tools.

Going Deeper

The dog picture by Lascelles is available online at http://s.ngm.com/your-shot/img/faked-blue-dog-615.jpg. Let's see what can be found with real image analysis...



Disclaimer
Please keep in mind: I'm not analyzing the original submission that National Geographic received. I'm analyzing a resave that National Geographic likely scaled for use on their web page. I seriously doubt that this is the original submission. In effect, I'm looking at a bad photocopy of the original submission. As with any bad photocopies, some results may be inaccurate due to artifacts introduced during the reproduction process and some evidence of modification may be completely wiped out. And more importantly: the original submission was fake, so it also includes modifications and artificial artifacts.

Photo Ballistics
Lascelles' file was saved as a JPEG and includes a JPEG APP12 section labeled "Ducky". (If you search for strings in the image, you will see "Ducky".) What is Ducky? It is a section added by Photoshop's "Save For Web" and includes the saved quality level. In this case, someone used Photoshop's Save For Web and selected 83% quality. (The "someone" was probably National Geographic.)

However, the quantization tables do not match the stated quality level. Instead, the Quantization tables match 94% compression. The discrepancy is due to the saved settings. Specifically, the last save used Save For Web with "JPEG", "Very High" and 83% quality. (The "Very High" setting selected the 94% quantization tables. Photoshop's quality level does not represent the quantization tables.)

Principal Component Analysis
PCA is great at identifying JPEG artifacts from resaves. These appear as rectangular blocks that are either 16x16, 16x8, 8x16, or 8x8. The more extreme the blocks, the lower the JPEG quality. High quality or original images should have very few visible artifacts.



In this case, the PCA shows severe blocking in the sky -- this is a low quality image from multiple resaves. But there is a problem... The blocks are not 8x8; they are smaller. In this case, the big squares appear to be 7x7 and small squares appear to be about 3x3. This means that the image was low quality and then scaled smaller. (I wouldn't be surprised if National Geographic scaled the final image for presenting on their web page.) The final image is probably about 40% smaller than the previous version. Since this image is 201x134, the previous image was somewhere around 500x335 (or larger with multiple resaves that scaled it smaller).

The other thing to notice is the block quality. The sky has big chunks indicating a low quality image. The front of the house has small blotches with no visible grid. The dog shows grid-like blocks on his ear and face that match the sky but not the house. And the jets have no visible blocky artifacts. So while the dog may go with the sky, it does not match the house or the jets. The mottled pattern on the front of the house actually matches what I would expect from a picture of this quality. (I even ran a few tests with other pictures using Save For Web and "Very High" and 83% -- the tests generated the same blotchy pattern seen on the house.) This means that the dog, jets, and sky are wrong for this picture.

Error Level Analysis
Taken without any context, the ELA for this image identifies the dog and jets as being at a higher error level potential (newer) than the rest of the image. However, this difference could be explained due to a combination of Photoshop and scaling. Photoshop attempts to counteract the JPEG losslessness by over-emphasizing high frequency areas. (See my Alyson Hannigan write-up.) This image does have a large amount of rainbowing (the red/blue/purple coloring), so this certainly matches the meta data that identified Photoshop. With scaling, every pixel is modified and high frequency areas (like the dog's fur) can have pixel values altered more than the rest of the image.



However, there are two issues that really stand out. First, the jet planes appear to be uniform in color (low frequency) yet have a high error level. So this is a modification.

Second, ELA should identify similar 8x8, 16x16, etc. blocks as PCA; for low quality images, ELA identifies the chrominance subsampling. And this is a problem. The subsampling should be the same across the entire image. However, the sky clearly shows 16x8 subsampling (scaled to fit 7x3 grids). However, the house only has square subsampling (either 8x8 or 16x16 scaled to 7x7; you can easily see them on the roof). With the dog and the jets, I don't see the subsampling grid. This means that the image must be made from four separate components: sky, house, jets, and dog.

Blur Detection
Blur detection identifies subtle, high frequency edges created from artificial blurs. Ideally, each edge should either consist on one thin line, or two parallel lines that are one pixel apart (1 pixel wide double line). Anything else indicates an artificial blur.



In this picture, the dog has 1-pixel wide double lined edges -- it is real. However, the house and jets both have wide double edges; the jets and house have artificial blurs.

Color Distance
A new algorithm that I've been working on is based on color distances. Basically, real pictures blend colors along edges. When pictures are spliced (one pasted onto another), there is no blending. This algorithm measures the amount of blend. If you see a thin black line outlining anything, then it was spliced.



In this case, the dog has a thin, black outline against the sky. He was spliced into the picture. It is a little more difficult to see, but the upper 4 jets also have thin black lines. They were pasted into the picture.

Observation

Alright, so it really looks like the dog and jets were spliced into the image. How many jets were there originally?



In this example, I have shifted the picture down and to the left a little. This allows me to overlay the top three jets onto the bottom three jets. Guess what? Two of them are perfect matches. Let's number the jets for clarity:

4

 2

5 1

 3

6

From what I can tell jets 2, 3, 4, and 5 are all the same plane and all uniformly spaced. Similarly, jets 1, 3, and 6 are the same plane. That is a total of two unique planes.

In real life, jets flying in formation will not be perfectly at the same angle to the viewer and they will not be perfectly spaced. Instead, they will all be ever so slightly different.


DoD photo by Airman 1st Class Gul Crockett, U.S. Air Force/Released


Same image, shifted to align the top plane on the middle plane, and overlaid to show differences.

This uniqueness factor even holds when the image is scaled smaller, like when the wingspan is only 20 pixels across. They may be small, but they should all be different. I think Lascelles cloned some jets.

Also, notice how none of Lascelles' vapor trails look the same. If they are about the same thickness and same color and the sun is in the same place, then they should all look similar. Jet #3 has a much whiter vapor trail -- likely two trails pasted next to each other. Jet #5 has a dark edge, probably from blending an overlapping paste.

National Disaster

Even without specialized tools, National Geographic should have noticed the cloning of the jets, varying vapor trails, and the artificial blur. Since they have the original submission, they could have checked the meta data and quantization tables to see if it matched the digital camera. This alone would have identified the fraud. And most importantly: don't just accept the word of a photographer; it may be significantly altered even if they claim that it is "100% real".

I use FreeImage for image analysis because it supports loading dozens of different image file formats (and because I like the FIPL licensing). However, FreeImage does not support every format that I need. For example, it has problems with PNG files that use something other than 8-bit data. It also has problems with 16-bit PPM/PGM files and it does not support lossless JPEG compression. So for these formats (and a few others), I use my own rendering systems.

However, I seem to be coming across more odd file formats that FreeImage simply does not recognize. In these cases, I usually implement my own rendering system. For example, I recently encountered the Digital Imaging and Communications in Medicine (DICOM) file format. Wow... I used to think that JPEG was a horrible file format because it is inconsistent and cannot store true color. And TIFF is bad because you jump all over the file. And PDF is bad due to the complexity (objects reference objects reference objects...). But then comes along DICOM, and suddenly I have a newfound respect for JPEG.

Is there a doctor in the house?

Let's face it: hospitals have problems. Many hospitals and doctors offices still use paper records and digital records are not necessarily compatible between vendors. If you switch hospitals, then you either need to bring over your medical records or you need to start with no records on file.

So it should come as no surprise that the image file format used for CT scans, MR scans, PET, and other imaging systems uses a format that, on the surface appears complete, yet actually isn't.

Thinking Positively

With JPEG, PNG, and other file formats, the image usually comprises most of the file. However, many DICOM files store much more meta information that images. Each DICOM file includes information about the patient (name, age, gender), doctor (name, address, hospital, referring physician, etc.), and the image itself (date, time, sequence number, equipment, scanning duration, relative slice position, and much more). For the medical community, this is outstanding! You will never have a case of the image being separated from the relevant medical information.

Although DICOM is capable of storing multiple images in one file, this appears to be uncommon. Instead, each picture in a series is stored in an independent DICOM file. Each file stores one image plus all of the relevant meta data. I view this as a positive thing -- it makes it very easy to send one specific image to a coworker for a second opinion. (Since a DICOM can be a multi-megabyte file, why send a Gig of images when you can send a meg?)

Unlike PNG or JPEG, DICOM has literally thousands of pre-defined meta fields. This is beneficial since it guarantees that everyone will operate with compatible meta data. The list of meta data fields reads like a committee trying to guess about every possible need -- and covering it! For example, there is no generic "comment" field. But there are fields for Patient Comments, Image Comments, Frame Comments, Study Comments, Comments on Radiation Dose, Comments on Schedule Procedure Steps, and much more.

While DICOM overflows with detailed meta data, it totally lacks flexibility. For example, let's say that a new scanner appears on the market tomorrow. They want to be compatible, so they want to use DICOM to record their medical images. The doctor, patient, and related information is still relevant. However, there is no place to store the new, scanner-specific parameters. At this point, the vendor has three options: (1) use vendor-specific fields (incompatible with other vendors), (2) submit new fields to the Medical Imaging and Technology Alliance (committee that updates the standard almost annually), or (3) don't record the data (not useful).

Moreover, every meta data field is defined by a 4-byte numeric type. Thus, if you don't already know what 0018:9090 means, then you are out of luck -- I have yet to find a central repository that details each standard tag and type of data that goes with it.


Most DICOM seem to store 12-bit or 16-bit grayscale images. I applied a coloring algorithm.

Unnecessary Procedures

The problem with DICOM really doesn't come from the plethora of meta data. Rather, it comes from the file format itself. The basic DICOM record consists of tag-type-length-data blocks. However, they are a little more complicated.

Tags. The tags actually consist of two fields. The first two bytes define the class of data element. For example 0x0010 defines patient information, 0x0028 describes the image file format, and 0x300A seems to be for radiation therapy. The next two bytes describe the specific meta data. For example, 0028:0010 and 0028:0011 are the image width and height (called "Rows" and "Columns"), 0010:21C0 is the patient's pregnancy status, and 300A:0018 is for radiation dose reference point coordinates.

Type. Called the "Value Representation" (or VR), this two-byte field defines the type of data. Most data types are pretty standard: short string, unsigned integer, date, float, etc. The only real odd-balls are the "other" fields -- other byte (OB), other word (OW), and other float (OF). These define arbitrary data lists.

Length. This is the amount of data following the tag. Unlike JPEG or TIFF, this is actually the number of bytes not including the tag-vr-length. (Excellent! Zero actually means no data! With JPEG, no data is a length of 2.)

This seems simple enough to parse, right? Except that there are a few exceptions... For example:

• All length are 16-bits (2-bytes) long. Except for records with a VR of OB, OW, OF, sequences (SQ), unlimited text (UT), and unknown (UN). These exceptions use 4-byte lengths.
  
  
• Tag 0002:0010 defines the transfer encoding. One encoding method specifies no VR field. In this case, there may be no VR in records following this record and all lengths for undefined VR fields are 4-bytes long. Also, if you don't know the default VR for the particular record, then you cannot parse the data. (If you don't know that 0008:0092 should be a string, then you won't be able to parse the referring physician's address.)
  
  
• Assuming you can parse the lengths, some records have a length of 0xffffffff. This means that the length is unspecified and the subsequent records are nested under this record. (Just keep reading the next records.) Eventually there will be an FFFE:E0DD tag with length zero. This tag marks the end of the nested data sequence. Oh, and these undefined length records can be nested...
  
  
• Most tags only store one piece of data, but some have complex data formats. For example, text fields may use "^", "\", or "/" to separate strings. (Doctor names use "^" as in "REIL^TODD^D.^^M.D.", image type uses "\", and addresses use "/".)
  
  
• All values are stored in little endian format. Unless Tag 0002:0010 defines big endian. In that case, 16-bit image data should be processed in big endian but all other values remain in little endian.

Frankly, the data fields should be consistent. Either always specify the type of data or require the type to be known. Making the data type optional is just going to lead to parsing problems. While I'm not a big fan of hard-coded tag numbers (0028:0031 is hard-coded as the zoom factor), they could at least embed the type of data in the tag. For example, there are 27 different data types. They could reserve 5 bits in the Tag for specifying the data type, or 1 bit for indicating 2 or 4 bytes for the length.

I should also point out that you must be able to parse the entire file. Image data is almost always found at the very end. So if you cannot parse the file then you cannot find the start of the image data.

Wrong Prescriptions

So you finally found the image data itself (record 7FE0:0010). The image data is either stored in one big block (OB or OW), or in multiple fragments (frames).

Image data itself is even more complicated. For example, 0028:0100 defines the number of bits allocated per pixel data record (either 8 or 16). (I don't know why they use this when OB/OW also define the data size. But I like using 0028:0100 better.) 0028:00101 defines the number of bits stored, and 0028:0102 defines the position of the most significant bit (MSB). So, for example, they may allocate 16 bits per sample, use 12 bits, and have the MSB at position 14. This means bits 3-14 contain the data, and bits 0, 1, 2, and 15 are unused. (Unless there are overlays... yet more complexity.)

And what format is the data in? This is defined by Tag 0002:0010 (Transfer Syntax UID). The value is a sequence of numbers (that looks a lot like an SNMP MIB entry). For example, 1.2.840.10008.1.2 defines an implicit VR (VR field is removed) and the image is a bitmap in little endian. 1.2.840.10008.1.2.2 defines a bitmap in big endian. 1.2.840.10008.1.2.4.50 through 1.2.840.10008.1.2.4.93 are different JPEG formats (lossy and lossless, JPEG and JPEG-2000). 1.2.840.10008.1.2.5 uses a bitmap with run-length encoding while 1.2.840.10008.1.2.1.99 uses zlib compression. Exiftool identifies over 250 different possible values. (I currently support all JPEGs and bitmaps, but nothing else. I'll be adding RLE this weekend.)

Of course, this isn't even including the 3D image information... There's an entirely different section for storing voxel data.

Feeling Better

Consistency is always a good thing. With image formats, consistency simplifies parsing, reduces potential implementation problems, and increases compatibility. While DICOM includes an amazing number of pre-defined data fields, it lacks generic expansion and the hard-coded tag identifiers restrict the total number of possible tags (there is a finite amount of expansion). DICOM's meta fields are great for today, but could be problematic in the future.

The inconsistent file format options and overly complex 0002:0010 tag means that a parser must understand hundreds of very different variations before being able to render an arbitrary DICOM file. While I can understand the need for lossy, lossless, and 3D data storage, couldn't they decided on a single lossless method? I mean, all lossless will look the same anyway since they are lossless. So why provide a dozen different storage methods? (Alright, DICOM dates back to 1985 so some formats may be old residues. Yet none have been obsoleted. And this doesn't explain the dozens of system-specific formats like 1.2.840.10008.5.1.4.1.1.2, which is a CT Image Storage format, or 1.2.840.10008.5.1.4.1.1.1.2, which is Digital Mammography X-Ray Image Storage for presentations as opposed to 1.2.840.10008.5.1.4.1.1.1.2.1 which is the same thing for processing.)

While DICOM is very detailed, it lacks consistency in file structure, meta data format, and image format. All of the problems with the medical community have clearly been passed into their image file format. And with all the implementation complexity, medical software is bound to be more expensive. Perhaps the medical community should consult a computer scientist before creating any future image formats. I don't prescribe medicines and they should not design file formats.