Whenever people hear that I analyze pictures, they always ask the same thing: Do you analyze videos?
While I do a little bit with videos (there's no real way to get away from it), I mainly focus on static pictures. As critical a I am toward image formats, video formats are on an entirely different (and chaotic) level.
Back to Basics
If you want to support image formats, how many different formats do you need to support? Keep in mind, there are literally hundreds of proprietary and lesser-used formats. However, there are only a handful of widely used image formats. If you support these, then you will support more than 90% of the images you will come across on the Internet: GIF, PNG, and JPEG.
Other formats are used in certain fields, but are rare outside of those fields. For example, BMP is common on older Windows systems, but you are unlikely to run into them on Linux or Unix systems. The bitmap formats (PPM, PGM, PBM, and XPM) are used in certain niches where people fear lossy formats (even though PNG is lossless and compressed). And in certain cases, there is a need to edit image data -- raw bitmaps can be better than PNG.
Matryoshka Images
Beyond the basic image formats are container formats. There are really three types of containers: mono-format, multi-format, and any-format. (My terms for them.)
The mono-format containers can hold many images, but all internal images are the same format. For example, the ICO format contains only BMPs. Windows Thumbs.db files only contain JPEGs (but depending on the version of Windows, they are either standard or non-standard JPEGs) and animated GIFs are really just a file containing a series of GIF images.
The multi-format containers hold many images, but permit a limited number of different formats. For example, the JPEG format (the JFIF and EXIF containers) can hold bitmaps, JPEG, and lossless JPEG images. TIFF can contain bitmaps or JPEGs. And PDFs can contain bitmaps, JPEG, and PNG images as well as vector graphics. While each of these formats permit extensions to store other image formats, you are very unlikely to come across a JPEG with an embedded GIF or a PDF that contains a TIFF.
Finally, there are a few any-format containers. These are file formats that can hold anything. Like Word and PPT documents -- anything can be stored in them.
Frankly, if you can support a dozen image and container formats, then you are likely to support that vast majority of image formats that you are ever going to run across.
Moving Pictures... Moving Targets
So supporting GIF, PNG, JPEG, BMP, TIFF, and a handful of other formats is enough to easily support more than 90% of the image formats you will come across. What about videos?
Video formats are a nightmare. There is a huge variety of formats and subformats that can be mixed and matched. What do you need to support in order to claim that 90% compatibility mark? MP4, MPG, AVI, FLV, WMV, RealMedia, DIVX... Oh, and don't forget DVDs! And even then, you might not be at 90%.
Each of the video formats I listed are actually container formats. There are a huge number of audio and video formats that can exist inside these containers. The
MPlayer video player (MPlayer-1.0rc2)
supports at least 279 different video codec variations and 131 different audio formats! And that's only counting the ones that are "working" -- there are dozens more that are under development.
Simply trying to support "MPG" means you still need to support dozens of different audio and video codecs. Whereas, if you support PNG then you only need to support one format.
Most open source applications use MPlayer or FFmpeg as the basis for video format support (and MPlayer uses FFmpeg). This will easily get you over that 90% support mark. However, I've found the open source code to be poorly documented and very complicated to follow. If you happen to hit some of the poorly supported formats, then your program is likely to crash and even if you are a hard-core developer, you are unlikely to be able to trace the code and identify or fix problems. More importantly, there isn't a great way to tell these applications to
not support some file formats.
Commercial applications actually have more control here. While QuickTime and Windows Media Player don't support every possible combination that MPlayer/FFmpeg support, they provide stronger support options. (For example, you can rewind and fast forward with QuickTime and Media Player, but
seeking with FFmpeg is very inexact.)
Basically, there are so many video formats and codec combinations, that you really have to make a choice: do you want to use open source and kind of support everything, or do you want to go closed source and support fewer formats but do them really well?
Better than Better
For people old enough, you know that records were replaced by CDs virtually overnight. While a few people still claim that records are better, the fact is, CDs are better in almost every way. They can store more audio, last longer, and offer good-enough sound (better than records when you take pops and scratches into account). Switching from records to CDs was a no-brainer.
With image formats, we still have a handful of competing formats but the battle for dominance is pretty much settled. Proprietary formats exist because they are proprietary, but few people are developing new formats for the purpose of being "better".
Unfortunately, the same cannot be said for video formats. There is no strongly dominant format and newer (but not better) formats keep coming out. Right now we're stuck with a hodgepodge of audio, video, and container formats.
