Pete Steege’s Storage Effect is commenting today on an ABI report that highlights the explosion of video content on the web, which expected to increase to one billion viewers by 2013. Steege’s response is that the report ignores the “digital home,” which will no doubt become ubiquitous in the coming years.

I agree, and would add that there are still other things driving video storage growth as well, such as a drastic increase in the number of video surveillance cameras and their resolution. But mainly, what I see is that the storage problem itself could actually be solved to a great extent with the proper optimization. For video, since video files are already compressed for transmission, the proper storage optimization has to include both video-specific recompression and video-specific deduplication.

For video on the internet, you have two related but different problems. One is to store the vast amount of content that is being generated. The second is provide the bandwidth needed for high-definition viewing of hot content.

Most video content is not hot. People upload thousands of hours of video per day to popular sites like YouTube, but only a small fraction of that gets wide viewership. It all needs to be stored, but the key thing for most of it is to store it cheaply. That's going to mean not just cheap disks, but video-specific storage optimization that greatly reduces the size of the video files.

The relatively few videos (meaning, a couple hundred a day) that do become popular won't be so aggressively compressed, or they'll be compressed for bandwidth rather than for storage optimization. That is, solving the speed problem for the hot stuff that everyone is watching is easy - it will be replicated and cached, and people will get access to their hot shows and user-contributed videos. Solving the "store 900 Petabytes of user-generated video really cheaply" problem is not so easy to solve.

Another major optimization of video storage is that most videos that most people want access to is duplicated across many homes. Today, a blockbuster movie, a hit TV show, a TiVo of the big game - these are all stored hundreds of thousands of times across millions of households.

As video storage moves to cloud storage services, a lot of that can be deduplicated. For entire licensed content (e.g., a studio movie) that's relatively easy - you'd say, here are 10,000,000 users uploading their copy of the Lion King…let's just save one. But to get real optimization, cloud storage providers are going to want to be able to find and compress video at finer granularity than that. Let's say there's a football game broadcast on ABC in some markets, and carried by ESPN (with different commercials) in another market. User A records it in standard def. User B records it in high def. The user in Atlanta records it from ABC. The user in Portland records in from ESPN. To be efficient, you'll want storage optimization that recognizes that those users are all uploading versions of the same thing, and takes out the redundant information as part of the compression / deduplication process.

Without aggressive storage optimization - including video-specific compression and dedupe - the explosive growth of video content is going to overwhelm storage capability.

Yesterday's Mashable post looking at online file storage providers caught my eye. Right now, online "cloud" storage providers are all targeting different markets, but the competition is fierce in all segments. Some are going after the consumer - such as AOL X-Drive -, some are going after online backup, and some are going after web site data. Actually, that article doesn't even mention Amazon's S3, for example, which is a huge online repository.

The obvious benefits are basically twofold: ease-of-use and - for most of them - the fact that they manage your data for you in terms of backing it up, replicating it, etc. The biggest drawback here is that you have to be connected to a network to get to your files.

Most customers will still look at cost/Gigabyte as the main motivator to use a service like this and, at the right price and benefit point, people will put their files online. Since all these storage service providers all buy their disks from the same small number of companies that actually make disk drives, the costs are all roughly the same for the physical infrastructure needed to build an online storage service and compete.

I think that the real solution here is that, for anyone to breakthrough and get some separation from the crowd, they are going to have to incorporate breakthrough storage optimization in their offering - and do so in a way that's transparent to the end user. That could be dedupe, that could be compression, or it could be something more sophisticated like Ocarina. The main thing is that if you can get 5:1 or 10:1 ratios on how much logical space you can provide via the cloud to how much physical space you, as a provider, have to buy, then you can have a compelling proposition. The competition is fierce in this market and in order to grow and thrive in any business that offers online storage, the providers are going to have to develop a strategy to significantly increase their online storage capacity without increasing cost and overhead in step.

Last week, Storage Switzerland Analyst George Crump penned a guest column for Information Week about the optimization of primary storage, a topic close to our hearts.

The article looks at the various approaches to online storage optimization and how these solutions can help reduce the footprint of online data and help companies effectively respond to the massive increase in information that lives and is accessed via the Internet.

George is working on a series of articles looking at vendors in the space, for the first installment on Ocarina Networks click here.

Disclosure, I am a co-founder of Ocarina.