Back up of Streaming server

Question

I want to take a new streaming server for my website which generally holds videos and audio files. But how do we maintain backup of the streaming server if storage size is increasing day by day.

Generally on Database servers, like Sql Server, backups can be easily taken and restored very easily as they do not occupy much space for medium range applications.

On the other hand how can we take backup of streaming server? If the server fails, the there should be an alternative server / solution that should decrease downtime of the server.

How is the back-end architecture of YouTube built to handle this?

Answer 1

What we do is to have multiple FC SANs, each in sync with each other across different datacentres, each is connected to banks of servers acting as 'origin' servers translating FC storage out to either NFS or CIFS/SMB. These servers are then split into a load-balanced VIP blocks which in turn feed similarly-VIP'ed blocks of web servers which are then presented via FW/LBs to the outside world.

The actual content is periodically snapped from one or more of the FC SAN boxes onto a dedicated SAN box which is then backed up to disk at another site, tapes are then stored with Iron Mountain. I'm in the streaming business :)

There's no shortcutting with content, it's big and you just have to deal with it. If I were you I'd setup a dedicated backup machine with a big chunk of disk available to it and use rsync to ensure you have a copy of every file on the main content store, even though this will inevitably end up as a superset of your live data. Then take disk or tape backups of that machine and periodically delete aged data to keep it manageable.

Oh and youtube don't properly backup any regular-user content, their design ensures that they have multiple copies distributed around the world but that's more for performance than restorative capabilities. They do backup their own content or any other content that they're paid to deploy but that's a tiny drop in the ocean compared to all the content they have no contractual obligation to store.

Answer 2

Now you are finding out why having a streaming video/audio service, and keeping it reliable, isn't easy to set up. To have a full backup solution you need:

• Additional servers to hold the backed up content.
• Sufficient available bandwidth (on a separate back-end network) to handle the backup.
• Scripts to copy the content on a timely basis, or an automatic backup as files are being added to the server.

If you are looking at decreasing downtime, you need to add in at least twice the amount of servers you have for the original solution, as well as a way to manage this network. Cost will be at least 2x of the original solution.

As Chopper3 has noted, you can build into the infrastructure the need to do 'backups' because when content is added it is automatically mirrored.

Answer 3

Assuming one of your questions is "How the back-end architecture of YouTube built to handle this." even though you never used a question mark in your post, the answer to that is that Google is incredibly huge, and has tons of servers dispersed across the globe, and you can be sure that the data is stored on more than one machine, so in case one of them goes down the data can keep on streaming.

Usual backup plans include having off-site backup, but if you need to have high uptime, you may need both off-site and local backups so you can restore quickly from the local, though if there is any disaster ever in the DC you will have to use the off-site ones.

Someone mentioned tape backups, though I'd advise against them, as you don't seem to actually need to have archives of the data, and you probably just want to be able to sync your data to another server to have a backup. There are useful tools like rsync which can keep data in sync and will only upload modified files, sparing you from doing a full backup.

There are ways you can overcomplicate it and ways to burn money setting up a lot of redundancy, but something tells me you can't afford it and you don't need the hassle of managing too many machines.