How To Safely Back-Up Your Data on Location - Simple Data Wrangling
Everyone can back-up data. It's simple and you do it most days when copying photo's from your camera or phone to your computer (or online). Backing up footage from film shoots is not that different but a little more care should be taken. So here is a guide on how to do it safely, and to insurance bonded standards. This is not a guide to becoming a DIT but it will make sure you don't loose your footage! I'm going to keep it really simple and straight-forward. There are many equally valid ways to do this, and every job is different but most of the techniques used here are becoming fairly standard. Ok...... AN OVERVIEW OF EQUIPMENT
A Simple Data-Wrangling Kit
The above picture shows a Macbook Pro - pretty much the standard for straight forward data back-ups. Many things in Film and TV work on a left to right principle - which is a really good way to train your brain to work. On the left we have our reader - in this case a Lexar USB3.0 CF / SD card reader (we will get into speeds and connectors later). In the middle is the Macbook (obviously) and on the right are our back-up drives. The data goes from the left into the reader, through the computer and onto the drives. It's simple. Don't be cheap with drives! Generally the more you spend the safer your data will be. Do you trust your £4000+ film to be stored on a £30 drive from Tesco? No matter how good the quality of your drive is, your data is not safe until it is on at LEAST TWO drives. Drives fail easily, get lost or stolen. No insurer would ever pay out for data being backed up to a single drive. And as drives are relatively cheap you should never take the risk. Your Back-Up Drives
The above set-up is based on a multi-day shoot. We have a shuttle drive (can be Lacie, G-Tech Mini etc) which is intended to go to the editor/post to be backed up further -it's a fairly simple drive. The silver drive is a RAID (multiple drives inside). This is going to be the drive that stays with the unit. Above all it needs to be safe and reliable so build quality is of high importance. It should live in a Pelicase or similar for protection when travelling. It is built as RAID5 so if a drive fails the data can be restored. It also needs to have a large storage capacity to hold the data. It's expensive but depending on the shoot you can get away with much cheaper drives. I sometimes use something like this for onset and at the end of the shoot transfer onto a cheaper drive. The Shuttle drive is called Shuttle (or Transit) as it's purpose is to come back to you after being at post. Again this travels in a small pelicase for protection (sometimes shuttled very far). There is no standard for labelling but it must be clear (I usually prefer to use a Dymo for labels). Everything should be idiot proof. If a drive is lost, you fall ill - whatever someone must be able to know at a glance what is contained on it. So each drive has a unique identifier. Shuttle A (there will be a Shuttle B, Shuttle C etc) and RAID_1 etc. This is all part of your 'system' which is the deciding factor on whether your data is safe or not. A good system means it's very difficult to loose data, a bad system is....bad! Paperwork is highly important, which will be covered later but a great deal will correspond with you drive labelling. The Shuttle drive also must contain the name of the shoot (which shouldn't really need to be said), the Day, Date, Camera ID (whether it's A,B,C or all cameras). I'd also put on the production company name and contact details (phone number is fine but the address is helpful if being posted). The box should also contain the same information and a 'to address'. Card Readers for CF/SD, Alexa SxS/Sony F55 and Red Epic/Scarlet
Obviously you will need the correct card reader for the camera you are using. For this walk-through we will be using a simple Panasonic GM1 which uses SD Cards, so the Lexar Reader is perfect. ----------------------------------------------------------------------- SOFTWARE AND HOW TO USE YOUR LAPTOP We now know a little about what hardware is needed, so now we need to look at the software. You likely copy your photos by dragging them from the card into a folder on your desktop or external hard drive. This is known as 'dragging and dropping' which is fine in some situations but can lead to not copying everything or on some camera cards corrupting the data. This is where copy verification programs help. Copy verification programs such as R3D Data Manager, Shotput and Silverstack do many things. They can copy the data sequentially (which is not the case in dragging and dropping).This is very useful for more complicated file types (such as Red's R3D). They can also provide you with file verification - which is extremely useful and will save your neck one day. File verification works in many ways (file size, MD5 etc) but basically means that after you copy a file the program compares it with the original. If it matches you get a successful copy, if it fails you get an error. But the most useful...er use of the Copy verification software is that it can allow you to offload to multiple destinations at the same time. For us that means we can copy our card onto two drives at the same time. Time management is the only complex part of this job and anything that saves time is good. We're going to be using ShotPut Pro 5
First though, we have to setup our drives! Drive format is very important. You need to speak to the Post House, or editor/assistant editor to ask which format they use. Mac's use HFS+, PC's Use NTFS. And there is handily ExFAT which both can use (on newer OS's). Not doing this can cause problems so it must be known in advance. There are ways to read and write using different formats but they each have their own problems. We're using HFS+ There are some important naming conventions. Again different post houses use different variations so you must check. One of the most common conventions is what we will demonstrate:
This is the route structure of our MASTER RAID Drive. The Drive is named RAID_1 The First folder, where we will be keeping our project is named after the shoot title - 'GRANT_SHOOT' As it is a multi-day shoot the first folder within 'GRANT_SHOOT' is '20141115_DAY1_A' This is the date backwards, the day and the camera designation Some post houses like an underscore (from FAT32 days) some hate it. Ask! The date backwards is used mostly but some prefer it in other ways. The day obviously is the day you are shooting And the letter at the end lets us know which camera was being used. Some post houses like different folders for different cameras. THIS IS ALL VERY IMPORTANT. This is where little mistakes are going to crop up and you must get it right. Imagine a 200 day shoot. Everything must be stored in the correct place and be easy to navigate. Coming back to a project in a years time is a high possibility so your project management is very important. It's also likely this will be stored on an LTO tape where little mistakes being changed become big problems. Your shuttle drive should follow the same structure, though it will be 'SHUTTLE A' instead of RAID_1 You must speak to your post house before the shoot day for clarification. A very good idea is to hand around a workflow document so everyone knows what is expected and needed.
In Shotput you need to set up your destinations - your back-up drives! Navigate to the day folder for each drive. Shotput will make a folder for each card you copy. Again this folder will start with the date backwards. You can set up a prefix to save you doing this for each card. And give the card a unique name! Project Management at this stage is the most important part of the job. You must know what your cards are called, and importantly everyone else must. If the card is from A Camera, the first Roll number will be A001, the second A002 and so on. So the folder '20141115_A001' will contain the contents of card 1 from A camera and so on. Now we're ready to go.......
Boxes! With Pretty Colours!
What are these? As I've mentioned I like to make everything idiot proof because I get confused easily. These boxes use a traffic light system. I'm not going to explain as your clever enough to know. Card in a box
It's very likely you may get more than one card at a time, or the card you already have has not finished copying when you get a new one. Use your boxes!
When you get a card from the camera department put it in the 'To Be Copied' Red Box. You will notice there is a tab with the card. THIS MUST STAY WITH THE CARD UNTIL IT HAS BEEN FULLY COPIED AND READY TO RETURN TO THE CAMERA DEPARTMENT 99% of all issues are due to human error. If you have 5 cards and no system you will loose data! Card In Reader (as you can see!)
When you are ready to start copying your card - place it in the reader. If it has a safety tab then use it!
And the tab goes onto the reader. If your called away you may forget what card is in. Nothing is too obvious. Don't feel silly by making everything look simple. Your Desktop
When you insert the card it will appear on your desktop. I like to arrange the drives in the order they are physically by your computer - Card to copy on the left, Drives to Copy To on the right.
Drag the SD card (or card to copy) onto the big blue ball thing then hit BEGIN! And your card will copy onto your two destinations! This is where drive speed, connecter type and how you've connected your drives/cards factors in. It is vitally important you choose the correct drives, drive speed and connector for the task in hand. We are copying from a small SD card onto a USB3 Shuttle Drive and an eSATA RAID connected by Thunderbolt. The SD Card reader is very fast. It is no problem to copy a full SD card, it will take seconds/minutes. It only becomes a problem if you are dealing with 20 cameras. If you have USB2 drives and a Red Reader connected by FW800 you can quickly bottle neck. And not having cards to give to the camera department on a busy shoot will not make you popular. Drive Speeds and Connecters are for another article but this should be factored in. VERY IMPORTANT - DO NOT RUN OUT OF MEDIA. IT IS YOUR RESPONSIBILITY! BING!
When your card has copied you will see either a green tick - which means you have a successful copy or a red cross. Red cross means something has gone wrong and the cards needs to be copied again. We used a MD5 checksum which is fairly standard (other checksums can be better but slower, or faster but not so careful). We now have a MD5 Checksum verified copy of our card! A very useful piece of evidence. But that's still not good enough! Byte for Byte Comparison
I would suggest you also do a 'control I' for your source and destinations. Make sure the bytes match on each copy. Shotput or similar software is not infallible so this is another piece of evidence to support a successful copy. The most foolproof way to check a card has been copied is to watch a copy back in realtime. Depending on the file format use software such as Quicktime or VLC for a quick 'eyeball'. This is very basic QC-ing. All your evidence so far shows is that the card was copied properly. If the card contains errors then this is how you flag them up. *****Full QC-ing is different and won't be covered in this article****** As a last piece of evidence take a snapshot of the first frame (or slate) of the card. This shows you have watched it. It is very important for you to collate all the evidence and other details of the card into a document. I sometime use Bento. It's not as pretty as other programs (you can even use Word or Excel) but is has a good database. Date, Roll Number, File size, number of clips on the card, Slate, Lens details are all good things to add and very helpful for the camera crew, yourself and the post team. A great reference (although a little metaphysical with the snapshot).
At this stage your card is likely to be in the 'To Be Verified' Box. When your happy that everything has been copied you can format the card to be put in the green 'Return To Set' Box. Keep the tab from the roll number in a safe place. Formatting the card has caused a little controversy. But it's generally thought the pro's outweigh the cons at this stage. A card given back to the camera still needs to be formatted on the camera, but if you have done your job properly the camera crew know the card has been safely backed up. It removes any doubt that it could have been muddled up and not copied. It also means that if they (and it happens) put the card they have just taken out of the camera back in to format it will be flagged up immediately. Your Bento Report, Byte-Bytre verification, Screengrab of first slate and MD5 Checksum will all go into a DIT Docs folder with the drive for that day. More cards copied
Bento Report
Hopefully that will keep your data safe. There will be some follow up articles on Simple Colour Grading, Applying LUTs, Transcoding and using Livegrade in the near future! A few more tips... Use a UPS - Uninterruptible Power Supply. Your best friend. When you loose power it automatically switches to a battery supply without affecting your copy. Power goes off a lot on shoots! Time management is key - you should never be sitting watching progress bars - you should work your system so you are always working and not wasting your own time. Don't ever work from the card original - ALWAYS use a copy. MAKE A NOTE OF CARD SERIAL NUMBERS. If there is an ongoing error it helps identify a problem.
