2019 Photo & Video Data Management

 

2019: Year of Growth

I have a lot of big plans for 2019. I am focused on growing my own business in commercial photography and video production. I have also started creating daily content for Instagram along with shooting weekly YouTube videos for myself and my wife’s channel. There is even a podcast in the works. As someone with an engineering background, I am always looking ahead to identify potential bottlenecks in whatever project I am working on.

Looking at 2019, the most obvious problem which is going to come up sooner rather than later is…DATA! There is going to be a lot of it. So I put a plan into action to upgrade my current data management system to find a solution that will have enough capacity, be scalable, be responsive, be accessible, and be secure. You can find links for the exact gear I used in this project in the Links section at the end of this post.

So what’s the current situation?

My current IT setup consists of a PC I built almost 5 years ago now. I will expand upon this in future blog posts but building your own PC is definitely the way to go. The system is still going strong 5 years later and runs the latest Adobe suite with no hiccups. I have added graphics cards and hard drives over time. Flexibility is the key term here, being able to adjust as your needs change is a big benefit of your own build.

My hard drive setup consists of a 500 GB SSD for operating system and programs, 120 GB SSD as a scratch drive for Premiere Pro editing, a 1 TB WD Black drive for documents, and another 1 TB WD Black drive for my media files only. Both of my 1 TB hard drives are reaching max capacity and, with 2019 on the horizon, need a serious upgrade.

What specs do I care about?

There are a few key specs you will want to consider here, mainly capacity and speed. This new Media drive will be the main storage location for my photo and video assets, which is continuously growing. I am targeting around 8 TB of capacity for this new media storage solution. I will also be accessing them directly to edit in Adobe CC applications, which means I need good read/write speeds. There are many different ways you can approach this problem, such as:

  • Use a single larger SSD

    • No SSD available in the 8 TB capacity I was targeting

    • Still too expensive for large scale storage needs

    • While this would have great read/write speeds, it’s probably unneccesary

  • Use a single large HDD

    • Single 8 TB spinning hard drives are available, but at higher $/GB costs than smaller drives

    • Speed is just OK, same as standard hard drive speeds which I would prefer to be faster

  • Store all files on a larger network attached storage (NAS) and connect to PC

    • Using a RAID configuration on a NAS solves the capacity problem since you can use multiple, smaller, more cost-effective hard drives

    • Speed is limited by network connection, which is lower than any internal drive and will cause problems with Adobe CC

    • I currently have a NAS for shared storage and archiving, but speeds are not high enough to edit directly in Adobe CC

  • Use an internal RAID in the PC itself

    • Solves the capacity problem

    • Depending on RAID type, can utilize disk striping to speed up performance much higher than a single drive

    • For example, two hard drives in RAID0 will read/write at 2x the standard hard drive speed which means the more you add the closer you can get to super high SSD speeds

After this analysis, it was obvious to go with an internal RAID. I chose to use two Western Digital 4 TB Red hard drives in a RAID0 configuration. RAID0 essential writes data across both drives at the same time so, while your data is split between the drives your speeds essentially double in a two disk setup. Additional hard drives can be added later to expand capacity as well.

But..is it safe?

The biggest knock against a RAID0 system is there is no automatic data backup or redundancy. I solved this by purchasing an 8 TB external hard drive which will be setup to backup the internal 8 TB drive volume. This gives me two copies of any file on two different media. A true backup also has a third copy, and I won’t expand upon it here but I also use a QNAP NAS to backup files, which covers my bases.

How do we get it working?

Luckily, Windows 10 comes with a utility to create your own hard drive RAID configurations. It is called Storage Spaces, which you can find in the Control Panel. After installing both drives, you can combine them together as a striped volume and you are done. Now I just have to transfer my media from my old hard drive to my new 8 TB volume and I can get back to creating content.

In Summary

In short, here is my 2019 data workflow

  1. Content created on camera SD card

  2. SD card data transferred onto internal Media hard drive volume which is it’s main living area

  3. Internal Media hard drive routinely backed up to matching External USB hard drive

  4. Internal Media hard drive also routinely backed up across home network to NAS

So there you have it. An overview of how I am managing my content storage for 2019 in all it’s glory. I will be making some additional blog posts on some of the finer points of this setup going forward so look for those.

Of course, if you have any questions about this setup leave a comment below, or you the Contact Justin button at the top of the page to send me a message.

Links

Here you will find affiliated links to the gear I have used in this project. Clicking and purchasing sends a little back to me to keep this site going. Huge thanks for all the support!

WD 8TB MyBook External Hard Drive: https://amzn.to/2W10UP9

WD 4TB Red Internal Hard Drive: Non-Affiliate Link

SATA Cables for new hard drives: https://amzn.to/2W14B7m

 
TechJustin CejkaWD, PC, DataComment