Storage of video files

[SteelFlyer]

You've gotten a wealth of knowledge on storage solutions. Just to add another one - I use two 6TB RAID 10 setups for my long-term backups and use a 4TB thunderbolt drive for working copies. My two RAID units are in different geographic locations and I use a program called Chronosync to keep them synchronized.

To lessen my space needs, I've started creating a "selects" reel with the best pieces from the raw footage. Let's face it, every shoot has a bunch of garbage that you'd never use. I make my final edit using some of the "selects" shots. The rest I just save in case I ever want more footage. But I trash all the rest of the stuff.

 

[chrismorace]

Yeah - there's a pretty distinct jump in space/usage I think. Stemming in part from if you keep all originals, but that's more of an outcome of what you're intending to do with the video as well as how much you're willing to invest.

After a couple of TB or so it starts to get more expensive - not just bc you're buying more disk but there's a point at which if you are keeping that much data it doesn't make sense to do so unless you're protecting it better and redundancy costs $$.

I'm a photographer, I still have far far more photo data than video. But it's impressive how fast the video is able to close the gap! I'm running a 1TB SSD for live/project data (which is nearly full right now) and 12TB RAId10 (6x 4tb drives) as my large storage (also almost full now!). I've got a 4 bay readynas with 4x 4tb drives for backup - unfortunately that actually now has too small of a capacity to back up everything now, so I'm backup up the ssd only nightly since what's in there isn't backed up elsewhere. Then all of this is disaster recovery backup with Backblaze. They mean it when they say unlimited! (Not a shill, I'm just super happy with the service, been using it for 8 years for myself and implemented at two businesses and never had a single hiccup).

I haven't traveled for >1 day since I got the new drones, so 4x 64gb cards should be enough to cover a full day's shooting if needed. But otherwise it's basically the same setup as Brian above. 512gb ssd for live, a couple of externals as needed for backup. When I travel with my camera gear I copy it off, back it up, but still try to not touch the card until it's home and onto the array. But that's not always feasible unfortunately for a longer trip. Heck, I've been making a bunch of HDR panos on t eh drone and those alone can eat up 9gb for a single pano.

This was super helpful perspective, DaRana. Thanks for sharing it!

 

[Raptorman0909]

While it's hard to find an exact number of copies that Amazon makes for your data, they do store it in multiple facilities. If one center goes down, you still have access to your data. If I was doing photography for business, I would include what ShammyH is doing: Uploading new files to AWS S3 and having it automatically age into Glacier storage.

Amazon is not doing data mining against your data stored in AWS. As for hacking, your data is a secure as your credentials. You are the weak link for access to Azure and AWS, they are both very secure. Never store your keys where they can accessed from a public web site. I've read horror stories about developers who included their access keys in source code that was stored in Github. People regularly scan Github looking for credentials.

Here's a quick quote...

Verizon confirmed on Wednesday the personal data of 6 million customers has leaked online.
The security issue, uncovered by research from cybersecurity firm UpGuard, was caused by a misconfigured security setting on a cloud server due to "human error."


So, no matter what your credentials are if the company has there head up there @ss then you're still vulnerable no matter what your credentials are.

Verizon data of 6 million users leaked online


Brian

 

[excalibur1814]

I always have my media in two places.

I built a media server, with 4x 6TB drives which, at that time, was more than enough for all my music, videos, tv etc, but the move to 4k droning will eventually cause an issue. I then have an external 8TB portable drive as a backup just in case as you never know what'll happen. if you really, really, want to keep your media then two places is a must. I dump the raw vids onto the server, as a project (Example: Project0021) and also on my main editing rig. Create the video, then move the media onto the server and then backup. At all points the media is in two places.

Plus I can, at any point, re-edit the media in the case of a sudden copyright claim on the music. (I do use royalty free but places like CDBaby may suddenly file a claim).

 

[anotherlab]

Here's a quick quote...

Verizon confirmed on Wednesday the personal data of 6 million customers has leaked online.
The security issue, uncovered by research from cybersecurity firm UpGuard, was caused by a misconfigured security setting on a cloud server due to "human error."


So, no matter what your credentials are if the company has there head up there @ss then you're still vulnerable no matter what your credentials are.

Verizon data of 6 million users leaked online


Brian

If you read that article closely, Verizon stored the data in a public S3 instance.

The incident stemmed from NICE security measures that were not set up properly. The company [Verizon] made a security setting public, instead of private, on an Amazon S3 storage server -- a common technology used by businesses to keep data in the cloud. This means Verizon data stored in the cloud was temporarily visible to anyone who had the public link.​

Amazon's security was not in question, it's 100% Verizon's fault for putting the data out in the open where anyone can get it. You can have a really good lock and deadbolt on your front door, but if you leave for work and forget to shut the door, then you were the one that suffered from a cranial-rectal inversion, not the lock company.

Amazon and Microsoft have taken strong measures to ensure the security of their cloud services. I am part of a team that designed a large application that runs entirely in the Azure Cloud. We had to document to our customers the security certifications that Microsoft had earned for Azure.

You are taking a security risk when you store data in the Cloud because you do not control the infrastructure. That risk is so much smaller than the risk of losing your images due to hardware failure, accidental erasure, catastrophic loss, etc, that the risk is essentially negligible.

The bigger risk that you have with cloud storage is connectivity. The data is secure, but you just can't get to it. If your ISP goes down, you can't access your data. And while it's rare, Azure and S3 do have outages. It's usually minutes or hours, but it can happen. Which is why you want redundancy with your backups.

 

[anotherlab]

Not to go too much OT, there is a great site for monitoring if your email address shows up in a data breach. It's called "Have I been pwned" and it's run by a security consultant named Troy Hunt. You can search for your email and subscribed to be notified if your email shows up in a new breach, for free.

 

[Raptorman0909]

Not to go too much OT, there is a great site for monitoring if your email address shows up in a data breach. It's called "Have I been pwned" and it's run by a security consultant named Troy Hunt. You can search for your email and subscribed to be notified if your email shows up in a new breach, for free.

Thanks for the link -- afraid to look!


Brian

 

[Shammyh]

If you read that article closely, Verizon stored the data in a public S3 instance.

The incident stemmed from NICE security measures that were not set up properly. The company [Verizon] made a security setting public, instead of private, on an Amazon S3 storage server -- a common technology used by businesses to keep data in the cloud. This means Verizon data stored in the cloud was temporarily visible to anyone who had the public link.​

Amazon's security was not in question, it's 100% Verizon's fault for putting the data out in the open where anyone can get it. You can have a really good lock and deadbolt on your front door, but if you leave for work and forget to shut the door, then you were the one that suffered from a cranial-rectal inversion, not the lock company.

Amazon and Microsoft have taken strong measures to ensure the security of their cloud services. I am part of a team that designed a large application that runs entirely in the Azure Cloud. We had to document to our customers the security certifications that Microsoft had earned for Azure.

You are taking a security risk when you store data in the Cloud because you do not control the infrastructure. That risk is so much smaller than the risk of losing your images due to hardware failure, accidental erasure, catastrophic loss, etc, that the risk is essentially negligible.

The bigger risk that you have with cloud storage is connectivity. The data is secure, but you just can't get to it. If your ISP goes down, you can't access your data. And while it's rare, Azure and S3 do have outages. It's usually minutes or hours, but it can happen. Which is why you want redundancy with your backups.

Exactly.

S3 is probably the single largest data-store on Earth and has never been hacked.
As you say... companies like Microsoft, Amazon, IBM, etc... take security VERY seriously. If your data is compromised, it's almost certainly a fault of the customer, not of the service.

S3 also has trillions of objects stored, and to my knowledge, has lost like a couple dozen in total, and that was mostly in the early days. They replicate data in three locations simultaneously... so short of nuclear war, very unlikely to be an issue with storage durability. There are infrequent, but occasional outages, for sure. But when S3 goes down... so does the rest of the internet... so does it really matter? ;-)

~~~

Again, as you stated... the viability of cloud storage for very large files really is a question of connectivity. Not so much downtime/inaccessibility, but simply bandwidth.
I have Comcast in the Boston area, and my line is 330/12... so uploading large amounts of data does take days... but also means I'm more free to locally store data in RAID 0, as opposed to RAID 1.

 

[chrismorace]

When I was building PC's years ago when the Western Digital 10K RPM Raptor SATA drives came out and they had a 5yr warranty and even with cooling fans blowing on the HDD's and a well ventilated case it was rare to see one last 5yrs.
JarrodF you are 100% correct on the when, not the if a mechanical HDD will fail that is for sure!

I designed WD's supply chain back in the day. The whole point of "the cloud" is it is the most redundant RAID array ever built an accessible from everywhere. For the average person buying hardware is wasted money.

The limitation of the cloud is simply the bandwidth. If your use case involves massive movement of files quickly or editing on planes then you are better off with hardware still.

 

[anotherlab]

If you use AWS and have to upload a massive amount of data very quickly (switching over, just filmed/edited a move. etc), Amazon has their Snowball service to transfer huge amounts of data. Microsoft has something similar for Azure, but it's not as clever.

 

[Raptorman0909]

If you read that article closely, Verizon stored the data in a public S3 instance.

The incident stemmed from NICE security measures that were not set up properly. The company [Verizon] made a security setting public, instead of private, on an Amazon S3 storage server -- a common technology used by businesses to keep data in the cloud. This means Verizon data stored in the cloud was temporarily visible to anyone who had the public link.​

Amazon's security was not in question, it's 100% Verizon's fault for putting the data out in the open where anyone can get it. You can have a really good lock and deadbolt on your front door, but if you leave for work and forget to shut the door, then you were the one that suffered from a cranial-rectal inversion, not the lock company.

Amazon and Microsoft have taken strong measures to ensure the security of their cloud services. I am part of a team that designed a large application that runs entirely in the Azure Cloud. We had to document to our customers the security certifications that Microsoft had earned for Azure.

You are taking a security risk when you store data in the Cloud because you do not control the infrastructure. That risk is so much smaller than the risk of losing your images due to hardware failure, accidental erasure, catastrophic loss, etc, that the risk is essentially negligible.

The bigger risk that you have with cloud storage is connectivity. The data is secure, but you just can't get to it. If your ISP goes down, you can't access your data. And while it's rare, Azure and S3 do have outages. It's usually minutes or hours, but it can happen. Which is why you want redundancy with your backups.

Again, that doesn't change anything -- the fact is even with good credentials and otherwise tight security you are ALWAYS vulnerable to idiocy on the part of anyone with access and control. A chain is as weak as it's weakest link and although that is usually credentials it is sometime idiocy somewhere else in the chain that can compromise security. Most often it's lack of concern on the part of the company not wanting to spend a penny more than absolutely necessary but sometimes one of the many people that are permitted access does something that provides the open door. Verizon is not a little guy in the game and if they can f'up so can anyone.


Brian

 

[Shammyh]

Again, that doesn't change anything -- the fact is even with good credentials and otherwise tight security you are ALWAYS vulnerable to idiocy on the part of anyone with access and control. A chain is as weak as it's weakest link and although that is usually credentials it is sometime idiocy somewhere else in the chain that can compromise security. Most often it's lack of concern on the part of the company not wanting to spend a penny more than absolutely necessary but sometimes one of the many people that are permitted access does something that provides the open door. Verizon is not a little guy in the game and if they can f'up so can anyone.


Brian

True that the chain is only strong as the weakest link. And true that major corporations do mess up.

What's far more likely though... is that you eff up. Verizon also does not have the best track record for security, or generally highly ethical and cogent behavior, so they're not a great example.

Amazon datacenters are hard to even physically locate... let along get access to. Whereas... your home network is both more easily accessed and located.
Also... I can't speak for all cloud vendors... but AWS is EXTREMELY security conscious. They have an entire GovCloud for the freaking CIA/NSA/etc... just to show you how much certain clients trust them.
Yes, some people at Amazon certainly have low-level access to sensitive stuff, like your S3 storage... but that number is very limited, and if you don't trust Amazon employees, use your own encryption keys, simple

How good is your home network security? Are you using an IDS/IPS or just a dinky off-the-shelf router? Or even worse, a Comcast/Verizon/ISP provided router?
Is every single computer on your network fully up-to-date and patched for any known vulnerabilities? From phones, to tablets, to potentially your fridge? ;-)

Do you have an alarm running 24/7 with cameras streaming to offsite backup? Do you have security guards? Do you have a secured vault to keep your tape library? etc etc etc

~~~

Long story short... "cloud storage" within a reputable cloud, properly configured and utilized, is significantly more robust to both hacking and data-loss than anything you can setup at home. Full stop.

Doesn't change the fact that "the cloud" is not a good solution for everyone's storage needs... but again, that's more due to bandwidth and other logistical concerns, not the security or durability of the files in question.

 

[anotherlab]

Again, that doesn't change anything -- the fact is even with good credentials and otherwise tight security you are ALWAYS vulnerable to idiocy on the part of anyone with access and control. A chain is as weak as it's weakest link and although that is usually credentials it is sometime idiocy somewhere else in the chain that can compromise security. Most often it's lack of concern on the part of the company not wanting to spend a penny more than absolutely necessary but sometimes one of the many people that are permitted access does something that provides the open door. Verizon is not a little guy in the game and if they can f'up so can anyone.
Brian

It gets back your previous statements

....But, the other thing I dislike about cloud storage is the data mining that's done and the fact that there's a crap ton of hackers out there that spend all day everyday looking to break into and destroy peoples data. I'd guess the cloud companies have better security than many others, but you the user has little control over that....
Brian

Amazon and Microsoft are not data mining their bulk storage products. And they haven't been hacked. The user has full control over his own security and is the weak link in the security chain. We've been talking about the S3 and Azure products (and Dragonfly) in this thread. With both services, you pay for what you use. You have 4 TB to store, you pay for 4 TB. You have a petrabyte, you pay for a petrabyte.

As for Verizon being a big company, yes they shouldn't have made that mistake. Being a large player doesn't mean that you have the best and brightest people working on every project. And as Verizon customer, I am more that a little annoyed that they have not notified me that my data was exposed. Or maybe it wasn't, they have more than 6 million accounts.

If you don't trust Cloud storage, then don't use it. It's your data, you need to be comfortable with what you are using. I just disagree with the assumptions that you are making about it. Your data is at a higher risk on your home PC than in the cloud.

 

[anotherlab]

You might want to try Dragonfly (Home). Just launched. Special promotions going on right now to build the community including discounted storage pricing, free give away of Spark, Mavic Pro, and DJI Goggles.

Upsides are:

- No hardware required other than your phone. You can transfer video in the field after a flight without removing the SD card
- Web uploader for batch moving collection to the cloud for safe backup/enjoyment
- App to browse your collection, grab files when you need them, and view your flights on the map
- Ability to browse other friends or the community if they give you permission
- Automatic creation of mixes that can be shared to email, text, social media, or saved

Since I referred to Dragonfly, can you talk about where the data is stored and what level of redundancies are provided with the videos?

I hope that your company is successful, but it's a market with razer thin margins. Let's say 8 months down the road, something happens and you have to shut down the cloud storage. It doesn't matter why. Dragonfly could have failed or it could have been wildly successful and then acquired and the new owners want to do in a new direction. What sort of protections are there for the customers so that they would not lose any data?

 

[Raptorman0909]

It gets back your previous statements



Amazon and Microsoft are not data mining their bulk storage products. And they haven't been hacked. The user has full control over his own security and is the weak link in the security chain. We've been talking about the S3 and Azure products (and Dragonfly) in this thread. With both services, you pay for what you use. You have 4 TB to store, you pay for 4 TB. You have a petrabyte, you pay for a petrabyte.

As for Verizon being a big company, yes they shouldn't have made that mistake. Being a large player doesn't mean that you have the best and brightest people working on every project. And as Verizon customer, I am more that a little annoyed that they have not notified me that my data was exposed. Or maybe it wasn't, they have more than 6 million accounts.

If you don't trust Cloud storage, then don't use it. It's your data, you need to be comfortable with what you are using. I just disagree with the assumptions that you are making about it. Your data is at a higher risk on your home PC than in the cloud.

I use cloud storage so I'm not opposed to it, but to suggest they are somehow immune from human failure is naive. I'd argue that the best and brightest work for the NSA and we all know about Snowden. I can accept that Amazon MAY be better at this than others but no companies products are exploited more than Microsoft. I wonder how many M$ zero day vulnerabilities the NSA has up there sleeve...


Brian

 

[anotherlab]

I use cloud storage so I'm not opposed to it, but to suggest they are somehow immune from human failure is naive. I'd argue that the best and brightest work for the NSA and we all know about Snowden. I can accept that Amazon MAY be better at this than others but no companies products are exploited more than Microsoft. I wonder how many M$ zero day vulnerabilities the NSA has up there sleeve...


Brian

No code is immune from human failure. But AWS and Azure are much more secure than Windows/Mac OS/Linux. The zero day releases from NSA were for Windows, not Azure. They are not the same products.

 

[Ing Zimmerman]

New P4P owner with a question. How are you guys storing all your video files. Those things are huge. I just bought a 2 TB external thunderbolt hard drive. I'm not sure that will be enough.

The video on this thing is awesome.

I have 8 3-TeraByte external hard drives for all my videos. Left the DVD route for the HD's.

 

[chrismorace]

Since I referred to Dragonfly, can you talk about where the data is stored and what level of redundancies are provided with the videos?

I hope that your company is successful, but it's a market with razer thin margins. Let's say 8 months down the road, something happens and you have to shut down the cloud storage. It doesn't matter why. Dragonfly could have failed or it could have been wildly successful and then acquired and the new owners want to do in a new direction. What sort of protections are there for the customers so that they would not lose any data?

Hi anotherlab,

Thanks for asking the question and your positive comments! We definitely are excited about building something the drone community loves. We are founded by executives from Jive (leading collaboration platform that went public in 2011), Facebook, the SF 49ers, and HP. Early investors include Ron Conway (Google, Twitter, Dropbox, AirBNB...) as well as executives & founders from Snap, Facebook, Google, Quora, and Jive. That's a long way of saying we "aren't two kids and a dog" in a garage

We are built on top of Amazon's AWS. We selected them after an extensive security and innovation review. We believe they are going to allow us to consistently offer our customers better features faster at a regularly decreasing price, which we can then pass on to our customers.

As far as protections go you can visit our Terms of Service which specify that should anyone's account be terminated for any reason we guarantee they will have 30 days to retrieve their content. You can compare this to Dropbox's policy which specifies only that users will be given a "reasonable amount of time". We also post our Privacy Policy and our Community Guidelines there, which were crafted with an intent to protect your privacy and assure good conduct on the platform.

If you have any additional questions or thoughts about what we can do better, please let me know.

 

[Richard Walker]

Quit using DVDs years ago. The substrate on most of them starts to fail at about 7 to 10 years. Use external HDs and BluRays now.

 

[chrismorace]

New P4P owner with a question. How are you guys storing all your video files. Those things are huge. I just bought a 2 TB external thunderbolt hard drive. I'm not sure that will be enough.

The video on this thing is awesome.

This month the Dragonfly drone video app is having a promotion that will let you earn free storage. Admittedly, you'd have to hustle to get 2 free TB, but it is possible Might be worth checking it out. It is always backedup up, easy to view, and always available when you need it. Promotion