datahoarder
Who are we?
We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.
We are one. We are legion. And we're trying really hard not to forget.
-- 5-4-3-2-1-bang from this thread
view the rest of the comments
If you want longevity and shelf stability, tape drives are the way to go. You can get them in very large capacities, even into the hundreds of TB.
Their benefit is that they have no internal motorized components, they are a lot like VHS videocassettes - two spools with tape. This makes them very shelf-stable, unlike hard drives which can have their spindles seize up over time.
They also have absolutely epic data densities. You could store on one tape the contents of dozens of the largest hard drives currently available.
Their downside is that you need highly specialized hardware to read and record them. And this makes the hardware quite expensive.
So why don’t we use tape drives to store data? Because they store said data linearly - great for writing once, terrible for finding or updating said data - and because they are slow. You want to get to a file 20Tb in? Enjoy scrolling past every single byte up until that point.
But for cold backups, there ain’t nothing better.
We recently researched these for work.
They tick a lot of boxes - lots of space, reasonable speed, great cold storage figures. Reasonably priced tapes. Agree, they're the best thing. The slow read speed isn't quite as bad as expected (They can go extremely fast in seek mode), but definitely something to consider. We were okay with that for our needs.
But damn, the price of the hardware was horrendous - we got priced (I think) close to £20k for a suitable drive that met our needs. Completely killed the project. And remember that if you're doing site replication for DR, you'll need at least two of them. Sadly, it looks like we'll be using external HDD's for a while longer...
exciting but def not for me :)
Is there a go-to budget tape system that's worth looking at? In the past it seemed like getting anything with reasonable density like 10TB+ per tape was so expensive I could buy like 200TB of HDDs for the same price as the drive alone.
Don't buy into tape. It is costly and is inferior to hard drives by most metrics for smaller scale operations. You can easily get 8TB hard drives for less than $20/TB. While tape is cheaper than that, the drive to actually use it is expensive, plus you get all the disadvantages of the tape itself.
Fun fact: you can probably buy a whole server, external sas card and disk shelf for less than the cost of a somewhat modern tape drive.
If you are wanting to store less than 100TB of data, it would probably be cheaper to use drives, then in 3-5 years buy another set of disks and still be ahead compared to tape.
Yeah that's what it seems like, used SAS HDD are around $7/TB right now and I don't think anything is going to be cheaper than that.
Yeah but what about in 9000 years?
Where do you buy these? What's the cost per space?
The hardware to read the tapes are calibrated to the maximum size they are configured to accept. So when you hit up eBay, you will need to know the maximum amount of data you will need, and either the size of the largest tape drive to hold all that - if you are not getting a machine with an auto-loader - or the maximum number of drives that the machine’s autoloader can take, so you can size the tapes properly for the data.
Say you need to back up 25Tb. You are unlikely to ever need more backup than that. So you either look for a machine that takes 25Tb tapes, or you get a machine that can take max. 5Tb tapes, but has an autoloader that can hold at least 4 additional tapes (in addition to the one in the drive) such that all five will automagically cycle through the backup process. That way, all 25Tb will be backed up in either case without your direct and immediate involvement, all you have to do is rotate the tapes off-site after the backups are done, and slot the next ones in for the next backup run.
Obviously, incremental backups are a no-go, as backups are stored off-site. So it’s an all-or-nothing process. And as such, this is usually done both on your entire primary data set (for fast total-disaster restores) once in a while, with a different set of tapes focusing on your local/on-site warm backups and backing up only the atomic/incremental locally-stored backups for the day/week/month.