A new report has shed light on the extent of the dilemma facing organizations as the demand for data storage capacity continues to skyrocket.
Published jointly by Fujitsu and Twist Bioscience, both of which operate in the archival storage market, the report (opens in new tab) predicts the gap between available storage capacity and demand will exceed 7.8 million petabytes by 2030.
In this scenario, businesses will be left with no choice but to delete large swathes of old data to make room for the new, which is enough to send a shiver down the spine of any firm with aspirations in areas such as artificial intelligence.
Data dilemma
As the volume of data produced by internet activity, digital devices and IoT sensors continues to expand at an aggressive rate, businesses are running out of time to solve a critical problem: where to put it all.
While hard disk drives (HDDs) and solid state drives (SSDs) do an excellent job of holding and supplying the quantities of data that servers and client devices need to function, neither are well-suited to storing information en masse and for long durations.
When it comes to archival storage, Linear Tape-Open (LTO) magnetic tape rules the roost, with the lowest cost per capacity of any technology. The current generation of tape, LTO-9, has a native capacity of 18TB and can be purchased for as little as $150 (or roughly $8.30/TB).
According to the report, large companies will have to invest heavily in tape and other archival media, as the volume of data produced by business operations continues to rise. The alternative would be to discard old data, but to do so would be to lose out on its potential value as a source of insight; the most advanced AI products are typically informed by the largest, most exhaustive pools of data.
“We believe most of this [new] enterprise data will be unstructured, ‘cold,’ infrequently accessed and will have to be maintained at minimal cost,” explained report author Jon Monroe, who says the spread of storage spend will need to reflect that fact.
However, while cost-effective, tape has its weaknesses too; data can only be accessed serially, making it hard to locate specific files, and companies also need to migrate to fresh tape on a semi-regular basis to avoid data loss.
In light of these issues, researchers are hunting for new ultra-dense and ultra-durable storage technologies. A few different candidates have emerged, but one concept looks particularly promising: DNA.
DNA storage
DNA, the foundational material of living organisms, is made up of four molecular building blocks: adenine (A), guanine (G), cytosine (C) and thymine (T). These compounds connect in pairs (A-T & G-C) to form the rungs of the famous double helix ladder.
This structure can be utilized as an extremely dense and durable form of data storage, by converting binary 1s and 0s into the four-lettered genetic alphabet. A single gram of DNA has been found to be capable of storing 215 PB (220,000 TB) of data.
“DNA holds the promise of offering the magic three in storage: ultra-high density, reasonable cost, and sustainability,” said Emily Leproust, CEO and co-founder of Twist Bioscience, which is investing heavily in bringing the technology to fruition.
“We expect that new media will be needed to address the $7 billion-plus of unmet storage demand projected in the years ahead.”
As it stands, the technology remains unusable at scale, as a result of the time it takes to write data to DNA and various other challenges. Naturally, the report also needs to be taken with a pinch of salt, produced as it was by two organizations with vested interests in an increase in spending on archival storage.
However, there is no denying that the rise in capacity of traditional data storage technologies is failing to keep pace with the rate of data production, which means a reorientation of the storage stack is inevitable.
“The datacenters of the future will need everything the SSD, HDD, and tape industries can manufacture and deliver, as well as requiring new DNA and optical and perhaps other enterprise storage technologies, to cost-effectively and reliably preserve the priceless artefacts of our personal, corporate, and cultural history,” added Monroe.
“Availability and sustainability challenges, combined with the costs of managing our multi-millionfold-petabyte dataverse over increasingly lengthy time periods, will create new use cases for old storage technologies and demand the creation of new, more cost-effective, and power-efficient storage technologies.”
Via Blocks and Files (opens in new tab)
