What is Data Deduplication?

Deduplication is one of the key storage technologies enterprises rely on to deliver optimal storage efficiency and reduced infrastructure costs. Deduplication is one of the most valued storage technologies in enterprise storage products because it’s designed to eliminate the need to store multiple copies of identical files. This reduces the amount of storage capacity required to store any given amount of data. Deduplication is a space efficiency feature that can be found in many enterprise storage products. However, because of a lack of standards, the effectiveness of data reduction ratios and cost savings are difficult to quantify when comparing different vendors. There is simply not always an apples-to-apples comparison. This is mainly because vendors may use different types of implementations and individual techniques that will produce different results. I think it’s safe to say that, now more than ever, having deduplication as one of the capabilities in a storage product has become more than just another checkbox. It’s crucial for enterprises to be educated about a product’s implementation details in order to understand the advertised deduplication ratios and cost savings.

What are some data deduplication challenges?

Deduplication is the technology where duplicate blocks of data present in multiple files are identified and only the unique blocks are stored. This task relies on having a metadata database that stores fingerprints of the unique blocks, then used for comparisons when finding duplicates. The deduplication task becomes quickly non-trivial when systems become large and complex. The metadata alone could go into terabytes, and hence can’t be handled by a single node. Also, it taxes the system resources as incoming data has to be parsed and compared with all the existing blocks.

How is the value of deduplication measured?

The value of deduplication is measured in a couple of ways: Effectiveness of data reduction capabilities and maximum achievable deduplication ratios. Overall impact on storage infrastructure cost and other data center resources and functions, such as network bandwidth, data replication, and disaster recovery requirements. Certain deduplication traits can determine the level of data reduction efficiency and overall value based on the implementation details.

What is fixed length deduplication?

This dedupe algorithm divides data into fixed-size chunks. Once the blocks are compared, only the unique blocks are stored. When compared to the alternative, fixed-length is far less efficient, but easier to implement.

What is variable length deduplication?

This dedupe algorithm uses advanced context-aware anchor points to divide data into blocks based on the characteristics of the data itself and not on a fixed size. With fixed-length dedupe, small offsets in the data set can result in significant loss of space savings.

What is local deduplication?

This approach limits data deduplication to just within a single node. There is no visibility to other nodes within the environment, where duplicate data will be found.

What is global deduplication?

This approach is the most effective process for data reduction and increases the dedupe ratio which helps to reduce the required capacity to store data. When a piece of data is written on one node, immediately after the write is acknowledged on node1, if the same data is written onto multiple additional nodes, the implementation is able to identify that data has already been written and will not write it again.

What is integrated data compression?

Compression works well on a single file, but across files, there is a need for some macro-level data compression. For example, when two identical copies of a file are stored, compression can individually compress the files. On the other hand, deduplication can completely eliminate the need to store any data for the second copy. Adding compression to the deduped data you further reduce the data size. This works by finding small byte patterns common between the deduplicated blocks. Based on the type of data being ingested, compression can give no benefit for encrypted or random data, or up to 5-10x compression for common log files. Deduplication ratios for VMs, databases, file shares all lie somewhere in between that range.

How can Cohesity help with deduplication?

Our storage platform was designed from the ground up to provide a powerful and flexible global deduplication architecture. Cohesity utilizes a unique, variable-length data deduplication technology that spans an entire cluster, resulting in significant savings across the entire storage footprint.

data protection Archives

Tag: data protection

July 2, 2018

Why Global Deduplication Matters

Today’s enterprises are storing roughly double the amount of data every two years from before. They are using it for a variety of workloads such as development and testing, file shares, analytics, backup, cloud archival, etc.…

June 19, 2018

Cohesity CMO Talks Funding and Data Protection at Cisco Live 2018

Cohesity CMO Lynn Lucas sits down with theCube hosts John Furrier and Stu Miniman to talk about data protection and many other good things happening at Cohesity. The chat took place at Cisco Live 2018 in…

June 7, 2018

Why Modern Data Protection is Better than Legacy Backup

Many organizations today still rely on legacy backup and recovery products. These traditional, multipoint solutions are fragmented, and consist of backup software, target storage, media and master servers, and bolt-on cloud gateways — all from different…

May 25, 2018

For data protection, GDPR compliance is more positive than pitfall

Sometimes it feels like technology evolves at the speed of light. With that rapid change comes a predictable cycle of security pitfalls and the need for increased consumer protections. This is especially true when it comes…

May 23, 2018

Cohesity DataPlatform Named as CODiE Finalist in Two Categories

The SIIA CODiE Awards are getting closer, which makes for exciting times here at Cohesity! Adding to the suspense, Cohesity DataPlatform was named as a finalist in TWO (yes, two!) categories: Best Software Defined Infrastructure and Best Storage…

May 21, 2018

Customers Ditch Legacy Backup for Pure Storage with Cohesity’s Modern Approach

Whether you are an individual consumer or a large enterprise, digital transformation has really pushed the limits of the amount of data we create, consume, and store. But, the exponential growth of data is only half…

May 17, 2018

How Cohesity Copy Data Management Enables Efficient Dev/Test

Last week I wrote about how Cohesity helps organizations protect their SQL databases against accidental data loss. Today we look at how Cohesity’s copy data management capabilities enable efficient test/dev datasets. One of the biggest trends in…

May 15, 2018

Accelerate Azure Cloud Secondary Data Workloads

I recently joined Cohesity to build on the existing relationship with Microsoft and take that to the next level. My prior experience working with Microsoft as a partner for over eight years gave me a front…

May 8, 2018

Protecting SQL Databases: Cohesity as an Alternative to Dumps and Copies

Last week I wrote about how easy it is to protect a SQL environment with Cohesity. This week we continue the conversation. Organizations that use Microsoft SQL Server sometimes struggle with how best to protect their SQL…

April 19, 2018

Configuring Backup Policies to Protect Cohesity File Services

I blogged recently about simplifying data protection, and consolidating files and objects on a single platform. In that blog, I included the first of three video demos about how Cohesity consolidates file & object workloads. In the…