A closer look at speed and scale results when backing up Amazon S3

As millions of organizations adopt the simplicity of Amazon S3 to store and manage business-critical data, the need for high-performance backup and recovery is becoming increasingly clear. The applications, data lakes, and machine learning models that run on Amazon S3 are often powered by critical and sensitive data that needs to be backed up on a continuous basis, and retrievable near-instantly. Clumio has pioneered backup and recovery for Amazon S3 (we even wrote a book on it) for 2 years now, and it’s exciting to see other players join the market.

As you can see, object count and overall data volume both play a role in the backup time of a given bucket. This is illustrated by the two examples of buckets containing about 24 billion objects, but at very different sizes, resulting in very different backup times. Generally speaking, the object count should be your first consideration in estimating backup time, with data volume following. While other factors will always influence backup speed, you can count on Clumio’s backup speed being relatively predictable from the highest to the lowest object counts and data volumes. Notice that Clumio’s performance does not reduce for the lowest object count example, and also scales to well above 20 billion objects per bucket, resulting in exceptional backup speed regardless of the customer’s S3 environment. Indeed, we have customers backing up buckets that are a few GB to tens of PB.

In addition to being fast, Clumio also detects when buckets being backed up are in demand, and throttles the backup processing to allow the primary application to perform optimally even while backup is in process. This is especially important during large seeding backups, which can get large enough to take up significant resources and API quotas. This means if an application wasn’t making any calls to a given bucket, its backup time could be as much as 50% faster. However, that’s not a typical scenario. Many applications, especially those with a global user base, need to run continuously. It’s important to preserve their functionality, especially for large buckets whose initial backups will take several days.

For large S3 buckets, waiting for a full restore of your backup to resume operations might be too disruptive. On the other hand, maintaining a warm failover (such as a replicated bucket) of your data all the time can be expensive (and doesn’t offer point in time recovery, air gap, and other features of backups). Clumio’s Instant Access feature exposes an S3 endpoint for applications to directly access data from backups without having to restore the objects at all. This reduces the RTO to minutes, even for buckets containing tens of petabytes.

Hopefully this has been a helpful exploration of backup and restore speeds for Amazon S3. When assessing a backup solution for Amazon S3, be sure to test restore performance! If you’d like to learn how Clumio’s performance can help you better manage your data estate, request a personalized demo.