Feed your Appetite for Reduction. Meet us at booth #605 at AWS re:Invent.
As millions of organizations adopt the simplicity of Amazon S3 to store and manage business-critical data, the need for high-performance backup and recovery is becoming increasingly clear. The applications, data lakes, and machine learning models that run on Amazon S3 are often powered by critical and sensitive data that needs to be backed up on a continuous basis, and retrievable near-instantly. Clumio has pioneered backup and recovery for Amazon S3 (we even wrote a book on it) for 2 years now, and it’s exciting to see other players join the market.
AWS Backup recently announced a performance improvement for S3 that speeds up their backup times for buckets containing over 300 million objects. The improvement additionally allows backup of buckets containing more than their previous upper limit of 3 billion objects. Below is the average expected performance with these improvements as published in the AWS Backup Developer Guide, and we have added a column calculating the objects per second figure for each. Restore performance metrics were notably absent from the AWS Backup published results.
As published in Amazon S3 Backup Developer Guide
|Number of objects||Bucket size||Estimated time to complete initial backup||Objects per second (calculated by Clumio)|
|135 million||425 GB||31 hours||1,210|
|670 million||800 TB||38 hours||4,898|
|5 billion||6 PB||100 hours||13,889|
|7.5 billion||370 TB||180 hours||11,574|
We thought this might be a good opportunity to share some results of Amazon S3 backups with Clumio. Architected as a serverless data processing pipeline, speed and scale of Amazon S3 backups have been the primary focus at Clumio since we started supporting it in 2021.
|Number of objects||Bucket size||Actual time to complete initial backup||Objects per second|
|100 million||5 TB||80 minutes||20,833|
|3.38 billion||1.54 PB||53 hours||17,715|
|6.14 billion||4.24 PB||95 hours||17,953|
|24.4 billion||26.7 PB||231 hours||29,341|
|24.69 billion||759 TB||184 hours||37,274|
As you can see, object count and overall data volume both play a role in the backup time of a given bucket. This is illustrated by the two examples of buckets containing about 24 billion objects, but at very different sizes, resulting in very different backup times. Generally speaking, the object count should be your first consideration in estimating backup time, with data volume following. While other factors will always influence backup speed, you can count on Clumio’s backup speed being relatively predictable from the highest to the lowest object counts and data volumes. Notice that Clumio’s performance does not reduce for the lowest object count example, and also scales to well above 20 billion objects per bucket, resulting in exceptional backup speed regardless of the customer’s S3 environment. Indeed, we have customers backing up buckets that are a few GB to tens of PB.
In addition to being fast, Clumio also detects when buckets being backed up are in demand, and throttles the backup processing to allow the primary application to perform optimally even while backup is in process. This is especially important during large seeding backups, which can get large enough to take up significant resources and API quotas. This means if an application wasn’t making any calls to a given bucket, its backup time could be as much as 50% faster. However, that’s not a typical scenario. Many applications, especially those with a global user base, need to run continuously. It’s important to preserve their functionality, especially for large buckets whose initial backups will take several days.
While backup speed is an important benchmark, restore speed is even more important. After all, the point of backups is to restore data quickly when you need it most. Even though the AWS Backup performance results didn’t showcase their restore metrics, let’s take a look at the restore speeds that you can expect with Clumio for Amazon S3 data.
|Number of objects||Bucket size||Actual time to complete restore||Objects per second|
|1.6 million||236 MB||13 minutes||2051|
|1.1 million||1.03 TB||29 minutes||632|
|22.15 million||2.64 TB||2h 37m||2351|
|24 million||410 GB||2h 44m||2439|
|30 million||34 TB||56 minutes||8929|
S3 restores most commonly go to new buckets, which start with lower API rate limits. AWS scales up the API limits automatically as more objects are created in the destination bucket. This delayed scaling impacts the rate at which objects are created in the destination bucket during the initial phases when API limits are low. The restore process can be accelerated by working with AWS to proactively increase the limits before starting.
For large S3 buckets, waiting for a full restore of your backup to resume operations might be too disruptive. On the other hand, maintaining a warm failover (such as a replicated bucket) of your data all the time can be expensive (and doesn’t offer point in time recovery, air gap, and other features of backups). Clumio’s Instant Access feature exposes an S3 endpoint for applications to directly access data from backups without having to restore the objects at all. This reduces the RTO to minutes, even for buckets containing tens of petabytes.
Hopefully this has been a helpful exploration of backup and restore speeds for Amazon S3. When assessing a backup solution for Amazon S3, be sure to test restore performance! If you’d like to learn how Clumio’s performance can help you better manage your data estate, request a personalized demo.