124. S3 Performance

In this episode, we discuss some tips and tricks for optimizing performance when working with Amazon S3 at scale. We start by giving an overview of how S3 works, highlighting the distributed nature of the service and how data is stored redundantly across multiple availability zones for durability. We then dive into specific tips like using multipart uploads and downloads, spreading the load across key namespaces, enabling transfer acceleration, and using S3 byte-range fetches. Overall, we aim to provide developers building S3-intensive applications with practical guidance to squeeze the most performance out of the service.

💰 SPONSORS 💰

AWS Bites is brought to you by fourTheorem an AWS consulting partner with tons of experience with S3. If you need someone to work with to optimize your S3-based workloads, check out at fourtheorem.com !


🔖 Chapters:

00:00 Intro
01:29 Some S3 usage data and how S3 works under the hood
05:25 S3 Performance tip #1 - Split into chunks & leverage concurrency
08:37 S3 Performance tip #2 - Spread the load across prefixes
12:09 S3 Performance tip #3 - Use multiple connections
14:11 S3 Performance tip #4 - Use the "edge"
17:41 S3 Performance tip #5 - Select only the data you need
19:29 Closing notes

In this episode, we mentioned the following resources:

- Our previous episode on S3 best practices: https://awsbites.com/83-bucket-list-o...
- “Deep dive on Amazon S3” (re:Invent talk from 2021):    • AWS re:Invent 2021 - Deep dive on Ama...  
- More recent re:Invent talk on Amazon S3 (with updated data) - We discovered this one just after the recording:    • AWS re:Invent 2023 - Dive deep on Ama...  
- Multi-part upload user guide: https://docs.aws.amazon.com/AmazonS3/...
- Code examples from the SDK (high-level and low-level APIs): https://docs.aws.amazon.com/AmazonS3/...
- Node.js official helper library (@aws-sdk/lib-storage): https://docs.aws.amazon.com/AWSJavaSc...
- Example on how you can implement byte-range fetches: https://docs.aws.amazon.com/AmazonS3/...
- s3-getobject-accelerator library: https://github.com/widdix/s3-getobjec...
- aws-c-s3 library: https://github.com/awslabs/aws-c-s3
- S3 storage lens: https://aws.amazon.com/s3/storage-lens/
- Docs on S3 Transfer Accelerator: https://docs.aws.amazon.com/AmazonS3/...
- Performance Guidelines for Amazon S3: https://docs.aws.amazon.com/AmazonS3/...
- Performance Design Patterns for Amazon S3: https://docs.aws.amazon.com/AmazonS3/...


You can listen to AWS Bites wherever you get your podcasts:
- Apple Podcasts: https://podcasts.apple.com/us/podcast...
- Spotify: https://open.spotify.com/show/3Lh7Pzq...
- Google: https://podcasts.google.com/feed/aHR0...
- Breaker: https://www.breaker.audio/aws-bites
- RSS: ​​https://anchor.fm/s/6a3312a0/podcast/rss

Do you have any AWS questions you would like us to address?
Leave a comment here or connect with us on X, formerly Twitter:
-   / eoins  
-   / loige  

#aws #cloud #s3 #storage #performance #amazon