Скачать или смотреть How to Safely Upload Large Backup Files to AWS S3 for SQL Server Migration

Learn effective strategies to upload large SQL Server backup files to AWS S3, including troubleshooting connection issues and using multipart uploads.
---
This video is based on the question https://stackoverflow.com/q/66296331/ asked by the user 'Kris' ( https://stackoverflow.com/u/13404763/ ) and on the answer https://stackoverflow.com/a/66296430/ provided by the user 'AWS PS' ( https://stackoverflow.com/u/12615044/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: AWS RDS SQL Server Migration - Upload large backup file to AWS S3

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/l...
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/... ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
A Comprehensive Guide to Uploading Large SQL Server Backup Files to AWS S3

Migrating SQL Server databases to AWS RDS (Relational Database Service) can be a daunting task, especially when dealing with large backup files. If you've ever tried to upload a sizable backup file—like a 20GB SQL Server backup—to Amazon S3 and faced connection issues, you’re not alone. In this guide, we’ll explore an effective solution to ensure that your valuable database backups reach their destination without running into errors.

Understanding the Issue

You attempted to upload a large backup file using AWS CLI but encountered an error after uploading a portion of the file. The issue specifically stated:

[[See Video to Reveal this Text or Code Snippet]]

This error indicates that the connection was interrupted before the upload was completed. Moreover, the command you used is:

[[See Video to Reveal this Text or Code Snippet]]

This method may not be optimal for large files. But don't worry! There are several strategies to efficiently transfer your large SQL Server backups to S3.

Solution: Multipart Uploads to the Rescue

What is Multipart Upload?

Amazon S3 supports a feature called Multipart Upload, which allows you to split large files into smaller parts and upload them individually. This process offers several advantages:

Faster Uploads: Smaller parts can be uploaded in parallel, enhancing upload speed.

Reliability: If an upload fails, you can re-upload only the failed parts rather than the entire file.

Enhanced Control: You can commence, pause, or cancel the upload based on your requirements.

How Multipart Upload Works with AWS CLI

With AWS CLI, Multipart Upload is initiated automatically when you use commands designed for large file transfers. Here's how to go about it:

Chunking the File: S3 will automatically break down large files when necessary. Use the aws s3 cp or aws s3 sync command, and it will manage the multipart upload behind the scenes.

Using an EC2 Instance: If you are experiencing connection issues from your on-premises machine while uploading the file directly to S3, consider the following workaround:

Launch an EC2 Instance: Create an EC2 instance close to your S3 bucket’s region.

Transfer the File: Upload the backup file to your EC2 instance using FTP or SCP.

Run the Upload Command: From your EC2 instance, run the same aws s3 cp command to transfer your backup file to S3:

[[See Video to Reveal this Text or Code Snippet]]

Best Practices for Large Backups

Split Large Databases: If you are working with even larger databases (e.g., 200GB), consider splitting them into smaller, manageable backup files if possible.

Check Your Networking Configurations: Ensure that there are no firewall or network security settings that might be impeding the upload process.

Regular Upload Attempts: If you continue to face issues, attempting the upload during off-peak hours might lead to better connection stability.

Conclusion

Uploading large SQL Server backup files to AWS S3 doesn't need to be overwhelming. By taking advantage of multipart uploads and strategically using an EC2 instance to manage the transfer, you can effectively overcome the common challenges associated with large file uploads.

By following the steps outlined above, you’ll be well on your way to achieving a successful SQL Server migration to AWS RDS. Happy migrating!