Describe the bug
Amplify Android has been investigating an issue (aws-amplify/amplify-android#2869) where the root cause is that when doing an S3 transfer with chunked encoding, we read the number of bytes transferred as larger than the actual file size itself.
I understand that there's technically byte transfer overhead holistically when it comes to chunked encoding passing additional headers however the numbers don't really add up. e.g. for a 10,000,000 byte file, there's an additional 13,941 bytes that get transferred which seems way higher than the headers overhead size for chunked encoding. Additionally, other SDKs don't have this issue. I cross-referenced with Amplify Swift and even the legacy AWS SDK for Android's S3 transfer and their progress is always reported accurately -- for a 10,000,000 byte file, 10,000,000 bytes are reported as transferred.
If we disable chunked encoding, the number of bytes that are reported is accurate.
Regression Issue
Expected behavior
We understand that for situations where the length of the content is unknown, progress listening doesn't make sense -- we're not asking about that. We have a known file size, so we expect that we should know when the file has finished transferring.
Current behavior
If you read the number of bytes transferred for an S3 download/upload with chunked encoding enabled, the number of bytes transferred is larger than the file size.
Steps to Reproduce
See our implementation of a progress listener interceptor:
https://github.com/aws-amplify/amplify-android/blob/e31b961e97c5896f200ff5939fa6d249832c0d27/aws-storage-s3/src/main/java/com/amplifyframework/storage/s3/transfer/ProgressListenerHttpInterceptor.kt#L53-L72
That gets added as an interceptor to the S3 client like this:
https://github.com/aws-amplify/amplify-android/blob/e31b961e97c5896f200ff5939fa6d249832c0d27/aws-storage-s3/src/main/java/com/amplifyframework/storage/s3/transfer/worker/SinglePartUploadWorker.kt#L45-L51
From there, just initiate an S3 download or upload and read the number of bytes and it'll read a higher number of bytes with encoding than the file size itself.
Possible Solution
No response
Context
No response
Smithy-Kotlin version
1.3.23
Platform (JVM/JS/Native)
Android
Operating system and version
Android 11
Describe the bug
Amplify Android has been investigating an issue (aws-amplify/amplify-android#2869) where the root cause is that when doing an S3 transfer with chunked encoding, we read the number of bytes transferred as larger than the actual file size itself.
I understand that there's technically byte transfer overhead holistically when it comes to chunked encoding passing additional headers however the numbers don't really add up. e.g. for a 10,000,000 byte file, there's an additional 13,941 bytes that get transferred which seems way higher than the headers overhead size for chunked encoding. Additionally, other SDKs don't have this issue. I cross-referenced with Amplify Swift and even the legacy AWS SDK for Android's S3 transfer and their progress is always reported accurately -- for a 10,000,000 byte file, 10,000,000 bytes are reported as transferred.
If we disable chunked encoding, the number of bytes that are reported is accurate.
Regression Issue
Expected behavior
We understand that for situations where the length of the content is unknown, progress listening doesn't make sense -- we're not asking about that. We have a known file size, so we expect that we should know when the file has finished transferring.
Current behavior
If you read the number of bytes transferred for an S3 download/upload with chunked encoding enabled, the number of bytes transferred is larger than the file size.
Steps to Reproduce
See our implementation of a progress listener interceptor:
https://github.com/aws-amplify/amplify-android/blob/e31b961e97c5896f200ff5939fa6d249832c0d27/aws-storage-s3/src/main/java/com/amplifyframework/storage/s3/transfer/ProgressListenerHttpInterceptor.kt#L53-L72
That gets added as an interceptor to the S3 client like this:
https://github.com/aws-amplify/amplify-android/blob/e31b961e97c5896f200ff5939fa6d249832c0d27/aws-storage-s3/src/main/java/com/amplifyframework/storage/s3/transfer/worker/SinglePartUploadWorker.kt#L45-L51
From there, just initiate an S3 download or upload and read the number of bytes and it'll read a higher number of bytes with encoding than the file size itself.
Possible Solution
No response
Context
No response
Smithy-Kotlin version
1.3.23
Platform (JVM/JS/Native)
Android
Operating system and version
Android 11