How to increase s3 upload speed

[hardy-luo]

i need upload 200MB/s， but with aws m6in.8xlarge machine and disk io2, it only can reach 120~140MB/s。 so ,i changed the code of S3MultiPartOutputStream.java
Unfortunately, the log uploaded to S3 was parsed using the kafka command and an error was reported

bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /tmp/00000000000000000000.log | head Dumping /tmp/00000000000000000000.log Log starting offset: 0 Found 1072768950 invalid bytes at the end of 00000000000000000000.log

S3MultiPartOutputStream.java.txt

[jeqo]

@hardy-luo thanks for reaching out.

Could you share a bit more on the need to increase the throughput on the upload side? We were considering to allow this by parallelizing the upload part requests but decide against it given that we want to be conscious about the resources used when uploading files in the background--e.g if there are too many files to upload, it will consume as much bandwidth/cpu to upload them.
Because of this, the upload side now support a rate limiter to bound this per broker.

Reviewing the txt file will be difficult. Could you create a draft PR instead with the changes you're looking to make?