Process deadlock while waiting for s3 file transfer

[ferrouswheel]

Despite updating our aws-sdk-cpp libs recently to ensure we had fixes #511 and #520 - we are still seeing infrequent deadlocks when transferring to s3. Presumedly it's still related to TransferHandle::WaitUntilFinished() being called and m_waitUntilFinishedSignal never receiving a signal, but I am unable to see how.

We only have a single thread interacting with the AWS libraries, so we are reasonably confident it's not anything specific to our implementation.

It is difficult to debug as we only see this on EC2 instances (whether this is because we only run things at scale on EC2 or because of the reduced latency I don't know). Debugging requires some setup to remote attach and catch it before we terminate the process. If we encounter it again I will try update with the exact location it blocks.

For now I just wanted to document the issue in case others have seen it or have ideas where the trouble might be occurring.

[singku]

@ferrouswheel Hi, if possible, could you please describe the scenario of your use case, so that for us to better debugging this problem?

[singku]

@ferrouswheel Looking at transferHandle.cpp. There is a potential that currentStatus changed after acquisition, then If test actually do a wrong test. Will this affect your logic?

[ferrouswheel]

@singku The only place m_status.store is called is within that if statement, here - so I'm not sure that the cause of the deadlock.

[ferrouswheel]

Our use case is:

• we have many ec2 instances accessing s3
• the instances run multiple jobs, and they are reading hundreds of megabytes and writing hundreds of megabytes.
• for a single job. it will: read many small files, read and seeking to random positions in a large file, and usually write one single large file at the end. often we have seen the deadlock when writing the large file.

I'll try to provide further information as we can.

[singku]

Yes, but since your service write large file at the end. Multipart upload is applied, there are multiple executors(threads) help uploading file. it's possible some thread acquired m_status but it was changed(updateStatus call) by other thread before it go into if condition.

[marcomagdy]

One thing you could do, that could help us pin the problem down, is hook up the TransferStatusUpdatedCallback.

Something like:

    auto statusUpdatedCallback = [](const TransferManager* manager, const TransferHandle& 
    handle) 
    {
        // log sequence of events here
    }
    TransferManagerConfiguration tmConfig;
    tmConfig.s3Client = <your s3 client>;
    tmConfig.TransferStatusUpdatedCallback = statusUpdatedCallback;
    TransferManager transferManager(tmConfig);

[singku]

@ferrouswheel Hi, we have pushed out another fix for this bug. Could you please verify it? If the problem has been solved. Could you please close this issue?

[ferrouswheel]

I will integrate this in our next release. Unfortunately due to the nature of these bugs I won't be able to confirm if it has been solved until we've been running without deadlocks for a few weeks, but if it occurs again I'll reopen this issue.

Thank you for investigating and applying a fix!