-
Notifications
You must be signed in to change notification settings - Fork 3.5k
Fix static mutex destruction order issues causing crashes at program termination #25770
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
During program termination, static mutexes can be destroyed while other static objects still reference them, causing crashes. This commonly occurs when logging during cleanup or when cleanup happens in an unexpected order. This fix uses heap-allocated mutexes (intentionally leaked) for critical static mutexes to ensure they remain valid throughout program lifetime: 1. DefaultLoggerMutex() in logging.cc - Used by logging infrastructure 2. GetOrtEnvMutex() in ort_env.cc - Used by ORT environment management 3. GetActiveSessionsMutex() in inference_session.cc - Windows-specific session tracking The intentional memory leak is negligible (a few bytes per mutex) and only occurs at program termination. This is a common pattern for avoiding static destruction order issues in C++. Fixes crashes during program termination when: - Logging occurs during static destruction - Multiple static objects have interdependencies - Cleanup order varies between platforms/configurations
c4d4c8e to
25a2444
Compare
|
@microsoft-github-policy-service agree [company="{your company}"]
@microsoft-github-policy-service agree [company="{Neekware Inc.}"] |
@microsoft-github-policy-service agree company="Neekware Inc." |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR addresses crashes during program termination caused by static mutex destruction order issues in C++. The fix replaces static mutexes with heap-allocated mutexes that are intentionally leaked to ensure they remain valid throughout the program's lifetime.
- Converts static mutexes to function-local heap-allocated mutexes using
new std::mutex() - Updates all mutex references to use getter functions instead of direct static member access
- Removes static mutex member declarations from header files
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| onnxruntime/core/session/ort_env.h | Removes static mutex member declaration and adds explanatory comment |
| onnxruntime/core/session/ort_env.cc | Implements GetOrtEnvMutex() function and updates all mutex usage |
| onnxruntime/core/session/inference_session.h | Removes Windows-specific static mutex declaration and adds explanatory comment |
| onnxruntime/core/session/inference_session.cc | Implements GetActiveSessionsMutex() function and updates all mutex usage |
| onnxruntime/core/common/logging/logging.cc | Updates DefaultLoggerMutex() to use heap-allocated mutex |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| // (and not shutting down). | ||
| static OrtEnv* p_instance_; | ||
| static std::mutex m_; | ||
| // Mutex is now accessed through GetOrtEnvMutex() function |
Copilot
AI
Aug 19, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The comment should explain why the mutex was moved to a function (to fix static destruction order issues) rather than just stating where it's accessed from.
| // Mutex is now accessed through GetOrtEnvMutex() function | |
| // The mutex is accessed through GetOrtEnvMutex() function to avoid static destruction order issues. | |
| // In C++, static objects (like a global mutex) may be destroyed before other static objects that still need them, | |
| // leading to undefined behavior. By moving the mutex into a function, it is constructed on first use and destroyed | |
| // after all other static objects, ensuring safe access during shutdown. |
| static std::map<uint32_t, InferenceSession*> active_sessions_; | ||
| #ifdef _WIN32 | ||
| static std::mutex active_sessions_mutex_; // Protects access to active_sessions_ | ||
| // Mutex is now accessed through GetActiveSessionsMutex() function to avoid static destruction issues |
Copilot
AI
Aug 19, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[nitpick] This comment is more informative than the one in ort_env.h but could be more consistent. Consider using similar phrasing across both files for consistency.
| // Mutex is now accessed through GetActiveSessionsMutex() function to avoid static destruction issues | |
| // Mutex is accessed via GetActiveSessionsMutex() |
|
/azp run Linux QNN CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows GPU Doc Gen CI Pipeline, Windows x64 QNN CI Pipeline |
|
Azure Pipelines successfully started running 5 pipeline(s). |
|
There's no need to leak as we can use the placement static std::mutex& DefaultLoggerMutex() noexcept {
alignas(std::mutex) static std::byte storage[sizeof(std::mutex)];
static std::mutex* const m = ::new (static_cast<void*>(storage)) std::mutex();
return *m;
} |
Description
This PR fixes crashes that occur during program termination due to static mutex destruction order issues. When static
mutexes are destroyed while other static objects still reference them, it causes undefined behavior and crashes.
Motivation and Context
Static destruction order in C++ is undefined across translation units. This causes crashes when:
Changes
The fix uses heap-allocated mutexes (intentionally leaked) for critical static mutexes to ensure they remain valid
throughout program lifetime:
DefaultLoggerMutex()inlogging.cc- Used by logging infrastructureGetOrtEnvMutex()inort_env.cc- Used by ORT environment managementGetActiveSessionsMutex()ininference_session.cc- Windows-specific session trackingTechnical Details
new std::mutex()without correspondingdelete(intentional leak)Testing
Note
This issue was discovered while working on the Rust-based TTS library which uses ONNX Runtime. The crashes were occurring during cleanup when the library was unloaded.