Skip to content

Conversation

@Hazhzeng
Copy link
Contributor

@Hazhzeng Hazhzeng commented Dec 14, 2020

Description

Fixes #789

The new feature is implemented under PYTHON_ISOLATE_WORKER_DEPENDENCIES feature flag.

This PR is to introduce a capability to separate customer's site-packages from worker's dependencies.
We create a wrapper for loading the modules and ensure the sys.path and sys.modules have consistent content (invalidate overlapped cache) as the worker begin loading customer's code.

This will permanently address the issue when we bump up protobuf version in the future Python worker when customer is using an old version of protobuf.

When the new flag is set, we should see the following logging entries:

  1. Linux Consumption: Reloaded azure google namespace from customer dependencies
  2. Linux Dedicated: Start using customer dependencies {cls.cx_deps_path}

PR information

  • The title of the PR is clear and informative.
  • There are a small number of commits, each of which has an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, see this page.
  • If applicable, the PR references the bug/issue that it fixes in the description.
  • New Unit tests were added for the changes made and CI is passing.

Quality of Code and Contribution Guidelines

@Hazhzeng Hazhzeng marked this pull request as draft December 14, 2020 20:01
@Hazhzeng Hazhzeng changed the title [WIP] Implemented capability to isolate worker dependencies Implemented capability to isolate worker dependencies Dec 21, 2020
@Hazhzeng Hazhzeng marked this pull request as ready for review December 21, 2020 21:13
@Hazhzeng Hazhzeng force-pushed the hazeng/isolate branch 2 times, most recently from ea329f1 to 3b4c737 Compare December 22, 2020 19:37
@Hazhzeng
Copy link
Contributor Author

I tried many ways to isolate the environment variable changes impact on other tests, but only find separating the testing file works. Let me know if you have any questions on this one.

@Hazhzeng Hazhzeng merged commit 8404b64 into dev Jan 13, 2021
@Hazhzeng Hazhzeng deleted the hazeng/isolate branch January 13, 2021 06:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AttributeError: module 'google.protobuf.descriptor' has no attribute '_internal_create_key'

3 participants