ENOENT: no such file or directory, stat chardet package

[wjohnson]

When deploying a Queue Trigger app and having the azure-storage-blob package installed, the app fails to deploy because the KuduSync process is looking for a file in the chardet lib that doesn't appear to exist in my .python_packages folder that is generated by the --build-native-deps pre deployment task.
7:06:54 AM wjtestqueutrig: Error: ENOENT: no such file or directory, stat '/tmp/zipdeploy/extracted/.python_packages/lib/python3.6/site-packages/chardet-3.0.4.dist-info/chardetect,sha256=fadXTQgqo6bB2Ofhhd0rG4HjgXzoiXwYp2Dtv3EI80c,229

Edit: When deploying to a new Consumption plan, the requirements.txt / init below work successfully. I am still experiencing this issue when deploying to brand new app service plan.

Investigative information

Please provide the following:

• Timestamp: 7:06:54 AM 2019-10-11 CST
• Function App name: wjtestqueutrig
• Function name(s) (as appropriate): QueueTrigger
• Core Tools version: 2.7.1704

Repro steps

Provide the steps required to reproduce the problem:

Expected behavior

Expected to successfully deploy the Azure Function.

Actual behavior

I receive the following error and the contents of my QueueTrigger function never makes it up to Azure / the App Service as observed in Kudu / SSH.

The most interesting line in this error is 7:06:54 AM wjtestqueutrig: Error: ENOENT: no such file or directory, stat '/tmp/zipdeploy/extracted/.python_packages/lib/python3.6/site-packages/chardet-3.0.4.dist-info/chardetect,sha256=fadXTQgqo6bB2Ofhhd0rG4HjgXzoiXwYp2Dtv3EI80c,229 I don't know where this file is coming from. I looked in my .python_packages and it doesn't exist there so perhaps this is an issue with the KuduSync command?

I'm able to run the python function locally without any issue.

7:05:46 AM wjtestqueutrig: Starting deployment...
7:06:36 AM wjtestqueutrig: Fetching changes.
7:06:36 AM wjtestqueutrig: Cleaning up temp folders from previous zip deployments and extracting pushed zip file /tmp/zipdeploy/7736f54c-c12c-42c1-87a1-f7d40d878375.zip (5.81 MB) to /tmp/zipdeploy/extracted
7:06:43 AM wjtestqueutrig: Updating submodules.
7:06:43 AM wjtestqueutrig: Preparing deployment for commit id 'fc5b500bed'.
7:06:44 AM wjtestqueutrig: Generating deployment script.
7:06:45 AM wjtestqueutrig: Using the following command to generate deployment script: 'azure site deploymentscript -y --no-dot-deployment -r "/tmp/zipdeploy/extracted" -o "/home/site/deployments/tools" --basic --sitePath "/tmp/zipdeploy/extracted"'.
7:06:48 AM wjtestqueutrig: Generating deployment script for Web Site
7:06:48 AM wjtestqueutrig: Generated deployment script files
7:06:48 AM wjtestqueutrig: Running deployment command...
7:06:48 AM wjtestqueutrig: Command: "/home/site/deployments/tools/deploy.sh"
7:06:48 AM wjtestqueutrig: Handling Basic Web Site deployment.
7:06:49 AM wjtestqueutrig: Kudu sync from: '/tmp/zipdeploy/extracted' to: '/home/site/wwwroot'
7:06:49 AM wjtestqueutrig: Copying file: 'host.json'
7:06:49 AM wjtestqueutrig: Copying file: 'proxies.json'
7:06:49 AM wjtestqueutrig: Copying file: 'requirements.txt'
7:06:49 AM wjtestqueutrig: Copying file: '.python_packages/requirements.txt.md5'
7:06:49 AM wjtestqueutrig: Copying file: '.python_packages/lib/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so'
7:06:49 AM wjtestqueutrig: Copying file: '.python_packages/lib/python3.6/site-packages/six.py'
7:06:49 AM wjtestqueutrig: Copying file: '.python_packages/lib/python3.6/site-packages/.libs_cffi_backend/libffi-ae16d830.so.6.0.4'
[SNIPPED FOR BREVITY]
7:06:51 AM wjtestqueutrig: Omitting next output lines...
7:06:54 AM wjtestqueutrig: Error: ENOENT: no such file or directory, stat '/tmp/zipdeploy/extracted/.python_packages/lib/python3.6/site-packages/chardet-3.0.4.dist-info/chardetect,sha256=fadXTQgqo6bB2Ofhhd0rG4HjgXzoiXwYp2Dtv3EI80c,229
7:06:54 AM wjtestqueutrig: '
7:06:54 AM wjtestqueutrig: An error has occurred during web site deployment.
7:06:54 AM wjtestqueutrig: Kudu Sync failed
7:06:54 AM wjtestqueutrig: \n/opt/Kudu/Scripts/starter.sh "/home/site/deployments/tools/deploy.sh"
7:06:54 AM wjtestqueutrig: App container will begin restart within 10 seconds.
7:07:05 AM wjtestqueutrig: Waiting for long running command to finish...
Deployment to "wjtestqueutrig" completed.

Known workarounds

If I don't need azure-storage-blob, the function works fine. In this simplified example, I don't need it (as the init never calls azure.storage.blob) but I DO need it for the actual application I'm building.

Related information

Provide any related information

• Links to source
• Contents of the requirements.txt file
• Bindings used

Source

# __init__.py

import logging

import azure.functions as func

def main(msg: func.QueueMessage) -> None:
    logging.info('Python queue trigger function processed a queue item: %s',
                 msg.get_body().decode('utf-8'))

asn1crypto==1.0.1
azure-common==1.1.23
azure-functions==1.0.4
azure-storage-blob==2.1.0
azure-storage-common==2.1.0
certifi==2019.9.11
cffi==1.12.3
chardet==3.0.4
cryptography==2.7
idna==2.8
pycparser==2.19
python-dateutil==2.8.0
requests==2.22.0
six==1.12.0
urllib3==1.25.6

[priyaananthasankar]

@Hazhzeng : At a customer level Azure DevOps will be used to deploy things. Func core tools is used for local testing/debugging and beyond that it will be Azure DevOps.
Does the DevOps pipeline support remote build? For Appservice vs Consumption? If not, it should be targeted to first get some support there.

[PrateekVachher]

I am facing the same issue on Kudu with Azure App Service

[Hazhzeng]

@Hazhzeng : At a customer level Azure DevOps will be used to deploy things. Func core tools is used for local testing/debugging and beyond that it will be Azure DevOps.
Does the DevOps pipeline support remote build? For Appservice vs Consumption? If not, it should be targeted to first get some support there.

From my knowledge, no, currently DevOps for Linux Consumption is still using WEBSITE_RUN_FROM_PACKAGE deployment.

[anirudhgarg]

@hanzheng - they are asking about App Service plan

[ankitkumarr]

Would you mind trying to add an App Setting WEBSITE_RUN_FROM_PACKAGE = 1 to your function app and then retrying.

[PrateekVachher]

@ankitkumarr @anirudhgarg Yeah, adding that Variable in the App Settings fixed the problem. Do I have to do this for every app I deploy? Can I expect this error to re-appear even if I don't make any changes to my cloud architecture or configuration in production?

[anirudhgarg]

@PrateekVachher - We have seen Kudu Sync be sometimes problematic when we have a running app if you are not using Run from Package. So for this scenario yes it is recommended to use WEBSITE_RUN_FROM_PACKAGE=1 @priyaananthasankar @wjohnson please also try and let us know

[bmurphy92]

@ankitkumarr @anirudhgarg We are experiencing this as well, except with an existing production app that we have successfully deployed to many times. Setting WEBSITE_RUN_FROM_PACKAGE = 1 failed as well (Kudu said to check diagnostic logs). This is an urgent problem - please advise?

[anirudhgarg]

@bmurphy92 Are you seeing the exact same symptoms and is the app also using azure-storage-blob. If it is affecting production I would suggest to open a support ticket.

[bmurphy92]

@anirudhgarg yes, we are experiencing the exact same error message.

2019-10-18T22:48:05  Fetching changes.
2019-10-18T22:48:05    Cleaning up temp folders from previous zip deployments and extracting pushed zip file /tmp/zipdeploy/5ebee4e9-ba68-4bca-acb6-822e8a1d481d.zip (52.19 MB) to /tmp/zipdeploy/extracted
2019-10-18T22:48:15  Updating submodules.
2019-10-18T22:48:15  Preparing deployment for commit id '0a1f9ee145'.
2019-10-18T22:48:15  Generating deployment script.
2019-10-18T22:48:15    Using cached version of deployment script (command: 'azure -y --no-dot-deployment -r "/tmp/zipdeploy/extracted" -o "/home/site/deployments/tools" --basic --sitePath "/tmp/zipdeploy/extracted"').
2019-10-18T22:48:15  Running deployment command...
2019-10-18T22:48:15    Command: "/home/site/deployments/tools/deploy.sh"
2019-10-18T22:48:15    Handling Basic Web Site deployment.
2019-10-18T22:48:16    Kudu sync from: '/tmp/zipdeploy/extracted' to: '/home/site/wwwroot'
2019-10-18T22:48:16    Copying file: '.dockerignore'
2019-10-18T22:48:16    Copying file: 'Dockerfile'
2019-10-18T22:48:16    Copying file: 'README.md'
2019-10-18T22:48:16    Copying file: '__init__.py'
2019-10-18T22:48:16    Copying file: 'host.json'
2019-10-18T22:48:16    Copying file: 'requirements.txt'
2019-10-18T22:48:16    Copying file: '.python_packages/requirements.txt.md5'
[SNIPPED FOR BREVITY]
2019-10-18T22:48:17    Omitting next output lines...
2019-10-18T22:48:18    Error: ENOENT: no such file or directory, stat '/tmp/zipdeploy/extracted/.python_packages/lib/python3.6/site-packages/chardet-3.0.4.dist-info/chardetect,sha256=fadXTQgqo6bB2Ofhhd0rG4HjgXzoiXwYp2Dtv3EI80c,229
2019-10-18T22:48:18    '
2019-10-18T22:48:18    An error has occurred during web site deployment.
2019-10-18T22:48:18    Kudu Sync failed
2019-10-18T22:48:18    
/opt/Kudu/Scripts/starter.sh "/home/site/deployments/tools/deploy.sh"
2019-10-18T22:48:18  
2019-10-18T22:48:18  App container will begin restart within 10 seconds.

It isn't realistic to provide a reproducible example, as this Azure function is a fairly complex process that hits several SQL server databases and Azure Blob storage containers.

Some information:

• Unlike @wjohnson, this isn't a new Azure function, it has been running for about 4 months and we have deployed to it (whether to staging or production) over 20 times without issue
• We use Azure CLI via powershell to deploy, with the command func azure functionapp publish <name> --build-native-deps. We have not been running from a package
• It just started failing on Friday. but we unfortunately did not catch it because it fails silently through Azure CLI in Powershell (it looks like a success). It was not until we were doing final testing on our staging environment over the weekend that we realized the function had not been updated and checked the Kudu deployment logs and discovered the issue

Questions:

• What is the best way to create a support ticket for this issue? Is there a specific team we should reach out to?
• In the meantime, I'd like to try to publish the function with WEBSITE_RUN_FROM_PACKAGE = 1. However, there is very little documentation for how to do this with a Python Azure function
• Do we need to create a new Azure function to run from package? What Azure CLI command should we use to deploy? func azure functionapp <name> --build-native-deps, or something else? It is confusing because when we deploy via the Azure CLI, we receive a message that says Assuming --nozip (do not run from package) for publishing to Linux dedicated plan. which seems to contradict your and @ankitkumarr guidance on this bug report to run the Python function from package

Thanks.

[ankitkumarr]

@bmurphy92, I think you are most likely using an old version of azure-functions-core-tools. You can check using func --version. The current release is 2.7.1724.

Would you mind trying the new version of core-tools? We made a ton of improvements recently when Python was announced GA.