Unable to deploy a service using local packages

[mzmiric5]

This is a Bug Report

Description

When attempting to package or deploy a service which uses local packages either referenced via file: or yarn-workspace definitions, the deploy/package step that triggers the webpack build throws an error with

Command failed: yarn install --frozen-lockfile --non-interactive
error An unexpected error occurred: "Your lockfile needs to be updated, but yarn was run with `--frozen-lockfile`.".

Running yarn install --frozen-lockfile --non-interactive manually works just fine, so that points to an issue with serverless-webpack.

For bug reports:

• What went wrong?
  sls deploy and sls package fail with "Your lockfile needs to be updated, but yarn was run with `--frozen-lockfile`." error when using a local package
• What did you expect should have happened?
  The service should have been packaged and deployed successfully
• What was the config you used?
• What stacktrace or error message from your provider did you see?

@HyperBrain as requested in #369 I've opened a separate issue. I also went ahead and prepared a minimal repro repo with the config and dependencies set up that cause these issues to surface. https://github.com/mzmiric5/sls-wp-build-error-repro
Some notes:

• services/sls-only is a template without webpack, which deploys correctly and has no problems running on AWS
• shared/third-party-private is a placeholder for a package we are unable to share, but the fact that the package we are using has this one specific dependency is causing npm repro to fail with an error on npm install inside the sls package/deploy step
• our team primarily uses yarn, so this is our primary concern, and services/sls-wp-yarn shows the issue the best, as it fails with both the third-party-private package and the moment-wrapper example package (which the npm packager doesn't have an issue with)

For feature proposals:

• What is the use case that should be solved. The more detail you describe this in the easier it is to understand for us.
• If there is additional config how would it look

Similar or dependent issue(s):

Additional Data

• Serverless-Webpack Version you're using: 5.1.1
• Webpack version you're using: 4.5.0
• Serverless Framework Version you're using: 1.26.1
• Operating System: macOS Sierra 10.12.6
• Stack Trace (if available):

[nicholas-gates]

I ran into this exact same problem today. My yarn.lock file is up-to-date but the build still reports that it isn't. The command that fails in the build process runs without issue when executed manually.

[HyperBrain]

Hi @nickelstar ,

Can you check the package.json files that are generated by the plugin during deployment? You should find them after the error happens in the .webpack/dependencies and .webpack/service or .webpack/---function-name-- folders. It might be that a dependency you have is wrongly put into devDependencies and as a consequence is not installed and packaged during the deployment.

The frozen-lockfile error should reliably detect such errors, even if it does not show the exact reason without deeper inspection.

[nicholas-gates]

Thanks for the helpful info. When I looked at .webpack/dependencies/package.json, I noticed a package listed with an empty version number:

"dependencies": {
    "aws-sdk": "",

This struck me as odd since I don't use aws-sdk and it's not listed in my package.json. So I added it to my package.json via 'yarn add aws-sdk'. After that, the error went away and deployments started working with yarn specified as the webpack packager.

[HyperBrain]

@nickelstar Good to hear that. I think you can even move the aws-sdk to devDependencies as it will be found but eventually omitted when packaging, because AWS already provides it in theiy Lambda environment. That will reduce your package sizes again.

[franciscocpg]

@HyperBrain
I have this dependencies:

"dependencies": {
    "moment": "^2.20.1",
    "moment-timezone": "^0.5.14",
    "mongodb": "^2.2.33",
    "mssql": "^4.1.0",
    "object.values": "^1.0.4",
    "papaparse": "^4.3.6",
    "slug": "^0.9.1",
    "source-map-support": "^0.5.0",
    "sprintf-js": "^1.1.1",
    "underscore": "^1.8.3"
  },
  "devDependencies": {
    "aws-sdk": "^2.172.0",
    "babel-core": "^6.26.0",
    "babel-eslint": "^8.0.3",
    "babel-jest": "^22.1.0",
    "babel-loader": "^7.1.2",
    "babel-plugin-transform-runtime": "^6.23.0",
    "babel-preset-env": "^1.6.1",
    "babel-preset-flow": "^6.23.0",
    "eslint": "^4.16.0",
    "eslint-config-airbnb-base": "^12.1.0",
    "eslint-plugin-flowtype": "^2.40.1",
    "eslint-plugin-import": "^2.8.0",
    "eslint-plugin-jest": "^21.7.0",
    "flow-bin": "^0.63.1",
    "flow-typed": "^2.4.0",
    "glob": "^7.1.2",
    "jest": "^22.2.0",
    "serverless": "^1.25.0",
    "serverless-log-forwarding": "^1.1.6",
    "serverless-offline": "^3.16.0",
    "serverless-secrets": "https://github.com/franciscocpg/serverless-secrets.git#e623994662366b6fcfe7cc40f1d85fd19b5029cd",
    "serverless-webpack": "^5.1.1",
    "webpack": "^3.10.0",
    "webpack-node-externals": "^1.6.0",
    "webpack-plugin-copy": "^1.0.1"
  }

and then I see this .webpack/dependencies/package.json generated:

{
  (...)
  "dependencies": {
    "mongodb": "^2.2.33",
    "mssql": "^4.1.0",
    "object.values": "^1.0.4",
    "source-map-support": "^0.5.0",
    "moment": "^2.20.1",
    "lodash": "",
    "slug": "^0.9.1",
    "moment-timezone": "^0.5.14",
    "papaparse": "^4.3.6",
    "underscore": "^1.8.3",
    "sprintf-js": "^1.1.1"
  }
}

This empty lodash ("lodash":"") is causing the Your lockfile needs to be updated, but yarn was run with --frozen-lockfile error for me.

If I remove it yarn install --frozen-lockfile --non-interactive runs successfully. Do you have some idea what is causing this empty lodash?

Also when I run sls package I got a lot of warnings like this: WARNING: Could not determine version of module lodash.

[HyperBrain]

@franciscocpg That is strange. The lodash in the package.json points out that webpack deteceted it as a primary dependency of your code, but it seems to be missing in your original package.json.

This can happen if you use it directly in your code (by using import or require lodash) but forgot to add it as dependency. If any of your dependencies brings lodash in, you won't recognize the error locally, because it is implicitly in the node_modules folder. NPM might swallow such issues, but yarn - with it's better optimization - will bring it to the surface.
So I'm sure that the root cause is, that you use lodash somewhere but did not add it as explicit dependency.

[franciscocpg]

@HyperBrain
I've search for all occurrences of "lodash" in my entire project and I'm not using it anywhere.
I guess is some problem related to transitive dependencies and how yarn is resolving it.
Anyway I was able to workaround it by adding "lodash": "^4.17.10" to my dependencies.

[HyperBrain]

@franciscocpg Yes, you're right. This looks like a bug in the Yarn dependency tree transform (packager/yarn.js) and it seems to fail to retrieve the version numbers of 2nd level dependencies.
Cannot be much. I'll have a look later this week.
However, the case that a version cannot be retrieved can still happen even if the bug is fixed. Would it be wise and safe to assume @latest in these cases (additionally to the printed warning)?

[damonmaria]

Just struck this too (the original issue with local packages, not the lodash problem).

My local package is called shared. In the project directory it's referred to in files as:

• package.json: "shared": "file:../shared/dist/",
• yarn.lock: "shared@file:../shared/dist":

But in .webpack/dependencies it's referred to as:

• package.json: "shared": "file:../../../shared/dist/",
• yarn.lock: "shared@file:../shared/dist":

So the relative link in package.json has been fixed up with the move, but not the lock file. So yarn correctly fails but that might be tricky to correct for...

[HyperBrain]

@damonmaria Yes, that's exactly the problem. For npm we implemented a lock file rebasing for file references, for yarn I did not, because I missed it.

In theory the same approach as with npm should work (that's an easy one). I'll prepare a PR soon and handle this with high prio so that you can try and test that later.

[HyperBrain]

I'm working on a solution right now. However, there are 2 cases how local files can be integrated with yarn, so both need a proper handling:

case 1: yarn add my-module ../../somedir/mymodule
case 2: yarn add my-module file:../../somedir/mymodule
which lead to different representations in package.json as well as in yarn.lock

I'll do a first try to rebase any version reference that is file-like. If we encounter issues with that, we could - at least for now - restrict file references to use only the file: notation to make the use case work.

[HyperBrain]

@franciscocpg I think we should separate the 2 issues contained in this one into 2 issues.

I will fix the "local file references not handled" here and close it with the PR, but I think thet the dependency issue with yarn and transient dependencies is important too and should be solved and discussed properly.