Error when packager is yarn

[bazaglia]

This is a Bug Report

Description

When using nodeExternals in webpack config, deployment throws an error. The issue is only reproduced with includeModules: true + packager: 'yarn'.

 Error --------------------------------------------------
 
  Command failed: yarn install --frozen-lockfile --non-interactive
error An unexpected error occurred: "Your lockfile needs to be updated, but yarn was run with `--frozen-lockfile`.".

It may help because the deployment only include production packages, which causes a difference in yarn lockfile.

Additional Data

• Serverless-Webpack: 5.1.1
• Webpack version: 4.5.0
• Serverless Framework: 1.26.1
• webpack-node-externals: 1.7.2

[mzmiric5]

I've also noticed the same behavior, but only when I use local packages with the file: reference in my package.json

[HyperBrain]

@bazaglia @mzmiric5 This is intended behavior. The plugin sets --frozen-lockfile to prevent the build/deployment server install any new versions on its own and so change the module dependencies. You have to make sure that you commit a yarn.lock file in your projects that matches the package.json.

You should check, if your yarn.lock file is up-to-date and matches everything that is in your package.json. If you change your package.json manually, they are out of sync and the packaging will fail.

As long as you do a yarn install after changing the files and use yarn add to add linked of local (file) packages, it will generate a matching yarn.lock file and should work.

However, I can add an option like custom: webpack: packagerOptions: noFrozenLockFile: true that can be set in the serverless.yml. Imo it must default to freeze because of CI/CD stability but then you could disable the behavior easily for unstable testing environments.

@mzmiric5 :

but only when I use local packages with the file: reference [...]

That might be a small different bug in the plugin because local references are rebased. I think the package lock file is not rebased properly in that case.

[mzmiric5]

@HyperBrain oh yeah, I mean I totally agree that frozen lockfile should be a feature that is used for build stability, and in my case, if I just run the exact same yarn command outside of the serverless build process, I don't get this error. I guess I can make a separate issue regarding this since it seems to be affecting npm as well for me (which I was about to do anyway but I saw this issue, and it was describing the same final output I was getting so I tagged along).

[HyperBrain]

I guess I can make a separate issue regarding this since it seems to be affecting npm as well

Yes, please do that. Then the file reference/lock issue can be solved separately.

[bazaglia]

In my case, I'm not using local references. But while reading the output from serverless deployment, I just found out what was happening. In the list of modules in Packing external modules stage, there was a peer dependency trying to be packaged that was actually missing from my package.json. Then it was throwing the error.

Regarding the discussion above, the --frozen-lockfile completely makes sense to me.

Thank you @mzmiric5 and @HyperBrain.

[hollg]

I'm getting this same issue, but with no local modules and with a yarn.lock file that's in sync with package.json. I tried deleting yarn.lock and running yarn install to generate a new one, but it generated an identical file to the one I was already using.

If I remove nodeExternals() from externals in webpack.config.js then I no longer get the problem.

[HyperBrain]

If I remove nodeExternals() from externals in webpack.config.js then I no longer get the problem.

That's expected because then you'll end up without external modules and everything will be packaged. Can you check if you have some dependencies in devDependencies that are actually used by your deployed code instead of having them in dependencies?
This also leads to a "fresh" install of the modules on deployment and yields the lock file error. If that's the case the error is expected and points to such issues.

[hollg]

@HyperBrain thank you for the unbelievably quick response!

Perhaps I'm confused about what should be considered a devDependency. I'm only used to using Webpack for bundling React apps, where it is a devDependency. In this case, should Webpack and the related modules be dependencies instead?

[HyperBrain]

I don't mean the tools, but other dependencies that are runtime dependencies. A sample for that is e.g. source-map-support which becomes a runtime dependency if you use the webpack plugin for source map support. If you now put that into devDependencies, your code will run locally, but it might lead to a changed lock file when sls-webpack generates the final package.json for your service that contains the needed dependencies.

To further debug your problem, you could inspect the package.json files that were created in .webpack/dependencies or within the .webpack/service or .webpack/---function name--- folders. This should give insight into what happened exactly.

[phstc]

Command failed: yarn install --frozen-lockfile --non-interactive

I was having the same problem ☝️

Something that I noticed, is that the Packing external modules was outputting babel-runtime, without a version:

Serverless: Packing external modules: babel-runtime, @aws/dynamodb-data-mapper@0.4.0, serverless-dynamodb-client@^0.0.2, nodemailer@^4.4.2, wkhtmltopdf@^0.3.4, memorystream@^0.3.1, joi@12.0.0, bson-objectid@^1.2.2

all other modules have a version.

Adding babel-runtime in my package.json "fixed" the problem.

But adding babel-runtime didn't feel right, so I ended removing all my dependencies, then added one-by-one back, to find out that babel-plugin-transform-object-entries was causing the problem, but since I no longer need it, I removed it and the problem was gone 🙏

[Vadorequest]

I'm running into a similar issue, so I checked my .webpacl/dependencies/package.json VS my package.json and there are some differences:

{
  "name": "sib-helpers-server",
  "version": "1.0.0",
  "description": "Packaged externals for sib-helpers-server",
  "private": true,
  "scripts": {},
  "dependencies": {
    "source-map-support": "",
    "babel-runtime": "",
    "axios": "0.18.0",
    "dotenv": "6.0.0",
    "moment": "2.22.2",
    "@google-cloud/storage": "2.1.0",
    "aws-sdk": "2.334.0"
  }
}

Here is the package.json:

"devDependencies": {
    "@jimdo/serverless-dotenv": "1.0.0",
    "babel-cli": "6.26.0",
    "babel-core": "6.26.3",
    "babel-loader": "7.1.5",
    "babel-plugin-source-map-support": "2.0.1",
    "babel-plugin-transform-runtime": "6.23.0",
    "babel-preset-env": "1.7.0",
    "babel-preset-stage-2": "6.24.1",
    "nodemon": "1.18.4",
    "serverless": "1.32.0",
    "serverless-offline": "3.27.0",
    "serverless-plugin-aws-alerts": "1.2.4",
    "serverless-webpack": "5.2.0",
    "webpack": "4.20.2",
    "webpack-node-externals": "1.7.2"
  },
  "dependencies": {
    "@google-cloud/storage": "2.1.0",
    "adm-zip": "0.4.11",
    "aws-sdk": "2.334.0",
    "axios": "0.18.0",
    "copy-webpack-plugin": "4.5.4",
    "dotenv": "6.0.0",
    "fast-crc32c": "1.0.4",
    "jest": "23.6.0",
    "json2csv": "4.2.1",
    "jszip": "3.1.5",
    "lodash.map": "4.6.0",
    "lodash.uniq": "4.5.0",
    "moment": "2.22.2",
    "yargs": "12.0.2"
  }

I don't understand why source-map-support and transform-runtime are empty.

Here is my .babelrc:

{
  "plugins": [
    "source-map-support",
    "transform-runtime"
  ],
  "presets": [
    "env",
    "stage-2"
  ]
}

Any idea? I tried deleting/recreating the yarn.lock as well but didn't fix anything.

--

Edit: We fixed it by using babel-runtime and source-map-support as dependencies, I don't really understand why it works this way, it's very confusing to say the least.

  "devDependencies": {
    "aws-sdk": "2.354.0",
    "babel-cli": "6.26.0",
    "babel-core": "6.26.3",
    "babel-loader": "7.1.5",
    "babel-plugin-source-map-support": "1.0.0",
    "babel-plugin-transform-runtime": "6.23.0",
    "babel-preset-env": "1.7.0",
    "babel-preset-stage-2": "6.24.1",
    "nodemon": "1.18.4",
    "serverless": "1.32.0",
    "serverless-offline": "3.27.0",
    "serverless-plugin-aws-alerts": "1.2.4",
    "serverless-webpack": "5.2.0",
    "webpack": "4.20.2",
    "webpack-node-externals": "1.7.2"
  },
  "dependencies": {
    "@google-cloud/storage": "2.1.0",
    "@jimdo/serverless-dotenv": "1.0.0",
    "adm-zip": "0.4.11",
    "axios": "0.18.0",
    "babel-runtime": "6.26.0",
    "copy-webpack-plugin": "4.5.4",
    "dotenv": "6.1.0",
    "fast-crc32c": "1.0.4",
    "jest": "23.6.0",
    "json2csv": "4.2.1",
    "jszip": "3.1.5",
    "lodash.map": "4.6.0",
    "lodash.uniq": "4.5.0",
    "moment": "2.22.2",
    "source-map-support": "0.4.18",
    "yargs": "12.0.2"
  }

[clarsen]

FWIW i've also noticed this if I'm using a dependency that a module included but I didn't explicitly add as a dependency. E.g. I have in one of my files:

import { DataSource } from 'apollo-datasource'

it isn't in my package.json, but I get this ok because it comes via apollo-server-lambda -> apollo-server-core -> apollo-datasource. I use serverless offline to run the code locally.

On serverless deploy, I see:

Serverless: WARNING: Could not determine version of module apollo-datasource
Serverless: Adding explicit peers for dependency apollo-server-lambda
...
  yarn install --frozen-lockfile --non-interactive failed with code 1
  error Your lockfile needs to be updated, but yarn was run with `--frozen-lockfile`.

but if i do yarn add apollo-datasource, it deploys ok.

I suspect it is good practice to have as package dependencies the modules you explicitly use in your package, e.g. you shouldn't be able to import 'lodash' just because some module you use has it as a dependency. However, why serverless deploy doesn't warn or fail outright because the dependency isn't explicitly added is a a puzzle to me.