Unable to build from source, trying to build with sqlcipher

[wedwards99]

I was having problems building node-sqlite3 with sqlcipher on Windows 7 64bit, it appeared that it was not a sqlcipher issue but a build-from-source issue, note: "npm install sqlite3" works just fine for me.

When I run the install with build from source I get the following output with the spawn ENOENT error.

Thanks, Will

C:\will> npm install sqlite3 --build-from-source

npm http GET https://registry.npmjs.org/sqlite3
npm http 200 https://registry.npmjs.org/sqlite3
npm http GET https://registry.npmjs.org/sqlite3/-/sqlite3-2.2.7.tgz
npm http 200 https://registry.npmjs.org/sqlite3/-/sqlite3-2.2.7.tgz
npm http GET https://registry.npmjs.org/nan
npm http GET https://registry.npmjs.org/set-immediate
npm http 200 https://registry.npmjs.org/nan
npm http GET https://registry.npmjs.org/nan/-/nan-1.1.2.tgz
npm http 200 https://registry.npmjs.org/nan/-/nan-1.1.2.tgz
npm http 200 https://registry.npmjs.org/set-immediate
npm WARN deprecated set-immediate@0.1.1: Use setimmediate instead
npm http GET https://registry.npmjs.org/set-immediate/-/set-immediate-0.1.1.tgz
npm http 200 https://registry.npmjs.org/set-immediate/-/set-immediate-0.1.1.tgz

sqlite3@2.2.7 install C:\will\node_modules\sqlite3
node-pre-gyp install --fallback-to-build

C:\will\node_modules\sqlite3>node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin....\node_modules\node-gyp\bin\node-gyp.js" rebuild --name=sqlite3 --configuration=Release --module_name=node_sqlite3 --version=2.2.7 --major=2 --minor=2 --patch=7 --runtime=node --node_abi=node-v11 --platform=win32 --target_platform=win32 --arch=x64 --target_arch=x64 --module_main=./lib/sqlite3 --host=https://mapbox-node-binary.s3.amazonaws.com/ --module_path=C:\will\node_modules\sqlite3\lib\binding\node-v11-win32-x64 --remote_path=./sqlite3/v2.2.7/ --package_name=node-v11-win32-x64.tar.gz --staged_tarball=build\stage\sqlite3\v2.2.7\node-v11-win32-x64.tar.gz --hosted_path=https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v2.2.7/ --hosted_tarball=https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v2.2.7/node-v11-win32-x64.tar.gz
gyp ERR! configure error
gyp ERR! stack Error: spawn ENOENT
gyp ERR! stack at errnoException (child_process.js:998:11)
gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:789:34)
gyp ERR! System Windows_NT 6.1.7601
gyp ERR! command "node" "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" "rebuild" "--name=sqlite3" "--configuration=Release" "--module_name=node_sqlite3" "--version=2.2.7" "--major=2" "--minor=2" "--patch=7" "--runtime=node" "--node_abi=node-v11" "--platform=win32" "--target_platform=win32" "--arch=x64" "--target_arch=x64" "--module_main=./lib/sqlite3" "--host=https://mapbox-node-binary.s3.amazonaws.com/" "--module_path=C:\will\node_modules\sqlite3\lib\binding\node-v11-win32-x64" "--remote_path=./sqlite3/v2.2.7/" "--package_name=node-v11-win32-x64.tar.gz" "--staged_tarball=build\stage\sqlite3\v2.2.7\node-v11-win32-x64.tar.gz" "--hosted_path=https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v2.2.7/" "--hosted_tarball=https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v2.2.7/node-v11-win32-x64.tar.gz"
gyp ERR! cwd C:\will\node_modules\sqlite3
gyp ERR! node -v v0.10.28
gyp ERR! node-gyp -v v0.13.0
gyp ERR! not ok
node-pre-gyp ERR! build error
node-pre-gyp ERR! stack Error: Failed to execute 'node-gyp.cmd rebuild --name=sqlite3 --configuration=Release --module_name=node_sqlite3 --version=2.2.7 --major=2 --minor=2 --patch=7 --runtime=node --node_abi=node-v11 --platform=win32 --target_platform=win32 --arch=x64 --target_arch=x64 --module_main=./lib/sqlite3 --host=https://mapbox-node-binary.s3.amazonaws.com/ --module_path=C:\will\node_modules\sqlite3\lib\binding\node-v11-win32-x64 --remote_path=./sqlite3/v2.2.7/ --package_name=node-v11-win32-x64.tar.gz --staged_tarball=build\stage\sqlite3\v2.2.7\node-v11-win32-x64.tar.gz --hosted_path=https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v2.2.7/ --hosted_tarball=https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v2.2.7/node-v11-win32-x64.tar.gz' (1)
node-pre-gyp ERR! stack at ChildProcess. (C:\will\node_modules\sqlite3\node_modules\node-pre-gyp\lib\util\compile.js:76:29)
node-pre-gyp ERR! stack at ChildProcess.EventEmitter.emit (events.js:98:17)
node-pre-gyp ERR! stack at maybeClose (child_process.js:753:16)
node-pre-gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:820:5)
node-pre-gyp ERR! System Windows_NT 6.1.7601
node-pre-gyp ERR! command "node" "C:\will\node_modules\sqlite3\node_modules\node-pre-gyp\bin\node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR! cwd C:\will\node_modules\sqlite3
node-pre-gyp ERR! node -v v0.10.28
node-pre-gyp ERR! node-pre-gyp -v v0.5.22
node-pre-gyp ERR! not ok
Failed to execute 'node-gyp.cmd rebuild --name=sqlite3 --configuration=Release --module_name=node_sqlite3 --version=2.2.7 --major=2 --minor=2 --patch=7 --runtime=node --node_abi=node-v11 --platform=win32 --target_platform=win32 --arch=x64 --target_arch=x64 --module_main=./lib/sqlite3 --host=https://mapbox-node-binary.s3.amazonaws.com/ --module_path=C:\will\node_modules\sqlite3\lib\binding\node-v11-win32-x64 --remote_path=./sqlite3/v2.2.7/ --package_name=node-v11-win32-x64.tar.gz --staged_tarball=build\stage\sqlite3\v2.2.7\node-v11-win32-x64.tar.gz --hosted_path=https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v2.2.7/ --hosted_tarball=https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v2.2.7/node-v11-win32-x64.tar.gz' (1)
npm ERR! sqlite3@2.2.7 install: node-pre-gyp install --fallback-to-build
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the sqlite3@2.2.7 install script.
npm ERR! This is most likely a problem with the sqlite3 package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-pre-gyp install --fallback-to-build
npm ERR! You can get their info via:
npm ERR! npm owner ls sqlite3
npm ERR! There is likely additional logging output above.

npm ERR! System Windows_NT 6.1.7601
npm ERR! command "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "install" "sqlite3" "--build-from-source"
npm ERR! cwd C:\will
npm ERR! node -v v0.10.28
npm ERR! npm -v 1.4.9
npm ERR! code ELIFECYCLE
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! C:\will\npm-debug.log
npm ERR! not ok code 0

[Mithgol]

• What's in your PATH system environment variable?
• Do you have all the things that node-gyp requires on your system? (Python, Visual Studio, etc.)
• Does anything improve if you run npm update npm -g beforehand?

[wedwards99]

I debugged this some more and as I suspect you were thinking it was a problem with the Microsoft SDK, apparently a service pack update had removed the 64 bit compileres, I went here: http://www.microsoft.com/en-us/download/details.aspx?id=4422 to get the MS patch to restore them.

Then when I ran --build-from-source using the Windows SDK 7.1 Command Prompt the build went through fine.

However, when I then expanded that to build with SQLCipher:

npm install -dd --build-from-source --sqlite_libname=sqlcipher64 --sqlite=/sqlitedriver/usr

The final linker step contained invalid linker options: "-L\sqlitedriver\usr\lib.lib" sqlcipher64.lib I could not figure out how to make those right so in the end I copied the linker command from the debug output into a text editor replaced the invalid part with \sqlitedriver\usr\lib\libeay64.lib \sqlitedriver\usr\lib\sqlcipher64.lib \sqlitedriver\usr\lib\ssleay64.lib and ran that edited link command directly from the build dir.

Thanks,
Will

[springmeyer]

@wedwards99 - thanks for this report - the issue, as you found, is that the --sqlite_libname option was never tested on windows and needs to be handled differently on that platform.

[springmeyer]

@wedwards99 47f3812 starts addressing this. It looks like it alone is not a complete solution since it appears ssleay64.lib and libeay64.lib also need to be linked. Can you tell me more about these libraries? Are they windows system libraries or part of sqlcipher?

[wedwards99]

They are part of openssl which is in turn part of sqlcipher. I followed this to build the sqlcipher libraries: https://github.com/CovenantEyes/sqlcipher-windows

Initially when building node-sqlite3 I only included sqlcipher64.lib but got a ton of unresolved externals when linking, all the resolution failures were openssl related and the inclusion of the openssl libs ssleay64.lib and libeay64.lib fixed that issue.

Thanks, Will

[McPo]

Is there anymore substance that can be added to this issue?

I have successfully built SQLCipher for Windows and am able to execute it. (Used this guide http://www.jerryrw.com/howtocompile.php)
I then attempted to compile it against node.

The issue I ran into is that the node build looks for a .lib, I initially only had a .a and .dll
As such using the lib tool I made an importable .lib file.

I then compiled this against node using a command similar to the above npm one (Obviously file names are different). It successfully created a .node file.

As such I went to execute the electron project. It then complains

"The specified module could not be found. \?C:......."

If I delete the .node file it complains.

"Cannot find module C:......."

A few interesting things to note. The error is different when I remove the .node file. So its definitely finding it. In the case that it loads the .node file, it complains that the specified module could not be found, however the path is prepended with \?.

Another interesting thing is that the file size is substantially smaller than the previous node file I was using SQLite (1080KB) vs SQLCipher (180 KB). Although the file size is line with SQLCipher for the other projects, so maybe not that odd.

The only thing I can think of, is that the .lib should be a static .lib and not the import .lib which it appears to be. However Im unable to make the vcproj reference the .a file as its generated during compilation.

Is the .lib supposed to be static, and is there a way I can forever that, that is the issue. Finally are you aware of a method to turn an .dll or .a into a static .lib, or a way in which I can make the vcproj load the .a file instead.

I have already managed to generate a working .node file with SQLCipher for both Linux and OSX. The Windows "node install" also generates another .exp and .lib file.

Thanks,
Emmet

[Mithgol]

@McPo Could it be that the .node file that you created was (somehow) not compatible with Electron's ABI? After reading a bit of http://electron.atom.io/docs/tutorial/using-native-node-modules/ I suspect that might be the cause.

[McPo]

Apologies I should've updated this.

The issues was that I had to drop the DLL for both OpenSSL and SQLCipher into the bin folder.
I didn't want to do this, so I had to find a way to statically link against OpenSSL and SQLCipher.

In the end I used.
https://github.com/karlosp/sqlcipher-visual-studio

It provides a more up to date SQLCipher version than the previous linked project, and it has OpenSSL out of the box. It also provides instruction on how to up date to the latest SQCipher which I can confirm works (3.4.0).

Few tips.
You need to use Visual Studios 2013. 2012 wont work with node-gyp and 2015 has issues with the SQLCipher project at the final stage of compiling it against node. So use Visual Studios 2013. The community edition is fine. It also has to be Visual Studios, the Build Tools alone, isn't good enough.

If your updating the SQLCipher version used in the project, then compile it on a Linux machine and copy across the .c and .h files. There are no issues and it works out of the box. OSX has some issues finding the OpenSSL headers. Compiling it on Windows is gonna require MYSYS2.

If your updating the OpenSSL implementation, there will be several libea32.lib files. You need to copy the one from the 32dll folder (I think thats the correct name). Otherwise you'll be using the proxy lib file for the dll and not the actual static lib file. This will replace the libea32MT.lib file in the Visual Studio project.

Im planning on making a blog post with more details.

• This is from memory so the specifics might be slightly off. If anyone has issues, feel free to contact me.

[onkarnaik511]

@McPo
I too successfully built SQLCipher for Windows and ran it and while running it in node, ran into the same issues that you mentioned regarding the node_sqlite3.node file not being found.
So I checked the link https://github.com/karlosp/sqlcipher-visual-studio.
Once again I built SQLCipher for Windows, which was easy compared to the previous try but once again i ran into issues while trying to build for node.
Could you give more details on how you built SQLCipher for node environment, after building it for windows.

[fritx]

related: #587

[daniellockyer]

Would you be able to try with the latest version of this library? A lot of build issues have been fixed since and it's difficult to compare versions 6 years apart 😀

If there's still a problem, please open an issue with the new templates