Support for binary deployment

[springmeyer]

This pull changes the behavior of node-sqlite3 builds to prefer installation from pre-built binaries. Closes #67

This makes it easier and faster to run npm install sqlite3 on OS X, Linux, and Windows without any compiler being available and also even without python and node-gyp. When a user runs npm install sqlite3 or npm install from the source checkout first a binary node-sqlite3.node file will be fetched, if available, from a remote repository of binaries managed by @springmeyer on s3/cloudfront. If a binary is not available that matches what the user needs then the build system will fall back to a source compile.

If a developer wants a source compile instead of a binary then she can pass --build-from-source or --build-from-source=sqlite3 to the args passed to npm install or node-gyp.

Binaries are versioned based on:

{configuration folder}/node-sqlite3-{MAJOR}.{MINOR}.{ABI}-node-v{ABI}-{platform}-{arch}

Details:

1. Release/Debug configuration
2. node-sqlite3 version, specifically MAJOR.MINOR.ABI where MAJOR and MINOR match the semver version of the module but ABI is a special letter that is manually incremented by node-sqlite3 developers when the c++ code in src changes enough to warrant new builds. A letter was chosen to avoid confusion and conflict with the PATCH number. This allows the semver PATCH to be incremented independently of binary compatibility and will allow for easy tagging of new PATCH versions without needing to create new binary builds.
3. Node process.versions.modules number, which is a manually incremented (by node.js core developers) version that indicates ABI breakages in node's C++ API. For older node versions (before v0.10.x) where this is not available the v8 MAJOR.MINOR version that node.js included is used instead
4. platform: e.g. win32, linux, darwin
5. arch: eg. 32 bit (ia32, aka i386) or 64 bit (x64, aka x86_64)

Currently these are the Release builds available:

s3cmd ls s3://node-sqlite3/Release/*
2013-09-08 22:54    383667   s3://node-sqlite3/Release/node_sqlite3-v2.1.a-node-v11-darwin-ia32.tar.gz
2013-09-08 22:54    400503   s3://node-sqlite3/Release/node_sqlite3-v2.1.a-node-v11-darwin-x64.tar.gz
2013-09-08 22:32    435705   s3://node-sqlite3/Release/node_sqlite3-v2.1.a-node-v11-linux-x64.tar.gz
2013-09-08 22:32    379754   s3://node-sqlite3/Release/node_sqlite3-v2.1.a-node-v11-win32-ia32.tar.gz
2013-09-08 22:32    460186   s3://node-sqlite3/Release/node_sqlite3-v2.1.a-node-v11-win32-x64.tar.gz
2013-09-08 22:32    410606   s3://node-sqlite3/Release/node_sqlite3-v2.1.a-v8-3.11-linux-x64.node.gz
2013-09-08 22:32    436081   s3://node-sqlite3/Release/node_sqlite3-v2.1.a-v8-3.11-linux-x64.tar.gz
2013-09-08 22:32    347648   s3://node-sqlite3/Release/node_sqlite3-v2.1.a-v8-3.14-darwin-ia32.node.gz
2013-09-08 22:32    347244   s3://node-sqlite3/Release/node_sqlite3-v2.1.a-v8-3.14-darwin-ia32.node.tar.gz
2013-09-08 22:32    364251   s3://node-sqlite3/Release/node_sqlite3-v2.1.a-v8-3.14-darwin-x64.node.gz
2013-09-08 22:32    363995   s3://node-sqlite3/Release/node_sqlite3-v2.1.a-v8-3.14-darwin-x64.node.tar.gz
2013-09-08 22:32    410518   s3://node-sqlite3/Release/node_sqlite3-v2.1.a-v8-3.14-linux-x64.node.gz
2013-09-08 22:32    435760   s3://node-sqlite3/Release/node_sqlite3-v2.1.a-v8-3.14-linux-x64.tar.gz
2013-09-08 22:32    379038   s3://node-sqlite3/Release/node_sqlite3-v2.1.a-v8-3.14-win32-ia32.node.gz
2013-09-08 22:32    459215   s3://node-sqlite3/Release/node_sqlite3-v2.1.a-v8-3.14-win32-x64.node.gz
2013-09-08 22:32    508678   s3://node-sqlite3/Release/node_sqlite3-v2.1.a-v8-3.6-linux-x64.node.gz

[springmeyer]

Going to merge this now to get more people testing. If anyone has comments, suggestions, or ideas for improvement please comment here.

[commadelimited]

Hate to comment on a closed PR but there's a number of people that are running into trouble trying to install the new blogging platform Ghost:

https://en.ghost.org/forum/installation/35-installation-error-sqlite3-is-not-a-valid-knex-client?page=1

It seems as if some of the troubles that people are encountering (including myself) it might be related to this particular set of changes. Could you please offer input?

[springmeyer]

Hi @commadelimited - this set of changes was driven/encouraged by the ghost team and has helped ease the node-sqlite3 install by providing binaries for many systems. But its certainly not perfect. It seems like the people that are having trouble likely are running old (older than v0.10.x) node versions and therefore are not benefiting from this change (pre-built binaries). So, then a source compile is attempted and they likely lack the dependencies necessary for a successful source compile which can be seen at https://github.com/TooTallNate/node-gyp#installation. To make matters worse currently a failed source compile does not output the actual error due to this bug: #215

[juristr]

I am using Linux 2.6.18-238.19.1.el5 x86_64 x86_64 x86_64 GNU/Linux with Node v0.10.21. Any attempt to install Ghost fails as the binary it searches for http://node-sqlite3.s3.amazonaws.com/Release/node_sqlite3-v2.1.a-node-v11-linux-x64.tar.gz doesn't seem to be present.

A source compile fails as well. I tried to do a yum groupinstall "Development Tools" followed by a npm install sqlite3 --build-from-source with no success. Any ideas?

Update:
strange...the file actually exists, but still, npm install sqlite3 gives

[sqlite3]: Checking for http://node-sqlite3.s3.amazonaws.com/Release/node_sqlite3-v2.1.a-node-v11-linux-x64.tar.gz
[sqlite3]: http://node-sqlite3.s3.amazonaws.com/Release/node_sqlite3-v2.1.a-node-v11-linux-x64.tar.gz not found, falling back to source compile (Error: connect ETIMEDOUT)
Build failed
[sqlite3]: 1
npm ERR! weird error 1
npm ERR! not ok code 0

Update 2:
Seems due to the fact that I'm behind a company proxy and seems like the node http object that is used to perform the download (in build-tools/tools.js) of the latest sqlite packages does not take my proxy settings??

[springmeyer]

@juristr - thanks for the report. So, you are hitting two problems. One, yes I'm missing support for proxies, which I will work to fix asap, see #219. The other is that due to #215 any errors at the node-gyp compile stage (the fallback) were not visible. This is now fixed in master, so can you try again by doing:

git clone https://github.com/mapbox/node-sqlite3.git
cd node-sqlite3
npm install

That should output a more helpful error and likely will indicate some missing dependency for running node-gyp. If you need help interpreting it please comment back here.

[juristr]

Hi,
nice to hear that. Yes, I have seen #215 before and did therefore already a manual clone of the node-sqlite3 repo for getting the exact build error which worked just fine. Thx anyway