Package pre-compiled binaries with node-canvas for downstream consumers

[eonarheim]

I propose that for major OS's (windows, mac, ubuntu, fedora, etc) node-canvas have a pre-compiled set of binaries for consumers to avoid install/configuration/build woes.

I think this was proposed in an older comment in the issue tracker:
#113 (comment)

I've noticed in the issues list that there are quite a number of people having trouble with running/installing/building node-canvas (23 labeled with the installation label https://github.com/Automattic/node-canvas/labels/Installation). That being said, I think the manual install/build for people actively working on and developing node-canvas directly is okay.

However, for people who want to consume node-canvas in a downstream system this is a tremendous hassle. Currently, I have been trying to use js-imagediff which takes a dependency on node-canvas. Now I have to go through a multi-step install process for node-canvas to get js-imagediff to work. I feel like this breaks the compactness and the "it just works" promise of npm modules.

[eonarheim]

Additionally on first glance https://github.com/mapbox/node-pre-gyp seems to provide these features for relatively low cost, since node-canvas already uses node-gyp. Even outline stems for automating Linux/OSX builds with TravisCI and Windows builds with Appveyor.

Also node-pre-gyp has a "fallback-to-build" when the appropriate binary cannot be located.

[LinusU]

This would be very nice to have. It might not be as straight forward since we probably want to package with binaries of libcairo as well, otherwise you still need to have that installed for stuff to work. I'm not entirely sure how that would work to be honest.

[eonarheim]

I think you would package all the appropriate binary dependencies with the pre-compiled distribution. In the windows build all of the necessary binaries land in the ./build/Release directory. This is all you need to run node-canvas compactly. By default windows looks in the current directory for dependent assemblies, linux not so much.

However, this can be done for linux and mac distributions as well since we are well aware of the libraries that we are linking against in order to build the canvas.node native extension. Copy the shared libraries to the /build/Release directory which can be found with the following command, or have the dependent libraries checked into the repository under a "/deps" folder and copied on a build step.

~/node-canvas/build/Release$ readelf -d canvas.node | grep NEEDED
 0x0000000000000001 (NEEDED)             Shared library: [libcairo.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libpng12.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libpangocairo-1.0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libpango-1.0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgobject-2.0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libjpeg.so.8]
 0x0000000000000001 (NEEDED)             Shared library: [libgif.so.4]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6] # probably don't need to package these
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6] #
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6] #

Then the only thing needed would to link the canvas.node binary to look at the origin for shared libraries something like -Wl,-rpath \'-Wl,$$ORIGIN\' so canvas.node could be packaged with it's shared assemblies

/// binding.gyp
...
}, { # 'OS!="win"'
   'ldflags': [
       '-Wl,-rpath \'-Wl,$$ORIGIN\'',
   ],

...

Notice that the linker looks local now first, instead of the shared location which would require an install.

~/node-canvas$ objdump -p build/Release/canvas.node  | grep RPATH
  RPATH                $ORIGIN

~/node-canvas/build/Release$ ldd canvas.node
        linux-vdso.so.1 =>  (0x00007fffe5dc3000)
        libcairo.so.2 => /home/eonarheim/node-canvas/build/Release/./libcairo.so.2 (0x00007f5492e46000)
        libpng12.so.0 => /home/eonarheim/node-canvas/build/Release/./libpng12.so.0 (0x00007f5492c20000)
        libpangocairo-1.0.so.0 => /home/eonarheim/node-canvas/build/Release/./libpangocairo-1.0.so.0 (0x00007f5492a12000)
        libpango-1.0.so.0 => /home/eonarheim/node-canvas/build/Release/./libpango-1.0.so.0 (0x00007f54927c5000)
        libgobject-2.0.so.0 => /home/eonarheim/node-canvas/build/Release/./libgobject-2.0.so.0 (0x00007f5492574000)
        libjpeg.so.8 => /home/eonarheim/node-canvas/build/Release/./libjpeg.so.8 (0x00007f549231e000)
        libgif.so.4 => /home/eonarheim/node-canvas/build/Release/./libgif.so.4 (0x00007f5492115000)

Now with node-pre-gyp you could select the appropriate binary tarbar containing the finished products and dependencies to unpack and drop into /build/Release depending on platform/architecture

///package.json
 "scripts": {
        "install": "node-pre-gyp install --fallback-to-build",
    },
    "binary": {
        "module_name": "your_module",
        "module_path": "./lib/binding/",
        "host": "https://your_module.s3-us-west-1.amazonaws.com"
    }

~/myfork-node-canvas/node-canvas$ make
npm install

> canvas@1.2.9 install /home/eonarheim/myfork-node-canvas/node-canvas
> node-pre-gyp install --fallback-to-build

node-pre-gyp http GET https://github.com/eonarheim/node-canvas/raw/master/dist/c
anvas-v1.2.9-node-v11-linux-x64.tar.gz
node-pre-gyp http 200 https://github.com/eonarheim/node-canvas/raw/master/dist/c
anvas-v1.2.9-node-v11-linux-x64.tar.gz
[canvas] Success: "/home/eonarheim/myfork-node-canvas/node-canvas/build/Release/
canvas.node" is installed via remote

~/myfork-node-canvas/node-canvas$ make test

   canvas: 1.2.9
   cairo: 1.13.1


  ․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․․

  37 passing (207ms)

Complete!

I've done a little work on my fork to demonstrate the basic principle in linux, node-canvas builds and tests pass. Right now the tars are in github (a dubious place to host at best 😉) The full implementation would be integrating with TravisCI and AppVeyor to automatically build and put the binaries for all the major systems in a central location.

https://github.com/eonarheim/node-canvas

Thoughts?

[LinusU]

Wow, very good work! How does this work with different distributions of Linux? Where did you get the so files from?

We would also need to fix this for OS X.

Nice job! Glad someone is picking this up 👍

[inssein]

@eonarheim was just looking at this today for my fork of node-canvas. I would definitely like to see this get solved.

Where would these build files get hosted? We would probably need an S3 bucket. While it isn't expensive, how does an open source project handle that?

[eonarheim]

@LinusU I think a best effort approach is best. Target the most popular web server/dev platforms like: ubuntu server, redhat, fedora, windows, OSX etc. I think this is where we would get the most bang for the buck. Once that is done node-pre-gyp has a fall back to build when the requisite binary cannot be located for the platform.

@inssein I agree completely 😄

I think there are a few options available for open source projects to host binary files. Here are a couple I found after quickly searching (by no means an exhaustive list, I'm sure there are others).

• Amazon S3 - Fast, easy, and already node-pre-gyp compatible. Relatively inexpensive (pennies) but someone will need to pay...
• Github Raw - Relatively easy to do, (see my fork) Easy to wire in with TravisCI/AppVeyor. Github doesn't necessarily like hosting large binaries, but hopefully all of the dists together would be < 100MB considering my fork's packed dist was ~ 1MB https://github.com/eonarheim/node-canvas/blob/master/dist/canvas-v1.2.9-node-v11-linux-x64.tar.gz
  • https://help.github.com/articles/distributing-large-binaries/
  • https://help.github.com/articles/what-is-my-disk-quota/
• FOSSHUB - They are built to host open source binaries http://code.fosshub.com/about.html. However, they look relatively immature in terms of an API surface. We can only use SSH, but it looks possible to automate.

I think ideally the TravisCI/AppVeyor build automation for the various platforms and the chosen binary hosting option would be owned by the project maintainers. Also the associated accounts/passwords/api-keys etc.

Thoughts?

[eonarheim]

@LinusU The needed .so files could be lifted from the build system dynamically using readelf -d canvas.node | grep NEEDED then using those names locate the .so files.

Or the needed .so files could be explicitly identified up front and placed in a "deps" directory and linked to a project local directory like the above. (I used ldd canvas.node on the original distribution to find all of the linked libraries and their shared locations, then copied them locally). There would be some manual maintenance with this option, but less platform fragmentation (you know exactly the versions of the dependencies node-canvas is built and runs best against)

[piranna]

This would be very nice to have. It might not be as straight forward since we probably want to package with binaries of libcairo as well, otherwise you still need to have that installed for stuff to work. I'm not entirely sure how that would work to be honest.

I has a pull-request that add support to generate a statically linked version of node-canvas, so Cairo and all its dependencies get bundled inside the final cairo.node library file and there's no need to have installed any dynamic library. It's pending for aproval since last June without any answer...

[piranna]

Also, by doing a statically linked library, all the problems about OSX would disappear, and would make the distribution using node-pre-gyp fairly trivial.

[potmo]

👍

[chearon]

I've been thinking about this a lot too, having node-canvas as a dependency in your project can end up being hellish. When a new issue is posted here there's a 75% chance that the problem is in the build system.

Some things to add

• bintray is free for open source
• I think node-pre-gyp recommends using GitHub releases for hosting binaries
• One time I calculated how much S3 would cost for node-canvas, and given that Windows needs around 30MB in binaries it wasn't pennies

@LinusU it would be great to add this to the 2.0 release list

[piranna]

I have been using lately prebuild and prebuild-install and they are really easy to use, and use Github releases by default.

[chearon]

@piranna good, I'm going to try that... and (to your comment in January) wouldn't a single canvas.node mean you have to recompile dependencies?

[piranna]

wouldn't a single canvas.node mean you have to recompile dependencies?

If they are compiled statically no, since they are included inside the canvas.node file.

I have tried to update my pull-request to latest code from node-canvas, but due the changes on nan2 I'm having some problems (I've not used nan too much).