Thoughts on pre-packaged binaries (aka: Native Gems) for all platforms?

[luislavena]

Hello folks,

Wanted to reach out and check with you your thoughts about pre-packaged binaries (aka: Native Gems), similar to Nokogiri latest fat-binaries process, in order to address several issues around developer and deployment experienced around sqlite3.

The top reasons I can think of:

1. Some system-provided versions are too old (AFAIK 3.8 on macOS while 3.31 on Ubuntu 20.04)
2. Not able to have same set of features/fixes across systems.
3. XYZ doesn't work for some and works on my machine for others, increasing the debugging time.
4. Different available extensions configurations across systems.

Of course, moving to this increases certain aspects of the project maintenance, not to mention the potential of having to deal with CVE (not many) but still present.

Sending this out for comments. Perhaps @flavorjones or @larskanis can comment on the developer and maintainer experience pros/cons of this approach for Nokogiri.

Thank you.
❤️ ❤️ ❤️

[flavorjones]

I'd be happy to help out if y'all decide this is something you want to try. I think precompiled Nokogiri has been a huge success, both for users but also for me. Support issues are down, negative tweets are down, rates of installation and upgrade are trending up. And the gem release process is pretty repeatable -- my rate of new release has increased, too -- 13 releases so far in 2021 is on track to triple any previous year's output.

I recently helped the the team maintaining https://github.com/kwilczynski/ruby-magic to ship a precompiled extension, and it was helpful to learn which bits from Nokogiri's config were reusable. Trying this with a third gem might be a good excuse to figure out how to extract the common bits into a developer package of some sort that ties together rake-compiler, rake-compiler-dock, mkmf, and some guidelines and out-of-the-box tests for packaged gems.

I'm also giving a talk on precompiled extensions at RubyKaigi next month and RubyConf in November which is forcing me to articulate exactly how this works and how to do it yourself. Maybe November's talk could include notes from this experience.

[larskanis]

The past issues with bundler and it's broken cross platform support are solved. This was a precondition for Nokogiri's binary gems, but now bundler works pretty well in this regard.

I think it's crucial to properly test the compiled binary gems on various operation systems. Gitlab Actions are well suited to automate this, but it takes a good piece of craftsmanship to get everything working. Nokogiri is well suited as an example to copy from.

Also I'm not sure how much this affects SQLite, but I hesitated to publish binary gems of ruby-pg till now. It's library "libpq" has a lot of possible build time dependencies like authentication modules, Kerberos support, locale files, a hardcoded path to the default unix socket file and other more or less deep integrations into the operation system. Regarding the Windows binary gems the typical dependencies are compiled into, but on Linux there are more of them.

[larskanis]

The ffi gem has no external dependencies and would benefit from fat binary gems. To implement this is still on my todo list, but I didn't have the time to do it. Similarly these binaries require proper CI tests on all important operation systems and variations.

[luislavena]

Thank you both @flavorjones and @larskanis for the information and all these details! 😻

SQLite itself has no external dependency 🙏🏽, but comes bundled with several extensions that, depending on the distribution or packager, may or may not be enabled (Eg. FTS3-5, JSON, etc). Not to mention the runtime extension loading capabilities 🤯

flavorjones: I'm also giving a talk on precompiled extensions at RubyKaigi next month and RubyConf in November which is forcing me to articulate exactly how this works and how to do it yourself. Maybe November's talk could include notes from this experience.

Saw that and was eager to hear more about it! Looking forward we can experiment on this!

larskanis: Similarly these binaries require proper CI tests on all important operation systems and variations.

Indeed that is a critical point and will need to be corrected before any other work is started.

Once again thank you both for all this information, before we commit any work on this, would love to hear from @tenderlove, not for him to maintain such functionality, but to know we can get his or not.

❤️ ❤️ ❤️

[flavorjones]

Just wanted to drop a note here to say that I've been thinking about this.

For my upcoming RubyConf talk I want to try to extract as much of the precompilation configuration (outlined at https://github.com/flavorjones/ruby-c-extensions-explained as part of the RubyKaigi talk) and see how "easy" I make precompiled versions of sqlite3 and maybe one or two other gems.

But I only have about a month! We'll see how far I get.

[tenderlove]

Sounds great!

[flavorjones]

Taking this on, I've started a branch to do all the work at #320 and will be following most of the patterns established in https://github.com/flavorjones/ruby-c-extensions-explained/tree/main/precompiled

[flavorjones]

v1.5.0.rc1 is out, please let me know if it works for you!

https://github.com/sparklemotion/sqlite3-ruby/releases/tag/v1.5.0.rc1

[flavorjones]

v1.5.0 final is out.

[luislavena]

Amazing work @flavorjones 👏🏽 👏🏽 👏🏽

Thank you!
❤️ ❤️ ❤️