Allow extconf.rb to cancel building the extension

[casperisfine]

Ref: https://bugs.ruby-lang.org/issues/20152

There are various gems that ship with a native extension as a way to speedup part of the gem, but also ship with a pure Ruby version of these methods as a fallback. So they only want to compile the extension if the platform supports it, and if not, just fallback Ruby implementation.

Right now users rely on one of two hacks to do this. Either they create an empty Makefile, but then still depend on make being available and it feels very hacky, or they publish platform specific packages without any extension in them.

Examples include bootnsap, erb, hiredis-client, ddtrace, prism.

This changes attempt to make this use case a first class citizen by checking if the extconf.rb did generate a gem.build_skipped file. If it did, make isn't invoked at all.

[segiddins]

Seems reasonable to me, probably needs to be documented somewhere though 👍🏻

[casperisfine]

probably needs to be documented somewhere though

Is there an existing documentation for extconf.rb in Rubygems? I can't find it.

Also my plan is to add some skip_compilation method to mkmf on the Ruby side, that would be documented.

[eregon]

One problem with this is the one I explained in https://bugs.ruby-lang.org/issues/20152#note-11:

There could be a skip-make/skip-compilation or so file which does not include RUBY_ENGINE in the filename and is created dynamically by the extconf.rb, but this will take a lot longer to get adopted, as it means each gem skipping compilation would need to use that.
But it is error-prone, because e.g. then if one builds (e.g. rake compile) a gem locally on JRuby and then on CRuby then compilation will be unexpectedly skipped on CRuby, unless the code also takes care of removing that file (I guess mkmf create_makefile could remove it, it seems too annoying if extconf.rb or Rakefile needs to remove it manually).

i.e. specifically it is very error-prone to use this feature until the oldest Ruby a gem supports would have this skip_compilation in mkmf, which will be in a while.
In the meantime, doing things like File.write("gem.build_skipped", "") would cause this problem.

[deivid-rodriguez]

In principle the idea of not trying to make if there's nothing to make sounds more appealing to me? Is there any downside to that? Could it potentially hinder bugs in extconf.rb files?

[casperisfine]

Could it potentially hinder bugs in extconf.rb files?

Not that I can think of. I mean if you generate an empty make file by accident today, running a noop make or not running anything shouldn't make a difference.

I went with a different file because it seemed more deliberate, but I'm fine with trying to detect empty Makefile too.

[headius]

This came up again in gsamokovarov/skiptrace#8 and I think everyone would be ok with simply detecting an empty Makefile and not running make at all. That would eliminate the need for make for JRuby extensions that don't require a build.

There's still a remaining issue in #3520: even fake builds still emit files to indicate that something was built for the current JDK + JRuby platform, and switching JDK complains about missing build files even though nothing is built. (@deivid-rodriguez Can we get that issue re-opened too?)

[deivid-rodriguez]

Sure, I understand better that issue now so happily to reevaluate it. Not sure when I'll get to it though, but happy to keep it open as a remainder to give it another look after we fix this first issue.

[simi]

I understand the need and like the idea, but I'm not sure it is good idea to rely on file presence. 🤔

[deivid-rodriguez]

@simi Do you mean that you prefer to detect if the Gemfile is empty in order to decide to not run make, rather than checking a sentinel file like in this PR? Or you don't like either idea?

[simi]

@deivid-rodriguez I have no clear idea for now. Checking for empty Makefile seems also just bandaid.

[deivid-rodriguez]

It's definitely not first class citizen solution, but seems like an easy thing to implement on our side with no downside? And wouldn't be in conflict with potential future approaches.

[eregon]

At the risk of repeating myself, I think detecting the empty Makefile generated from dummy_makefile is by far the best solution:

• Gems already use dummy_makefile it's effectively the current API to skip compilation based on some condition
• It would be a transparent optimization in latest RubyGems, on older RubyGems it would run make (we can't change anything about existing releases of RubyGems anyway).

Adding a new API could only be used in gems once current RubyGems version is EOL or so, or would need to check a condition + change many gems which I don't think is worth the time or effort, File.write("Makefile", dummy_makefile($srcdir).join("")) is already explicit enough.

[byroot]

I agree with @eregon

[simi]

OK, 👍 on empty Makefile.

[deivid-rodriguez]

I was thinking more about this, and I realized that even if we fix the "make not available" problem by detecting if the Makefile is empty, we still don't have a good way to fix #3520. Every time platform changes (like when upgrading Java, or macOS), or when the Ruby ABI changes and you have a shared GEM_HOME, RubyGems will ignore these gems because their extensions are missing and will print warnings.

As an alternative, I was thinking of introducing a new extension builder, OptionalExtConfBuilder, that works like ExtConfBuilder, but does not try to run make at all when a Makefile was not generated. So no more dummy_makefile, just do nothing when the extension does not apply. Also, this kind of gems with extensions would not get ignored nor print warnings if their extensions are missing.

This builder would get used when the extension is named optional_extconf.rb, and in order to provide backwards compatibility, RubyGems could also run this builder when it finds extconf.rb in the extension list, but there's also a optional_extconf.rb file alongside. So gems that want to support both new and old RubyGems would keep Specification#extensions list unchanged, so things work normally in older RubyGems versions, but add an optional_extconf.rb file that can be used by newer RubyGems.

[headius]

even if we fix the "make not available" problem by detecting if the Makefile is empty, we still don't have a good way to fix #3520.

A quick idea: a top-level metadata that indicates an installed gem does not need missing ext check because the ext is optional and not being used? The check would first see if it's a gem with an omitted extension, and not go further. This could be localized to the RUBY_ENGINE since CRuby would rarely omit extension and JRuby frequently will.

This builder would get used when the extension is named optional_extconf.rb, and in order to provide backwards compatibility, RubyGems could also run this builder when it finds extconf.rb in the extension list, but there's also a optional_extconf.rb file alongside.

It seems to me this would work but it's also a little cumbersome. Can you explain why OptionalExtConfBuilder wouldn't just become the default now?

[deivid-rodriguez]

A quick idea: a top-level metadata that indicates an installed gem does not need missing ext check because the ext is optional and not being used? The check would first see if it's a gem with an omitted extension, and not go further. This could be localized to the RUBY_ENGINE since CRuby would rarely omit extension and JRuby frequently will.

What would "top level metadata" mean in this case? A gemspec attribute? Rolling out gemspec API changes seems quite complicated. I don't think fixing this is worth going through all the trouble if we can avoid it.

It seems to me this would work but it's also a little cumbersome. Can you explain why OptionalExtConfBuilder wouldn't just become the default now?

We could make it the default, but the warnings that extensions are not built in my opinion are useful. A different "optional builder" would allow skipping those warnings when not needed but keep them when the standard more strict extconf builder is used.

[deivid-rodriguez]

Also note it's not only about the warnings being annoying. Gems with missing extensions are currently ignored when activating gems, so making the optional builder I'm proposing the default would mean sometimes activating broken gems, for those gems where the extension is actually compulsory.

[deivid-rodriguez]

A quick idea: a top-level metadata that indicates an installed gem does not need missing ext check because the ext is optional and not being used? The check would first see if it's a gem with an omitted extension, and not go further. This could be localized to the RUBY_ENGINE since CRuby would rarely omit extension and JRuby frequently will.

Oh, you probably meant gemspec metadata, which can technically include anything, so it's a backwards compatible way of adding stuff to the gemspec. So, something like metadata = { optional_ext: [jruby] }, maybe? That feels a bit cumbersome to me too, and not as flexible as having this logic in this extconf.rb file.

[headius]

@deivid-rodriguez sorry I don't know the right terminology here. I mean rather than always going to the platform and architecture-specific directory to look for confirmation that the extension was built, introduce a higher level location that can provide metadata for the installed gem not specific to a given platform or architecture. So first it would check for this gem on this Ruby engine, is it expected to have an extension built, and only after that is confirmed does it proceed to the current check.

It would be a way to indicate that a given gem on the current Ruby engine does not require an extension to be built.

[deivid-rodriguez]

Oh, I see! So you're essentially proposing something similar to what this PR is proposing. I thought of that but things did not square, but after another thought maybe that works?

We could:

• Detect empty Makefile like proposed for this problem, and avoid running make if that's the case as mostly agreed for this issue.

• For Extension gems built for JRuby warn when switching Java version #3520, let extconf.rb leave a gem.build_optional file in a location not specific to either ABI or platform (say, in extensions/erb-1.0.0, rather than extensions/arm64-darwin-23/3.3.0/erb-1.0.0), indicating that the extension is not required for the gem to function. RubyGems will still try to build the extension normally when installing if it does not find a gem.build_complete file in the platform/ABI specific location, but will not print warnings or ignore the gem at gem activation time regardless of whether the gem is missing extensions.

[headius]

Detect empty Makefile like proposed for this problem

Yes something like you proposed but I do not know if that is the best solution or not! I will go with whatever the majority thinks on that regard, so long as I can eliminate make from our JRuby requirements some day.

let extconf.rb leave a gem.build_optional file in a location not specific to either ABI or platform (say, in extensions/erb-1.0.0, rather than extensions/arm64-darwin-23/3.3.0/erb-1.0.0), indicating that the extension is not required for the gem to function

Yeah that sounds right! I would love for there be a way to say something like spec.extension_required["jruby"] = false to cut it off at that point (and avoid all building and checking), but we can accomplish the same with a marker to indicate that "missing" extension is expected behavior.

[eregon]

For #3520, building on what @deivid-rodriguez said we could do this automatically without requiring any changes in gems by, after RubyGems calls extconf.rb, it detects if empty/dummy Makefile, if it is:

• it skips running make
• it creates a gem.build_complete file in a platform-independent directory (e.g. extensions/erb-1.0.0 or maybe even at the gem's root)

That way the check for "are extensions builds" just needs to be extended to check that file on top of checking the platform-specific gem.build_complete file.

If we want to apply this new logic for already-installed gems when updating RubyGems that's also possible by having the "are extensions builds" check, if it doesn't find any gem.build_complete, look for the empty/dummy Makefile and if so create the platform-independent gem.build_complete file.

IOW, we already have a marker for "no extension library is expected", it's the empty/dummy Makefile.

[deivid-rodriguez]

If I understood correctly, I had that same idea but I don't think that works? If we put a gem.build_complete file in a platform independent directory for gems with optional extensions, that would mean it would not be rebuilt on any platform, even on those who can benefit from the extension.

I think these are just two different things which need different markers:

• Whether the extension needs to be rebuilt or not at install time. For that, current marker works and we just need the optimization to not run make when not necessary.
• Whether it matters or not that the extension is not built at gem activation time. For that, I propose to introduce a new marker that would apply to all platforms, and thus should live in a platform independent location.

[eregon]

If I understood correctly, I had that same idea but I don't think that works? If we put a gem.build_complete file in a platform independent directory for gems with optional extensions, that would mean it would not be rebuilt on any platform, even on those who can benefit from the extension.

For a gem with an extension that gets built on CRuby, the extconf.rb would generate a non-dummy Makefile, so there would be no platform-independent gem.build_complete, and same behavior as before.

If you are thinking about sharing a gem home between CRuby and some other Ruby implementation, that has never been supported or worked AFAIK (except maybe for pure-Ruby gems with no -java variants, which are not concerned here).

It's the same case for both, everything can be decided as soon as the extconf.rb has finished running, and that records "no extension to build" if it's a dummy Makefile, and "there is some extension, same behavior as before" otherwise.

[deivid-rodriguez]

If you are thinking about sharing a gem home between CRuby and some other Ruby implementation, that has never been supported or worked AFAIK (except maybe for pure-Ruby gems with no -java variants, which are not concerned here).

Yes, that's what I'm thinking. That may not have worked well sometimes, but RubyGems does try to act reasonably when that happens. In fact, I think all these "extensions are not built" warnings and logic to rebuild extensions when necessary was added many years ago to support that kind of situation, and many people do it in real life anyways. Recently we have shipped some fixes related to that and as far as I know it currently works just fine. Also for situations where people set a custom GEM_HOME (not scoped to ABI or platforms), we want to handle Ruby upgrades gracefully.

[eregon]

Also for situations where people set a custom GEM_HOME (not scoped to ABI or platforms), we want to handle Ruby upgrades gracefully.

How does that work?
Extension libraries still seem copied under some-gem/lib/ which prevents any meaningful sharing to work since there can be only one file there, no?
For example:

$ ruby -v                                   
ruby 3.4.0dev (2024-10-13T13:00:20Z master cf8388f76c) +PRISM [x86_64-linux]
$ gem --version
3.6.0.dev
$ gem i json
$ ruby -e 'require "json"; puts $:, nil, $"'
/home/eregon/.rubies/ruby-dev/lib/ruby/gems/3.4.0+0/gems/json-2.7.2/lib
/home/eregon/.rubies/ruby-dev/lib/ruby/gems/3.4.0+0/extensions/x86_64-linux/3.4.0+0-static/json-2.7.2
/home/eregon/.rubies/ruby-dev/lib/ruby/site_ruby/3.4.0+0
/home/eregon/.rubies/ruby-dev/lib/ruby/site_ruby/3.4.0+0/x86_64-linux
/home/eregon/.rubies/ruby-dev/lib/ruby/site_ruby
/home/eregon/.rubies/ruby-dev/lib/ruby/vendor_ruby/3.4.0+0
/home/eregon/.rubies/ruby-dev/lib/ruby/vendor_ruby/3.4.0+0/x86_64-linux
/home/eregon/.rubies/ruby-dev/lib/ruby/vendor_ruby
/home/eregon/.rubies/ruby-dev/lib/ruby/3.4.0+0
/home/eregon/.rubies/ruby-dev/lib/ruby/3.4.0+0/x86_64-linux

...
/home/eregon/.rubies/ruby-dev/lib/ruby/gems/3.4.0+0/gems/json-2.7.2/lib/json/version.rb
/home/eregon/.rubies/ruby-dev/lib/ruby/gems/3.4.0+0/gems/json-2.7.2/lib/json/common.rb
/home/eregon/.rubies/ruby-dev/lib/ruby/gems/3.4.0+0/gems/json-2.7.2/lib/json/ext/parser.so
/home/eregon/.rubies/ruby-dev/lib/ruby/gems/3.4.0+0/gems/json-2.7.2/lib/json/ext/generator.so
/home/eregon/.rubies/ruby-dev/lib/ruby/gems/3.4.0+0/gems/json-2.7.2/lib/json/ext.rb
/home/eregon/.rubies/ruby-dev/lib/ruby/gems/3.4.0+0/gems/json-2.7.2/lib/json.rb

(that's using the RubyGems shipped with ruby head)
We see gems/json-2.7.2/lib is before extensions/x86_64-linux/3.4.0+0-static/json-2.7.2, and that gems/3.4.0+0/gems/json-2.7.2/lib/json/ext/parser.so is the loaded one.

And indeed it's unsafe to share it:

$ chruby 3.1.3
$ GEM_HOME=/home/eregon/.rubies/ruby-dev/lib/ruby/gems/3.4.0+0 ruby -e 'require "json"'
/home/eregon/.rubies/ruby-dev/lib/ruby/gems/3.4.0+0/gems/json-2.7.2/lib/json/common.rb:86: [BUG] Segmentation fault at 0x0000000000000008
ruby 3.1.3p185 (2022-11-24 revision 1a6b16756e) [x86_64-linux]
...

These days TruffleRuby and CRuby-head check the ABI version of these .so files before fully loading them (but CRuby releases do not!), otherwise it would just segfault to use that parser.so on a different CRuby version (as shown above, unless they share the same ABI, of course). Getting the ABI version from the .so to compare does mean loading the library though as the ABI version is stored as a symbol in the .so file, and that may have non-trivial side effects (e.g. some global variables might be declared and picked up by another compiled version of that extension). IOW, rescue-ing the LoadError from these mismatched ABI version is likely dangerous.

And even then, if somehow there was no more .so under gems' lib, sharing between different CRuby versions is one thing (which doesn't work well as that's basically guaranteed to have non-built extensions, which will be warned/error but still sounds very impractical to manually rebuild on every warning/error), but sharing a gem home between CRuby and JRuby or TruffleRuby I think that's even more problematic.

That said, if you do want to try to support that, the platform-independent gem.build_complete could be under a path containing both the RUBY_ENGINE and ABI version (RbConfig::CONFIG['ruby_version']). (the same thing Bundler uses to avoid mixing gems of different Rubies when using the Bundler path config).

Whether it matters or not that the extension is not built at gem activation time. For that, I propose to introduce a new marker that would apply to all platforms, and thus should live in a platform independent location.

The problem is that's dependent on the RUBY_ENGINE for example, on CRuby it's typically a problem if the extension is missing, as it would fail with LoadError (or for a few gems use a much slower pure-Ruby backend for some gems, still undesirable). OTOH e.g. for the fiddle gem on JRuby & TruffleRuby there is no need for the C extension and a pure-Ruby backend is used.

[deivid-rodriguez]

Sorry, I didn't mean sharing GEM_HOME like that. That of course fails and nobody does that in real life.

I mean using something like GEM_HOME=foo, installing your gems there, and then upgrading Ruby. In that case, RubyGems checks if the extension has been built for the specific ABI being run, and refuse to activate the gem if it has not been explicitly built for it.

For example, run GEM_HOME=foo gem install json with Ruby 3.3, now switch to Ruby 3.2. Then ruby -e 'require "json" will not try to activate the version of json compiled for Ruby 3.3 and will instead use the one that comes with Ruby 3.2 by default.

[deivid-rodriguez]

The problem is that's dependent on the RUBY_ENGINE for example, on CRuby it's typically a problem if the extension is missing, as it would fail with LoadError (or for a few gems use a much slower pure-Ruby backend for some gems, still undesirable). OTOH e.g. for the fiddle gem on JRuby & TruffleRuby there is no need for the C extension and a pure-Ruby backend is used.

Exactly, that's why it would be an "activation time" thing that would allow no warnings on JRuby and a graceful fallback on CRuby, but extensions would still get reinstalled when it matters at "install time".

[deivid-rodriguez]

Actually I just tried your exact steps (except with Ruby 3.3 and Ruby 3.4-dev), and I got the default json version for Ruby 3.3 activated and no crash. Not sure what's going in your case, maybe the RubyGems version in Ruby 3.1 did not handle this better 🤷‍♂️.

[eregon]

Actually I just tried your exact steps (except with Ruby 3.3 and Ruby 3.4-dev), and I got the default json version for Ruby 3.3 activated and no crash. Not sure what's going in your case, maybe the RubyGems version in Ruby 3.1 did not handle this better 🤷‍♂️.

The ABIs of 3.3 and 3.4-dev are closer, so there is more chances it doesn't fail immediately. If you try with Ruby 3.1 you should be able to reproduce. With 2.7 it even gives an undefined symbol.

[deivid-rodriguez]

The ABIs of 3.3 and 3.4-dev are closer, so there is more chances it doesn't fail immediately. If you try with Ruby 3.1 you should be able to reproduce. With 2.7 it even gives an undefined symbol.

We never try to load an extension compiled for a different ABI, no matter how close they are. I can't reproduce either with Ruby 3.1 either, the default json version compiled for 3.1 is loaded.

[deivid-rodriguez]

As I said before, your RubyGems version in Ruby 3.1 is probably old and does not have the latest improvements. This behavior of ignoring gems with extensions compiled for a different ABI has been there for a long time, but only recently we fixed an issue where it was not applying to default gems like json.

[eregon]

For example, run GEM_HOME=foo gem install json with Ruby 3.3, now switch to Ruby 3.2. Then ruby -e 'require "json" will not try to activate the version of json compiled for Ruby 3.3 and will instead use the one that comes with Ruby 3.2 by default.

I tried just that with 3.3 and 3.2 and bigdecimal:

$ export GEM_HOME=$PWD/foo
$ chruby 3.3.5
$ gem i bigdecimal     
$ ruby -e 'require "bigdecimal"; puts $:, nil, $"'
/home/eregon/tmp/foo/gems/bigdecimal-3.1.8/lib
/home/eregon/tmp/foo/extensions/x86_64-linux/3.3.0/bigdecimal-3.1.8
/home/eregon/.rubies/ruby-3.3.5/lib/ruby/site_ruby/3.3.0
/home/eregon/.rubies/ruby-3.3.5/lib/ruby/site_ruby/3.3.0/x86_64-linux
/home/eregon/.rubies/ruby-3.3.5/lib/ruby/site_ruby
/home/eregon/.rubies/ruby-3.3.5/lib/ruby/vendor_ruby/3.3.0
/home/eregon/.rubies/ruby-3.3.5/lib/ruby/vendor_ruby/3.3.0/x86_64-linux
/home/eregon/.rubies/ruby-3.3.5/lib/ruby/vendor_ruby
/home/eregon/.rubies/ruby-3.3.5/lib/ruby/3.3.0
/home/eregon/.rubies/ruby-3.3.5/lib/ruby/3.3.0/x86_64-linux

...
/home/eregon/tmp/foo/gems/bigdecimal-3.1.8/lib/bigdecimal.so
/home/eregon/tmp/foo/gems/bigdecimal-3.1.8/lib/bigdecimal.rb

$ chruby 3.2.2
$ ruby -e 'require "bigdecimal"; puts $:, nil, $"'
<internal:/home/eregon/.rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require': linked to incompatible � - /home/eregon/tmp/foo/gems/bigdecimal-3.1.8/lib/bigdecimal.so (LoadError)
	from <internal:/home/eregon/.rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from /home/eregon/tmp/foo/gems/bigdecimal-3.1.8/lib/bigdecimal.rb:4:in `<top (required)>'
	from <internal:/home/eregon/.rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from <internal:/home/eregon/.rubies/ruby-3.2.2/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from -e:1:in `<main>'

$ chruby 3.2.4
$  ruby -e 'require "bigdecimal"; puts $:, nil, $"'
<internal:/home/eregon/.rubies/ruby-3.2.4/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require': linked to incompatible l�r - /home/eregon/tmp/foo/gems/bigdecimal-3.1.8/lib/bigdecimal.so (LoadError)
	from <internal:/home/eregon/.rubies/ruby-3.2.4/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /home/eregon/tmp/foo/gems/bigdecimal-3.1.8/lib/bigdecimal.rb:4:in `<top (required)>'
	from <internal:/home/eregon/.rubies/ruby-3.2.4/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from <internal:/home/eregon/.rubies/ruby-3.2.4/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from -e:1:in `<main>'
$ ruby -e 'require "bigdecimal"; puts $:, nil, $"'
<internal:/home/eregon/.rubies/ruby-3.2.4/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require': linked to incompatible {� - /home/eregon/tmp/foo/gems/bigdecimal-3.1.8/lib/bigdecimal.so (LoadError)
	from <internal:/home/eregon/.rubies/ruby-3.2.4/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from /home/eregon/tmp/foo/gems/bigdecimal-3.1.8/lib/bigdecimal.rb:4:in `<top (required)>'
	from <internal:/home/eregon/.rubies/ruby-3.2.4/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from <internal:/home/eregon/.rubies/ruby-3.2.4/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
	from -e:1:in `<main>'

Which looks like a dynamic loader/linker error with some undefined memory being read, not a RubyGems exception.

---

But indeed with oj, installing on 3.3 and testing with master I do get the error from RubyGems:

$ chruby 3.3.5   
$ gem i oj
$ chruby ruby-dev
$ ruby -e 'require "oj"; puts $:, nil, $"'
Ignoring bigdecimal-3.1.8 because its extensions are not built. Try: gem pristine bigdecimal --version 3.1.8
Ignoring json-2.7.2 because its extensions are not built. Try: gem pristine json --version 2.7.2
Ignoring oj-3.16.6 because its extensions are not built. Try: gem pristine oj --version 3.16.6
<internal:/home/eregon/.rubies/ruby-dev/lib/ruby/3.4.0+0/rubygems/core_ext/kernel_require.rb>:136:in 'Kernel#require': cannot load such file -- oj (LoadError)
	from <internal:/home/eregon/.rubies/ruby-dev/lib/ruby/3.4.0+0/rubygems/core_ext/kernel_require.rb>:136:in 'Kernel#require'
	from -e:1:in '<main>'

It's a good safety measure RubyGems refuses to activate the gem in such a case.
It doesn't seem very practical though to gem pristine those, it seems much safer to remove the gem home and reinstall into it, otherwise there will likely be some leftovers from the previous version.

[eregon]

Exactly, that's why it would be an "activation time" thing that would allow no warnings on JRuby and a graceful fallback on CRuby, but extensions would still get reinstalled when it matters at "install time".

I think it shouldn't allow a graceful fallback on CRuby, it should fail as it currently does on CRuby by refusing to activate the gem (assuming a gem where the extension is built on CRuby, and noop on JRuby via dummy Makefile).
IOW, the extension is not optional on CRuby, it's required (for most extension gems, there is simply no fallback if the extension is not built; if there is a pure-Ruby fallback there is typically another gem without extensions at all like json-pure). And it's always a noop (dummy Makefile) on JRuby/TruffleRuby for a number of extension gems.
EDIT: That's why I'm not a big fan of the notion of "optional extension" because it's too vague/permissive when it's actually required non-optional on CRuby, not built on other Rubies.

So indeed, if sharing a GEM_HOME should be supported between different Ruby implementations I think we should do this:

That said, if you do want to try to support that, the platform-independent gem.build_complete could be under a path containing both the RUBY_ENGINE and ABI version (RbConfig::CONFIG['ruby_version']). (the same thing Bundler uses to avoid mixing gems of different Rubies when using the Bundler path config).

Agreed on activation time, it's already how this works, just the check would be extended to find the platform-independent but RUBY_ENGINE-dependent and ABI-dependent gem.build_complete (a gem extconf.rb could decide to not build an extension if have_func 'foo' is false, so that's why ABI-dependent).

I get your point now that we cannot look for a dummy Makefile at activation time, because the Makefile might not be the one corresponding to the current Ruby, it might be the one of another Ruby on which gem pristine was run or so, in case of a shared GEM_HOME.

[eregon]

FWIW on TruffleRuby we currently have this warning:

[ruby] WARNING gem paths: /home/eregon/tmp/foo are not marked as installed by TruffleRuby. They might belong to another Ruby implementation and break unexpectedly. Configure your Ruby manager to use TruffleRuby, or unset GEM_HOME GEM_PATH. See https://github.com/oracle/truffleruby/blob/master/doc/user/ruby-managers.md

So it's already very much discouraged and unsupported to reuse a GEM_HOME on TruffleRuby.
So maybe it's fair enough to take the assumption a GEM_HOME shouldn't be shared between different Ruby implementations.

[deivid-rodriguez]

I tried just that with 3.3 and 3.2 and bigdecimal

And it should work fine too with the latest RubyGems.

It doesn't seem very practical though to gem pristine those, it seems much safer to remove the gem home and reinstall into it, otherwise there will likely be some leftovers from the previous version.

It seems fine to me. It may leave some leftovers for other platform/ABIs, but it seems more practical than reinstalling all your gems. In any case, any change in this message or improvements to clean up artifacts for other platforms/ABIs seem unrelated here.

I think it shouldn't allow a graceful fallback on CRuby, it should fail as it currently does on CRuby by refusing to activate the gem (assuming a gem where the extension is built on CRuby, and noop on JRuby via dummy Makefile).
IOW, the extension is not optional on CRuby, it's required (for most extension gems, there is simply no fallback if the extension is not built; if there is a pure-Ruby fallback there is typically another gem without extensions at all like json-pure). And it's always a noop (dummy Makefile) on JRuby/TruffleRuby for a number of extension gems.
EDIT: That's why I'm not a big fan of the notion of "optional extension" because it's too vague/permissive when it's actually required non-optional on CRuby, not built on other Rubies.

Alright, so then we're back to almost what this PR proposes, except that:

• We put the gem.build_skipped file in a engine+ABI folder instead of in the current platform+ABI folder.
• We automatically write the gem.build_skipped file if we detect an empty Makefile, so that no changes in extconf.rb files are needed for now.
• We avoid building extensions at all at install time when we find a gem.build_skipped file in the proper location, and we never ignore gems with "build-skipped extensions" at activation time when we find a gem.build_skipped file in the proper location.

[eregon]

Yes, that sounds good to me (with we never ignore gems -> we ignore gems).

There is one more complication that a gem may have multiple extensions and some extension might be skipped and another extension of that gem might be not skipped.
This is the case for json for example: skipped extension and non-skipped extension.
That case can be safely treated as "does not skip all extensions, so has some non-skipped extension, so same behavior as before", it just needs the logic to handle that multiple extensions case.

[deivid-rodriguez]

Yes, that sounds good to me (with we never ignore gems -> we ignore gems).

I think we never ignore gems is correct. In JRuby or other implementations where extensions are not really built, then gems with extensions are always "require-ready" and we never need to filter out ("ignore") gems with missing extensions since there's no such thing.

There is one more complication that a gem may have multiple extensions and some extension might be skipped and another extension of that gem might be not skipped.

Good point, we'll need to handle that case carefully.

[eregon]

I think we never ignore gems is correct.

Ah yes, makes sense, agreed, I misunderstood what was meant there. In fact I was thinking about naming and whether it should be gem.build_skipped or gem.build_complete (or something else) for dummy Makefile cases. It seems good terminology to consider such cases "build/gem complete, the gem/extensions is considered built, the gem is ready to be activated/used". We didn't really "skip" anything when installing the gem, the gem just told us it doesn't need this extension (dynamically). But it's mostly details.