Skip to content

Conversation

flavorjones
Copy link
Collaborator

@flavorjones flavorjones commented Dec 27, 2024

#134 requested bumping Ruby 3.0.

Let's see what happens if we bump them all.

Bump rubies:

  • 3.4.1 → 3.4.1 1
  • 3.3.5 → 3.3.5 1
  • 3.2.0 → 3.2.6
  • 3.1.0 → 3.1.6
  • 3.0.0 → 3.0.7
  • 2.7.0 → 2.7.8
  • 2.6.0 → 2.6.10
  • 2.5.0 → 2.5.9
  • 2.4.0 → 2.4.10

Other changes:

  • set OSXCROSS_PKG_CONFIG_USE_NATIVE_VARIABLES=1 because we started running into build problems with the 3.0.x series
  • on darwin, when building 3.1.x, use a patched rake-compiler to explicitly build miniruby to address https://bugs.ruby-lang.org/issues/19239 (this fix appeared in 3.2.0 but was not backported)

cc @mudge

Footnotes

  1. unchanged because it's already the latest 2

@flavorjones flavorjones force-pushed the flavorjones-bump-rubies branch 6 times, most recently from 5eff8e1 to 7548777 Compare December 28, 2024 20:50
@flavorjones
Copy link
Collaborator Author

flavorjones commented Dec 29, 2024

@mudge Just to keep you in the loop here. I've got all of the ruby patch-level updates working except for 2.7 and 3.0 (the one you originally asked for!) which are failing on Darwin.

Good news: the issue seems to be the same in both:

dyld[4641]: missing symbol called

Bad news: I don't have any mac hardware that I can use to reproduce and diagnose, so I'm falling back to trying to debug this through repeated Github Action runs, which is going to take a while.

If you've got time to reproduce this on a local mac, then maybe we can pair on isolating what the missing symbol is. I previously fixed an issue with our osxcross build that was similar but not the same: #60

If not, well, with a little patience and time I'll figure it out.

@mudge
Copy link

mudge commented Dec 29, 2024

I have access to both an Intel and Apple silicon Mac so I’m happy to try to reproduce this locally: what would you like me to try running? We can try to coordinate pairing too: I might have time this evening (after 7:30pm UK time).

@flavorjones flavorjones force-pushed the flavorjones-bump-rubies branch from 9788bcb to ed8735d Compare December 29, 2024 16:51
@flavorjones flavorjones force-pushed the flavorjones-bump-rubies branch 2 times, most recently from ec6cca0 to eae6a5d Compare December 29, 2024 17:03
@flavorjones
Copy link
Collaborator Author

flavorjones commented Dec 29, 2024

@mudge Thanks. First see if you can reproduce it:

# checkout this PR's branch
gh pr checkout 135 # should be eae6a5d0

# build docker image: ghcr.io/rake-compiler/rake-compiler-dock-image:1.7.0-mri-x86_64-darwin
bundle install
bundle exec rake build:x86_64-darwin

# build the test gem: pkg/rcd_test-1.0.0-x86_64-darwin.gem
cd test/rcd_test
bundle install
bundle exec rake clean clobber
bundle exec rake gem:x86_64-darwin

# running ruby 3.0, install the test gem and run it:
ruby -v # please use 3.0.7 if you can
cd test/rcd_test
gem install --local pkg/rcd_test-1.0.0-x86_64-darwin.gem --verbose
ruby -rrcd_test -S rake test

Ideally you should see:

$ ruby -rrcd_test -S rake test
Run options: --seed 64061

# Running:

....

Finished in 0.002798s, 1429.7034 runs/s, 2859.4069 assertions/s.

4 runs, 8 assertions, 0 failures, 0 errors, 0 skips

But if you're reproducing the issue, you'll see something like:

dyld[4641]: missing symbol called

@mudge
Copy link

mudge commented Dec 29, 2024

Prior to seeing your latest reply, I just downloaded the built gem from your failing run, installed it, and when I tried to require rcd_test, I got the following error:

<internal:/Users/mudge/.rubies/ruby-3.4.0-rc1/lib/ruby/3.4.0+1/rubygems/core_ext/kernel_require.rb>:136:in 'Kernel#require': dlsym(0x399e2cfb0, ruby_abi_version): symbol not found - ruby_abi_version

Let me follow your steps above now.

@flavorjones
Copy link
Collaborator Author

@mudge I believe the error you're seeing should only happen when you're running a prerelease ruby version (e.g., 3.4.0-preview2 or something). Is that the case?

@flavorjones
Copy link
Collaborator Author

Meantime I'm running some experiments at #136 to try to isolate which of the changes I made in this PR is causing the problem.

@mudge
Copy link

mudge commented Dec 29, 2024

Unfortunately I can't get past the build stage which fails with the following error on macOS 15.2:

 > [ 7/11] RUN /tmp/mk_osxcross.sh:
0.078 + git clone -q --depth=1 https://github.com/tpoechtrager/osxcross.git /opt/osxcross
0.740 + rm -rf /opt/osxcross/.git
0.744 + set +x
0.744 + curl -L -o MacOSX11.1.sdk.tar.xz https://github.com/larskanis/MacOSX-SDKs/releases/download/11.1/MacOSX11.1.sdk.tar.xz
0.750   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
0.750                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 46.6M  100 46.6M    0     0  7495k      0  0:00:06  0:00:06 --:--:-- 8617k
7.139 + tar -xf MacOSX11.1.sdk.tar.xz -C .
10.53 + cp -rf /usr/lib/llvm-10/include/c++ MacOSX11.1.sdk/usr/include/c++
10.53 + cp -rf /usr/include/x86_64-linux-gnu/c++/9/bits/ MacOSX11.1.sdk/usr/include/c++/v1/bits
10.54 cp: cannot stat '/usr/include/x86_64-linux-gnu/c++/9/bits/': No such file or directory
------
common-5a9d5e864bca8a999b6437c9df2a47dcecbb35f7:41
--------------------
  39 |
  40 |     COPY build/mk_osxcross.sh /tmp
  41 | >>> RUN /tmp/mk_osxcross.sh
  42 |
  43 |
--------------------
ERROR: failed to solve: process "/bin/sh -c /tmp/mk_osxcross.sh" did not complete successfully: exit code: 1
rake aborted!

@mudge
Copy link

mudge commented Dec 29, 2024

@mudge I believe the error you're seeing should only happen when you're running a prerelease ruby version (e.g., 3.4.0-preview2 or something). Is that the case?

Yes, you’re right. If I switch to, e.g. 3.3.6, I can require rcd_test without issue.

@flavorjones
Copy link
Collaborator Author

flavorjones commented Dec 29, 2024

If I switch to, e.g. 3.3.6, I can require rcd_test without issue.

@mudge OK, then can you try that running Ruby 3.0.7?

@mudge
Copy link

mudge commented Dec 29, 2024

@mudge OK, then can you try that running Ruby 3.0.7?

It took a while to get it installed but it seems to run the tests fine locally on my Apple silicon Mac:

~/Downloads/rake-compiler-dock/test/rcd_test> ruby --version
ruby 3.0.7p220 (2024-04-23 revision 724a071175) [arm64-darwin24]
~/Downloads/rake-compiler-dock/test/rcd_test> gem list rcd_test

*** LOCAL GEMS ***

rcd_test (1.0.0 arm64-darwin)
~/Downloads/rake-compiler-dock/test/rcd_test> ruby -rrcd_test -S rake test
Run options: --seed 2340

# Running:

....

Finished in 0.000675s, 5925.9258 runs/s, 10370.3701 assertions/s.

4 runs, 7 assertions, 0 failures, 0 errors, 0 skips

@flavorjones
Copy link
Collaborator Author

Blurgh. OK, thanks for doing that. I'm not sure there's anything else you can do at this point, if we can't reproduce it outside of the Actions runner. I'll keep at it in #136 for a bit.

@mudge
Copy link

mudge commented Dec 29, 2024

I just tried the x86_64-darwin gem on an Intel Mac with macOS 12.7 and the tests pass there too.

@chadlwilson
Copy link

Just curious, have you tried to see if it fails the same way on GHA macos-15 runners as with the macos-13/x64 and macos-14/arm combinations here?

Not sure how much it helps, but I have Intel macOS 14/Sonoma hardware available to test with, if it's useful.

@flavorjones
Copy link
Collaborator Author

@chadlwilson Is there a particular reason you think this would work on macos-15? I have no reason to believe that this is macos-specific at this point.

@chadlwilson
Copy link

No particular reason to believe it will work. Since it seemed to be something specific to the GHA MacOS runners - if it did happen to pass on one runner but not another we could perhaps compare the inventory of the versions of everything pre-installed on those runners to see what might be causing the problem.

@flavorjones flavorjones force-pushed the flavorjones-bump-rubies branch from eae6a5d to e0ad34e Compare January 4, 2025 14:51
@flavorjones flavorjones force-pushed the flavorjones-bump-rubies branch 2 times, most recently from 3f54cfd to 3ab4666 Compare January 5, 2025 18:13
@flavorjones flavorjones force-pushed the flavorjones-bump-rubies branch from 2957fb3 to 3dc21cb Compare January 5, 2025 19:44
@flavorjones
Copy link
Collaborator Author

OK, pushed a fix for the dyld issue in later 2.7/3.0 patch releases, which required applying the fix from 0328204 to other rubies.

Also pushed a fix for the 'miniruby' issue in ruby ~> 3.1.3 which is applying the workaround I originally suggested in https://bugs.ruby-lang.org/issues/19239 to darwin builds of 3.1.x.

I think this should go green?

- 3.4.1 → 3.4.1 [^1]
- 3.3.5 → 3.3.5 [^1]
- 3.2.0 → 3.2.6
- 3.1.0 → 3.1.2 [^2]
- 3.0.0 → 3.0.7
- 2.7.0 → 2.7.8
- 2.6.0 → 2.6.10
- 2.5.0 → 2.5.9
- 2.4.0 → 2.4.10

And update the default rbenv from 3.1.3 to 3.1.6.

[^1]: unchanged
[^2]: not the latest 3.1.6 because ruby/ruby#6193 broke
      cross-compilation builds until it was fixed in 3.2.0.rc2 by
      ruby/ruby#6944 and ruby/ruby#6959 which weren't backported
      (see https://bugs.ruby-lang.org/issues/19239)
@flavorjones flavorjones force-pushed the flavorjones-bump-rubies branch from 3dc21cb to 38084cf Compare January 5, 2025 20:27
@flavorjones
Copy link
Collaborator Author

flavorjones commented Jan 5, 2025

This went green. Trying one more build without setting OSXCROSS_PKG_CONFIG_USE_NATIVE_VARIABLES, because it might be unnecessary? I added it while trying to get the 3.0.x series to build, so it might be addressing a red herring.

flavorjones and others added 3 commits January 5, 2025 16:21
Because otherwise we started to run into errors during configuration
of ruby with the 3.0.x series.
It looks like -bundle-loader was backported to 2.7.7, 3.0.5, and 3.1.3
in addition to being present in 3.2 and later.
@flavorjones flavorjones force-pushed the flavorjones-bump-rubies branch from 38084cf to b887a5b Compare January 5, 2025 21:21
@flavorjones
Copy link
Collaborator Author

looks like we need the PKG_CONFIG_USE_NATIVE_VARIABLES, though I don't fully understand why it's just Ruby 3.1. 🤷 moving on with my life.

@flavorjones flavorjones merged commit f83b50d into main Jan 5, 2025
191 checks passed
@flavorjones flavorjones deleted the flavorjones-bump-rubies branch January 5, 2025 23:02
@flavorjones
Copy link
Collaborator Author

@mudge See https://github.com/rake-compiler/rake-compiler-dock/releases/tag/v1.8.0. Let me know if you hit any snags.

@chadlwilson
Copy link

Impressive - great stuff @flavorjones !

@mudge
Copy link

mudge commented Jan 6, 2025

Thanks for all your hard work on this, @flavorjones. I've now released re2 2.15.0 precompiled with rake-compiler-dock 1.8.0 for Ruby versions 2.6, 2.7, 3.0, 3.1, 3.2, 3.3, and 3.4 and will report back if there are any issues.

stanhu added a commit to stanhu/rb-sys that referenced this pull request Jan 14, 2025
This reflects the changes made in
rake-compiler/rake-compiler-dock#135.

While this requires packages to update their `RUBY_CC_VERSION` to use
Ruby patches across the board, this is already needed for Ruby 3.3.5.
This change at least makes this consistent.

See
https://github.com/rake-compiler/rake-compiler-dock/releases/tag/v1.8.0.
stanhu added a commit to stanhu/rb-sys that referenced this pull request Jan 14, 2025
This reflects the changes made in
rake-compiler/rake-compiler-dock#135.

While this requires packages to update their `RUBY_CC_VERSION` to use
Ruby patches across the board, this is already needed for Ruby 3.3.5.
This change at least makes this consistent.

See
https://github.com/rake-compiler/rake-compiler-dock/releases/tag/v1.8.0.
ianks pushed a commit to oxidize-rb/rb-sys that referenced this pull request Jan 16, 2025
This reflects the changes made in
rake-compiler/rake-compiler-dock#135.

While this requires packages to update their `RUBY_CC_VERSION` to use
Ruby patches across the board, this is already needed for Ruby 3.3.5.
This change at least makes this consistent.

See
https://github.com/rake-compiler/rake-compiler-dock/releases/tag/v1.8.0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants