HTTPS `Clone` fails with `remote pointer not found` using Go transport

[hiddeco]

Given:

1. All tests (even Clone) initialize a Repository before they run a test
2. The tests that make use of Clone seem to be using a local file path
3. The limited number of examples available

I am not really sure if this is due to a misuse of the API, or a bug. In any case, the following used to work before not too long ago with git2go/v31 and backing crypto / transport dependencies:

func TestRemoteClone(t *testing.T) {
	t.Parallel()
	path, err := ioutil.TempDir("", "git2go")
	if err != nil {
		t.Fatalf("Failed to init temp dir")
	}
	defer os.RemoveAll(path)
	remote, err := Clone("https://github.com/libgit2/libgit2", path, &CloneOptions{})
	if err != nil {
		t.Fatalf("Clone() = %v, want success", err)
	}
	defer remote.Free()
}

While attempting to update to git2go/v32 however, and trying to depend on less C libraries, this now returns a remote pointer not found error:

=== RUN   TestRemoteClone
=== PAUSE TestRemoteClone
=== CONT  TestRemoteClone
    remote_test.go:94: Clone() = remote pointer not found, want success
--- FAIL: TestRemoteClone (0.00s)

Following the breadcrumbs, it seems to end on https://github.com/libgit2/git2go/blob/main/clone.go#L43 returning -7 (which makes me wonder even more).

[codexetreme]

I am facing the same issue. I wonder if this has something to do with compiling libgit2 with the openSSL and libSSH libraries.

The language around these is a bit confusing, so I wonder if this is one potential problem

[hiddeco]

Following the introduction of Go native transports, the configuration to link against these libraries was disabled in this repository: https://github.com/libgit2/git2go/blob/main/script/build-libgit2.sh#L76-L77.

However, even after spending hours trying various configuration options, we have not been able to get a working version of v32, with or without the crypto libs. All eventually failing in some step of the Clone operation.

[codexetreme]

I am not sure how go native transports work, but let me tinker around and see if I can get it to work. I am still relatively new to this CGO and cross-language compiling. I will keep posting my findings, lets hope something works :P

[codexetreme]

After a bit of tinkering, I have turned on the HTTPS flag
-DUSE_HTTPS=ON

now when I try to clone, I get the error

# github.com/codexetreme/monorepo-semver-cli/git_bindings.test
/home/codexetreme/sdk/go1.16/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: /home/codexetreme/src/monorepo-semver-cli/third_party/git2go/static-build/install/lib/libgit2.a(openssl.c.o): in function `shutdown_ssl':
/home/codexetreme/src/monorepo-semver-cli/third_party/git2go/vendor/libgit2/src/streams/openssl.c:54: undefined reference to `BIO_meth_free'
/usr/bin/ld: /home/codexetreme/src/monorepo-semver-cli/third_party/git2go/vendor/libgit2/src/streams/openssl.c:59: undefined reference to `SSL_CTX_free'
/usr/bin/ld: /home/codexetreme/src/monorepo-semver-cli/third_party/git2go/static-build/install/lib/libgit2.a(openssl.c.o): in function `bio_destroy':
/home/codexetreme/src/monorepo-semver-cli/third_party/git2go/vendor/libgit2/src/streams/openssl.c:238: undefined reference to `BIO_set_data'
/usr/bin/ld: /home/codexetreme/src/monorepo-semver-cli/third_party/git2go/static-build/install/lib/libgit2.a(openssl.c.o): in function `bio_create':
....

this leads me to believe that simply linking the OpenSSL library is the issue now

[codexetreme]

UPDATE: I got it to work, (ie, it is not throwing any more SSL errors)
but a few more errors with setting cert checks and verification remain.

Currently the new bug is

go test -v -tags static ./git_bindings
=== RUN   Test_clone
time="2021-10-08T17:26:48+05:30" level=info msg="starting commit: https://github.com/ThreeDotsLabs/wild-workouts-go-ddd-example"
time="2021-10-08T17:26:49+05:30" level=error msg=error error="user cancelled hostkey check"
--- PASS: Test_clone (1.33s)

EDIT: I will update my process of getting this whole thing to work once I iron out the kinks

[codexetreme]

@hiddeco using your test, I am able to get it working with recompiling the vendored libgit2 with
CMAKE -DUSE_HTTPS=ON in scripts/build-libgit2.sh

my setup is probably slightly different from yours since I am packaging gogit2 as a submodule, but I believe that is the minimal tweak required.

also, my tests were cached, so I had to use go clean -testcache and re run

[lhchavez]

yikes, this seems to be caused by the fact that libgit2 duplicates the remote while cloning, so git2go can't match the original Remote object to the one passed into the callback :/ we probably need a different way of matching the remote instead of by-pointer.

[codexetreme]

But why does it work with the openssl libraries activated?

…

________________________________ From: lhchavez ***@***.***> Sent: Monday, October 11, 2021 5:28:44 PM To: libgit2/git2go ***@***.***> Cc: Yashodhan Ghadge ***@***.***>; Comment ***@***.***> Subject: Re: [libgit2/git2go] HTTPS `Clone` fails with `remote pointer not found` using Go transport (#836) yikes, this seems to be caused by the fact that libgit2 duplicates the remote while cloning, so git2go can't match the original Remote object to the one passed into the callback :/ we probably need a different way of matching the remote instead of by-pointer. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#836 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACCHNM4CNQ73LKSYNNPUK2LUGLYBZANCNFSM5E6KTZ2A>.

[lhchavez]

But why does it work with the openssl libraries activated?

that uses a different codepath that doesn't involve the remote callbacks.

[codexetreme]

Oh, if you have some pointers (pun intended) could you let me know? I would like to work on this if it does turn out to be a bug.

[lhchavez]

Oh, if you have some pointers (pun intended) could you let me know? I would like to work on this if it does turn out to be a bug.

it is indeed a bug. what should happen is that given that we have no way of recovering the original Remote (due to the underlying pointers being different), if we receive something that is not in our list of remotes, maybe we can create a new Remote object backed by the *git_remote that's passed into the smartTransportCallback. but then we need to handle that with care because we can't just free it: that would break libgit2 and cause use-after-frees :S

and add a test, haha.

[codexetreme]

wow, I just tried to go through the code paths, and yes it's pretty complicated. My question is, if the pointer in the smartTransportCallback is in fact a copy, and has nothing to do with the original *git_remote, then, we can safely free it right? it won't affect libgit2's cloning process since that will be performed on the original *git_remote pointer. (I am super new to C bindings, and libgit2, so if it sounds stupid, it probably is)

solution 2, is it possible to fix this pointer allocation on libgit2's side? can we submit an upstream PR that returns the same objects that are passed to the cloning process?

[lhchavez]

wow, I just tried to go through the code paths, and yes it's pretty complicated. My question is, if the pointer in the smartTransportCallback is in fact a copy, and has nothing to do with the original *git_remote, then, we can safely free it right?

we can't ^^;; that pointer is owned by libgit2 and if we free it we will cause a double free D: https://github.com/libgit2/libgit2/blob/5bd49aeeeb7a69f85249b970ac4e2c7011b7127d/src/clone.c#L407-L421 (the last line is the call to git_remote_free on the copy).

it won't affect libgit2's cloning process since that will be performed on the original *git_remote pointer. (I am super new to C bindings, and libgit2, so if it sounds stupid, it probably is)

solution 2, is it possible to fix this pointer allocation on libgit2's side? can we submit an upstream PR that returns the same objects that are passed to the cloning process?

i don't think so, that copy was introduced very intentionally: libgit2/libgit2@3c60768

[codexetreme]

right, so it landlocks on all sides. What code path does it take when the OpenSSL / libssh2 is compiled against libgit2? cause it seems to work just fine along that code path. Can we try to mimic what happens along that path?

[lhchavez]

right, so it landlocks on all sides. What code path does it take when the OpenSSL / libssh2 is compiled against libgit2? cause it seems to work just fine along that code path. Can we try to mimic what happens along that path?

we can't: that codepath is done 100% in libgit2 D: anything that crosses the Go/C boundary needs to go through a very different codepath.