Skip to content

Fix GPU builds #385

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Sep 20, 2021
Merged

Fix GPU builds #385

merged 6 commits into from
Sep 20, 2021

Conversation

rnett
Copy link
Contributor

@rnett rnett commented Sep 20, 2021

Fixes GPU builds by updating .bazelrc and CI CUDA and CUDNN versions.

@karllessard can you tag this for CI build?

I just overwrote the .bazelrc file, if there's anything we want to add we should be doing it in a separate file, anyways.

@rnett
Copy link
Contributor Author

rnett commented Sep 20, 2021

I'm testing this locally, but it's still building.

@Craigacp Craigacp added the CI build Triggers a full native build on a pull request label Sep 20, 2021
@rnett
Copy link
Contributor Author

rnett commented Sep 20, 2021

I could copy the tensorflow bazelrc from the remote repo, and allow us to use our own for whatever we need (and prevent this kind of out-of-sync issue from happening again). It would make it much harder to edit though (you'd need to use a patch). Thoughts?

@rnett
Copy link
Contributor Author

rnett commented Sep 20, 2021

This is working on my machine, so we should be good to merge. tensorflow.bazelrc is now the bazelrc from the tensorflow repo, and .bazelrc can be whatever we want (I have readonly build cache in there for now, setting it to write in the CI will take precedence).

cc @karllessard @Craigacp

@rnett
Copy link
Contributor Author

rnett commented Sep 20, 2021

Would be good to figure out where the QuantizedConcatV2 on my machine is coming from first though.

@rnett
Copy link
Contributor Author

rnett commented Sep 20, 2021

The CI timing out since it can't reuse the cache.

@karllessard
Copy link
Collaborator

The CI timing out since it can't reuse the cache.

Yes, so let's merge it, anyway we know that the current build is not working properly on GPU devices, thanks @rnett

@karllessard karllessard merged commit 5e3fc49 into tensorflow:master Sep 20, 2021
@rnett
Copy link
Contributor Author

rnett commented Sep 20, 2021

It's timing out on master too, I think you have to manually re-run it a couple times to build the cache.

@karllessard
Copy link
Collaborator

Finally got the snapshots to build, I think it took 4 passes this time... you can try it out now

@rnett rnett deleted the rn_fix_gpu branch October 17, 2021 00:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI build Triggers a full native build on a pull request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants