Parallel quasi-newton optimizers for tensorflow-gpu #26100

timudk · 2019-02-25T19:21:46Z

Please make sure that this is a feature request. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template

System information

TensorFlow version (you are using): 1.12.0
Are you willing to contribute it (Yes/No): Yes

Describe the feature and the current behavior/state.
Is there any interest in implementing quasi-Newton methods for tensorflow-gpu? There has been some work on a parallel L-BFGS-B method (https://www.sciencedirect.com/science/article/pii/S0097849314000119) for GPus.

Will this change the current api? How?
New implementation.

Who will benefit with this feature?
I have had some recent success with using the BFGS optimizer for feedforward neural network. This approach, however, is limited to small networks and to tensorflow-cpu.

Any Other info.

timudk · 2019-03-08T20:19:12Z

Any updates?

JiaweiZhuang · 2019-04-16T20:28:12Z

Found this from a tweet. Second-order methods are very efficient for small & smooth problems. An example is Picking an Optimizer for Style Transfer that shows BFGS performs better than SGD-like methods (related: tensorflow/tfjs#127). Currently this is done through a SciPy wrapper without GPU support (keras-team/keras#5085)

Useful references:

PyTorch's L-BFGS implementation : source code and usage in style transfer.
Prototyping L-BFGS with TF-eager: TensorFlow meets PyTorch with Eager execution.

junpenglao · 2019-04-17T05:04:01Z

You can have a look at the implementation of BFGS and LBFGS in Tensorflow_probability, which do not wrap Scipy: https://github.com/tensorflow/probability/tree/master/tensorflow_probability/python/optimizer

junpenglao · 2019-04-17T05:04:50Z

It might be better to raise this feature request to Tensorflow_probability.

timudk · 2019-06-26T14:36:46Z

I finally got TFP's BFGS optimizer working for a supervised problem. Would adding an example to https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples be desirable?

sanjoy · 2019-12-09T21:18:04Z

I finally got TFP's BFGS optimizer working for a supervised problem. Would adding an example to https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples be desirable?

@jvdillon WDYT?

danieltsai0 · 2020-06-26T00:30:51Z

I finally got TFP's BFGS optimizer working for a supervised problem. Would adding an example to https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples be desirable?

Can you share a code-snippet?

rmothukuru · 2021-06-18T02:41:47Z

@timudk,
As per this comment this Feature seems to be implemented. Please find this working example in the Tensorflow Documentation.Thanks!

google-ml-butler · 2021-06-25T03:16:50Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

google-ml-butler · 2021-07-03T01:15:49Z

Closing as stale. Please reopen if you'd like to work on this further.

Imported from GitHub PR openxla/xla#26100 These are no longer needed, since computations now know their callers. I kept the utility function to check if a computation is async, since it is used from many places. I had to remove one check from the AsyncStart constructor. Previously, it didn't actually verify that the computation isn't already called by an async-start, just that the special field is unset. I don't see a way to have this check and make the code in CommandBufferScheduling work. After this, the only remaining special case is kFusion. Copybara import of the project: -- 0f99b6f6a2e8fab85a90c7a451e94713034a90e5 by Johannes Reifferscheid <[email protected]>: Clean up special cases for AsyncStart in HloComputation. These are no longer needed, since computations now know their callers. I kept the utility function to check if a computation is async, since it is used from many places. I had to remove one check from the AsyncStart constructor. Previously, it didn't actually verify that the computation isn't already called by an async-start. I don't see a way to have this check and make the code in CommandBufferScheduling work. After this, the only remaining special case is kFusion. -- 2dbc6e9d74e2b235303830a1247dcbecf5fde0a3 by Johannes Reifferscheid <[email protected]>: Fix HloParser. Merging this change closes #26100 FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#26100 from jreiffers:cg 2dbc6e9d74e2b235303830a1247dcbecf5fde0a3 PiperOrigin-RevId: 755257520

Imported from GitHub PR openxla/xla#26100 These are no longer needed, since computations now know their callers. I kept the utility function to check if a computation is async, since it is used from many places. I had to remove one check from the AsyncStart constructor. Previously, it didn't actually verify that the computation isn't already called by an async-start, just that the special field is unset. I don't see a way to have this check and make the code in CommandBufferScheduling work. After this, the only remaining special case is kFusion. Copybara import of the project: -- 0f99b6f6a2e8fab85a90c7a451e94713034a90e5 by Johannes Reifferscheid <[email protected]>: Clean up special cases for AsyncStart in HloComputation. These are no longer needed, since computations now know their callers. I kept the utility function to check if a computation is async, since it is used from many places. I had to remove one check from the AsyncStart constructor. Previously, it didn't actually verify that the computation isn't already called by an async-start. I don't see a way to have this check and make the code in CommandBufferScheduling work. After this, the only remaining special case is kFusion. Merging this change closes #26100 FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#26100 from jreiffers:cg 0f99b6f6a2e8fab85a90c7a451e94713034a90e5 PiperOrigin-RevId: 755258248

Imported from GitHub PR openxla/xla#26100 These are no longer needed, since computations now know their callers. I kept the utility function to check if a computation is async, since it is used from many places. I had to remove one check from the AsyncStart constructor. Previously, it didn't actually verify that the computation isn't already called by an async-start, just that the special field is unset. I don't see a way to have this check and make the code in CommandBufferScheduling work. After this, the only remaining special case is kFusion. Copybara import of the project: -- 0f99b6f6a2e8fab85a90c7a451e94713034a90e5 by Johannes Reifferscheid <[email protected]>: Clean up special cases for AsyncStart in HloComputation. These are no longer needed, since computations now know their callers. I kept the utility function to check if a computation is async, since it is used from many places. I had to remove one check from the AsyncStart constructor. Previously, it didn't actually verify that the computation isn't already called by an async-start. I don't see a way to have this check and make the code in CommandBufferScheduling work. After this, the only remaining special case is kFusion. -- 2dbc6e9d74e2b235303830a1247dcbecf5fde0a3 by Johannes Reifferscheid <[email protected]>: Fix HloParser. Merging this change closes #26100 FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#26100 from jreiffers:cg 2dbc6e9d74e2b235303830a1247dcbecf5fde0a3 PiperOrigin-RevId: 755257520

Imported from GitHub PR openxla/xla#26100 These are no longer needed, since computations now know their callers. I kept the utility function to check if a computation is async, since it is used from many places. I had to remove one check from the AsyncStart constructor. Previously, it didn't actually verify that the computation isn't already called by an async-start, just that the special field is unset. I don't see a way to have this check and make the code in CommandBufferScheduling work. After this, the only remaining special case is kFusion. Copybara import of the project: -- 0f99b6f6a2e8fab85a90c7a451e94713034a90e5 by Johannes Reifferscheid <[email protected]>: Clean up special cases for AsyncStart in HloComputation. These are no longer needed, since computations now know their callers. I kept the utility function to check if a computation is async, since it is used from many places. I had to remove one check from the AsyncStart constructor. Previously, it didn't actually verify that the computation isn't already called by an async-start. I don't see a way to have this check and make the code in CommandBufferScheduling work. After this, the only remaining special case is kFusion. Merging this change closes #26100 FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#26100 from jreiffers:cg 0f99b6f6a2e8fab85a90c7a451e94713034a90e5 PiperOrigin-RevId: 755258248

Imported from GitHub PR openxla/xla#26100 These are no longer needed, since computations now know their callers. I kept the utility function to check if a computation is async, since it is used from many places. I had to remove one check from the AsyncStart constructor. Previously, it didn't actually verify that the computation isn't already called by an async-start, just that the special field is unset. I don't see a way to have this check and make the code in CommandBufferScheduling work. After this, the only remaining special case is kFusion. Copybara import of the project: -- 0f99b6f6a2e8fab85a90c7a451e94713034a90e5 by Johannes Reifferscheid <[email protected]>: Clean up special cases for AsyncStart in HloComputation. These are no longer needed, since computations now know their callers. I kept the utility function to check if a computation is async, since it is used from many places. I had to remove one check from the AsyncStart constructor. Previously, it didn't actually verify that the computation isn't already called by an async-start. I don't see a way to have this check and make the code in CommandBufferScheduling work. After this, the only remaining special case is kFusion. Merging this change closes #26100 PiperOrigin-RevId: 755381030

ymodak added type:feature Feature requests comp:gpu GPU related issues labels Feb 26, 2019

ymodak assigned azaks2 Feb 26, 2019

ymodak added the stat:awaiting tensorflower Status - Awaiting response from tensorflower label Feb 26, 2019

azaks3 assigned sanjoy and unassigned azaks2 Dec 9, 2019

sanjoy assigned jvdillon Dec 9, 2019

tensorflowbutler removed the stat:awaiting tensorflower Status - Awaiting response from tensorflower label Dec 10, 2019

githubpsyche mentioned this issue Jul 27, 2020

Upgrade to GPU Programming githubpsyche/pcitpy#35

Closed

rmothukuru self-assigned this Jun 18, 2021

rmothukuru added the stat:awaiting response Status - Awaiting response from author label Jun 18, 2021

google-ml-butler bot added the stale This label marks the issue/pr stale - to be closed automatically if no activity label Jun 25, 2021

google-ml-butler bot closed this as completed Jul 3, 2021

copybara-service bot mentioned this issue May 6, 2025

PR #26100: Clean up special cases for AsyncStart in HloComputation. #92778

Closed

copybara-service bot mentioned this issue May 6, 2025

PR #26100: Clean up special cases for AsyncStart in HloComputation. #92780

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Parallel quasi-newton optimizers for tensorflow-gpu #26100

Parallel quasi-newton optimizers for tensorflow-gpu #26100

timudk commented Feb 25, 2019

timudk commented Mar 8, 2019

Uh oh!

JiaweiZhuang commented Apr 16, 2019 •

edited

Loading

Uh oh!

junpenglao commented Apr 17, 2019

Uh oh!

junpenglao commented Apr 17, 2019

Uh oh!

timudk commented Jun 26, 2019

Uh oh!

sanjoy commented Dec 9, 2019

Uh oh!

danieltsai0 commented Jun 26, 2020

Uh oh!

rmothukuru commented Jun 18, 2021 •

edited

Loading

Uh oh!

google-ml-butler bot commented Jun 25, 2021

Uh oh!

google-ml-butler bot commented Jul 3, 2021

Uh oh!

Parallel quasi-newton optimizers for tensorflow-gpu #26100

Parallel quasi-newton optimizers for tensorflow-gpu #26100

Comments

timudk commented Feb 25, 2019

timudk commented Mar 8, 2019

Uh oh!

JiaweiZhuang commented Apr 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

junpenglao commented Apr 17, 2019

Uh oh!

junpenglao commented Apr 17, 2019

Uh oh!

timudk commented Jun 26, 2019

Uh oh!

sanjoy commented Dec 9, 2019

Uh oh!

danieltsai0 commented Jun 26, 2020

Uh oh!

rmothukuru commented Jun 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

google-ml-butler bot commented Jun 25, 2021

Uh oh!

google-ml-butler bot commented Jul 3, 2021

Uh oh!

JiaweiZhuang commented Apr 16, 2019 •

edited

Loading

rmothukuru commented Jun 18, 2021 •

edited

Loading