[REQUEST] Inquiry about code for Domino

[s1ghhh]

I saw in Domino that the code would be released here. Could you let me know when will the code be released to the public?

[GuanhuaWang]

Hi @s1ghhh

Thanks for your interests! our code has been merged to master branch for both deepspeed and deespeedexample

• deepspeed code: Add Domino code #6733

• deepspeed-example code (tutorial and dependencies): Deepspeed-Domino DeepSpeedExamples#929

  • Feel free to try it out with our tutorial here https://github.com/microsoft/DeepSpeedExamples/tree/master/training/DeepSpeed-Domino

Our blog is still under some internal review process, but will be merged to deepspeed repo soon.

And we are more than happy to collaborate in any aspects (e.g. code refactoring, remove third-party dependency, new ideas, etc).

Thanks

[Quentin-Anthony]

@GuanhuaWang -- Great work on Domino! I looked through these PRs, but I'm struggling to find the lower-level kernel optimizations that you did for domino. Can you point me to those?

I'm looking to ablate figure 10 to nail down just how much benefit is from your optimizations in section 4.3 vs sections 4.1/4.2

[GuanhuaWang]

@GuanhuaWang -- Great work on Domino! I looked through these PRs, but I'm struggling to find the lower-level kernel optimizations that you did for domino. Can you point me to those?

I'm looking to ablate figure 10 to nail down just how much benefit is from your optimizations in section 4.3 vs sections 4.1/4.2

Hi @Quentin-Anthony ,

That's a very good question! In short, we are gradually adding only Necessary features on our updated code base.

For this code release, we did several modifications on top of our paper submission codebase.

Cuda_graph: We rewrite core-attention using pytorch native module scaled_dot_product_attention, which we do not see too much bubble time between adjacent kernels as megatron's attention module. Therefore, we did not add cuda graph on top of it. If you want these features, you can simply added it back via using torch.cuda.make_graphed_callables as below.

self.self_attention =  torch.cuda.make_graphed_callables(self.self_attention,(args,), 11)

please do note that 11 is a minimum warm-up number for module using pytorch ddp wrapper. Also, if needed, we may add cuda graph back in either python or c/cuda level.

torch.compile & multi-stream:
We were doing them both at python level. For some small kernel sequences like bias dropout add, we were doing torch compile to make it faster (simply add torch compile decorator). For multi-stream, since we break down data dependency (let's say we chunk tensor half-half as t0 and t1) so we were launching t0's kernels on one stream and t1's kernels on another stream.
But right now we want to do kernel fusion and multi stream (if necessary) at cuda level for maximum performance. @hwchen2017 is leading this line now. Once we finalize our kernel side implementation, we will do a code update.

For Figure 10 ablation, without kernel side optimization as sec4.3, we can reach around 90+% but never reach 100+%. But right now we see potential bigger gain, Stay tuned! Also feel free to email me if you are interested in contributing to domino project.

[Quentin-Anthony]

Thanks for this @GuanhuaWang!

[GuanhuaWang]

Close the issue