[RFC]: Reduce Unit Test to Speed Up CI

[yewentao256]

Motivation.

#21764

Currently CI spent a lot of time in kernel unit test, even longer than 3 hours, but you know they are just kernels. So we aim to simplify some unit tests to make CI faster.

Proposed Change.

1. Reduce Test Cases

For example, we can reduce the amount of shapes

MNK_FACTORS = [
    (1, 128, 128),
    (1, 512, 512),
    (1, 128, 7168),
    (1, 1024, 7168),
    (1, 4608, 128),
    (1, 4608, 512),
    (1, 4608, 7168),
    (83, 128, 128),
    (83, 512, 512),
    (83, 1024, 7168),
    (83, 4608, 512),
    (83, 4608, 7168),
    (128, 128, 128),
    (128, 512, 512),
    (128, 1024, 7168),
    (128, 4608, 512),
    (128, 4608, 7168),
    (2048, 128, 128),
    (2048, 1024, 7168),
    (2048, 4608, 512),
    (2048, 4608, 7168),
    (8192, 128, 128),
    (8192, 512, 512),
    (8192, 128, 7168),
    (8192, 1024, 7168),
    (8192, 4608, 512),
    (8192, 4608, 7168),
]
->
MNK_FACTORS = [
    (1, 128, 128),
    (1, 1024, 7168),
    (1, 4608, 128),
    (83, 512, 512),
    (128, 128, 128),
    (128, 1024, 7168),
    (2048, 1024, 7168),
    (2048, 4608, 512),
    (8192, 512, 512),
    (8192, 4608, 512),
    (8192, 4608, 7168),
]

2. Refactor test type

wentao@gpu66:~/vllm-source/tests/kernels/moe$ ls
__init__.py                      test_modular_kernel_combinations.py
__pycache__                      test_moe.py
modular_kernel_tools             test_moe_align_block_size.py
parallel_utils.py                test_moe_permute_unpermute.py
test_batched_moe.py              test_mxfp4_moe.py
test_block_fp8.py                test_nvfp4_moe.py
test_block_int8.py               test_pplx_cutlass_moe.py
test_count_expert_num_tokens.py  test_pplx_moe.py
test_cutlass_grouped_gemm.py     test_rocm_aiter_topk.py
test_cutlass_moe.py              test_silu_mul_fp8_quant_deep_gemm.py
test_deepep_deepgemm_moe.py      test_triton_moe_ptpc_fp8.py
test_deepep_moe.py               utils.py
test_deepgemm.py

Some communication kernels like deepep & pplx is time consuming and should be moved to other test folder. And only be triggered if necessary

3. Remove Test

Possibly remove some useless test entirely.

Feedback Period.

No response

CC List.

@mgoin

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[kfhfar]

Hey new to the VLLM community. Willing to pick this up if this is still relevant

[njhill]

That would be great, thanks @kfhfar ... we are coordinating this CI sprint in #ci-sprint channel in vllm slack

[kfhfar]

Thanks for assigning it. I will start taking a look. I will also try to keep the slack channel updated.

[yewentao256]

#22942
#22939
#22936
I think @mgoin has done some of the work, @kfhfar I think you can similarly take a look at the remaining tests and see if there are something we can reduce

[kfhfar]

Understood. Sorry for the late reply. Taking a look.

[yewentao256]

@kfhfar Hi, what is the state now?

[kfhfar]

@yewentao256 I was briefly focused on other tasks at work. However, I am prioritizing from late this week and will provide continuous updates. I apologize for the delay

[kfhfar]

Is there a strict need by date?

[yewentao256]

No, it is not prioritized

[kfhfar]

Had some time to look at it. Most of the kernel optimization test pruning are done in the PR requests mentioned. There is a couple of folders that are still left i.e. : tests/kernel/mamba and tests/kernel/core. Will try to prioritize a PR for these two in the next couple of days.