Fail to build tag v2.1.0.dev+cpu.llm

[kenplusplus]

Describe the issue

The line

intel-extension-for-pytorch/csrc/cpu/autocast/autocast_mode.cpp

Line 212 in 114cc75

_scaled_dot_product_attention, fallthrough, fp32)

may need delete as below:
MAKE_REGISTER_FUNC_TWO_POLICIES(
scaled_dot_product_attention, fallthrough, fp32)
MAKE_REGISTER_FUNC_TWO_POLICIES(
_scaled_dot_product_attention, fallthrough, fp32)
MAKE_REGISTER_FUNC_TWO_POLICIES(
_scaled_dot_product_attention_math, fallthrough, fp32)

[ZhaoqiongZ]

Hi @kenplusplus , thanks for report! Do you follow the instruction in the link to compile it? https://intel.github.io/intel-extension-for-pytorch/llm/cpu/ the v2.1.0.dev+cpu.llm branch need a specific pytorch nightly version. so please follow the instructions in the tutorial to install python -m pip install torch==2.1.0.dev20230711+cpu torchvision==0.16.0.dev20230711+cpu torchaudio==2.1.0.dev20230711+cpu --index-url https://download.pytorch.org/whl/nightly/cpu , or use the compile_bundle script to compile the wheels as in the tutorial.

[kenplusplus]

Hi @kenplusplus , thanks for report! Do you follow the instruction in the link to compile it? https://intel.github.io/intel-extension-for-pytorch/llm/cpu/ the v2.1.0.dev+cpu.llm branch need a specific pytorch nightly version. so please follow the instructions in the tutorial to install python -m pip install torch==2.1.0.dev20230711+cpu torchvision==0.16.0.dev20230711+cpu torchaudio==2.1.0.dev20230711+cpu --index-url https://download.pytorch.org/whl/nightly/cpu , or use the compile_bundle script to compile the wheels as in the tutorial.

Thanks, I used torch-2.1.0.dev20230730, so do you mean 20230711 provide additional function _scaled_dot_product_attention_math?

[jingxu10]

v2.1.0.dev+cpu.llm is only veriried with 20230711 nightly build. Since pytorch nightly build is still under development, there could be incompatible changes in different date nightly builds.

[kenplusplus]

v2.1.0.dev+cpu.llm is only veriried with 20230711 nightly build. Since pytorch nightly build is still under development, there could be incompatible changes in different date nightly builds.

OK, I will try 20230711, althoughI still think _scaled_dot_product_attention is a typo in code ( _ is no need)

[jingxu10]

@leslie-fang-intel would you check this concern?

[leslie-fang-intel]

Looks like _scaled_dot_product_attention already been deleted 3d5a1ab by @zhuhaozhe in master branch.

[jagane-infinstor]

The pytorch 20230711 nightly build is no longer available in the pytorch download site. Hence compile_bundle now fails with the following:

• python -m pip install torch==2.1.0.dev20230711+cpu torchvision==0.16.0.dev20230711+cpu torchaudio==2.1.0.dev20230711+cpu --index-url https://download.pytorch.org/whl/nightly/cpu Looking in indexes: https://download.pytorch.org/whl/nightly/cpu ERROR: Could not find a version that satisfies the requirement torch==2.1.0.dev20230711+cpu (from versions: 2.1.0.dev20230821+cpu, 2.1.0.dev20230822+cpu, 2.1.0.dev20230823+cpu, 2.1.0.dev20230824+cpu, 2.1.0.dev20230825+cpu, 2.1.0.dev20230826+cpu, 2.1.0.dev20230827+cpu, 2.1.0.dev20230828+cpu, 2.1.0.dev20230829+cpu, 2.1.0.dev20230830+cpu, 2.1.0.dev20230831+cpu, 2.1.0.dev20230901+cpu, 2.1.0.dev20230902+cpu, 2.1.0.dev20230903+cpu, 2.1.0.dev20230904+cpu, 2.1.0.dev20230905+cpu, 2.2.0.dev20230906+cpu, 2.2.0.dev20230907+cpu, 2.2.0.dev20230908+cpu, 2.2.0.dev20230909+cpu, 2.2.0.dev20230910+cpu, 2.2.0.dev20230911+cpu, 2.2.0.dev20230912+cpu, 2.2.0.dev20230913+cpu, 2.2.0.dev20230914+cpu, 2.2.0.dev20230915+cpu, 2.2.0.dev20230916+cpu, 2.2.0.dev20230917+cpu, 2.2.0.dev20230918+cpu, 2.2.0.dev20230919+cpu, 2.2.0.dev20230920+cpu, 2.2.0.dev20230921+cpu, 2.2.0.dev20230922+cpu, 2.2.0.dev20230923+cpu, 2.2.0.dev20230924+cpu, 2.2.0.dev20230925+cpu, 2.2.0.dev20230926+cpu, 2.2.0.dev20230927+cpu, 2.2.0.dev20230928+cpu, 2.2.0.dev20230929+cpu, 2.2.0.dev20230930+cpu, 2.2.0.dev20231001+cpu, 2.2.0.dev20231002+cpu, 2.2.0.dev20231003+cpu, 2.2.0.dev20231004+cpu, 2.2.0.dev20231005+cpu, 2.2.0.dev20231006+cpu, 2.2.0.dev20231007+cpu, 2.2.0.dev20231008+cpu, 2.2.0.dev20231009+cpu, 2.2.0.dev20231010+cpu, 2.2.0.dev20231011+cpu, 2.2.0.dev20231012+cpu, 2.2.0.dev20231013+cpu, 2.2.0.dev20231014+cpu, 2.2.0.dev20231015+cpu, 2.2.0.dev20231016+cpu, 2.2.0.dev20231017+cpu, 2.2.0.dev20231018+cpu, 2.2.0.dev20231019+cpu) ERROR: No matching distribution found for torch==2.1.0.dev20230711+cpu

[jingxu10]

Pls use the llm_feature_branch branch.

[kenplusplus]

Thanks!

[jagane-infinstor]

Hello folks - can anybody here point me to where I can help to run llama2 in hybrid (bfp16+int8) mode. This is on Intel Sapphire Rapids. I found this reference to the hybrid mode in a bug in the llama.cpp project: ggml-org/llama.cpp#2555 (comment) My larger goal is to create a MLflow model of a high performance llama2 for Sapphire Rapids. Thanks Jagane

…

On Thu, Oct 19, 2023 at 5:04 PM Lu Ken ***@***.***> wrote: Thanks! — Reply to this email directly, view it on GitHub <#395 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXXIT4QHACCPAD7NZQ6TWGDYAG5YXAVCNFSM6AAAAAA3BNDUIOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZRHA3DIOJWGY> . You are receiving this because you commented.Message ID: ***@***.***>

[jingxu10]

Would you check https://github.com/intel/intel-extension-for-pytorch/tree/v2.1.0%2Bcpu/examples/cpu/inference/python/llm?

[jagane-infinstor]

I checkout out the v2.1.0+cpu branch and tried quantizing llama2 70b huggingface model. The command I used was:

$ python run.py -m /home/sdp/working/llama/llama-2-70b-hf --dtype float32 --ipex-smooth-quant --output-dir out2 --int8-bf16-mixed

Some warnings and errors are printed out. Notably, it says 'Some weights from LlamaForCausalLM were not initialized from the model....'

Is this to be expected?
Log attached here.
errors.txt