How to adapt HuggingFace or other models for TorchTitan

[githubsgi]

Is there any thought on how to adapt HuggingFace or other models for pre-training with TorchTitan ?

[tianyu-l]

For HF models, there has been some work in #919 which needs some cleanup. Original issue is #903

For how to add models in general, I'll publish a PR of guidelines soon.

[githubsgi]

@tianyu-l , any progress on the HF model usage in TT guideline ?

[tianyu-l]

Right now we don't have bandwidth, but I probably can find someone to help work on it a month later. Also we'll be setting up the workflow of "load HFcheckpoint -> train in torchtitan -> save back to HF" soon. Does that sound helpful to you?

[githubsgi]

Thanks, would require a separate model file or the HF CP file has that ?

[tianyu-l]

It would load HF checkpoint (expecting HF model definition) -> convert the model definition to torchtitan model definition -> train in torchtitan -> convert the model definition back to HF model definition -> save in HF format

[rghadia]

Hi,
This would be super helpful for finetuning HF models. Is there a plan to have this anytime soon?

[tianyu-l]

@rghadia for llama3 it will be available after #1441

[rghadia]

Hi @tianyu-l ,

Thanks for pointing me to the PR #1441

I was able to refer to it to add support for HF model locally. However I feel the HF tokenizer wrapper is slightly different from the default HF tokenizer (with added wrappers for HF tokenizer's default methods).

IMHO this was inconvenient to use the default methods supported by HF tokenizer (such as tokenizer.add_tokens), so I wonder if there was a reason to NOT allow the HF tokenizer to be used as is ? (instead of making the HF tokenizer a class attribute: self._tokenizer.tokenizer?)

I might be missing something w.r.t. torchtitan at large, so feel free to enlighten me.

[tianyu-l]

IMHO this was inconvenient to use the default methods supported by HF tokenizer (such as tokenizer.add_tokens), so I wonder if there was a reason to NOT allow the HF tokenizer to be used as is ? (instead of making the HF tokenizer a class attribute: self._tokenizer.tokenizer?)

@H-Huang