We recently added support for fine-tuning the text encoder with LoRA here: https://github.com/huggingface/diffusers/pull/2918. Next up, we need to modify our [DreamBooth training example](https://github.com/huggingface/diffusers/tree/main/examples/dreambooth) to incorporate this support.