|
| 1 | + |
| 2 | +<!--Copyright 2025 The HuggingFace Team. All rights reserved. |
| 3 | +
|
| 4 | +Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with |
| 5 | +the License. You may obtain a copy of the License at |
| 6 | +
|
| 7 | +http://www.apache.org/licenses/LICENSE-2.0 |
| 8 | +
|
| 9 | +Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on |
| 10 | +an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the |
| 11 | +specific language governing permissions and limitations under the License. |
| 12 | +
|
| 13 | +⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be |
| 14 | +rendered properly in your Markdown viewer. |
| 15 | +
|
| 16 | +--> |
| 17 | + |
| 18 | +<div style="float: right;"> |
| 19 | + <div class="flex flex-wrap space-x-1"> |
| 20 | + <img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-DE3412?style=flat&logo=pytorch&logoColor=white"> |
| 21 | + <img alt="SDPA" src="https://img.shields.io/badge/SDPA-DE3412?style=flat&logo=pytorch&logoColor=white"> |
| 22 | + </div> |
| 23 | +</div> |
| 24 | + |
| 25 | +# Gemma3n |
| 26 | + |
| 27 | +## Overview |
| 28 | + |
| 29 | +Gemma3n is a multimodal model with pretrained and instruction-tuned variants, available in E4B and E2B sizes. While |
| 30 | +large portions of the language model architecture are shared with prior Gemma releases, there are many new additions in |
| 31 | +this model, including [Alternating Updates][altup] (AltUp), [Learned Augmented Residual Layer][laurel] (LAuReL), |
| 32 | +[MatFormer][matformer], Per-Layer Embeddings (PLE), activation sparsity, and KV cache sharing. The language model uses |
| 33 | +a similar attention pattern to [Gemma 3](./gemma3.md) with alternating 4 local sliding window self-attention layers for |
| 34 | +every global self-attention layer with a maximum context length of 32k tokens. Gemma 3n introduces |
| 35 | +[MobileNet v5][mobilenetv5] as the vision encoder, using a default resolution of 768x768 pixels, and adds a |
| 36 | +[Universal Speech Model][usm] (USM) as the audio encoder. |
| 37 | + |
| 38 | +The instruction-tuned variant was post-trained with knowledge distillation and reinforcement learning. |
| 39 | + |
| 40 | +You can find all the original Gemma 3n checkpoints under the [Gemma 3n][gemma3n-collection] release. |
| 41 | + |
| 42 | +> [!TIP] |
| 43 | +> Click on the Gemma 3n models in the right sidebar for more examples of how to apply Gemma to different vision, audio, |
| 44 | +> and language tasks. |
| 45 | +
|
| 46 | +The example below demonstrates how to generate text based on an image with [`Pipeline`] or the [`AutoModel`] class. |
| 47 | + |
| 48 | +<hfoptions id="usage"> |
| 49 | +<hfoption id="Pipeline"> |
| 50 | + |
| 51 | +```py |
| 52 | +import torch |
| 53 | +from transformers import pipeline |
| 54 | + |
| 55 | +pipeline = pipeline( |
| 56 | + task="image-text-to-text", |
| 57 | + model="google/gemma-3n-e4b", |
| 58 | + device=0, |
| 59 | + torch_dtype=torch.bfloat16 |
| 60 | +) |
| 61 | +pipeline( |
| 62 | + "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg", |
| 63 | + text="<start_of_image> What is shown in this image?" |
| 64 | +) |
| 65 | +``` |
| 66 | + |
| 67 | +</hfoption> |
| 68 | +<hfoption id="AutoModel"> |
| 69 | + |
| 70 | +```py |
| 71 | +import torch |
| 72 | +from transformers import AutoProcessor, Gemma3nForConditionalGeneration |
| 73 | + |
| 74 | +model = Gemma3nForConditionalGeneration.from_pretrained( |
| 75 | + "google/gemma-3n-e4b-it", |
| 76 | + torch_dtype=torch.bfloat16, |
| 77 | + device_map="auto", |
| 78 | + attn_implementation="sdpa" |
| 79 | +) |
| 80 | +processor = AutoProcessor.from_pretrained( |
| 81 | + "google/gemma-3n-e4b-it", |
| 82 | + padding_side="left" |
| 83 | +) |
| 84 | + |
| 85 | +messages = [ |
| 86 | + { |
| 87 | + "role": "system", |
| 88 | + "content": [ |
| 89 | + {"type": "text", "text": "You are a helpful assistant."} |
| 90 | + ] |
| 91 | + }, |
| 92 | + { |
| 93 | + "role": "user", "content": [ |
| 94 | + {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"}, |
| 95 | + {"type": "text", "text": "What is shown in this image?"}, |
| 96 | + ] |
| 97 | + }, |
| 98 | +] |
| 99 | +inputs = processor.apply_chat_template( |
| 100 | + messages, |
| 101 | + tokenize=True, |
| 102 | + return_dict=True, |
| 103 | + return_tensors="pt", |
| 104 | + add_generation_prompt=True, |
| 105 | +).to("cuda") |
| 106 | + |
| 107 | +output = model.generate(**inputs, max_new_tokens=50, cache_implementation="static") |
| 108 | +print(processor.decode(output[0], skip_special_tokens=True)) |
| 109 | +``` |
| 110 | + |
| 111 | +</hfoption> |
| 112 | +<hfoption id="transformers CLI"> |
| 113 | + |
| 114 | +```bash |
| 115 | +echo -e "Plants create energy through a process known as" | transformers run --task text-generation --model google/gemma-3n-e2b --device 0 |
| 116 | +``` |
| 117 | + |
| 118 | +</hfoption> |
| 119 | +</hfoptions> |
| 120 | + |
| 121 | +## Notes |
| 122 | + |
| 123 | +- Use [`Gemma3nForConditionalGeneration`] for image-audio-and-text, image-and-text, image-and-audio, audio-and-text, |
| 124 | + image-only and aduio-only inputs. |
| 125 | +- Gemma 3n supports multiple images per input, but make sure the images are correctly batched before passing them to |
| 126 | + the processor. Each batch should be a list of one or more images. |
| 127 | + |
| 128 | + ```py |
| 129 | + url_cow = "https://media.istockphoto.com/id/1192867753/photo/cow-in-berchida-beach-siniscola.jpg?s=612x612&w=0&k=20&c=v0hjjniwsMNfJSuKWZuIn8pssmD5h5bSN1peBd1CmH4=" |
| 130 | + url_cat = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg" |
| 131 | + |
| 132 | + messages =[ |
| 133 | + { |
| 134 | + "role": "system", |
| 135 | + "content": [ |
| 136 | + {"type": "text", "text": "You are a helpful assistant."} |
| 137 | + ] |
| 138 | + }, |
| 139 | + { |
| 140 | + "role": "user", |
| 141 | + "content": [ |
| 142 | + {"type": "image", "url": url_cow}, |
| 143 | + {"type": "image", "url": url_cat}, |
| 144 | + {"type": "text", "text": "Which image is cuter?"}, |
| 145 | + ] |
| 146 | + }, |
| 147 | + ] |
| 148 | + ``` |
| 149 | +- Text passed to the processor should have a `<image_soft_token>` token wherever an image should be inserted. |
| 150 | +- Gemma 3n accept at most one target audio clip per input, though multiple audio clips can be provided in few-shot |
| 151 | + prompts, for example. |
| 152 | +- Text passed to the processor should have a `<audio_soft_token>` token wherever an audio clip should be inserted. |
| 153 | +- The processor has its own [`~ProcessorMixin.apply_chat_template`] method to convert chat messages to model inputs. |
| 154 | + |
| 155 | +## Gemma3nAudioFeatureExtractor |
| 156 | + |
| 157 | +[[autodoc]] Gemma3nAudioFeatureExtractor |
| 158 | + |
| 159 | +## Gemma3nProcessor |
| 160 | + |
| 161 | +[[autodoc]] Gemma3nProcessor |
| 162 | + |
| 163 | +## Gemma3nTextConfig |
| 164 | + |
| 165 | +[[autodoc]] Gemma3nTextConfig |
| 166 | + |
| 167 | +## Gemma3nVisionConfig |
| 168 | + |
| 169 | +[[autodoc]] Gemma3nVisionConfig |
| 170 | + |
| 171 | +## Gemma3nAudioConfig |
| 172 | + |
| 173 | +[[autodoc]] Gemma3nAudioConfig |
| 174 | + |
| 175 | +## Gemma3nConfig |
| 176 | + |
| 177 | +[[autodoc]] Gemma3nConfig |
| 178 | + |
| 179 | +## Gemma3nTextModel |
| 180 | + |
| 181 | +[[autodoc]] Gemma3nTextModel |
| 182 | + - forward |
| 183 | + |
| 184 | +## Gemma3nModel |
| 185 | + |
| 186 | +[[autodoc]] Gemma3nModel |
| 187 | + - forward |
| 188 | + |
| 189 | +## Gemma3nForCausalLM |
| 190 | + |
| 191 | +[[autodoc]] Gemma3nForCausalLM |
| 192 | + - forward |
| 193 | + |
| 194 | +## Gemma3nForConditionalGeneration |
| 195 | + |
| 196 | +[[autodoc]] Gemma3nForConditionalGeneration |
| 197 | + - forward |
| 198 | + |
| 199 | +[altup]: https://proceedings.neurips.cc/paper_files/paper/2023/hash/f2059277ac6ce66e7e5543001afa8bb5-Abstract-Conference.html |
| 200 | +[attention-mask-viz]: https://github.com/huggingface/transformers/blob/beb9b5b02246b9b7ee81ddf938f93f44cfeaad19/src/transformers/utils/attention_visualizer.py#L139 |
| 201 | +[gemma3n-collection]: https://huggingface.co/collections/google/gemma-3n |
| 202 | +[laurel]: https://arxiv.org/abs/2411.07501 |
| 203 | +[matformer]: https://arxiv.org/abs/2310.07707 |
| 204 | +[usm]: https://arxiv.org/abs/2303.01037 |
0 commit comments