Tokens limits issue

[Billyroot]

Input validation error: inputs tokens + max_new_tokens must be <= 1512. Given: 603 inputs tokens and 1024 max_new_tokens

When deployed, the ui is working fine for like 2 or 3 promts, then every prompt we try we get a red line on top with a pop-up having this message. Please how can we remove this limitation on the code?

[nsarrazin]

Hi! Can I see your model definition in your .env ? Are you hosting your own endpoint ? 👀

[Billyroot]

Hi! Can I see your model definition in your .env ? Are you hosting your own endpoint ? 👀

Thanks for the reply.

We used Pythia epoch 3.5 and guanaco merged. We are relying on hugging face api inference,I don't know if it replies to your question about endpoints.

[nsarrazin]

Hi, sorry for the delay, is this issue still relevant ? Let me know if so!