Moving daily releases to -nightly

[krrishdholakia]

We're planning on moving our daily releases to be pre-releases with a -nightly tag.

This is so we can separate releases which haven't gone through a qa checklist, and therefore might have some instability (e.g. not being able to see keys on UI as an internal user).

Normal releases will therefore be more stable as they would have gone through manual QA (in addition to our ci/cd) before being published.

[krrishdholakia]

Stable Release QA Checklist

Proxy Admin

• create a team
• Invite a user to the team
• Create a key in the team
• Regenerate key
• Update key with new tpm/rpm limits
• delete key
• Perform a chat in playground with at least 2 different models
• Review logs page - verify playground requests got logged
• Review Usage Page, see Top API Keys Table. Click into any API Key view Key Info. Click Chart View Toggle on Top Keys, see chart
• Adding new models -> add specific model -> Click Test Connection with Valid credentials. Should work
• Adding new models -> add specific model -> Click Test Connection with Bad credentials. Should show raw curl request sent + error
• Adding new models -> add specific model -> Add model. Confirm it's added under Models Page
• Adding new models -> add wildcard route
• Add new model w/ reusable credential -> test if it works
• Add Azure AI Foundry mode, select custom model name, make test request on UI
• Add Azure Openai model, set custom public model name
• [TODO] Add VertexAI model with reusable credentials
• Add OpenAI-compatible model (perplexity/fireworks ai/xai) on UI , test request, confirm it works
• add team-only model → assign to team
• call model via anthropic pass-through route -> confirm spend tracked
• call bedrock anthropic on test key pane -> sanity check token counts
• Edit team model
• Edit team member for team - proxy admin does not belong to
• delete team
• Refresh team page -> teams should load
• Create key -> test if it works
• Open create key modal -> confirm all values match /key/generate on swagger
• [TODO] Create Key, Select Team, The models page should show the models for selected team.
• [TODO] Create Key, Select a different Team, The models page should show the models for newly selected team.
• search for user email on the team member add
• see all internal user created keys in team
• [TODO] Update RPM limit for org

Logs Page

• Go to Logs. Expand a log, verify you can view request details. Collapse, verify it collapses
• Expand a log, send 10 requests. Verify that the Expanded log does not change as you get traffic

Use this to generate 10 requests, just run on your terminal

#!/bin/bash

for i in {1..10}; do
  echo "Request $i:"
  curl -i http://localhost:4000/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-PIp1h0RekR" \
  -d '{
    "model": "us-gpt-4",
    "messages": [
      {
        "role": "user",
        "content": "hi who is this."
      }
    ]
  }'
  echo -e "\n\n------------------------\n"
done

Other Proxy Admin

• Invite a user, make them Admin and give them no access to models
• Create a Key, with no models
• Create a Key, with specific proxy model
• Use the new key to call proxy model
• Use the new key for the following API request. Expect it to pass

curl -X GET "http://localhost:4000/team/list" \
  -H "Authorization: Bearer sk-<key-you-just-created>"

Team Admin

• able to view all team keys
• able to add/remove member from team
• able to add / edit / delete model in team
• able to create team key with all team models
• able to see all team models in test key dropdown

Internal User

• Create key -> test if it works
• Refresh key page → check if created key still shown
• Click on teams tab → click on team → should be able to view team info
• Create new user -> doesn't belong to any teams -> create key -> test if it works
• [TODO] check keys page -> shouldn't see litellm-dashboard keys
• [TODO] As ‘user’ role within team → try creating key → ensure it fails

Internal User Viewer

• Admin, invite an internal user viewer
• Internal Viewer should only see the following pages, nothing out of this set
• Only the follow pages are visible
  - Keys, Usage, Teams, API Ref, Model Hub, Logs, New usage, API Playground
  - Nothing outside this set
• Should not Test Key page
• Keys Page: Should not see create key button
• Keys Page: Click key, should not see edit delete or regenerate on root or on the key specific page
• Teams Page: should not see Members or Team settings page

Security Tests

• Sign in as Admin, Invite internal user, follow onboarding link, sign in as internal user. Hover over user info -> This should be the new user (Note: Remember the password your entered here)
• Click Logout
• Sign in as Admin, hover over User Sign in info (corner right), current user should be proxy admin
• Sign in as the new user with your credentials from step 1, on sign in the ui should load only info for the new internal user (not Proxy Admin)
• Set PROXY_LOGOUT_URL="https://www.google.com" in your .env, start LiteLLM, sign in, click logout, confirm you got directed to www.google.com

SSO Tests

• Set MICROSOFT_CLIENT_SECRET, MICROSOFT_CLIENT_ID, MICROSOFT_TENANT. Start Proxy, sign in. Hover over user tab and confirm your email is the correct email

Server Tests

• Enable prometheus -> make request with key + team id -> check for any errors in server logs

---

Please let me know if there's any scenarios we should add to this.

[krrishdholakia]

PR with change is live here - #8499

[krrishdholakia]

This is now live from today onwards

[krrishdholakia]

Rollout plan:

1. Stage 1: Move daily docker releases to -nightly tag
2. Stage 2: Move pypi releases to -nightly or .dev tag (depending on what pypi supports)
3. Stage 3: Establish regular release cadence post QA (either daily or weekly)

[krrishdholakia]

Update pypi releases are now marked with a .post1

[krrishdholakia]

Stable releases are done on a weekly basis (every Sunday) - https://docs.litellm.ai/docs/proxy/release_cycle