Skip to content
This repository was archived by the owner on Jun 5, 2025. It is now read-only.

Conversation

@rdimitrov
Copy link
Member

The following PR adds malicious package test for Ollama.

Related to: #904

@rdimitrov
Copy link
Member Author

Judging by the logs it seems CodeGate finds the package and adds it to the context, but for some reason the response does not take it into account. Will debug further.

Signed-off-by: Radoslav Dimitrov <[email protected]>
@rdimitrov
Copy link
Member Author

I had to update the malicious package test case for this and the rest of the local providers because of the capabilities of the 0.5B model we are using for these tests. It looks like it cannot handle the system prompts as precisely as we expect which results in non-deterministic responses, i.e. sometimes it includes example code snippets even though it says the package is malicious, sometimes it doesn't include the link to insights. To handle all of this decided to simplify the test and just check if CodeGate was able to flag it.

Copy link
Contributor

@jhrozek jhrozek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small models are hard to predict

@rdimitrov rdimitrov merged commit a53187c into main Feb 5, 2025
9 checks passed
@rdimitrov rdimitrov deleted the malicious-ollama branch February 5, 2025 12:17
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants