How to get and modify weights between activated neurons?

[unwnstr]

Hello, I want to try simple live fine-tuning method, by direct and stupid downscale weights between activated neurons. How I expect this to work:

1. Model generates bad answer
2. User send command 'answer is bad'
3. Model direct scales down weights between activated neurons, during answer generation
4. Repeat until get good answer

Idk if it is possible to use this method for feeding 'right' answer and then upscaling weights.
*also may be i should scale only weights between 'new' neurons that activates during last answer generation to prevent damaging irrelevant knowledge.

*Isn't dopamine works kind of the same way?

I know why this method would not work for training models from scratch, but for fine-tuning it might work as well as lora, if not even better.

But I was not able to find any possible way of catching and modifying target weights, any suggestions where to look?

[KerfuffleV2]

This seems like it will be very, very hard to pull off in a way that's better than approaching it from the token sampling angle. You'll need a deep knowledge of the specific model's architecture.

It's not clear exactly what you mean by neurons here. The model is composed of a bunch of tensors, and some of them are stacked in layers. If you use the llama.cpp API you can manage running each layer and such on your own and do stuff like inspect the tensors between layers.

You can possibly change values in the tensors (or in the state). It's probably going to be pretty hairy if you're running in GPU mode since you'd have to manage getting the changes to the GPU. In CPU mode, this may work unless there are assumptions in GGML that the tensors are constant (in which case, making changes can lead to undefined behavior).

Labeling the result of evaluation as bad/good and penalizing everything that activated to contribute to the "bad" answer is probably way too blunt also. Assuming the model actually could produce a correct answer, it's likely most of the stuff that got activated was fine or part of the model being able to understand/relate the user's query. If you penalize all of it, you very likely will just make it impossible for the model to produce the correct answer.

Not sure if this will help, but I recently saw this article about intermediate activations in LLaMA2 models: https://www.lesswrong.com/posts/fJE6tscjGRPnK8C2C/decoding-intermediate-activations-in-llama-2-7b — maybe that's what you were talking about from the start, tweaking the state that eventually turns into the logits in between layers?

[unwnstr]

Thank you, this article helped me greatly develop my initial idea! (still figuring it out, but I would use it to determine more accurate, what weights I should modify)

Anyway, the way of accessing weights on live model still unclear. I could modify SiLU activation function to catch activations, store it for current and previous layer, and then somehow determine exact weights between only this 'neyrons'. And then, if user calls 'bad answer', model would run same prompt, but with modified SiLU, which would change target weights 'on fly'.

The idea is to change as little as possible parameters, to effect current output, just enough to change it slightly, so it could give another answer(set of answers).

I was looking to GGML, because on cpu it seems to be a lot easer task to do.

[klosax]

I like genuine ideas like this, and I think you should try your ideas, even if they seem to be doomed from the beginning.
That way you will learn from it, get other ideas and do new attempts. To find new working methods we have to do experimentation.

If your are going to down/up scale weights depending on the answer, I think you need to identify exactly which weights are active and responsible for both the wrong and the correct answer.

Take this example: "The digit for number one is". Now the correct answer is "1" and all other digits are wrong. And if you want it to answer "7" you will need a way to find exactly which weights are active and responsible for both digits. Feeding the model with many different sentences with the "1" digit and no other digits, you may find weights that are always activated, but not activated when feeding with other digits. Do this with all digits and using this statistic, you might be able to somehow fine-tune the model to answer "7" by down scaling the "1" weights and up scaling "7" weights..

[unwnstr]

Actually nice thought, I would play around this kind of modifications, if only find a way to do it XD
Probably this could work great, combined with article from previous answer, where you can watch, what's going on in intermediate layers, and determine exact 'neurons', where things gone wrong.

[SlyEcho]

There is something like this that I did in the steering vector experiment in PR #1472.

It reads and writes the embedding vectors on specific layers to influence the outcome.

(It is also very old and needs some work to bring up to the latest llama.cpp code, though)