Avoid client result invocation ID collisions

[BrennanConroy]

Fixes bullet 7 of #5280

Stephen had an interesting idea:

Couldn't we pass in a per-connection, IInvocationBinder that adds the connection id prefix back to the invocation id so the IHubProtocol doesn't need to be aware of it?

Unfortunately, we don't know the connection ID when IHubProtocol queries IInvocationBinder so the invocation id has to be the entire source of truth for the lookup.

After doing some local perf testing, the best approach I found was to just generate a GUID and base64 encode it for every invocation. There is only one allocation, the final string, and GUID generation is still fairly cheap, the most expensive part is likely getting the random bytes.

Other alternatives considered:

• generate a shorter random prefix at startup and append an incrementing ID every invoke. Then every 1000 or so invocations we generate a new random prefix and reset the ID to 0 (so the full invocation ID doesn't grow too big). Sadly, I couldn't figure out how to do this lock free, or even lock free until it needed to reset. So this approach becomes very slow in a multi-threaded environment.
• generate a shorter random prefix at startup and append an incrementing ID every invoke. If a collision occurs, reply to the original server that it collided and original server will try to use a new ID, repeat. This approach would require more noise over the backplane and complexities with coding.
• Use some form of randomness and then just throw if a collision occurs. Collisions should be rare and application code likely already is handling exceptions because of timeouts, connection closing, connection throwing error for a result, etc.

[davidfowl]

Do you have any of the perf results? This is something we can change in the future if performance is a problem right?

[davidfowl]

🤣 . Are we taking bets on if we ever see this error message filed in an issue?

[BrennanConroy]

This is something we can change in the future if performance is a problem right?

Yes, it's an internal implementation detail. Although if we wanted to do something co-operative it could be breaking if old server talks to new server.

100 threads, each thread doing 1,000,000 invocations:

lock	new id every time	hardcoded "random" prefix and increment id every time
21.3158182s	2.9508009s	3.6222541s

[BrennanConroy]

@Pilchie for RC2. This fixes a bug when using multiple servers and the redis backplane in SignalR in combination with the new client result feature where it was possible for 2 IDs to collide which would cause the caller to hang in their app code until the cancellation token (they hopefully passed in) is canceled.

[Pilchie]

Approved for .NET 7 RC2.