CP-52911: Negative scheduler for xapi threads.

[GabrielBuica]

This is a bit complex and it makes use of a few recently added changes:

• Thread Classification using ocaml\libs\tgroups;
• Thread Local Storage from xapi-stdext-threads\threadsext;
• ocaml's global timeslice configuration ocaml\libs\timeslice.

Summary:

We want a fair scheduling of different xapi threads, corresponding to the share allocated for each thread class inside tgroup.ml, but we cannot give more time to threads before the kernel decides to schedule a different one. Instead, we take away time from threads that have been running for longer than their defined share. This is achieved by a combination of yielding and sleeping depending on the thread class. Unix.sleelpf is used to throttle the requests from xe while critical working threads are just yielding more often.

1. Every global timeslice we want to yield and start calculating the ideal times for the next epoch. The resulting values are stored in Tgroup.ThreadGroups. At this step we also reset the time counters. We also increment the epoch counter.
2. Every time we hit the periodic hook, we update the time running for the current thread inside the epoch. If the time running has exceeded the expected ideal time, we decide if we want to yield or sleep for the remain time in the current timeslice epoch.
   If the time is exceeded, we also reset the counter for the thread_last_yield. This would give a good approximation for how long the next thread has been running.

The ideal time is calculated pretty straight forward: we look at ThreadGroups with more than 1 thread running. We add up the shares, updating the new total share, we normalize each share and we used the results as weights to get the ideal time for this ThreadGroup. The ideal time for the individual thread is the time for the ThreadGroup divided by the number of threads tunning inside the respective class.

This behaviour is behind 2 flags and both need to be enabled for this to take place. tgroups-enabled and runtime-sched.

Results:
While timing vm starts under different loads of xe vm-list, I managed to get the following results for different timeslices:

In the above, negative percentage are the improvement in time compared with the version without negative scheduling but under the same load and using the same timeslice values.

and these are the absolute vm start times in seconds under loads (with this configuration):

For this, results, I used slightly more skewed shares to illustrate that under load the amount of time to start a VM is can be close to that of starting a VM without a load. And the results are best visible when the load and test and clearly separable in terms of threads classification.

This is on top of ThreadLocalStorage PR, #6354.

Note: this will probably need changes to xs-opam (bechamel.monotonic_clock).

[lindig]

The results are encouraging but I am nervervous about this kind of code. How can we be sure that we are not creating a situation where the system does not make any progress because it deadlocks or does not schedule some jobs?

[lindig]

A recursive loop with a counter would be an easy way to track the attempts

[GabrielBuica]

I changed the array to a hashtable but I will keep in mind this approach.

[lindig]

Do we want to log this case? How would you ever know what went wrong?

[GabrielBuica]

This is already logged. It's a warning above when the task is failed to be written in cgroups.

[edwintorok]

But that warning doesn't reraise the exception, so if we get to this point, something else must've raised an exception. Adding a with e -> D.log_backtrace e might be useful to log this at debug level, at least initially (although you'll need my backtrace PR to get that function).

[GabrielBuica]

The results are encouraging but I am nervervous about this kind of code. How can we be sure that we are not creating a situation where the system does not make any progress because it deadlocks or does not schedule some jobs?

I understand the concern @lindig , I spent a lot of time manual testing this. All the functionality is guarded by 2 feature flags that are off by default. So we can thoroughly test this before enabling it.

I am also restricting the groups that are are sleeping to only one so that when the remaining work is done they can run. Moreover, if the amount of time calculated for sleeping is never bigger than the timeslice and if a thread has already run more than the remaining time in the timeslice, the sleep time is 0. It this scenario, the thread does not even yield. This is to make sure there are always threads that can run. Moreover, xapi core threads are only yielding (I am not using sleeps for those).

[edwintorok]

The results are encouraging but I am nervervous about this kind of code. How can we be sure that we are not creating a situation where the system does not make any progress because it deadlocks or does not schedule some jobs?

I understand the concern @lindig , I spent a lot of time manual testing this. All the functionality is guarded by 2 feature flags that are off by default. So we can thoroughly test this before enabling it.

I am also restricting the groups that are are sleeping to only one so that when the remaining work is done they can run. Moreover, if the amount of time calculated for sleeping is never bigger than the timeslice and if a thread has already run more than the remaining time in the timeslice, the sleep time is 0. It this scenario, the thread does not even yield. This is to make sure there are always threads that can run. Moreover, xapi core threads are only yielding (I am not using sleeps for those).

Now that we know that the underlying approach helps performance, we could perhaps move the sleeping code (which is really a ratelimit) elsewhere. E.g. we could move it to the top-level API call entrypoint, and keep only the yielding in the GC.Memprof callback.

We could have some strategically placed maybe_ratelimit() calls that invokes the potentially sleeping code (e.g. just before creating a new session, just before dispatching the API call -- after we know its classification, just before marshaling the reply). We'd need to check whether we still see the original perf improvements in that case, and add more maybe_ratelimit calls in that case.

That way we know we're not holding any locks and are not preventing other threads from making progress, instead of sleeping from a very deep GC callback. (for some locks we have an atomic counter that tells us whether we are holding any, but not in general for all Mutexes).

[mg12]

These lines that are repeated many times above, it feels like they should be a function like Tgroup.with_one_thread_of_unset_group tgroup f or similar inside the TGroup module.

[GabrielBuica]

I agree about the code duplication. I added a helper function at the top of the file. Unfortunately, I cannot call Threadext from Troups as there is a cycle dependency. i.e. the thread local storage contains the tgroup associated with the thread.

6e7a4b0.

[GabrielBuica]

Closing it for now as more investigation is needed to address the sleeping in the runtime concerns.