[SYCL] Reduce the time to get a kernel from cache

[alexanderfle]

The main idea of this patch is to fast get information from the cache.

Before this patch:

Regular cache requires too many syncs and many searches across multiple maps to get the information.

At first, regular cache requires to get program from cache by getBuiltPIProgram: in most cases, it requires 3 searches in getKernelSetId then under mutex get the program from the cache, checking atomic variable whether it is already built or in the progress. Then using the program under mutex, doing 2 searches, get the stored info from the cache, also checking the atomic variable whether it is already built or in the progress.

After this patch:

For fast cache, it is required under mutex to make 1 search, and if the info is not found it means that the info has not been prepared yet, and then you will need to follow the standard more expensive way.

In addition, the fast cache keeps the program close to the kernel to return the program and reduce search time spending in one PI call in the most common case in commands.cpp.

Signed-off-by: Alexander Flegontov [email protected]

[alexanderfle]

/summary:run

[s-kanaev]

@alexanderfle , could you, please, elaborate in patch description on the idea employed in this patch to enhance run-time of cache operations?

[s-kanaev]

For fast cache, it is required under mutex to make 1 search, and if the info is not found it means that the info has not been prepared yet, and then you will need to follow the standard more expensive way.

Is the following sequence correct here upon request to get a kernel?

1. Lock fast cache mutex
2. Check if data available in fast cache:
   2.1 If it is, return the data. FINISH
   2.2 If not, proceed to p.3
3. Unlock fast cache mutex
4. Create data bits
5. Lock regular cache mutex(-es)
6. Insert data into regular cache
7. Lock fast cache mutex
8. Insert data into fast cache
9. Unlock fast cache mutex
10. Unlock regular cache mutex(-es)
11. Return data bits created in p.4

[alexanderfle]

Is the following sequence correct here upon request to get a kernel?

No, because the implementation of that requires changes getOrBuild() because the fast cache should store also PiProgram. And, currently, PiProgram is unknown in getOrBuild() in the case when we call getOrBuild() to get the kernel.

[alexanderfle]

@s-kanaev , ping

[s-kanaev]

The changes look good.
Though, I believe, we need tests for fast cache here.

Also, this sequence involves two mutexes:

1. Lock fast cache mutex
2. Check if data available in fast cache:
   2.1 If it is, return the data. FINISH + Unlock fast cache mutex
   2.2 If not, proceed to p.3
3. Unlock fast cache mutex
4. Create data bits
5. Lock regular cache mutex(-es)
6. Insert data into regular cache
7. Unlock regular cache mutex(-es)
8. Lock fast cache mutex
9. Insert data into fast cache
10. Unlock fast cache mutex
11. Return data bits created in p.4

Hence, we need a thread safety test also.

[s-kanaev]

LGTM

[alexanderfle]

@bader, ping

[bader]

@bader, ping

@alexanderfle, pong?

[alexanderfle]

@bader, no:) I mean, could please have a look at the PR as only authorized user can handle this.

[bader]

could please have a look at the PR as only authorized user can handle this.

What do you mean by "handle this"?

[alexanderfle]

What do you mean by "handle this"?

It means: handle the merge

[bader]

could please have a look at the PR as only authorized user can handle this.

What do you mean by "handle this"?

It means: handle the merge

FYI: There are at least 8 people who can handle the merge (including @againull, who is assigned to review this pull request).
E.g. #4272 is merged by @vladimirlaz and #4281 is merged by @romanovvlad.

[alexanderfle]

FYI: There are at least 8 people who can handle the merge (including @againull, who is assigned to review this pull request).
E.g. #4272 is merged by @vladimirlaz and #4281 is merged by @romanovvlad.

ok, I see.

[alexanderfle]

@againull, could you take a look, please? Please, merge if it looks okay to you.

[againull]

LGTM