CP-48623: reduce XenAPI.py connection rate and drop 4 useless API calls #5533
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This test did not exercised the SR type expected by user, and notably failed when no default SR was defined. Adds the option of installing a VM in a non-default SR using VM.with_new. Signed-off-by: Yann Dirson <[email protected]>
Previously sub-second durations were printed as "ago", now they are pretty-printed using Mtime.Span's logic, for example "666ms ago". If the point in time is unexpectedly in the future, print "0ns ago". Durations lasting at least 1 seconds are printed as before. Signed-off-by: Pau Ruiz Safont <[email protected]>
Signed-off-by: Ming Lu <[email protected]>
…390570 CA-390570: Py3 socket.sendto needs bytes instead of a string
When creating a backup VDI on an SR we want to derive the VDI's UUID from the SR. This way we can't be later tricked into accepting a different VDI as a backup. Implement a hash that derives the VDI UUID and pass this UUID to SM to be used. Currently the creation of this VDI in Xapi is detected based on its name-label: "Pool Metadata Backup". The SM stack usually creates a new random UUID but will use a given UUID passed as "vdi_uuid" for its "vdi_create" command. The difficulty in the implementation is that "vdi_uuid" is used in other commands as well and we have to make sure to set it only in the intended context. It is not straight forward to pass the UUID from Xapi_vdi.create down to Sm. Signed-off-by: Christian Lindig <[email protected]>
We introduce deterministic UUID for backup VDIs to avoid searching for VDIs. Add this capability to the scripts that create and restore backups. In case the VDI with the expected UUID does not exist, let the user confirm probing for it. Signed-off-by: Christian Lindig <[email protected]>
Signed-off-by: Christian Lindig <[email protected]>
We have added an interactive prompt in case the backup VDI was created by an earlier version and does not match the expected UUID. To facilitate scripting, add a -y (yes) flag that assumes a confirmation. Signed-off-by: Christian Lindig <[email protected]>
|
Looks like I might have to fix some pytype warnings (but locally pytype says it doesn't support Python >3.11 yet, and I have Python 3.12). Lets see what the CI says. |
They look like genuine errors. I'll open a separate to fix those first though. |
|
Also apparently there is no cached_property in 3.6, just 3.8, I'll write it by hand instead.. |
edwintorok
force-pushed
the
private/edvint/xenapi
branch
from
6372e2b to
f490948
Compare
psafont
reviewed
Mar 27, 2024
robhoes
reviewed
Mar 27, 2024
robhoes
approved these changes
Mar 27, 2024
lindig
approved these changes
Mar 27, 2024
|
On sequential API calls (e.g. 500 VDI.destroy) there is an about 5% reduction in API execution times. I haven't done bootstorm or parallel VM start measurements yet, where I'd expect to see more improvements. |
edwintorok
force-pushed
the
private/edvint/xenapi
branch
2 times, most recently
from
d684a22 to
68112fb
Compare
Fixes this pytype report: ``` pytype_reporter: Expected: (self, uri, transport, encoding, verbose: bool = ..., ...) pytype_reporter: Actually passed: (self, uri, transport, encoding, verbose: int, ...) pytype_reporter: Expected: (self, uri, transport, encoding, verbose, allow_none: bool = ..., ...) pytype_reporter: Actually passed: (self, uri, transport, encoding, verbose, allow_none: int) ``` Signed-off-by: Edwin Török <[email protected]>
edwintorok
force-pushed
the
private/edvint/xenapi
branch
from
68112fb to
81fb164
Compare
pytype claims that there is no 'errno' in the socket module, but there is: ``` Python 3.6.8 (default, Nov 16 2020, 16:55:22) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import socket >>> socket.errno.ETIMEDOUT 110 ``` Also `session.local_logout` is a remote method that is proxied by `__getattr__`, I don't know why `pytype` complains about that one, and not about `logout`, but suppress it. Signed-off-by: Edwin Török <[email protected]>
Signed-off-by: Edwin Török <[email protected]>
…ication This is already the case for HTTP(S) connections, but XenAPI overrides the xmlrpclib implementation to provide a Unix socket transport, and it lacks this optimization. Otherwise we've noticed that SM makes at least 4 separate connections to XAPI to query the version number for example. On a busy system each of those connections can get delayed by 100ms: * OCaml's tick thread switches threads every 50ms, and you might need 2 switches when XAPI is busy with CPU-bound workloads: * one to accept the connection * another to handle the request on the newly spawned thread See: https://github.com/python/cpython/blob/v2.7.5/Lib/xmlrpclib.py#L1361-L1371 https://github.com/python/cpython/blob/v3.6.8/Lib/xmlrpc/client.py#L1237-L1245 Signed-off-by: Edwin Török <[email protected]>
Old version of XenAPI.py used to look at the API version to implement a compatibility layer, but that code got dropped and the API version is now completely unused: df9b539 ("CA-35286: Remove forward compatability code from the python XenAPI module in favour of the compatability layer in xapi") Remove the unused argument, this will allow us to avoid making 4 additional API calls after every login. Signed-off-by: Edwin Török <[email protected]>
The API version is not used anymore, so remove the 4 additional queries it'd make. For backward compatibility retain the field as a cached property: should a client of this code want to access the API version, then it'll make the 4 queries at that time, and cache the result for the duration of the API object. Cached properties are writable, so the fallback that writes the API version to 1.1 should keep working too Signed-off-by: Edwin Török <[email protected]>
|
I've fixed the |
pytype_reporter extracted 47 problem reports from pytype output. You can check the results of the job here |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
On a Unix domain socket (used by SM in Dom0 for example) we lacked the persistent connection optimisation, so we created a new connection for each API call, thus querying the version was done in 4 API calls and 4 separate connections
(with a single login thankfully).
Looking more deeply calling
_get_api_version()is not even needed, it was only ever used by some compat code that got dropped in 2010 (but the version queries remained).For backwards compatibility I kept the possibility to query the api version, but wrapped it in a cached property, and avoid calling it by default.
This shaves off 4 API calls after each login (i.e. every SM script call), and is in general useful (even for HTTP(S) clients).
I've only done some minimal testing on this PR, so a draft for now. Will report back when some wider testing and performance numbers are available.