Zephyr PM APIs confusion

[fabiobaltieri]

Introduction

Hi, I've been working with few chips using PM and PM_DEVICE recently and found some lack of clarity in the PM APIs and how/when they should be used.

Problem description

There's three main PM config options that control the system behavior, let's see if I got them right:

• PM: enables CPU power states, basically a callback that says "hey CPU, I won't need you for a while, go to sleep"
• PM_DEVICE: controls whether the device suspend/resume APIs are available and especially whether those xxx_pm_action driver callback are built or not
• PM_DEVICE_RUNTIME: adds reference counters on the stuff above, with a different API, if the device enables the runtime features either with pm_device_runtime_enable or zephyr,pm-device-runtime-auto
• PM_DEVICE_SYSTEM_MANAGED: enables the stuff in subsys/pm/device_system_managed.c, which basically means that now the system calls suspend and resume on all devices every time the CPU goes to sleep for too long, unless they are busy or flagged for pm runtime operation

So if you are running on a system that is using PM and PM_DEVICE, the default setting is PM_DEVICE_SYSTEM_MANAGED=y which means that all of your devices implementing PM are going to be suspended and resume continuously as the CPU goes to sleep and resume. I guess this would be fine if the PM device API were meant to be used for that reason, like preparing drivers to wake up the CPU early if needed, but the problem is that this is not how they are used all the time.

If one does a quick grep for PM_DEVICE_DT_INST_DEFINE in drivers/ there's a lot of usage of this for devices external of the CPU, displays, flash memories, GNSS, LED controllers, sensors, let's ignore all the input ones that I did while (clearly) not understanding what these were for. All of these clearly don't want to be suspended every time the CPU sleeps for a few milliseconds, but that's what happens, it's the same API. You may argue it's API misuse, but then if the device callback were meant to be hard bound to the CPU going in and out of sleep, why would PM_DEVICE be decoupled from PM?

Now one can disable PM_DEVICE_SYSTEM_MANAGED, but now some of the drivers that do want that callback when the CPU suspends are not going to get it anymore, and there's no way out.

So one can use PM_DEVICE_RUNTIME instead and ensure to all the devices that are NOT bound to the CPU have runtime PM enabled and are controlled by the application, but then this disables PM_DEVICE_SYSTEM_MANAGED by default... why?!

Now, PM_DEVICE_RUNTIME, how's that supposed to be used? There's load of drivers calling pm_device_runtime_enable on themselves so that they use the runtime API automatically once enabled, but then that means that once a driver gets that call introduced it changes from being active by default to suspended by default and MUST be enabled manually by the application, which inevitably causes a regression downstream. So maybe it should not be used in the driver? And we should rely on zephyr,pm-device-runtime-auto or the application calling pm_device_runtime_enable on the device? It seems to make more sense, but then how does one handle pm_device_init_suspended (which is often paired with self pm_device_runtime_enable).

Proposed change

These APIs and use cases needs to be clear as a start, right now it seems like there's a bit of a mess in the code base, the PM callback are used with different scope.

The whole PM_DEVICE_SYSTEM_MANAGED story seems to be the immediate problem, at least it is to me right now. I think that this should not be a single option that enables/disables cpu-device hooks, maybe it should be a PM_DEVICE_FLAG_ bit? PM_DEVICE_FLAG_SYSTEM_MANAGED or something?

Looking for initial feedback on this, trying to confirm if there's something I'm missing.

[fabiobaltieri]

cc @ceolin @nashif @teburd @tmleman @JordanYates @gmarull @carlescufi @keith-zephyr @bjarki-andreasen

[bjarki-andreasen]

Since I discovered this function pm_device_driver_init, at least from a device driver perspective it is pretty straight forward, just include #include <zephyr/pm/device.h> and implement the pm_action callback implementing ON and RESUME, while hiding the SUSPEND and OFF actions behind #ifdef CONFIG_PM_DEVICE. Run pm_device_driver_init() as the last thing before leaving the drivers init function. No idea if this is documented anywhere, I just listened to a talk on YouTube :)

[gmarull]

A partial attempt to split PM_DEVICE from PM_DEVICE_RUNTIME was made a while ago #62160. PM_DEVICE should be ditched entirely, it's just nonsense. We should also consider introducing a "PM v2", and deprecate "PM v1".

[fabiobaltieri]

A partial attempt to split PM_DEVICE from PM_DEVICE_RUNTIME was made a while ago #62160. PM_DEVICE should be ditched entirely, it's just nonsense. We should also consider introducing a "PM v2", and deprecate "PM v1".

Interesting, one of the point I wrote and then dropped was asking whether DEVICE and DEVICE_RUNTIME really make sense in the way they are currently implemented. So what was the idea? PM_DEVICE for CPU bound and PM_DEVICE_RUNTIME for application controlled?

[gmarull]

A partial attempt to split PM_DEVICE from PM_DEVICE_RUNTIME was made a while ago #62160. PM_DEVICE should be ditched entirely, it's just nonsense. We should also consider introducing a "PM v2", and deprecate "PM v1".

Interesting, one of the point I wrote and then dropped was asking whether DEVICE and DEVICE_RUNTIME really make sense in the way they are currently implemented. So what was the idea? PM_DEVICE for CPU bound and PM_DEVICE_RUNTIME for application controlled?

PM_DEVICE -> suspend/resume devices when CPU goes to idle (ie, nonsense the way it's done)
PM_DEVICE_RUNTIME -> suspend/resume devices based on consumer needs

They share code, but for no good reasons IMHO, it makes things confusing (and incompatible, how can the same callback be shared for both cases?)

[bjarki-andreasen]

PM_DEVICE from my understanding is just the resume/suspend hooks for devices? and PM_DEVICE_RUNTIME tracks usage, it is entirely "above" PM_DEVICE I believe. Maybe the source code paints a different story

[ceolin]

PM_DEVICE from my understanding is just the resume/suspend hooks for devices? and PM_DEVICE_RUNTIME tracks usage, it is entirely "above" PM_DEVICE I believe. Maybe the source code paints a different story

That is correct, PM_DEVICE is about enable the callback hook in devices. PM_DEVICE_SYSTEM_MANAGED and PM_DEVICE_RUNTIME control how device power management happens in devices.

[ceolin]

PM_DEVICE -> suspend/resume devices when CPU goes to idle (ie, nonsense the way it's done)
PM_DEVICE_RUNTIME -> suspend/resume devices based on consumer needs

That is simply wrong. PM_DEVICE_SYSTEM_MANAGED is the option that suspend and resume devices. Also this behavior can be tuned, shallow power states can decide to not trigger power management action in devices using zephyr,pm-device-disabled

They share code, but for no good reasons IMHO, it makes things confusing (and incompatible, how can the same callback be shared for both cases?)

The callback is for responding to an action (suspend / resume / off ).. Why can't it be shared ?

[ceolin]

Introduction

Hi, I've been working with few chips using PM and PM_DEVICE recently and found some lack of clarity in the PM APIs and how/when they should be used.

Problem description

There's three main PM config options that control the system behavior, let's see if I got them right:

• PM: enables CPU power states, basically a callback that says "hey CPU, I won't need you for a while, go to sleep"
• PM_DEVICE: controls whether the device suspend/resume APIs are available and especially whether those xxx_pm_action driver callback are built or not
• PM_DEVICE_RUNTIME: adds reference counters on the stuff above, with a different API, if the device enables the runtime features either with pm_device_runtime_enable or zephyr,pm-device-runtime-auto
• PM_DEVICE_SYSTEM_MANAGED: enables the stuff in subsys/pm/device_system_managed.c, which basically means that now the system calls suspend and resume on all devices every time the CPU goes to sleep for too long, unless they are busy or flagged for pm runtime operation

So if you are running on a system that is using PM and PM_DEVICE, the default setting is PM_DEVICE_SYSTEM_MANAGED=y which means that all of your devices implementing PM are going to be suspended and resume continuously as the CPU goes to sleep and resume. I guess this would be fine if the PM device API were meant to be used for that reason, like preparing drivers to wake up the CPU early if needed, but the problem is that this is not how they are used all the time.

If one does a quick grep for PM_DEVICE_DT_INST_DEFINE in drivers/ there's a lot of usage of this for devices external of the CPU, displays, flash memories, GNSS, LED controllers, sensors, let's ignore all the input ones that I did while (clearly) not understanding what these were for. All of these clearly don't want to be suspended every time the CPU sleeps for a few milliseconds, but that's what happens, it's the same API. You may argue it's API misuse, but then if the device callback were meant to be hard bound to the CPU going in and out of sleep, why would PM_DEVICE be decoupled from PM?

Now one can disable PM_DEVICE_SYSTEM_MANAGED, but now some of the drivers that do want that callback when the CPU suspends are not going to get it anymore, and there's no way out.

So one can use PM_DEVICE_RUNTIME instead and ensure to all the devices that are NOT bound to the CPU have runtime PM enabled and are controlled by the application, but then this disables PM_DEVICE_SYSTEM_MANAGED by default... why?!

This option was introduced to get rid of the atrocity that was PM_DEVICE_RUNTIME_EXCLUSIVE that was enabled by default when PM_DEVICE_RUNTIME was enabled and to facilitate the migration was asked to keep the old behavior, so this option is disabled when PM_DEVICE_RUNTIME is enabled.

The idea is having direct option to select which behavior you want.

• PM_DEVICE_SYSTEM_MANAGED -> Devices that are not doing device runtime pm are suspended when the SoC gets idle (and the power state used has not disabled it)
• PM_DEVICE_RUNTIME -> suspend / resume devices at runtime and not bound to system idle.

PM_DEVICE -> Enable the basic infra-struct, it is used by both behaviors.

This one we may remove or make a hidden option to avoid confusion and simplify things.

Now, PM_DEVICE_RUNTIME, how's that supposed to be used? There's load of drivers calling pm_device_runtime_enable on themselves so that they use the runtime API automatically once enabled, but then that means that once a driver gets that call introduced it changes from being active by default to suspended by default and MUST be enabled manually by the application, which inevitably causes a regression downstream. So maybe it should not be used in the driver? And we should rely on zephyr,pm-device-runtime-auto or the application calling pm_device_runtime_enable on the device? It seems to make more sense, but then how does one handle pm_device_init_suspended (which is often paired with self pm_device_runtime_enable).

Proposed change

These APIs and use cases needs to be clear as a start, right now it seems like there's a bit of a mess in the code base, the PM callback are used with different scope.

The whole PM_DEVICE_SYSTEM_MANAGED story seems to be the immediate problem, at least it is to me right now. I think that this should not be a single option that enables/disables cpu-device hooks, maybe it should be a PM_DEVICE_FLAG_ bit? PM_DEVICE_FLAG_SYSTEM_MANAGED or something?

Per device ? Per specific power states ?

Looking for initial feedback on this, trying to confirm if there's something I'm missing.

[ceolin]

A partial attempt to split PM_DEVICE from PM_DEVICE_RUNTIME was made a while ago #62160. PM_DEVICE should be ditched entirely, it's just nonsense. We should also consider introducing a "PM v2", and deprecate "PM v1".

There is nothing blocking the code to change. The problem was not ditching PM_DEVICE the problem was that you had tried to completely remove the support for suspend/resume devices when the system is idle (what is now controlled by PM_DEVICE_SYSTEM_MANAGED).

[henrikbrixandersen]

No matter how this all fits together, I think a good step would be to add more documentation on how the different options/behaviors are a) intended to be used by an application developer and b) intended to be implemented by the various driver types (on-chip/off-chip peripherals, SoCs).