Feature Request: MQTT trigger

[wsw70]

Note: the original tile of the question was "[question] how to run a always-on script?"

I am converting my automation from AppDaemon to pyscript and some of them depend on MQTT (as a client). A typical case is a switch emitting to MQTT topic I want to listen to and act accordingly.

This means that such a script is not triggered by anything but runs in the background, responds to some events it manages itself (MQTT messages in my case), and then contacts HA for some actions (switching on a light for instance).

Is it possible to run such a script? The only cases I saw in the documentation are ones that react to something (either time based, or coming from HA)

One solution would be to use HA to manage the MQTT messages, but there are problems with this in some cases (see https://community.home-assistant.io/t/how-to-get-the-name-of-the-actual-state-topic-in-an-mqtt-sensor/248541 for an example). Using HA is doable but I would prefer to avoid automations there and move everything to pyscript for consistency.

[dlashua]

You can use a @time_trigger of startup to indicate that the script should run immediately. Then, as long as the logic inside of that script never stops, it will continue to run.

However, if you set up a Home Assistant automation that subscribes to the topic and then fires an event with the topic and payload in it, you can easily use @event_trigger to process these and don't need the script to be running "all the time".

Something like:

- alias: MQTT Messages
  trigger:
    - platform: mqtt
      topic: mytopic/#
  action:
    - event: MQTT_MESSAGE
      event_data:
        topic: "{{ trigger.topic }}"
        payload: "{{ trigger.payload }}"

@event_trigger('MQTT_MESSAGE')
def mqtt_message(**data):
  log.info(f"Got an MQTT MESSAGE: {data}")

[wsw70]

Thank you for the quick and very detailed answer!

Since I finally (after giving a deeper thought) use some automations within HA (the simple ones), I will go for your second solution.

EDIT: it works great - thank you very much once again!

[dlashua]

@craigbarratt in order to prevent the need for the Home Assistant automation in this scenario, a @mqtt_trigger in pyscript would be awesome. Any thoughts about this?

[craigbarratt]

I don't know anything about mqtt. But it does seem popular and useful, so adding some native trigger support sounds like a great idea. Where should I start to learn about mqtt? I presume pyscript should natively trigger off mqtt messages (as you propose), and also be able to generate mqtt messages?

[wsw70]

@craigbarratt

First of all - thank you for pyscript. It is wonderful.

MQTT is a bus to which you connect to either send messages to a topic (a topic would be for instance home/garden/watering and the data on - data is a string), or you subscribe to one to get what was published there.

It is asynchronous and you can request some messages to be persistent (so when you subscribe to a topic, you get the last one - there is no stack).

I use it extensively in my microservices to get some data, and push them so that other services can make use of it. For instance I check Internet access and push it to a topic. If there is a problem, I have Home Assistant listening to it, but also a home made dashboard that will display an alert, and a monitoring system that sends notifications etc.

It is also typically used by IoTs to send their status, actions performed on them etc.

So this is a many-to-many distribution system (several programs can register to a topic to send to it, independently of others, but typically there is one source and several listeners).

It is implemented in Python via paho mqtt

[dlashua]

@craigbarratt since pyscript is entirely within Home Assistant, most of the work is already done for you. You don't need to worry about connecting to an MQTT server or how to publish payloads.

Put VERY simply, MQTT is a message queue. Messages are published to topics. And other processes subscribe to those topics to receive the messages.

Topics use "/" as a separator. Topic SUBSCRIPTIONS can contain two wildcard characters. "+" means one level and can appear anywhere. "#" means multiple levels and can only appear at the end.

Nothing is ever as simple as it seems, but, in Home Assistant, I think the basics for listening to MQTT messages looks a lot like @event_trigger. This untested code could be a starting point.

from homeassistant.components import mqtt

def message_handler(mqttmsg):
    """Listen for MQTT messages."""
    func_vars = {
        "topic": mqttmsg.topic,
        "payload": mqttmsg.payload,
        "qos": mqttmsg.qos,
    }

    try:
        func_vars["payload_json"] = json.loads(mqttmsg.payload)
    except ValueError:
        pass

    call_user_defined_method_here(func_vars)


remove = await mqtt.async_subscribe(
    hass, topic, message_handler, encoding=encoding, qos=qos
)

hass is the hass object you already have.

topic should be specified (and required) in @mqtt_trigger.

encoding and qos can be supplied in @mqtt_trigger though should have sensible defaults, "utf-8" and 0, respectively.

A second argument to @mqtt_trigger could work like @event_trigger does and allow you to make comparisons on payload, payload_json and topic. Because the topic can contain wildcards, being able to compare the topic is useful.

remove() should be called to destroy the subscription when the trigger function is destroyed.

Example user code might look like this:

@mqtt_trigger('test/#', 'payload == "ON" and topic.endswith("/set")', qos=2)
def dothething(topic=None, payload=None):
  # some code to turn the thing on
  log.info(f"{topic} is now {payload}")

This would match on topic test/mydevice/mysensor/set, for instance, and report that it is now ON.

[wsw70]

Put VERY simply, MQTT is a message queue.

For the sake of completeness, the MQTT protocol only supports one message per topic, new messages override the previous one (there is no stack). This is in case where persistence is enabled, otherwise messages are transient and if someone is not listening at the time it is sent, the message is lost.

There are other implementations of such buses (RabbitMQ for instance) where the history of messages is kept (and a listener can retrieve all of them for a given topic)

@dlashua thank you for the code, such a trigger would indeed be very useful. I have to learn someday async/await in Python (I use threading extensively but this is apparently old school).

[craigbarratt]

@wsw70 and @dlashua - thanks for the tutorial - that's really helpful.

I presume we'd also want some way of sending mqtt messages? Is there a service that does that already, and/or would there be more compact and expressive ways to do that?

[wsw70]

Is there a service that does that already,

Yes: mqtt.publish (https://www.home-assistant.io/docs/mqtt/service/)

[dlashua]

@craigbarratt another possible "gotcha" is that some people don't have MQTT established in Home Assistant. I have no idea what happens if you call mqtt.async_subscribe() without first setting up MQTT. Hopefully something easy to detect so pyscript can raise an error saying "um... maybe add mqtt as an integration first?".

MQTT has other features too (for instance, if you don't request a "clean_session" when you connect, MQTT will remember messages that you previously subscribed to and send them to you when you reconnect even if they aren't "retained"). This makes it really useful for not-constantly-connected IoT devices and such to be able to save battery and only check periodically for new messages.

But, Home Assistant handles all of those pieces already. All pyscript needs to do is allow subscriptions and allow publishing (which it already does because of the service call).

I can take a shot at a PR, if you'd like. But, I'll be copy-pasting the event_trigger bits and crossing my fingers that I did it right since I still haven't managed to work out how you implemented pseudo-decorators in pyscript. :) Though, in performing he copy-paste, I might actually finally figure it out.

[dlashua]

Yes: mqtt.publish (https://www.home-assistant.io/docs/mqtt/service/)

The only improvement I can think of compared to HASS's mqtt.publish deals with JSON.

Lots of MQTT consumers use JSON to format messages. Doing this (the hard way) means:

mqtt.publish(topic="mytopic", payload='{"a":1, "b":2}')

Since pyscript has python-power, we can do this instead:

import json

payload = {"a":1, "b";2}
payload_json = json.dumps(payload)
mqtt.publish(topic='mytopic', payload=payload_json)

But, it would be nice if mqtt.publish could take a payload_json kwarg and autoconvert it for us, thus eliminating the need for the json import, therefore allowing:

mqtt.publish(topic='mytopic', payload_json={"a":1, "b":2})

[wsw70]

I have updated the title of the question because it is not really relevant to the discussion after @craigbarratt reopened it.

[wsw70]

Thank you for the addition of this trigger - this is wonderful.

How is it going to appear in HA? At the next version? Should I forcefully bring in the new version (and if so - how?)

[craigbarratt]

If you installed pyscript using HACS, go to HACS -> Integrations -> pyscript and select "Reinstall" from the menu. Then instead of 1.0.0, select the master version.

[dlashua]

@wsw70 you will likely be the first real user of this feature, as I'm not currently using it except in my test cases. So, if you see any issues, please open an issue right away so we can get them all cleaned up before it lands in a release.

[wsw70]

@dlashua I just tested it for # and was flooded with messages → good :)

I will rewrite my switch automation tonight (when everyone is asleep as I test in production :)) and will raise issues if there are any.

The ability to have the actual topic when using wildcards is wonderful, thank you again for that!

[wsw70]

@dlashua @craigbarratt

I could not resist and rewrote the automations in the evening (I am in France), and everything works perfectly with my Zigbee and RF433 devices (all of them ultimately communicate via MQTT).

I am keeping an eye on my monitoring and the log console just in case but I think the code is top-notch. Thank you very much once again.