MY_MQTT_CLIENT_PUBLISH_RETAIN only works for I_BATTERY_LEVEL

[sburlot]

I have a RPi MySensors MQTT Gateway and a mosquitto broker: I use it to display the values received on a dashboard.
But since the gateway doesn't set the retain flag, my dashboard must wait for the sensors to send updated values. I save the last values received, but I was wondering why there isn't a possibility to set the retain flag on all messages.

[mfalkvidd]

Only retaining specific messages is by design, so the title of this issue is a bit misleading.

Perhaps add support for a new macro called MY_MQTT_CLIENT_PUBLISH_RETAIN_ALL ? Or does it need to be more granular?
What are the effects on a broker if all messages are being retained? Is there a clean-up mechanism? (I am not familiar with how retained messages work)

Edit: seems like brokers should not be affected very much, but clients might not be able to handle receiving many messages when connecting (using a wildcard subscribe for example). http://www.hivemq.com/blog/mqtt-essentials-part-8-retained-messages

[mannkind]

There's a good possibility something changed since PR #793 (I haven't kept up to date since the PR was accepted), but for what it's worth, it was originally written to support retaining incoming sensor values and battery levels -- C_SET and C_INTERNAL w/type=I_BATTERY_LEVEL.

I'd hazard a guess that you don't really want the gateway to set the retain flag on all messages; you might discover a lot of unintended consequences (I know that I did!)

[mfalkvidd]

@sburlot any thought on the scalability of retaining all types of messages? Are all messages what you want, or should it be possible to specify certain types of messages? Is there a reason your controller is unable to provide the values through its api?

[sburlot]

@mfalkvidd I have no information about the scalability of retaining all messages. I've googled and didnt find any information against this practice.
I moved from a serial gateway to an MQTT gateway to use the retain and publish/subscribe features, so I'm not dependent on one controller.
I built a dashboard that displays values received from the gateway directly, and I don't rely on the controller anymore. My dashboard stores the last values, but if it disconnects or die, it will display the last values received but I want to be able to get the last values sent by the gateway, without waiting for the sensors to send new values.

A cleanup for retained value can be done by sending a topic with a zero length payload, and the gateway should do it after some timeout (ie a sensor has not sent data recently).

Of course it's a complex issue, some sensors dont send informations at regular interval (a button/switch for example). Best way would be that each sensor sends a config to the gateway and ask him to retain or not the value, with an optional timeout.

Easiest way for my purpose is to put an MQTT client on the gateway machine, and this client will forward the MQTT received and alter the retain flag for some topics. And manage the optional timeout.

[mfalkvidd]

Thanks for your input!

I agree that letting a MQTT client (nodered for example) handle setting the retain would be an easy way. I'm not sure about how easy clearing of the messages is though. That would require keeping all used topics stored somewhere, otherwise it wouldn't be possible to know which topics to clear. MySensor gateways are designed to be stateless, so starting to store stuff will be a big change.

[Yveaux]

As far as I know mosquitto has no option to batch delete retained messages. As a workaround you can publish empty messages to retained topics to clear them.

[clclon]

There is no point in removing retain messages, each next message overwrites the previous one. If we consider the functionality of the timeout, then this function can be given to the controller. For example, if the host is not available for an hour, send a zero value to the host topic, or at your discretion ..