Mqtt Message Broker

Question

Anyone out there playing around with the MQTT messaging system for the Internet of Things? Just wondering if anyone else is connecting 'other' things to their 'Bot and Builder software.... - @tameion

Rich · Answer

All of my bots and all of my devices (that's everything from TV to washing machine to kettle) connect to each other. Everything can be aware of the status of everything else. Everything logs everything to a MySQL Database held on one of my servers, this gives the ability for everything else to look up whatever it needs to look up. My robots can know what's on TV, what's just been added to my media library, what's coming up soon, if lights are on or off, doors and windows open or closed, occupancy detected in rooms and outside, if the phone is ringing, if I missed a call, if I received a text, even who rung/is ringing, texting and what the text says. However, my system is a complex mess of virtual wires all going back to a central point however some taking a few extra steps to get there. The only thing that's not connected yet is the new DVR for the CCTV since it's pretty new and not set up properly yet. While it's "on the network" nothing else checks it yet. My previous CCTV (which is still in place too) was fed in to everything and had many improvements over the factory state i.e. face detection and recognition, motion detection and tracking etc. And all of this is reported via a simple web page, individual web pages with more detailed information for specific "parts", critical information relayed in real time to mobile devices, data logged and plotted on graphs and charts and some more fun stuff dynamically displayed as images (however the code needs optimising as they take a while to load the data)...   Yes I know my garden is overgrown... What is the MQTT messaging thing? Do you have a link? I'm always interested in seeing what else I can add in to the mess that is Jarvis

CochranRobotics · Answer

Message queuing I have used it before to file claims at federal courts for all of the major banks except for a few at a prior job. I haven't messed with it for about 4 years now though and haven't added it to any robots. It would be a cool addition though. Basically, it would allow you to place things into a centralized queue by multiple computers/robots. This queue would then be able to be processed in a first in/first out type process. I could see this used for things like multiple robots vacuuming a school based on messages in the queue. This can be done with a database. It is just more efficient to do with message queues, which frees your database to worry about storing data instead of a process queue. It is almost impossible to query what is coming up 5 steps down the queue, but if you don't need to know this, it's great. You could log all of the messages to a database while being added to the queue and then mark these database entries as complete after these tasks are completed, but that really defeats the benefits of using a message queue.

Tameion · Answer

Hi Rich.... long time no chat.... you'll remember me from other conversations about using ARC, EZ-B's and Eventghost in home automation. I have been researching for a couple of years now looking for an in-between solution similar to http put/get and other similar methods of moving sensor data around the network. I've tried many and been frustrated with the extra workload implementation has required and longed for a more robust solution with some form of quality of service (QoS). So here is my informed but not by any means complete explanation .  Quoting the net... "MQTT (formerly Message Queue Telemetry Transport) is a publish-subscribe based "light weight" messaging protocol for use on top of the TCP/IP protocol. It is designed for connections with remote locations where a "small code footprint" is required and/or network bandwidth is limited" And my bit: MQTT is a very small payload messaging protocol used in the background by big players like Facebook's Messenger and small players like the growing number of Internet of Things devices. It started life as a means of communicating in simple language, very quickly, with low powered devices like satellites, etc, - without a lot of extra baggage while maintaining quality of service level communication. It is the increasingly popular communication protocol for the internet of things (I.o.T.) and a plethora of devices of varying power and ability. There are three parts for MQTT.  a) the broker c) the message b) the client  The broker  acts like the server of any good RSS feed. It can be locally based inside your network or it can be on the www. It can be hosted on a Raspberry Pi or any other computer. The broker publishes, stores and keeps as many feeds as you would like to set up... unlimited. Everything from "Home\Lounge\camera\A" to "Home\master_bedroom\his_stuff\riches_heated_underwear" The Message  can be anything you like, in any format you like. as long as it is inside speech marks as below. The format is simple: mosquitto_pub -d -t hello/world -m "Hello, MQTT. This is my first message." It breaks down as Broker = mosquitto_pub Message rules = -d -t Feed = hello/world Type = -m Message ="Hello, MQTT. This is my first message." The client  is any I.o.T hardware device or software application. The client subscribes to the brokers feed and can then be told to change state according to the information it receives.... what's even more importantly is that it can also publish back to that same feed any change of state. Eg. ARC could potentially post to the feed the "Home\Garage\Roller Door\status" feed the message "Open". Now the fun starts.... ANY OTHER device also 'subscribed' to that particular feed gets immediately updated. Sensors no longer need to be one way or dumb devices. With MQTT you can have the change of state posted to the broker and the broker then post that change to all other subscribed devices. Brokers also have the ability to verify message delivery. It can deliver once and I don't care if you got it or not, message repeatedly until confirmation, or message with username and password for added security. A 'last will and testament' facility can be set for each feed that alerts any subscriber if the feed, device or application fails to respond.... ie. "It's dead Jim!" The beauty of it all becomes apparent when you realise that any device could subscribe to any number of feeds at any point in the fees tree. Eg. The home server might monitor the "Home\Garage" segment of the feed and receive all updates from all devices listed below that feed tree. Eg. "Home\Garage\window", "Home\Garage\sweeperbot", etc. While at the same time the sweeper robot only subscribes to the "Home\Garage\sweeperbot" feed for its instructions As long as your message can be parsed than you're good to go. Excellent article for further reading Getting Started with MQTT

Tameion · Answer

BTW Rich is that a Rainmeter Skin I see? Have you thought of calling the Rainmeter Bangs from within EZ_Builder and EventGhost? See screenshot of my set-up. The Rainmeter Meters are shown or hidden according to the status of various devices on my network. Mostly simple toggles from Eventghost but some from ARC... I have dual screens on a closeted pc acting as my homeserver. It runs ARC and connects to four EZ-B v3's. It acts as my home A.I. and is affectionately called V.I.K.I... lol (My wife says I spend more time with Viki than with her!)  Here is a closeup of the screens populated by a mixture of Rainmeter Skinz at boot up time  After sensors status has been checked and updated  It mostly used the Ping Alive/dead in Eventghost and the Xbox and Car Meters are permanent as the Xbox will not respond to a ping request and the Cars are an upcoming project. Screen shot one on the LH screen shows the internet is down while the R/H screen shows all networkable devices like computers, phones, routers, media servers, etc. Screen shot two on the LH screen shows the internet is now up. The R/H screen shows an unrecognised device on the network and all other known devices currently active and connected.

Rich · Answer

Not rainmeter, well the overlay may have been based/taken from a rainmeter skin but it doesn't use rainmeter, it's overlaid via PHP on to the image, the bars on the dials represent data too which dynamically update (however one is mains voltage and the other is USP charge, both are pretty much constant 24/7). The overlay is still work in progress and to be honest, may be scrapped since my CCTV system has changed from ZoneMinder to a dedicated 16 channel DVR with internet access which uses RTSP rather than jpg images. From within my network I have much more than those two images posted however external access has never been intended so I cannot show them short of a screen grab. I'll have to have a look in to MQTT and see if it'll improve what I have set up. Since Jarvis is about to have a huge overhaul (mainly moving hardware around and optimising the many different linux and windows boxes he uses) now is the time to slip in some major adjustments since everything needs re-writing or re-configuring anyway.

Tameion · Answer

I am confident that MQTT will be be a viable solution to the growing expanse of the Internet of Rich.... Here is an interesting read about the connectivity of I.o.T's which may help link all things together... harizanov.com/2014/09/mqtt-topic-tree-structure-improvements/ Anyway... Rainmeter is worth a look. See: rainmeter.net/ Main points are that you can use any image to overlay the screen at any level.... Each image (called a Meter) can me refreshed as often as you like and manipulated via a simple notepad script or from the command line. Manual is very informative too. docs.rainmeter.net/ Best of all you can call the commands from within ARC using the Exec command ... and it's FREE!

CochranRobotics · Answer

Message queueing has been around for a long time. The information available back to you about where you are in the queue has been the limitation that I have experienced with it. This will give you an idea of what I have used it for. Large banks would send xml files for bankruptcy claims against their customers. We would place messages into a queue as we read these xml files and validated these claims. We would then use a bank of computers wait for messages to be delivered and file the claim at the court that was provided in the message. We also pulled documents from the courts on a daily basis. We would use a bank of computers to process these also by using a message queue in the same way I just mentioned. We processed hundreds of thousands of items a day. It was effective because we didn't care about where we were in the queue very often. We also set things up to do batches of items by having a message represent 1000 items for processing. It operates off of a store and forward method of communication. First in, first out. As mentioned, it is very light weight which is a huge benefit especially when processing across slow connection speeds. If the queue dies, it can kill the messages. That's not good when it happens, but the queue can be reloaded and you can be processing again. There are some limitations with it which keep it light weight, which is its strength.

Tameion · Answer

If you want to connect an MQTT brokerage system to ARC so that you can send/receive messages from the Internet of Things (IoT) you'll need Python and the Mosquitto brokerage installed so this is not a project for the faint hearted but I've managed to get most everything talking to most everything else.... I use EventGhost to monitor activity in the background and manipulate my computer screens but you don't need to use it if you don't want to... you can send messages directly to the MQTT brokerage via the EXEC command in ARC and vice-versa via the Telnet interface. Anyway for those using Eventghost there is a link below this paragraph to the latest version of the Eventghost plugin ... (The website is a little unclear and you can accidentally install an older version) sto.hopto.org/Release/MQTT%20Client/Source/ The installation blurb with links to Python and Mosquitto is here... www.eventghost.org/forum/viewtopic.php?f=9&t=5728&hilit=mqtt&sid=3aae1f1169ad7ac6b5da348410f5f4b6#p28279 I've also written a bit of a help sheet for those who want to set a Raspberry Pi as the broker to broadcast to you robot enabled home. If you're keen let me know and I'll post a link. Chris Forsberg has a good video series about the IoT as a primer for those interested. See www.youtube.com/channel/UCGPzNDa4wsR_98ayoqpxkUA - Enjoy

Tameion · Answer

@DJSures I know you watch pretty much everything that gets posted to the forum.... historically proven from my point of view anyway.

Here's a thought: (Rare these days...) Have you considered adding the post, publish and subscribe commands for the lightweight MQTT protocol into Builder?

It is looking like being the basis for much of the Internet of Things IoT in terms of message transportation. Refer to previous posts to this thread for explanation but I have a feeling you've already seen my rantings.... From what I can work out the functionality is similar to the telnet/console ... but I'm just an apprentice...!

But.... Imagine a universe where your EZ-Robot has kindly subscribed to the MQTT feed from your fridge telling you off for leaving the the door open! Or your Robot coming to the door when the wireless doorbell is pressed because it is subscribed to the feed for the doorbell.

This will add additional real world connectivity to any other MQTT connected IoT device.

@Rich Eventghost plugin author William Krambriw has just released an update... looks real good... python and it is all open source.

See latest post www.eventghost.net/forum/viewtopic.php?f=9&t=5728&start=15

Interesting to see/hear your thoughts.

Wayne @Tameion

Tameion

Upgrade to ARC Pro