How to build an MQTT Payload Parser

How to build an MQTT Payload Parser

@Vitor Lima

As MQTT has the freedom of sending data in the format you want, you will need to normalize the data to the TagoIO format by using the Payload Parser.

This tutorial expects you to have read the In-depth guide to Payload Parser - How to - TagoIO Community.

Of course, you can skip all this tutorial by making sure you’ve already sent in the correct format, which is a json array with at least the parameters variable and value. Check out more about it here: Sending Data - TagoIO.


But if this is not your case, I’ll try to provide some guidance on the most common use cases.



Publishing hexadecimal payload only.

If you came to just publish a hexadecimal, such as “001100”, make sure that you’re sending it with the quotation marks so it doesn’t get interpreted as a number, the content of your payload parameter will be like:

  1. [{ "variable": "payload", "value": "001100", "metadata": { "mqtt_topic": "data" } } ]

I recommend you start with this code, it will transform it to a Buffer and you can use all the knowledge you got from this tutorial to parse it.

  1. /* This is a generic payload parser that can be used as a starting point for MQTT devices ** The code expects to receive a string hexadecimal data, and not JSON formatted data. ** ** Testing: ** You can do manual tests to the parse by using the Device Emulator. Copy and Paste the following JSON: ** [{ "variable": "payload", "value": "0109611395", "metadata": { "mqtt_topic": "data" } } ] */ // Prevents the code from running for other types data insertions. // We search for a variable name payload or a variable with metadata.mqtt_topic const mqtt_payload = payload.find((data) => data.variable === "payload" || (data.metadata && data.metadata.mqtt_topic)); if (mqtt_payload) { // Cast the hexadecimal to a buffer. const buffer = Buffer.from(mqtt_payload.value, 'hex') // Normalize the data to TagoIO format. // We use Number function to cast number values, so we can use it on chart widgets, etc. const data = [ { variable: 'protocol_version', value: buffer.readInt8(0) }, { variable: 'temperature', value: buffer.readInt16BE(1) / 100, unit: '°C' }, { variable: 'humidity', value: buffer.readUInt16BE(3) / 100, unit: '%' }, ]; // This will concat the content sent by your device with the content generated in this payload parser. // It also adds the field "serie" to be able to group in tables and other widgets. const serie = String(new Date().getTime()); payload = payload.concat(data).map(x => ({ ...x, serie })); }



Publishing a comma separated payload.

Comma separated values will require you to first split your string in an array, and then start to pick your values from the new generated array. Is it good to remember that an Array in Javascript starts from index 0.
If your data does have change on it’s length, you will need to resolve this problem by using IF’s or a For function.

  1. /* This is a generic payload parser that can be used as a starting point MQTT devices ** The code expects to receive comma separated data, and not JSON formatted data. ** ** Testing: ** Testing: ** You can do manual tests to the parse by using the Device Emulator. Copy and Paste the following JSON: ** [{ "variable": "payload", "value": "temp,12,hum,50", "metadata": { "mqtt_topic": "data" } } ] */ // Prevents the code from running for other types data insertions. // We search for a variable name payload or a variable with metadata.mqtt_topic const mqtt_payload = payload.find((data) => data.variable === "payload" || (data.metadata && data.metadata.mqtt_topic)); if (mqtt_payload) { // Split the content by the separator , const splitted_value = mqtt_payload.value.split(','); // splitted_value content will be ['temp', '12', 'hum', '50'] // index starts from 0 // Normalize the data to TagoIO format. // We use Number function to cast number values, so we can use it on chart widgets, etc. const data = [ { variable: 'temperature', value: Number(splitted_value[1]), unit: '°C' }, { variable: 'humidity', value: Number(splitted_value[3]), unit: '%' }, ]; // This will concat the content sent by your device with the content generated in this payload parser. // It also adds the field "serie" to be able to group in tables and other widgets. const serie = String(new Date().getTime()); payload = payload.concat(data).map(x => ({ ...x, serie })); }



Publishing a JSON payload, not TagoIO data format.

In this case we recommend you start with the snippet “Convert Raw JSON to TagoIO JSON”. It has all the comments you need and a quick function to transform the JSON for you.



Common mistakes

  • Payload Parser is not running

If you’re sending data through your MQTT device, but it’s not being stored and neither running your payload parser, it can be that you didn’t setup an Action for your device.


It looks like that:

You can either go to your Actions page and create an Action of type MQTT for your device, and set the action to Stored data to bucket, or change your topic to tago/data/post, as it will have the same effect. You can read the tutorial in the following link MQTT - TagoIO.


At the end, your Live Inspector will look like this:

  • Can’t display my variable in charts and some widgets

This is a very common mistake where you send a variable as a string and not a number. Data type does matter when using widgets on TagoIO.
A value of a variable can be a boolean, a number or a string. Inside the Payload Parser, it’s very common for data to be received as a string by default when using MQTT. You can easily fix that problem by casting your value to a number, as follows:

  1. const data = [ { variable: 'temperature', value: Number(splitted_value[1]), unit: '°C' }, { variable: 'humidity', value: Number(splitted_value[3]), unit: '%' }, ];

Instead of

  1. const data = [ { variable: 'temperature', value: splitted_value[1], unit: '°C' }, { variable: 'humidity', value: splitted_value[3], unit: '%' }, ];