The Payload Parser functionality at TagoIO is a very powerful tool that can be used in many ways. So, one way we can start this guide is by understanding what you can and can’t do by using it:

What you can do:

• It can decode the hexadecimal payload from your sensor.
• It can normalize data to the TagoIO format.
• It can convert values of variables.
• It can be used with any sensor and protocol.
• It can filter unwanted variables that you don’t want to store in your device’s bucket.
• It can add new variables other than what was sent by the device.
• It has access to device tags and device configuration parameters.

What you can’t do:

• Access data stored in your sensor in order to calculate values.
• Access any other type of TagoIO resource, such as dashboards, actions, etc.
• Have code that takes more than 2 seconds to run.
• Perform requests to external resources, such as HTTPs requests or MQTT requests.

With that in mind, the Payload Parser was built to perform operations in the data being added to your device’s bucket. It needs to be quick and to take into account only the information you have in the scope of what is being sent to the device at the moment the payload parser is running.

How to use it

Well, first thing you need to do is to go to your device, and then to the payload parser tab. Then you need to enable the feature for your device.

At the top of the editor, you will see a snippet dropdown menu. It has some code examples for you to start with.

• If you have a LoRaWAN device, you can select “Parse example for LoRaWAN”.
• If you have a Sigfox device, you can select “Parse example for Sigfox”.
• If you’re using an HTTP device and sending data to TagoIO in a JSON like format, you can select “Convert Raw JSON to TagoIO JSON”.
• You will not find any example for MQTT, as MQTT integrations is usually open to some freestyle on how you send your data. In this case, I advise you to read this guide and understand how to build your own parser.
  
  • If you want to have some guidance on how to send your MQTT Data, you can send it in the TagoIO JSON format.

In the Editor is where you want to write your code. So, let’s give you some introduction.

The Payload Parser is only available as Node.JS (Javascript). It is a very easy to understand language and you only need to understand the basics about the syntax and functions in order to start using it.

Here is a flowchart regarding showing how your data is processed when sent to TagoIO. It does have more steps, but I’ll simplify for now:

To summarize, your sensor sends the Uplink with all the information it wants to store in your device. It gets normalized by a Network server, if you’re using any, and then goes through your Payload Parser. In this step, the data can be transformed to whatever you want. Only the result of your Payload Parser is what will be stored.

Coding the Payload Parser

The code step is where you will write the content of your function. What that means is that the Payload Parser you write is actually inside a function that will be called when your data is sent to the device.

I added some comments to the following code to help you visualize it:

You can see that the payload parser includes some packages (lines 1-2-3) such as moment, dayjs and loraPacket, which you can use freely in your code.

You can also notice that the function payloadParser at line 5 receives two parameters: the payload parameter and the device parameter.

• payload: This parameter will contain anything that was sent to your device.
• device: This parameter will contain information regarding your device, such as its ID, configuration parameters, and tags.
  
  • device.tags: array of tags in your device, such as [{ key: “tag_key”, value: “tag_value”}]
  • device.params: array of configuration parameters of your device, such as [{ key: “param_key”, value: “param_value”, sent: false }]

For a list of all global and context parameters, check the documentation Context & global variables.

Line 7 and 8 is the content that you will be writing. You can see that we are filtering variables from the payload parameter, removing what we don’t want to be stored.

The most important is at line 10 as this function returns your payload parameter. This means that the content of this variable, whatever it is, is what TagoIO will try to store in the device’s bucket.

The content of the payload needs to be normalized to the TagoIO data format. It is a JSON of variable and values, and also other parameters if you want to include them, but they’re optional. Learn more in our documentation.

Regarding the code you’ll be writing in Javascript, there are some considerations that you need to have in mind:

• There is no support for Asynchronous functions, such as: forEach, setTimeout, setInterval…
• Use functions such as: for, map and filter instead.
• There is no support for external package importing.
• You can use “console.log” for debugging values. It will show up in the Live Inspector as we will see later in this tutorial.
• The code needs to run in at least 2 seconds. But this is more so for your information, as usually that is plenty of time for any decoder to run.

Working with hexadecimals and bytes

Usually when we talk about decoders, we are talking about hexadecimal values and bytes that are sent by the sensor. To work with that, Javascript supports some functions and bitmask to help us with decoding; we’ll take a close look on how it works.

If you got the LoRaWAN Parse example from the snippets, you will notice that it already has some code handling hexadecimals for you, and converting it to a Buffer in line 17.

It is good to notice that the TagoIO support for the Networks will always try to setup the content of your device’s data as hexadecimal. That means that if a network works with base64, TagoIO will convert it to hexadecimal for your payload parser. This only takes place for TagoIO official supported networks. If you do your own network integration, it probably will work the way you integrate it.

Moving back to the LoRaWAN example, it converts the payload to a buffer. The Buffer is an Array, starting from 0, with each position being one byte of your device’s data. For example:

Hex: "08FB383FFFFFFFFF"
Buffer: [08, FB, 38, 3F, FF, FF, FF, FF]

If you try to access the position 0 of the buffer, such as buffer[0] the value will be the decimal of your byte, in this case 8. Same goes for any position, buffer[1] results in 251.

There are some methods available that help us when reading bytes from a buffer:

• buffer.readInt8(0) will read the first byte
• buffer.readInt8(1) will read the second byte
• buffer.readInt16BE(0) will read the first and second byte, (16 bits = 2 bytes). Signed Int and Big Endian.
• buffer.readUInt16BE(0) will read the first and second byte, (16 bits = 2 bytes). Unsigned Int and Big Endian.
• buffer.readInt16LE(0) will read the first and second byte, (16 bits = 2 bytes). Signed Int and Little Endian.
• buffer.readUInt16LE(0) will read the first and second byte, (16 bits = 2 bytes). Unsigned Int and Little Endian.
• buffer.readInt32BE(0) will read from the first to the fourth byte, (32 bits = 4 bytes). Signed Int and Big Endian.
• buffer.readUInt32BE(0) will read from the first to the fourth byte, (32 bits = 4 bytes). Unsigned Int and Big Endian.
• buffer.readInt32LE(0) will read from the first to the fourth byte, (32 bits = 4 bytes). Signed Int and Little Endian.
• buffer.readUInt32LE(0) will read from the first to the fourth byte, (32 bits = 4 bytes). Unsigned Int and Little Endian.

You can see that we are sending 0 in the method parameter. It is actually the index from where the method will start to read the bytes. If you want to read 2 bytes starting from the third index, such as byte 38 and 3F from our previous example, we can run buffer.readInt16LE(2) for example.

Bitmask is also available. That means that you can do a AND, XOR and OR operators. And also unsign the byte by using a bitmask:

Here an example with an AND, and another with an OR operator.

• (buffer[0] & 0x0F) is equal to 8
• (buffer[0] | 0x0F) is equal to 15

Debugging with Live Inspector and Emulator

An important step when running your payload parser script is to be able to test and debug it for any issue. In your device’s page, you should also see the Emulator and the Live Inspector section.

Live Inspector

Any data that goes in or out of your device’s bucket will show up on the Live Inspector tab. You must click on the play button to start it, and it doesn’t store logs, which means that you can only see live data going here.

• You can see data that is being posted (stored) to your device.
• You can see data that is being pulled (get) from your device.
• You can see any console.log that is being used in your Payload Parser or Connector.
• You can see MQTT connections and disconnections.
• You can see MQTT publish from this device.
• You will see content of other steps of the data uplink, such as the raw value that was received, the parser step of a Network and its result, etc.

By using a console.log inside your payload parser, you can print content of variables and arithmetic results when the payload parser runs for any data that being stored to your device. This is very helpful when trying to understand why you’re receiving errors or unwanted results.

Example: console.log(device)

Will print to Live Inspector:

Emulator

The emulator is another way to send data to your device and emulate what your sensor would send. If you already know how your data will arrive, you can go ahead and start using it. Otherwise, you can send data through your sensor and copy the Raw Payload content of your Live Inspector and then use the emulator to send it again for your tests.

Usually, most of the networks and all LoRaWAN networks will send the payload and port as follow:

Example: [{ "variable": "payload", "value": "001100" }, { "variable": "port", "value": 100 }]

While MQTT will have its own way depending on how you’re sending data. If you’re not using the TagoIO data structure, it is common for it to be a variable called payload with some metadata on it.

Example:

[
    { "variable": "payload", "value": "0109611395", "metadata": { "mqtt_topic": "data" } }
]

Moving your Payload Parser to a Connector

Once you finish the Payload Parser for your sensor, and have it tested and working, you should be asking yourself if you will need to copy and paste the parser for each new sensor you install in your account.

To prevent that, and to also make it easier for anyone to update thousands of sensors with a payload parser once it’s needed, TagoIO created a new resource called Connector.

A connector is just a boilerplate that will contain your payload parser and run to any device that was created using it. It can also have a description, image and documentation sections if you want to use and make your connector public for others to use on their account.

You can create a connector by going to Integrations on the top left of your screen, after pressing on your profile name, or by accessing the following link: Integrations (tago.io)

To create one, just follow these steps:

1. Click the Create button.
2. Select the networks you expect this connector to support. You can have more than one and change it later.
3. Go to Payload Parser and paste the code you created for your device.
4. Press save and start using it. You can fill up the other fields if you want to, but as the connector is private, it’s your choice.

More information regarding the connectors can be found at Connector Overview - TagoIO

LoRaWAN and MQTT Examples

Now that you understand how the Payload Parser works in-depth, that should make it easier to understand the specific tutorials for LoRaWAN and MQTT that we have available.

Click on the links below to be redirected to the tutorials:

• LoRaWAN/Sigfox Payload Parser tutorial
• MQTT Payload Parser Tutorial