Overview
The Payload Parser is a data transformation tool in TagoIO that processes incoming device data before it’s stored in your device’s bucket. It acts as an intermediary layer that can decode, normalize, and modify data from IoT devices, ensuring it’s in the correct format for storage and analysis.
When IoT devices send data to TagoIO, they often transmit it in formats that aren’t immediately usable—such as hexadecimal strings, raw bytes, or proprietary encoding schemes. A Payload Parser is a JavaScript function that transforms this raw data into TagoIO’s standardized JSON format, making it accessible for dashboards, analytics, and other platform features.
Supported Operations
- Data Decoding: Convert hexadecimal payloads and byte arrays into readable values
- Format Normalization: Transform any data structure into TagoIO’s standard JSON format
- Value Conversion: Perform calculations, unit conversions, and data type transformations
- Protocol Support: Works with any communication protocol (LoRaWAN, Sigfox, MQTT, HTTP)
- Data Filtering: Remove unwanted variables before storage
- Variable Addition: Create new variables based on calculations or static values
- Device Context Access: Read device tags and configuration parameters during parsing
Technical Constraints
- No Historical Data Access: Cannot retrieve previously stored data from the device’s bucket
- Resource Isolation: Cannot access other TagoIO resources (dashboards, actions, other devices)
- Execution Time Limit: Code must complete within 2 seconds
- No External Requests: Cannot make HTTP requests or connect to external services
- Synchronous Only: No support for asynchronous JavaScript operations
How Payload Parsers Work
Device → Network Server → Payload Parser → Bucket Storage
When data arrives at your device:
- The network server (LoRaWAN, Sigfox, etc.) normalizes the data if applicable
- Your Payload Parser receives the normalized data
- The parser transforms the data according to your code
- Only the parser’s output is stored in the bucket
Execution Context
The Payload Parser runs as an evaluated function with two parameters:
// Your code runs inside this context:
function payloadParser(payload, device) {
// payload: Array of data sent to the device
// device: Object containing device metadata
// Your transformation code here
payload = [] // Replaces the Payload to return the transformed data
}
Getting Started
Enabling Payload Parser
- Navigate to your device in the TagoIO platform
- Select the Payload Parser tab
- Toggle the feature to Enable
- Choose a template from the snippet dropdown menu:
- Parse example for LoRaWAN: For LoRaWAN devices
- Parse example for Sigfox: For Sigfox devices
- Convert Raw JSON to TagoIO JSON: For HTTP devices sending JSON data
- Parse Example for MQTT Hexadecimal: If you’re sending a hexadecimal using MQTT
Code Structure
// Available packages (pre-imported)
// - dayjs: Lightweight date library
// - timeUtils: TagoIO proprietary library to handle timezones
// Function parameters
// payload: Array of incoming data objects
// device: Object with device metadata
// - device.tags: Array of device tags [{key, value}]
// - device.params: Array of config parameters [{key, value, sent}]
// Example: Filter unwanted variables
payload = payload.filter(item => item.variable !== 'unwanted_var');
// Payload is automatically returned to the system with the new data packet
Working with Binary Data
Hexadecimal to Buffer Conversion
Most IoT sensors send data as hexadecimal strings. Convert them to buffers for easier manipulation:
const payload_raw = payload.find(x => x.variable === 'payload');
const buffer = Buffer.from(payload_raw.value, 'hex');
// Example hex: "08FB383F"
// Buffer array: [0x08, 0xFB, 0x38, 0x3F]
// Decimal values: [8, 251, 56, 63]
Buffer Reading Methods
There are some methods available that help us when reading bytes from a buffer:
| Method | Description | Bytes Read | Endianness |
|---|---|---|---|
buffer.readInt8(offset) |
Signed 8-bit integer | 1 | N/A |
buffer.readUInt8(offset) |
Unsigned 8-bit integer | 1 | N/A |
buffer.readInt16BE(offset) |
Signed 16-bit integer | 2 | Big Endian |
buffer.readInt16LE(offset) |
Signed 16-bit integer | 2 | Little Endian |
buffer.readUInt16BE(offset) |
Unsigned 16-bit integer | 2 | Big Endian |
buffer.readUInt16LE(offset) |
Unsigned 16-bit integer | 2 | Little Endian |
buffer.readInt32BE(offset) |
Signed 32-bit integer | 4 | Big Endian |
buffer.readInt32LE(offset) |
Signed 32-bit integer | 4 | Little Endian |
buffer.readUInt32BE(offset) |
Unsigned 32-bit integer | 4 | Big Endian |
buffer.readUInt32LE(offset) |
Unsigned 32-bit integer | 4 | Little Endian |
Bitwise Operations
Extract specific bits using masks:
// Extract lower 4 bits
const lowerNibble = buffer\[0\] & 0x0F;
// Extract upper 4 bits
const upperNibble = (buffer\[0\] & 0xF0) >> 4;
// Check if specific bit is set
const isBitSet = (buffer\[0\] & 0x80) !== 0;
Debugging Tools
An important step when running your payload parser script is to be able to test and debug it for any issue. In your device’s page, you should also see the Emulator and the Live Inspector section.
Live Inspector
The Live Inspector provides real-time visibility into data flow:
- Incoming data: Raw payloads received by the device
- Parser execution: Console output and errors
- Stored data: Final data written to the bucket
- MQTT events: Connection status and publishes
By using a console.log inside your payload parser, you can print content of variables and arithmetic results when the payload parser runs for any data that being stored to your device. This is very helpful when trying to understand why you’re receiving errors or unwanted results.
console.log('Raw payload:', payload);
console.log('Device tags:', device.tags);
console.log('Parsed temperature:', temperature);
Emulator
The Emulator allows you to test your parser without sending actual device data:
- Copy a raw payload from Live Inspector or craft test data
- Paste into the Emulator input field
- Execute to see parser results
Example emulator inputs:
// LoRaWAN format
[
{ "variable": "payload", "value": "001100" },
{ "variable": "port", "value": 100 }
]
// MQTT format
[
{
"variable": "payload",
"value": "0109611395",
"metadata": { "mqtt_topic": "data" }
}
]
Complete Example
Here’s a practical parser for a temperature/humidity sensor:
// Find the payload in incoming data
const payload_raw = payload.find(x =>
x.variable === 'payload_raw' ||
x.variable === 'payload' ||
x.variable === 'data'
);
if (payload_raw) {
try {
// Convert hex to buffer
const buffer = Buffer.from(payload_raw.value, 'hex');
// Decode according to device protocol:
// Byte 0: Protocol version
// Bytes 1-2: Temperature (int16, big-endian, scaled by 100)
// Bytes 3-4: Humidity (uint16, big-endian, scaled by 100)
const data = [
{
variable: 'protocol_version',
value: buffer.readInt8(0)
},
{
variable: 'temperature',
value: buffer.readInt16BE(1) / 100,
unit: '°C'
},
{
variable: 'humidity',
value: buffer.readUInt16BE(3) / 100,
unit: '%'
}
];
// Preserve metadata from original payload
payload = payload.concat(
data.map(x => ({
...x,
group: String(payload_raw.serie || payload_raw.group),
time: String(payload_raw.time)
}))
);
} catch (e) {
console.error('Parser error:', e);
payload = [{ variable: 'parse_error', value: e.message }];
}
}
Scaling with Connectors
What are Connectors?
Connectors are reusable Payload Parser templates that can be applied to multiple devices. Instead of copying parser code to each device, you create a connector once and reference it from any compatible device.
Creating a Connector
- Navigate to Integrations in your account menu
- Click Create Connector
- Select supported networks (LoRaWAN, Sigfox, MQTT, etc.)
- Paste your tested Payload Parser code
- Add documentation and metadata (optional)
- Save the connector
Benefits
- Centralized Updates: Modify parser logic in one place
- Consistency: Ensure all devices use the same parsing logic
- Sharing: Make connectors public for community use
- Documentation: Include setup instructions and examples
More information regarding the connectors can be found at Connector Overview - TagoIO