Payload Parser - Timestamp in arrived data

#1

Hi,

I’m using the Payload Parser to extract data from the binary array that I sent from my devices.

Most of my devices sent only one buffer that contains chunks of data that include each one a timestamp and the some variables, like this:

<timestamp><varx><vary><varz><timestamp><varx><vary><varz>…

Everything is binary and I’am already parsing the first chunk correctly. I don’t know exactly how to extract all the chunks, and use each timestamp as the actual timestamp in the bucket…

Is there a way to do this?

Thanks!

#2

Hi @bruno,
Could you share your code here?

#3

Sure:

const ignore_vars = [ 'payload' ];

function parsePayload(payload) {
  try {
    const buffer = Buffer.from(payload,'hex');
    const data = [
      { variable: 'id', value: buffer.readInt8(1) },
      { variable: 'timestamp',  value: buffer.readUInt32LE(2) },
      { variable: 'battery',  value: buffer.readUInt16LE(6) / 100 },
      { variable: 'temperature',  value: buffer.readInt16LE(8) / 100 },
      { variable: 'humidity',  value: buffer.readUInt16LE(10) / 100 },
      { variable: 'pulses',  value: buffer.readUInt16LE(12) },
    ];
    return data;

  } catch (e) {
    console.log(e);
    return [{ variable: 'parse_error', value: e.message }];
  }
}

const payload_raw = payload.find(x => x.variable === 'payload_raw' || x.variable === 'payload' || x.variable === 'data');
if (payload_raw) {
  const { value, serie, time } = payload_raw;
  if (value) {
    payload = payload.concat(parsePayload(value).map(x => ({ ...x, serie, time: x.time || time })));
    payload = payload.filter(x => !ignore_vars.includes(x.variable));
  }
}

Basically the code is getting only the first chunk, I need to get a variable amount of chunks.
And each chunk has its own timestamp.

#4

Hi, @bruno .

Is it ok for you to share the buffer that you are receiving from your device? It will be easier for me to help you if I could understand the way your data is arriving.

One thing that can help you is passing the time attribute for each chunk of data.
You just need to add it as you are doing with variable and value.

Thank you.

#5

Hi @gclenz,

Yes, this is the log of one of the latest arrived data:

14:09:25:
[MQTT] Device publish
{ "topic": "x", "payload":     "6d0068643d5fcc021a0b771400003b6d00a4643d5fce021a0b601400003b6d00e0643d5fcf02390b501400003b6d001c653d5fcf023d0bbb1300003b6d0058653d5fd002260b111400003b6d0094653d5fd1023d0b2d1400003b6d00d0653d5fd1023a0bea1300003b6d000c663d5fd102370be31300003b6d0048663d5fd2023c0be61300003b6d0084663d5fd202430b571400003b6d00c0663d5fd202540ba21300003b6d00fc663d5fd2025f0b8f1300003b6d0038673d5fd3026d0b721300003b6d0074673d5fd302740b6c1300003b6d00b0673d5fd3027a0b101300003b6d00ec673d5fd3027d0b851300003b6d0028683d5fd302880b121300003b6d0064683d5fd302820b381300003b6d00a0683d5fd302820b311300003b6d00dc683d5fd302820bf41200003b6d0018693d5fd402700b0e1300003b6d0054693d5fd402780b621300003b6d0090693d5fd402780bda1200003b6d00cc693d5fd402620bf01200003b6d00086a3d5fd402600bd91200003b6d00446a3d5fd4027b0b0a1300003b6d00806a3d5fd402920bd11200003b6d00bc6a3d5fd4029c0ba31200003b6d00f86a3d5fd4028d0b911200003b", "qos": 0 }

The chunks are fixed size, begin with 0x6D and ends with 0x3B.

I’m already passing the ‘timestamp’ attribute and is being recorded in the database, but I would like to use it as the time that is already recorded in Tago, and not as a separate column/variable in the database… Is this that you are suggesting?

#6

Hi, @bruno

Yes, that’s what I am suggesting.
You can pass a ‘time’ attribute like this:

const data = [
  { variable: 'id', value: buffer.readInt8(1), time: buffer.insertHere },
  { variable: 'timestamp',  value: buffer.readUInt32LE(2), time: buffer.insertHere },
  { variable: 'battery',  value: buffer.readUInt16LE(6) / 100, time: buffer.insertHere },
  { variable: 'temperature',  value: buffer.readInt16LE(8) / 100, time: buffer.insertHere },
  { variable: 'humidity',  value: buffer.readUInt16LE(10) / 100, time: buffer.insertHere },
  { variable: 'pulses',  value: buffer.readUInt16LE(12), time: buffer.insertHere },
];

Let me know if this solves your question.

#7

Hi @gclenz,

Seems like the correct timestamp is appearing right on the log, but I’m not able to see the data in the points where it should be…

I’ve changed the parser to:

function parsePayload(payload) {
  try {
    const buffer = Buffer.from(payload,'hex');
    const t = buffer.readUInt32LE(2);
    console.log(t);
    const data = [
      { variable: 'id', value: buffer.readInt8(1), time: t },
      { variable: 'battery',  value: buffer.readUInt16LE(6) / 100, time: t },
      { variable: 'temperature',  value: buffer.readInt16LE(8) / 100, time: t },
      { variable: 'humidity',  value: buffer.readUInt16LE(10) / 100, time: t },
      { variable: 'pulses',  value: buffer.readUInt16LE(12), time: t },
    ];
    return data;

If I already have a data point for time 14:00:00, and send a data point witch time 13:30:00 for example, this point in the past would appear normally in the widgets?

#8

Hi @gclenz,

The timestamp that I send doesn’t include milliseconds, I needed to multiply by 1000 to get it right. That’s why I wasn’t seeing the data, the points were going to 1970…

Part of the problem solved!

One more thing, what would be the best way to implement the parser to process all the chunks? A ‘for’ loop inside the function would impact too much the performance? The arrived buffers will not exceed 1500 bytes.

#9

Hi, @bruno.

Yes, you can use a for loop inside a function.

I did an example so you can use it as a starting point. I used the payload that you sent me.

const buffer = Buffer.from(payload, 'hex')
const total_bytes = payload.length / 2;
const time = buffer.readUInt32LE(2)

const data_array = [];

for (let index = total_bytes; index > 0; index -= 9) {
  const item_from_buffer = payload.substr(total_bytes - index, 18);
  const item_hex_to_dec = Buffer.from(item_from_buffer, 'hex')
  console.log(item_hex_to_dec)

  data_array.push([
   { variable: 'id', value: item_hex_to_dec.readInt8(0, 1), time },
   { variable: 'battery',  value: item_hex_to_dec.readUInt16LE(1, 3) / 100, time },
   { variable: 'temperature',  value: item_hex_to_dec.readInt16LE(4, 4) / 100, time },
   { variable: 'humidity',  value: item_hex_to_dec.readUInt16LE(5, 1) / 100, time },
   { variable: 'pulses',  value: item_hex_to_dec.readUInt16LE(6, 12), time },
  ])
}

console.log(data_array)

I hope that this will give you a north.

Let me know if this can help you.

#10

Hi @gclenz,

I did a bit different:

var data= [];
    for( i = 0; i < Buffer.byteLength(buffer); i += 15 ) {
      t = buffer.readUInt32LE(i + 2) * 1000;
      const m = [
        { variable: 'id', value: buffer.readInt8(i + 1), time: t },
        { variable: 'battery',  value: buffer.readUInt16LE(i + 6) / 100, time: t },
        { variable: 'temperature',  value: buffer.readInt16LE(i + 8) / 100, time: t },
        { variable: 'humidity',  value: buffer.readUInt16LE(i + 10) / 100, time: t },
        { variable: 'pulses',  value: buffer.readUInt16LE(i + 12), time: t },
      ];
      data = data.concat(m);
    }
    return data;

It seems that everything is working now, I’ll add some sanity checks to it and watch closely if something wrong happens…

Thank you for your support!

#11

Hi, @bruno

Great to know that you got what you wanted.
Perhaps, you did it in a more elegant way than I, hahaha.

Thank you.

1 Like
#12

As a low-level programmer, it is a bit desperating dealing with these untyped variables in javascript… haha but at least is working