Bulk Upload Setup Tutorial using CSV Files

General Tutorials
1

This tutorial will guide you through setting up an Analysis and Dashboard on the TagoIO Platform to enable bulk uploading of CSV files for creating multiple devices simultaneously.

By following these steps, you will achieve:

  • A Dashboard to upload CSV files.
  • An Analysis that handles device creation.
  • A Dummy Device to work with dashboard widgets and store error information.

1700×990 49.3 KB

Prerequisites

  • Access to the TagoIO Platform.
  • Basic understanding of TagoIO components: Devices, Analysis, Actions, Dashboards.

Step 1: Create a Dummy Device

A device is required to store your data and interact with the dashboard widgets.

  1. Navigate to Devices:

  2. Add a New Device:

    • Click the “Add device” button located at the top right corner.

  3. Select Connector:

    • Choose the “Database” connector and click on it.

  4. Name Your Device:

    • Provide a name for your device and click Create.

  5. Device Confirmation:

    • You will be redirected to the device’s detail page.

Step 2: Setting Up the Analysis

The Analysis processes the CSV data to create devices automatically.

  1. Access the Analysis Template:

    a. Visit Analysis Template.

  2. Configure the Analysis:

    a. Clone the Template: Click on “Install Template” to clone the code to your own Analysis.

Step 3: Configuring Access Policies

Instead of using a Profile Token, you will now configure Access Policies to grant necessary permissions for device creation.

  1. Navigate to Access:

    • Click on the Access button in the sidebar.

  2. Add a New Policy:

    • Click the “+ Add Policy” button.

  3. Select the Target:

    • Choose your imported Analysis as the target for the policy.

  4. Set Permissions:

    • Add a new Permission for Devices with the following rules:

      • Access
      • Create
      • Edit

    • Add a new Permission for Services with the following rules:

      • Create Profile Notification

    • Set Field to Any

  5. Save the Policy:

    a. Click Save to apply the new access policy.

1702×1180 163 KB

Step 4: Setting up the Dashboard

The Dashboard provides a user interface to upload CSV files and manage devices.

  1. Access the Dashboard Template:

  2. Clone the Dashboard:

    • Click on “Install Template” to create your own Dashboard.

  3. Select the Device:

    • Choose the device you created earlier when prompted.

  4. Select the Analysis:

    • Choose the Analysis you imported earlier when prompted.

  5. Save Configuration:

    • Save your changes to finalize the Dashboard setup

Step 5: Preparing the CSV File

Your CSV file should contain the necessary information to create devices. An example file is available for download.

  1. Download Example CSV:

  2. CSV Structure:

    • Device Name: The name of the device.

    • Device Serial: Represents the EUI or ID, depending on your network. Do not change the CSV header.

    • Data Retention Value: Specify as forever, 1 day, 1 month, or 1 year.

  3. Customize Your CSV:

    • Populate the CSV with your device details following the structure above.

Example:

devicename,deviceserial,devicetype,chunkperiod,chunkretention
MyDevice,123,mutable,,
MyDevice2,123,immutable,day,2
MyDevice3,123,immutable,week,5
MyDevice4,123,immutable,month,1
MyDevice5,123,immutable,quarter,4

Using the dashboard

Interact with the Dashboard to upload your CSV and create devices.

  1. Access the Dashboard:

    a. Open your configured Dashboard on TagoIO.

  2. Download CSV (Optional):

    a. Use the provided link to download the example CSV if needed.

  3. Upload CSV File:

    a. Follow the on-screen instructions within the Dashboard to obtain your network and connector ID.

  4. Create Devices:

    a. Ensure you are uploading one type of device per CSV file (e.g., HTTP or MQTT).

    b. Press the “Create Devices” button to initiate the bulk upload.

  5. Verify Creation:

    a. Check for any errors returned by the widget as feedback.

    b. Confirm that the devices have been created successfully in your TagoIO account.

Troubleshooting

  • Access Issues: Ensure that your Access Policies are correctly configured to allow device creation.

  • CSV Errors: Verify that your CSV file follows the required structure and contains all necessary fields.

  • Widget Configuration: Double-check that the widget is correctly linked to the Analysis and that the Analysis has the appropriate permissions.

For further assistance, feel free to comment on this topic!

1 Like
2

@Andreas Gudmundsson

Thank you for this Vitor, very well documented and exactly what I’ve been looking for!

3

@Gary Howell

Hi @vitor.

I tried this script using Custom HTTPS device type, but nothing was reported in the error log and no devices were created. Any ideas where to check?

I was going to work through the analysis script as an exercise, but I notice that the script is weirdly formatted - snippet below. Is this essentially a compresses JS script from the tago builder - I have not seen this before? Interesting.

    /*!
     * @tago-builder
     * Tago (https://tago.io/)
     * Tago Builder V2.2.3 (https://git.io/vhEW5)
     * 
     * Generated at: 1565299425350 (Thu Aug 08 2019 18:23:45 GMT-0300 (Brasilia Standard Time))
     * Machine     : Vitors-MacBook-Pro.local - darwin
     * 
     */
    /******/ (function(modules) { // webpackBootstrap
    /******/ 	// The module cache
    /******/ 	var installedModules = {};
    /******/
    /******/ 	// The require function
    /******/ 	function __webpack_require__(moduleId) {
4

@Douglas Abreu

Hello @vitor!
Is it possible to update this example to include tags for each device in CSV file (preferably) or by dashboard?
Thanks in advance!

5

@Vitor Lima

Yes, that is an analysis compressed by tago-builder.

6

@Vitor Lima

I can look at that once I have time, but I can’t guarantee it will be soon.

If you want to do your own changes to the code, it is available in the github tago-io/analysis-example-bulkupload (github.com)

As it does make use of the json-2-csv library, you will be required to use the analysis-builder in order to upload it to TagoIO analysis.

Running Analysis as External using Node.JS - TagoIO

7

@Christos Mourouzis

Hello,

I have followed this post in order to create a bulk of devices. Following these instructions I was able to successfully create many devices using the csv template from the Dashboard.

What I would like to do is to create also devices with a csv containing many tags. I tried to change the code (uncomment and edit the tags line (line 107 of the template analysis script):



eval("/\* eslint-disable no-loop-func _/\\n/_ eslint-disable no-await-in-loop _/\\nconst { Analysis, Account, Utils, Device } = **webpack\_require**(/_! /sdk _/ “/sdk”);\\nconst axios = **webpack\_require**(/_! axios _/ “./node\_modules/axios/index.js”);\\nconst json2csv = **webpack\_require**(/_! json-2-csv \*/ “./node\_modules/json-2-csv/src/converter.js”);\\n\\nfunction validation(validation\_var, device, show\_markdown) {\\n return function \_(message, type) {\\n if (!message || !type) throw ‘Missing message or type’;\\n device.sendData({\\n variable: validation\_var,\\n value: message,\\n metadata: {\\n type: \[‘success’, ‘danger’, ‘warning’\].includes(type) ? type : null,\\n color: !\[‘success’, ‘danger’, ‘warning’\].includes(type) ? type : null,\\n show\_markdown,\\n },\\n });\\n\\n return message;\\n };\\n}\\n\\nasync function getDevice(account, device\_id) {\\n const customer\_token = await Utils.getTokenByName(account, device\_id);\\n const customer\_dev = new Device({ token: customer\_token });\\n return customer\_dev;\\n}\\n\\n// Use Axios package to get the file from the URL.\\nasync function getCSV(url) {\\n const options = {\\n method: ‘GET’,\\n url,\\n };\\n\\n const { data } = await axios(options);\\n const json = await json2csv.csv2jsonPromisified(data);\\n\\n // just making sure template is a boolean.\\n return json.map(x => ({ …x, template: x.template === ‘true’ }));\\n}\\n\\nasync function createDevices(context, account = new Account(), validate, device = new Device(), { connector\_id, network\_id, origin }, csv\_json = \[\]) {\\n const ignore\_serial = \[‘5bbd0d144051a50034cd19fb’, ‘5bbd0de44051a50034cd19fc’\];\\n validate(‘Starting to create the devices, it can take several minutes depending how much devices in the CSV file.’, ‘success’);\\n const itemList = csv\_json.slice(0, 20);\\n const nextList = csv\_json.slice(20);\\n if (nextList.length) {\\n setTimeout(() => {\\n account.analysis.run(context.analysis\_id, \[{ nextList, connector\_id, network\_id, origin }\]);\\n }, 40000);\\n }\\n\\n let errors = 0;\\n for (const \[index, dev\_line\] of itemList.entries()) {\\n // Create the device.\\n context.log(`[Error] Line ${dev_line.key1}: ${dev_line.tag1}` );\\n const result = await account.devices.create({\\n name: dev\_line.devicename,\\n serie\_number: !ignore\_serial.includes(network\_id) ? dev\_line.deviceserial : null,\\n connector: connector\_id,\\n network: network\_id,\\n **tags: \[ { key: dev\_line.key1, value: dev\_line.tag1 }, { key: dev\_line.key2, value: dev\_line.tag2 }, { key: dev\_line.key3, value: dev\_line.tag3 }, { key: dev\_line.key4, value: dev\_line.tag4 }, { key: dev\_line.key5, value: dev\_line.tag5 }, { key: dev\_line.key6, value: dev\_line.tag6 }, { key: dev\_line.key7, value: dev\_line.tag7 }, { key: dev\_line.key8, value: dev\_line.tag8},\],\\n** }).then((r) => { context.log(`Line ${index}: ${dev_line.devicename} succesfully created.`); return r; })\\n .catch(async (e) => {\\n context.log(`[Error] Line ${index}: ${dev_line.devicename} ${e}.`);\\n errors += 1;\\n await device.sendData({ variable: ‘log\_message’, value: `Error: ${dev_line.devicename} - ${dev_line.deviceserial}, message: ${e}`, metadata: { color: ‘pink’ } });\\n });\\n\\n if (result.bucket\_id && dev\_line.dataretention) {\\n account.buckets.edit(result.bucket\_id, { data\_retention: dev\_line.dataretention });\\n }\\n\\n if (index === itemList.length - 1) {\\n await device.sendData({ variable: ‘log\_message’, value: `${itemList.length - errors} device(s) created, ${errors} number of errors`, metadata: { color: errors ? ‘yellow’ : ‘lightgreen’ } });\\n if (!nextList.length) {\\n device.sendData({ variable: ‘log\_message’, value: ‘No more devices to be created from the CSV file.’, metadata: { color: ‘blue’ } });\\n }\\n }\\n }\\n}\\n\\n// Starting function for the analysis\\nasync function initAnalysis(context, scope) {\\n // Get the environment variables from TagoIO Analysis.\\n const env\_vars = Utils.envToJson(context.environment);\\n if (!env\_vars.account\_token) throw context.log(‘Missing account\_token in Environment Variables’);\\n else if (env\_vars.account\_token.length !== 36) return context.log(‘Invalid “account\_token” in the environment variable’);\\n\\n // create the Tago Account object.\\n const account = new Account({ token: env\_vars.account\_token });\\n const device = await getDevice(account, scope\[0\].origin);\\n\\n const validate = validation(‘bulk\_upload\_validation’, device);\\n\\n if (scope\[0\].nextList) {\\n return createDevices(context, account, validate, device, { connector\_id: scope\[0\].connector\_id, network\_id: scope\[0\].network\_id, origin: scope\[0\].origin }, scope\[0\].nextList);\\n }\\n\\n validate(‘Reading the CSV File, please wait…’, ‘warning’);\\n\\n // Get the variables from the Form.\\n const file = scope.find(x => x.variable === ‘csv\_file’ && x.metadata);\\n if (!file) throw validate(‘No file to upload. Please upload a file to csv\_file variable’, ‘danger’);\\n\\n let network\_id = scope.find(x => x.variable === ‘network\_id’);\\n if (!network\_id) throw validate(‘You must enter a Network ID’, ‘danger’);\\n network\_id = network\_id.value;\\n\\n let connector\_id = scope.find(x => x.variable === ‘connector\_id’);\\n if (!connector\_id) throw validate(‘You must enter a Connector ID’, ‘danger’);\\n connector\_id = connector\_id.value;\\n\\n // Check if connector and network are valids.\\n const connector = await account.integration.connectors.info(connector\_id).catch(() => null);\\n if (!connector) {\\n throw validate(‘Invalid Connector ID’, ‘danger’);\\n }\\n\\n const network = await account.integration.networks.info(network\_id).catch(() => null);\\n if (!network) {\\n throw validate(‘Invalid Network ID’, ‘danger’);\\n }\\n\\n const csv\_json = await getCSV(file.metadata.file.url).catch((e) => { throw validate(e.message || JSON.stringify(e), ‘danger’); });\\n\\n const issue\_index = csv\_json.findIndex(dev\_line => !dev\_line.devicename || !dev\_line.deviceserial);\\n if (issue\_index !== -1) {\\n const dev\_line = csv\_json\[issue\_index\];\\n if (!dev\_line.devicename) throw validate(`Missing device name, line ${issue_index}`, ‘danger’);\\n if (!dev\_line.deviceserial) throw validate(`Missing device serial, line ${issue_index}`, ‘danger’);\\n if (dev\_line.dataretention) {\\n const retention = String(dev\_line.dataretention).toLowerCase().split(’ ');\\n if (retention\[0\] !== ‘forever’ && Number.isNaN(Number(retention\[0\]))) throw validate(`Invalid data retention, line ${issue_index}`, ‘danger’);\\n if (!\[‘day’, ‘week’, ‘year’, ‘month’\].includes(retention\[1\]) && retention\[0\] !== ‘forever’) throw validate(`Invalid data retention, line ${issue_index}`, ‘danger’);\\n }\\n }\\n\\n await createDevices(context, account, validate, device, { connector\_id, network\_id, origin: scope\[0\].origin }, csv\_json);\\n}\\n\\nmodule.exports = new Analysis(initAnalysis, { token: ‘798a369f-1438-491c-8a49-677c682ee40e’ });\\n\\n\\n//# sourceURL=webpack:///./csvUpload/deviceBatch.js?");

but when I try to upload a csv file with columns as:



devicename, deviceserial, dataretention, key1, tag1, key2, tag2, key3, tag3, key4, tag4, key5, tag5, key6, tag6, key7, tag7, key8, tag8

I get an “Invalid Tag” error in the template Dashboard log.

Is there an example of bulk creating devices also with tags?

Thank you very much.

Christos

8

@Vitor Lima

From an Analysis perspective, it does like everything is correct.

Would be good if you can console.log the tags, so you can check exactly what you’re sending to TagoIO.
Invalid Device Tags in your case probably means that a key or a value is empty, or it is not a string.

Maybe casting everything to a string can fix the problem for you, example:

 [ { key: String(dev_line.key1), value: String(dev_line.tag1) }, ...]
9

@Christos Mourouzis

Hello Vitor,

Yeah that works. Converting everything to String did the job.

Thank you very much.

Christos

10

@CĂĄdmo Dias

Dear Vitor,

I was able to add Tags editing the code, but is there a way to add Configuration Parameters too, please?

As did with Tags, I tried to do this by configuration_parameters: [ { key: 'my_key', value: 'my_value' }], but unfortunetally it didn’t work.

Thank you!

11

@Fabiano Eger

hi @diascadmo,

You are missing a field on the configuration parameters object.

Look:
[ { key: 'my_key', value: 'my_value', sent: false }]

sent is a boolean required field on configuration parameters.

12

@CĂĄdmo Dias

Thank you for your answer!

It does make sense. Unfortunately, still any configuration parameters objects are being set.

My code:
configuration_parameters: [ { key: 'my_key', value: 'my_value', sent: false }],\n tags: [ { key: 'myTagKey', value: 'myTagValue' }],\n

13

@CĂĄdmo Dias

By the way, I just found configuration_params refering to the configuration parameters object in the documentation, but it still didn’t work.

14

@Fabiano Eger

hi @diascadmo,

You can’t create a param for the device directly.

You need to have a device created and then you can create a param for this device.
see this cURL example:

curl --location --request POST 'https://api.tago.io/device/YOUR_DEVICE_ID_HERE/params' \
--header 'Content-Type: application/json' \
--header 'token: YOUR_ACCOUNT_TOKEN_HERE' \
--data-raw '{
    "key": "param_key",
    "value": "param_value",
    "sent": false
}'
15

@CĂĄdmo Dias

I understand, but how would you implement this cURL example, please?

For better understanding and in case there are better suggestions, I’m trying to generate, via code, the configuration parameters so that in a list of devices I can click on a device’s link and then change the displayed data of that blueprint device (such as this example). I can do this manually, but as I will have many devices, I needed to automate this task.

Thank you again for you attention!

16

@Vitor Lima

The code for analysis is described here Devices | TagoIO SDK for JavaScript and TypeScript

The usage is pretty simple:
account.devices.paramSet(device_id, { key, value, sent })

You need to repeat for every parameter you want to add. If need to edit, you just need to include the parameter id in the JSON, with value being the parameter id you want to edit.

17

@CĂĄdmo Dias

It finally worked! Thank you @eger and @vitor.

18

@CĂĄdmo Dias

Dear @vitor,

Is there a newer version of this code or is the 2019 version of GitHub the latest, please?

Thank you!

19

@Vitor Lima

Yes, the code is the most updated version.

Until now, there is no relevant update to be done in the script. Would be good to update the SDK version for the package, but it won’t actually change anything in the code itself.

20

@CĂĄdmo Dias

I see. I ask because I tried to run it and found some errors such as missing deviceimei and I also didn’t find the part of the script that deals with the network and the connector. I will take a closer look, thanks for your attention as always!