Data Pipeline overview

iWorkflow operates at a high level by passing data from one action to the other. Each action is allowed to access, manipulate, and append to the data object before passing it to the next action. This allows for complex workflow scenarios where each action can build on the one before it.

The following is an example of a workflow that can be built build, as well as what is happening on the data pipeline behind the scenes:

Action	Data Changes
Fetch IQA Data	IQA dataset is added.
Delta Hash	IQA dataset is filtered to only changed records.
Gate	Check the count of records - if count = 0, exit the workflow.
Select	Filter the IQA data to only include 2 columns, and add a third hardcoded column to every row.
Convert to CSV	Convert the IQA dataset to CSV format.
FTP Upload	Upload the contents of the CSV file to a remote FTP server.
HTTP Request	Post a notification in a Teams channel (via a webhook) that the upload was successful and contained n record(s).

Viewing the data on the pipeline

The data that is passed between each action is available for inspection on the workflow run screen. This allows you to see the data that exists between each action to ensure it is accurate.

Data and step sequencing

Step sequencing has an effect on the data pipeline. The current action’s input / context will always be the previous action or trigger that ran. This may not be the action immediately preceding the current one in terms of defined order in the workflow definition.

If your action is expecting certain data as input, be sure that any step sequencing settings are set correctly so that data always exists, or use an expression to check for it first.

Example

The following is a step-by-step example to examine what the data looks like after each action has run:

Fetch IQA Data - The IQA runs, and inserts a dataset onto the pipeline. Give the property a name. For example, “myDataset”.

{
  "myDataset": [
    {
      "userID": "user123",
      "name": "Alice Smith",
      "age": 30,
      "email": "alice.smith@example.com"    },
    {
      "userID": "user456",
      "name": "Bob Johnson",
      "age": 45,
      "email": "bob.johnson@example.com"    },
    {
      "userID": "user789",
      "name": "Charlie Brown",
      "age": 25,
      "email": "charlie.brown@example.com"    }
  ]
}

Delta Hash - Remove any records that were processed from a previous workflow run and have not changed. One record that was identified as unchanged was removed:

{
  "myDataset": [
    {
      "userID": "user123",
      "name": "Alice Smith",
      "age": 30,
      "email": "alice.smith@example.com"    },
    {
      "userID": "user456",
      "name": "Bob Johnson",
      "age": 45,
      "email": "bob.johnson@example.com"    }
  ]
}

Select - In this specific example, age is excluded so filter it out by the clicking Select. Input the dataset expression into the select statement as:

{{ input.myDataset | array }}

Set up our select columns like the following:

Name	Value
`userID`	`{{ this.userID }}`
`name`	`{{ this.name }}`
`email`	`{{ this.email }}`
`verified`	`yes`

The select action will output the following data to the pipeline:

{
  "myDataset": [
    {
      "userID": "user123",
      "name": "Alice Smith",
      "email": "alice.smith@example.com",
      "verified": "yes"    },
    {
      "userID": "user456",
      "name": "Bob Johnson",
      "email": "bob.johnson@example.com",
      "verified": "yes"    }
  ]
}

Convert to CSV - Convert that dataset to a CSV file. Name the output property myCsvData.

{
  "myDataset": [
    {
      "userID": "user123",
      "name": "Alice Smith",
      "email": "alice.smith@example.com",
      "verified": "yes"    },
    {
      "userID": "user456",
      "name": "Bob Johnson",
      "email": "bob.johnson@example.com",
      "verified": "yes"    }
  ],
  "myCsvData": "userID,name,email,verified\nuser123,Alice Smith,alice.smith@example.com,yes\nuser456,Bob Johnson,bob.johnson@example.com,yes"}

The original dataset remains intact - most actions (with few exceptions) are additive and will pass through any previous data on to the next action. Also, actions that do not add any properties or manipulate the data pipeline (e.g., the Gate or Workflow Management actions) are also set up to pass the dataset on to the next action verbatim.

Note: Template expressions allow you to specify certain properties on the data pipeline that previous actions have passed in. For more information, check out the Template Engine documentation.