In the constantly evolving digital world, JSON and XML have become the standard data interchange formats. They are widely used in data exchange between client and server, APIs, and various other data communication scenarios.
Whether we’re dealing with JSON or XML, the nature of these formats includes nested data structures, arrays, and complex fields that can be challenging to navigate and update. We often encounter situations where we need to modify default API responses, combine data from different sources, or transform data in a specific format. However, it is a common requirement, especially for MuleSoft developers and those working with APIs, to dynamically populate, manipulate, and update these fields in response to various business needs.
JSON and XML are two of the most common formats for data exchange and since these formats often come with complex nested structures, it is challenging to update specific fields efficiently. In this article we will explain how to dynamically navigate and update fields in JSON and XML data.
JSON and XML Payload Solution
Within this code, you’ll find a collection of functions that work together to accomplish the task of updating a particular field in JSON and XML payloads.
%dw 2.0
import * from dw::core::Strings
import * from dw::util::Values
fun getIndex(field) = (flatten(field scan /\[(.*?)\]/)[1] default 0) as Number
fun extractKeyAndIndex(field) = (
{
"key": if(field contains("[")) substringBefore(field, "[") else field,
"index": getIndex(field)
}
)
fun getPath(field) = (
if(field contains("."))
extractKeyAndIndex(substringBefore(field, "."))
else
extractKeyAndIndex(field)
)
fun getRest(field) = (
substringAfter(field, ".")
)
fun addField(field, value) = (
if(field contains ("."))
(extractKeyAndIndex(substringBefore(field, ".")).key): addField(getRest(field), value)
else
(field): value
)
fun updateKey(object, key, value, id) = do {
var k = extractKeyAndIndex(key).key
---
if(isEmpty(object[k]))
object ++ addField(id, value)
else
object update field(k) with value
}
fun extractMuleEvent(field) = (
if(startsWith(field, "payload"))
substringAfter(field, "payload.")
else
field
)
fun updateField(data, id, value) = do {
var key = getPath(id).key
var index = getPath(id).index
var rest = getRest(id)
var val = data[key]
---
if(!isEmpty(key))
if(val is Object)
data update field(key) with updateField(val, rest, value)
else if(val is Array)
data update field(key) with (val update index with (updateField(val[index], rest, value)))
else
updateKey(data, key, value, id)
else
data
}We will provide a detailed explanation for every function.
The code begins by importing these modules from the DataWeave standard library for string manipulation and value handling:
- dw::core::Strings
- dw::util::Values
The getIndex function extracts the index value from a field that is enclosed in square brackets. We used the “scan” function that returns an array with all of the matches found in an input string. It uses regular expressions to accomplish this.
fun getIndex(field) = (flatten(field scan /\[(.*?)\]/)[1] default 0) as Number
The extractKeyAndIndex function extracts the key and index from a field. If the field contains square brackets, it splits the field and assigns the key and index accordingly. It returns an object containing the key and the index of the field.
fun extractKeyAndIndex(field) = (
{
"key": if(field contains("[")) substringBefore(field, "[") else field,
"index": getIndex(field)
}
)The getPath function determines the key and index for the field. If the field is nested, so if it contains a dot, it extracts the key and index from the substring before the dot. Otherwise, it extracts the key and index from the entire field. This enables us to traverse and locate the desired field efficiently.
fun getPath(field) = (
if(field contains("."))
extractKeyAndIndex(substringBefore(field, "."))
else
extractKeyAndIndex(field)
)The getRest function retrieves the remaining part of the field after the dot.
fun getRest(field) = (
substringAfter(field, ".")
)The addField function adds a new field to an object. If the field contains a dot, it means that it’s nested and it recursively adds the field to the nested object.
fun addField(field, value) = (
if(field contains ("."))
(extractKeyAndIndex(substringBefore(field, ".")).key): addField(getRest(field), value)
else
(field): value
)The updateKey function updates a key-value pair in an object. If the key is empty, it adds the field and value as a new key-value pair. Otherwise, it updates the existing key with the provided value.
fun updateKey(object, key, value, id) = do {
var k = extractKeyAndIndex(key).key
---
if(isEmpty(object[k]))
object ++ addField(id, value)
else
object update field(k) with value
}The extractMuleEvent function extracts the payload field from a given field. If the field starts with “payload”, it removes “payload.” from the beginning.
fun extractMuleEvent(field) = (
if(startsWith(field, "payload"))
substringAfter(field, "payload.")
else
field
)The updateField function updates a field in the payload. It recursively traverses the structure and updates the corresponding field with the new value.
fun updateField(data, id, value) = do {
var key = getPath(id).key
var index = getPath(id).index
var rest = getRest(id)
var val = data[key]
---
if(!isEmpty(key))
if(val is Object)
data update field(key) with updateField(val, rest, value)
else if(val is Array)
data update field(key) with (val update index with (updateField(val[index], rest, value)))
else
updateKey(data, key, value, id)
else
data
}JSON and XML Functionality Demo
Let’s dive into a hands-on demonstration to illustrate the functionality of the code.
This code can be used for JSON and XML data. We are going to use JSON input for the demonstration. We have this data that represents three companies and their properties.
{
"total": 3,
"companies": [
{
"name": "MuleSoft",
"founded": "2007",
"headquarters": "San Francisco, California, USA",
"website": "https://www.mulesoft.com",
"employeeCount": 1188
},
{
"name": "Salesforce",
"founded": "1999",
"headquarters": "San Francisco, California, USA",
"website": "https://www.salesforce.com",
"employeeCount": 79390
},
{
"name": "IWConnect",
"founded": "2004",
"headquarters": "Bitola, Macedonia",
"website": "https://www.iwconnect.com",
"employeeCount": 325
}
]
}Let’s say IWConnect has welcomed a new member in their company, so the number of employees increased from 325 to 326. We need to change the employeeCount field to 326. We are doing that by using our updateField function:
updateField(payload, extractMuleEvent("payload.companies[2].employeeCount"), 326)This is the output we get:
{
"total": 3,
"companies": [
{
"name": "MuleSoft",
"founded": "2007",
"headquarters": "San Francisco, California, USA",
"website": "https://www.mulesoft.com",
"employeeCount": 1188
},
{
"name": "Salesforce",
"founded": "1999",
"headquarters": "San Francisco, California, USA",
"website": "https://www.salesforce.com",
"employeeCount": 79390
},
{
"name": "IWConnect",
"founded": "2004",
"headquarters": "Bitola, Macedonia",
"website": "https://www.iwconnect.com",
"employeeCount": 326
}
]
}We’ve successfully updated the employeeCount field to 326.
Summary and Takeaways
With this code we performed a common use case for MuleSoft developers. If you encounter the need to dynamically modify fields in your JSON and XML data, you can use the above DataWeave expression in order to perform traversing of the desired field. You can efficiently perform precise data format transformations, customize default API responses and manipulate fields.
In the process, we’ve demonstrated how to use DataWeave to manipulate JSON and XML data, be it a simple array or more complex nested structures.
Understanding how to dynamically update and traverse fields in JSON and XML is an invaluable skill that can empower you to perform precise data transformations, customize default API responses, and manipulate fields more effectively. This exploration of data manipulation in JSON and XML highlights the immense power of DataWeave. Being able to handle data with precision and flexibility can significantly elevate your programming and data management capabilities. So, whether you are handling an XML document or a JSON response from an API, understanding and effectively manipulating these data structures can streamline your data interchange processes and lead to more efficient and effective solutions.
