n8n Advanced Data Transformation: Mastering JSON, Arrays, and Loops for Workflow Brilliance

Are your n8n workflows feeling a bit… stiff? 🤔 Do you find yourself wrestling with data that just doesn’t quite fit the mold of your next API call or database entry? You’re not alone! Data transformation is the unsung hero of powerful automation, and in n8n, mastering JSON, arrays, and loops is your key to unlocking truly dynamic and intelligent workflows.

This guide will dive deep into n8n’s capabilities, moving beyond basic $json.key access to give you the tools to elegantly manipulate complex data structures. Get ready to turn your data challenges into triumphs! 🚀

1. The Foundation: Understanding n8n’s Data Structure 💡

Before we start bending data to our will, let’s clarify how n8n handles information.

1.1 Items and Batches: The Core Concepts

• Item: In n8n, an “item” is a single piece of data flowing through your workflow. Think of it as one row in a spreadsheet or one record from a database. Each item is a JSON object.

  {
    "id": 123,
    "name": "Alice",
    "email": "alice@example.com"
  }

• Batch: A “batch” is a collection of one or more items. Most n8n nodes process data item by item within a batch. If a node outputs multiple items (e.g., fetching several records from a database), it sends them as a single batch, and subsequent nodes will process each item within that batch sequentially.

1.2 Accessing Data: The Basics

You’re likely familiar with accessing properties within an item using expressions:

• {{ $json.propertyName }}: Accesses a property directly from the current item’s JSON data.
• {{ $node["NodeName"].json.propertyName }}: Accesses a property from an item output by a previous node.
• {{ $item(0).$json.propertyName }}: Accesses a property from the first item of a batch. Useful when you expect only one item or need a specific one.

2. Mastering JSON Transformation: Shaping Your Data ✨

JSON (JavaScript Object Notation) is the language of n8n data. Learning to manipulate it effectively is paramount.

2.1 Simple Transformations with the Set Node

The Set node is your first stop for common JSON manipulations. It’s incredibly versatile for:

• Adding New Properties:
  • Example: Adding a status to an order.
    • Input Item: {"orderId": "ORD001", "amount": 100}
    • Set Node:
      • Add Key: status
      • Value: Processed (or {{ $json.amount > 50 ? 'Large' : 'Small' }})
    • Output Item: {"orderId": "ORD001", "amount": 100, "status": "Processed"}
• Modifying Existing Properties:
  • Example: Converting an email to lowercase.
    • Input Item: {"email": "USER@EXAMPLE.COM"}
    • Set Node:
      • Add Key: email (overwrites existing)
      • Value: {{ $json.email.toLowerCase() }}
    • Output Item: {"email": "user@example.com"}
• Renaming Properties:
  • Use “Rename Key”.
  • Example: product_id to productId.
    • Input Item: {"product_id": "P001", "name": "Widget"}
    • Set Node:
      • Rename Key: product_id to productId
    • Output Item: {"productId": "P001", "name": "Widget"}
• Removing Properties:
  • Use “Remove Key”.
  • Example: Removing sensitive password field.
    • Input Item: {"user": "Alice", "password": "securepwd"}
    • Set Node:
      • Remove Key: password
    • Output Item: {"user": "Alice"}

2.2 Advanced JSON with the Code Node (JavaScript Power!)

When the Set node isn’t enough, the Code node is your ultimate weapon. It allows you to write custom JavaScript to perform any imaginable JSON manipulation.

• Accessing Input Data: The input data to the Code node is available as items. Each item has a json property containing its data.

  // Example: Accessing the first item's JSON
  const firstItemData = items[0].json;

• Modifying Data and Returning: You must return an array of objects, where each object has a json property for the output data.

  // Example: Add a new property to each item
  for (const item of items) {
    item.json.newProperty = "Hello n8n!";
  }
  return items; // Important: Return the modified items

• Practical Example: Flattening Nested JSON Imagine you receive data like this from an API:

  [
    {
      "orderId": "ORD001",
      "customer": {
        "id": "CUST001",
        "name": "Bob Smith",
        "contact": {
          "email": "bob@example.com",
          "phone": "123-456-7890"
        }
      },
      "items": [
        {"productId": "P001", "qty": 1},
        {"productId": "P002", "qty": 2}
      ]
    }
  ]

  You need to flatten it for a spreadsheet, turning nested customer details into top-level properties.

  Code Node Script:

  const outputItems = [];
  
  for (const item of items) {
    const originalJson = item.json;
  
    // Create a new flattened object
    const flattenedJson = {
      orderId: originalJson.orderId,
      customerId: originalJson.customer.id,
      customerName: originalJson.customer.name,
      customerEmail: originalJson.customer.contact.email,
      customerPhone: originalJson.customer.contact.phone,
      // For array items, you might want to stringify or handle separately
      itemsDetails: JSON.stringify(originalJson.items)
    };
  
    outputItems.push({ json: flattenedJson });
  }
  
  return outputItems;

  Output Item (example):

  [
    {
      "orderId": "ORD001",
      "customerId": "CUST001",
      "customerName": "Bob Smith",
      "customerEmail": "bob@example.com",
      "customerPhone": "123-456-7890",
      "itemsDetails": "[{\"productId\":\"P001\",\"qty\":1},{\"productId\":\"P002\",\"qty\":2}]"
    }
  ]

  This demonstrates how you can access nested properties and reconstruct the JSON as needed.

3. Harnessing the Power of Arrays: Collections of Data 📦

Arrays are ordered lists of values or objects. They are fundamental when dealing with lists of products, users, line items, etc.

3.1 What are Arrays in n8n?

An array in n8n can be a property within a single item, or you might have a batch of items that effectively represents an array of objects.

• Array as a property:

  {
    "orderId": "ORD002",
    "products": [
      {"id": "P003", "name": "Keyboard"},
      {"id": "P004", "name": "Mouse"}
    ]
  }

• Batch of items (effectively an array of objects): Item 1: {"user": "Alice"}, Item 2: {"user": "Bob"} (This is how Split In Batches works!)

3.2 Common Array Operations with the Code Node

While some n8n nodes implicitly handle arrays (like Split In Batches), for powerful filtering, mapping, and reduction within a single item’s array, the Code node is essential.

• Filtering Arrays (.filter()): Selects elements that meet a condition.

  • Example: Keep only products with price > 50.
    • Input Item:

      {
        "products": [
          {"name": "Laptop", "price": 1200},
          {"name": "Mouse", "price": 25},
          {"name": "Monitor", "price": 300}
        ]
      }

    • Code Node Script:

      for (const item of items) {
        item.json.highValueProducts = item.json.products.filter(p => p.price > 50);
      }
      return items;

    • Output Item:

      {
        "products": [ /* original array */ ],
        "highValueProducts": [
          {"name": "Laptop", "price": 1200},
          {"name": "Monitor", "price": 300}
        ]
      }

• Mapping Arrays (.map()): Transforms each element in an array into a new form.

  • Example: Extract only product names.
    • Input Item: (same as above)
    • Code Node Script:

      for (const item of items) {
        item.json.productNames = item.json.products.map(p => p.name);
      }
      return items;

    • Output Item:

      {
        "products": [ /* original array */ ],
        "productNames": ["Laptop", "Mouse", "Monitor"]
      }

• Reducing Arrays (.reduce()): Aggregates array elements into a single value.

  • Example: Calculate total price of products.
    • Input Item: (same as above)
    • Code Node Script:

      for (const item of items) {
        item.json.totalPrice = item.json.products.reduce((sum, p) => sum + p.price, 0);
      }
      return items;

    • Output Item:

      {
        "products": [ /* original array */ ],
        "totalPrice": 1525 // (1200 + 25 + 300)
      }

3.3 Item Lists Node: Creating & Manipulating Item Arrays

The Item Lists node is great for converting an array within a single item into multiple separate items (similar to Split In Batches, but with more control over the array path) or vice-versa.

• Split Out Items: Converts an array within an item into separate items.
  • Example: If you have {"data": [{"id":1}, {"id":2}]}, it can turn this into two separate items.
• Merge All to One: Aggregates all incoming items into a single array under a specified key.
  • Example: After processing multiple items, you can collect them into one item’s array for a final step.

4. Looping for Iteration & Control Flow: Processing Collections 🔄

Loops are essential for processing each element in a collection, whether it’s an array within an item or a batch of items.

4.1 Implicit vs. Explicit Loops

• Implicit Loops: Many n8n nodes inherently “loop” over the items they receive in a batch. If you send 5 items into a Set node, the Set node will process each of those 5 items sequentially. This is the default behavior.
• Explicit Loops: When you need to iterate over an array within a single item and perform operations that require sending each element as a separate “item” through subsequent nodes, or when you need more fine-grained control, you use nodes like Split In Batches or custom Code node logic.

4.2 The Split In Batches Node: Unleash the Power of Parallel Processing (or Sequential Iteration)!

This is perhaps the most crucial node for processing arrays or batches of data. Split In Batches takes a single input item containing an array (or a batch of items), and splits it into individual items, sending each one down the workflow path.

• How it Works:

  • You specify the “Array field” (e.g., products, lineItems).
  • It takes that array, and for each element in the array, it creates a new item with that element as its JSON payload.
  • These newly created items are then sent one by one (or in smaller batches, if configured) to the next node.

• When to Use It:

  • When you retrieve a list of records (e.g., users from a CRM, orders from an e-commerce platform) and need to perform an action for each record (e.g., update a separate system, send an email, create a file).
  • When an API returns a list of items within a single response, and you need to process each item individually.

• Example: Processing a List of Users

  Imagine you fetch a list of users from a database, and you need to call a separate API endpoint for each user to update their profile.

  Workflow Structure:

  1. DB Read Node (e.g., Postgres, MySQL): Fetches users.

     • Output: (single item with users array, or multiple items if configured)

       [
         {
           "users": [
             {"id": 1, "name": "Alice", "email": "alice@example.com"},
             {"id": 2, "name": "Bob", "email": "bob@example.com"},
             {"id": 3, "name": "Charlie", "email": "charlie@example.com"}
           ]
         }
       ]

       OR if it outputs multiple items: Item 1: {"id": 1, "name": "Alice"}, Item 2: {"id": 2, "name": "Bob"}, etc.

  2. Split In Batches Node:

     • If the previous node outputs a single item with an array (like the first example above), configure “Mode” to Split into individual items and “Array Field” to users.
     • If the previous node outputs multiple items already, you might not need Split In Batches unless you want to re-batch them.
     • Output: Each user becomes a separate item. Item 1: {"id": 1, "name": "Alice", "email": "alice@example.com"} Item 2: {"id": 2, "name": "Bob", "email": "bob@example.com"} Item 3: {"id": 3, "name": "Charlie", "email": "charlie@example.com"}

  3. HTTP Request Node: Calls an API for each user.

     • URL: https://api.example.com/update-user/{{ $json.id }}
     • Body: {"name": "{{ $json.name }}", "email": "{{ $json.email }}"}

  4. (Optional) Merge Node: After processing, you might want to merge the results back together.

4.3 The Merge Node: Rejoining Your Data 🔗

After splitting your data with Split In Batches, you often need to bring it back together. The Merge node is crucial for this.

• Merge (Append): Simplest merge. It takes all incoming items from its two inputs and combines them into a single batch. Useful for collecting results from a loop.

• Merge (Join): More powerful. It joins items from its two inputs based on a common key (like a SQL JOIN). You can choose Left Join, Inner Join, Right Join, Full Join. Essential for enriching data after a loop.

  • Example (continued from user processing):
    • Let’s say the HTTP Request node returns a status for each user update.
    • Input 1 (from Split In Batches): Original user data.
    • Input 2 (from HTTP Request): API response for each user.
    • Merge (Join) Node:
      • “Node to Merge From”: HTTP Request
      • “Data to Join”: Item
      • “Join On”: id (from original user data) and user_id (from API response)
    • Output: An item containing both the original user data and the API response status, merged into one JSON object.

4.4 Looping with the Code Node for Finer Control

Sometimes you don’t need to split items, but just iterate through an array within a single item and perform an action, or build up a new array. The Code node is perfect for this with standard JavaScript loops (for...of, forEach, map, filter).

• Example: Building a Summary String from an Array of Items You have an order with multiple products, and you want a single string summary for an email.

  • Input Item:

    {
      "orderId": "ORD003",
      "products": [
        {"name": "Pen", "quantity": 5},
        {"name": "Notebook", "quantity": 2}
      ]
    }

  • Code Node Script:

    for (const item of items) {
      const products = item.json.products;
      let summary = `Order ${item.json.orderId} includes: \n`;
    
      for (const product of products) {
        summary += `- ${product.name} (Qty: ${product.quantity})\n`;
      }
      item.json.orderSummary = summary;
    }
    return items;

  • Output Item:

    {
      "orderId": "ORD003",
      "products": [ /* original array */ ],
      "orderSummary": "Order ORD003 includes: \n- Pen (Qty: 5)\n- Notebook (Qty: 2)\n"
    }

5. Practical Use Cases & Advanced Tips 🛠️

• Data Cleaning and Normalization: Convert inconsistent date formats, trim whitespace, standardize casing, remove null values before sending data to another system. The Set and Code nodes are your best friends here.
• API Request Preparation: Transform complex internal data structures into the exact JSON payload required by external APIs. This often involves flattening, renaming, and sometimes aggregating data.
• Aggregating Data: Use Code node with .reduce() or Merge (Append/Join) to combine data from multiple sources or summarize a list of items.
• Error Handling in Loops:
  • If you’re using Split In Batches and an HTTP Request node fails for one item, you can configure the HTTP Request node to “Continue On Error” to prevent the entire workflow from stopping.
  • Within a Code node, use try...catch blocks to handle potential errors gracefully during array processing.
• Performance Considerations (Batching): When dealing with thousands of items, sending them one by one through HTTP requests can be slow and hit API rate limits.
  • Split In Batches allows you to define a “Batch Size” (e.g., 100). This sends items in chunks, which can be more efficient for some APIs.
  • Consider using the Code node to prepare a single, larger array of items to send in one API request if the API supports bulk operations.

Conclusion: Your Workflow, Reimagined! 🎉

Mastering JSON, arrays, and loops in n8n isn’t just about technical proficiency; it’s about unlocking the true potential of your automation workflows. By being able to precisely shape, filter, and iterate over your data, you gain the power to:

• Integrate with virtually any API, regardless of its data requirements.
• Clean and prepare data for analysis or storage.
• Automate complex business processes that involve lists and collections of information.
• Build resilient workflows that can handle varying data structures.

The Set node is your quick-fix superhero, the Split In Batches and Merge nodes are your control flow maestros, and the Code node is your Swiss Army knife for anything else.

Don’t be afraid to experiment! Start with small datasets, use the “Execute Workflow” and “View Output” features frequently to see how your data changes at each step, and leverage the fantastic n8n community for inspiration and support. Happy automating! 🚀✨ G