jq cookbook

Merging JSON files recursively in the command-line

Have you ever need to merge two (or more) JSON files and you wondered if you can do it in the command-line? In this blog post, I will show you how you can use jq command-line JSON processor to merge recursively multiple JSON files.

Exemplary input files

Below you can find two exemplary JSON files we are going to merge recursively, including combining unique elements from all arrays.

Listing 1. product1.json file
{
  "id": "324a9e6f-44e8-4d70-9645-7247e5385d05",
  "name": "Product old name",
  "prices": [
    {
      "amount": 19.99,
      "currency": "USD"
    }
  ],
  "tags": [
    "tech",
    "book",
    "programming"
  ],
  "prototype": false,
  "meta": {
    "test": 1
  },
  "nested": {
    "map": {
      "1": "2",
      "4": 0
    },
    "list": [
      1,
      2,
      3
    ]
  }
}
Listing 2. product2.json file
{
  "id": "324a9e6f-44e8-4d70-9645-7247e5385d05",
  "name": "Product new name",
  "vendor": "Vendor Name Inc.",
  "prices": [
    {
      "amount": 19.99,
      "currency": "CAD"
    }
  ],
  "tags": [
    "programming",
    "learning"
  ],
  "meta": {
    "beta": 0
  },
  "nested": {
    "map": {
      "1": 3,
      "2": 1
    },
    "list": [
      2,
      3,
      4
    ]
  }
}

Merging multiple files recursively

And here is the jq code that does the job:

$ jq -s 'def deepmerge(a;b):
  reduce b[] as $item (a;
    reduce ($item | keys_unsorted[]) as $key (.;
      $item[$key] as $val | ($val | type) as $type | .[$key] = if ($type == "object") then
        deepmerge({}; [if .[$key] == null then {} else .[$key] end, $val])
      elif ($type == "array") then
        (.[$key] + $val | unique)
      else
        $val
      end)
    );
  deepmerge({}; .)' product1.json product2.json > merged.json

We define deepmerge(a;b) function that merges two JSON objects recursively.

Output

Listing 3. merged.json file
{
  "id": "324a9e6f-44e8-4d70-9645-7247e5385d05",
  "name": "Product new name",
  "prices": [
    {
      "amount": 19.99,
      "currency": "CAD"
    },
    {
      "amount": 19.99,
      "currency": "USD"
    }
  ],
  "tags": [
    "book",
    "learning",
    "programming",
    "tech"
  ],
  "prototype": false,
  "meta": {
    "test": 1,
    "beta": 0
  },
  "nested": {
    "map": {
      "1": 3,
      "4": 0,
      "2": 1
    },
    "list": [
      1,
      2,
      3,
      4
    ]
  },
  "vendor": "Vendor Name Inc."
}

Did you like this article?

Consider buying me a coffee

0 Comments