How to catch curl response in Jenkins Pipeline?

I guess you may ask this question: "why to bother with curl while there is Java or Groovy HTTP client I can use in the pipeline script block?". This question might look reasonable. However, there is one important fact you need to consider. Any Groovy (or Java) code you put into the script block is always executed on the Jenkins master server. No matter what agent/node you use to execute the specific stage(s). Now imagine the following scenario. Let’s say you have a dedicated node in your Jenkins infrastructure to run some REST API calls to perform a production deployment. And let’s assume that only this specific node can make such an HTTP call for security reasons. If you implement the deployment part by using some Groovy code (e.g., something as simple as java.net.HttpURLConnection), it will fail. The server that de facto makes an HTTP call, in this case, is not your dedicated node, but the Jenkins master server. The same one that is not allowed to make any connection to your production server(s). However, if you use the pipeline’s sh step to execute the curl command instead, the HTTP request will be sent from the designated Jenkins node. This way, you can design a topology where only a specific node can perform strategic REST API calls.

Now when you know "why", let’s take a look at the "how" part.

The way to execute curl command is to use sh (or bat if you are on the Windows server) step. You need to know that the sh step by default does not return any value, so if you try to assign it’s output to a variable, you will get the null value. To change this behavior, you need to set the returnStdout parameter to true. In the following example, I use my local Jenkins installation to get the build information of one of my Jenkins pipelines using the JSON API endpoint.

The endpoint I was trying to call in the previous example requires authentication. We can extend this example by adding -u username:password option and using Credentials Binding plugin to store those credentials.

In this example, I use the user’s API Token instead of a real password. It works like a regular password, but it can be easily revoked, and the new one can be re-generated if needed. Also, you can use usernameColonPassword to bind username:password to the specific variable easily.

In many cases, you want to react to different HTTP response status codes. For instance, you want to check if the response returned 200 or 400 status code. Let’s extend the previous example to fit into such a use case. We can pass -w' \n%{response_code}' option to change the output format to the one that contains the response body in the first line, and the status code in the second line. Then we can split the output by the new line character, and we can use {multiple-assignment}[Groovy multiple assignment] feature to assign the response body and the status to the two separate variables.

Once we have the response body and the status code extracted, we can parse the JSON body and extract the value we are interested in. The Jenkins build’s JSON API response contains a list of different actions. Let’s say we want to extract the value of executingTimeMillis stored as an action of type jenkins.metrics.impl.TimeInQueueAction.

If you know Groovy, I’m guessing you might already think about using groovy.json.JsonSlurper to solve the problem. It is not a terrible idea, but it has two downsides you need to take into account. Firstly, it makes Jenkins master server involved in parsing the JSON response, even if the stage gets executed on a different node. And secondly, groovy.json.JsonSlurper produces a lazy map which is not-serializable. The first thing might or might not be a problem - it mainly depends on your Jenkins topology architecture. If you use multiple agents to execute stages and have a lot of different Jenkins jobs on your server, it makes sense to utilize Jenkins master server resources more effectively. Using groovy.json.JsonSlurper to parse small JSON documents might not be a bottleneck. Still, if hundreds or thousands of Jenkins jobs start to parse large JSON documents, it may become an issue. The second problem can be solved easily - instead of using groovy.json.JsonSlurper, use groovy.json.JsonSlurperClassic. This class produces a regular hash map that can be serialized, and that way, you can avoid pipeline serialization problems.

There is an alternative, and it is usually a more recommended way to parse and extract values from the JSON data - the jq command-line tool. All you have to do is to make sure that the jq is installed on the Jenkins node that executes the pipeline.

Let’s take a look at how we can use it to extract the same information from the JSON response.

The list of all available options and parameters that the curl command supports is enormous. You can find all of them in the man curl manual page. Below you can find a few more useful options you may want to use in your Jenkins pipeline.