curl is exceedingly powerful – unfortunately, much of this power is somewhat hidden in the purposeful non-verbosity and the underlying complexity of its numerous flags, configurations, and options. Once understood, curl boasts a wide range of powerful options. Today, we’re going to cover some of these options and discuss the hidden power behind one of the most popular and beloved open-source projects.

What is curl?

curl: A command-line tool and library for transferring data with URLs.

curl is an open-source project that’s composed of two parts – curl proper, and libcurl. curl is a command-line tool that leverages the URL syntax to receive or send data and has a wide range of features and flags that allow it to do this with varying purposes and focuses. libcurl, on the other hand, is a free client-side URL transfer library which enables a good deal of this functionality. curl is often referred to as a Swiss Army Knife solution, as it has a wide range of powerful flags and options that delivers a wide range of awesome effects.

curl is portable, backward compatible, and by their own estimates, has millions of build combinations, allowing for a highly fluid combinatory set of functions and options. curl currently boasts support for 24 protocols and has 226 command-line options. For our purposes in this piece, we’ll stick to the hidden powers for HTTPS manipulation in curl, and largely utilize a smaller subset of flags.

One thing to note about curl – it is minimal by default and design. It really only does what it needs to do (or what it understood its purpose to be). A basic curl request provides only basic headers when defined, and boasts no fancy functionality. That may seem to imply that curl is simple in practice, but the reality is that curl operates on a series of levers and toggles – features can be turned on and off one by one, and the combination of these levers and toggles can deliver awesome power, indeed.

Before we dive into the cool hidden powers within curl, we should first look at what a regular curl request looks like.

Standard Requests

In its most basic form, we can make a curl request as follows:

curl example.com

Unfortunately, such a call does not have any response headers (remember, curl is by design minimal). In order to get those response headers, we can use the -i flag:

curl -i https://example.com

To take this a step further, we can pass the output into a rendering system in order to reap more complex responses:

curl https://example.com/json | jq

More Complex Standard Requests

Up to this point, it might seem like curl gives some relatively simplistic outputs. Even though we were able to view the JSON output and render it readable using jq, we’re still making a very basic request. For a more complex use case, we can look at redirects. Redirects are extremely common in HTTP APIs, so handling this can be foundational to our flow. In order to follow a redirect, we can simply make the following (slightly more complex) request:

curl -I -L https://example.com/redirected

Hybrid Requests

One interesting use case of the request system is the utilization of hybrid requests. In curl, different requests can be combined on the same command line as follows:

curl -d user=daniel https://example.com https://another.example.com

Of note is the fact that there’s no real limit to the number of URLs that can be used in such a scheme, meaning you could theoretically do a great number of requests to a great number of API endpoints on a single CLI line. This is especially useful when multiple requests have to be made to coalesce content – in fact, in this use case, you could use the same hybrid approach noted here, and then pipe these contents to a filename for combinatory efforts.

You can also set the specific order of this process by using --next as follows:

curl -d user=daniel https://example.com --next https://another.example.com

Conversely, if you wanted to do different things on the same line, you can do that too!

curl https://example.com --next -d user=daniel https://another.example.com

Advanced POSTing

As an example of more complex requests that can be made with basic curl requests, we can take a look at some POST options. First, let’s look at a very simple POST request:

curl -d @file https://example.com receiver -o saved

In this case, we have posted a very simple file by using the -d flag and the @file file naming scheme. What if we wanted to name this file using a more complex method? We can actually use curl to name a POSTed file using a piped output. Pipes in curl work much the same way as they do in any CLI, in that they take the output of something and utilize it as the input for something. In this case, we’ll use ls -l, a file listing command, to generate the file name used:

ls -l | curl -d @- https://example.com/receiver -o saved

This request takes the results of ls -l, and outputs it as the filename for the POST process. This is a wonderful way to synchronize the upload of files on a one-to-one basis as we work in the CLI. More than that, it represents the power behind the pipe itself – it bridges between the interior and exterior experience and makes the CLI itself a strong bridge between disparate data types and locations.

Increase Verbosity

Requests are useful of course, but the process of issuing that request and seeing the process as a whole can be difficult to see. Thankfully, curl supports a wide range of options for transparent curl requests.

One of the easiest ways to see more about what curl is doing is to simply generate a more verbose record of the interaction. In order to do that, we can easily pass the -v flag to require greater verbosity:

curl -v https://example.com -o /dev/null

This request will certainly show you a far larger amount of information, but even this information is limited to the command line itself. Thankfully, there are two incredibly strong methods to inspect this activity even deeper.

The first, and arguably most dense, is to simply output everything that occurs in curl to the --trace-ascii function. This function will take every single thing that happens during the process of the request, and output it as an ASCII output for parsing and viewing. This can be done by leveraging the following request:

curl https://example.com/ -d sendthisdata --trace-ascii

Another great option is to use the environmental variable SSLKEYLOGFILE. Using this variable will allow you to save encrypted traffic activity to a defined file, opening up the tracked data for perusal by a solution such as Wireshark. In order to do this, we must set our variable:

export SSLKEYLOGFILE=$HOME/tmp/tlskey

From here, we can make any curl request we’d want to. Once this request has been made, we need to tell Wireshark where the secrets are stored by using its settings file to point towards the tlskey file. From here, we can use Wireshark (or any snooping tool) to read everything that occurred during the request process in extreme detail.

Be More Flexible

One of the things that curl does really well is that it allows the user to be very flexible with both how they make requests, and how they appear to make requests. For example, curl allows you to pass custom HTTP headers – in this case, we could even use this to obscure a specific header such as user-agent:

curl https://example.com -H "User-agent: removed"

Headers aren’t the only thing that can be changed either – we can even change the specific method we are using for our specific interaction with pretty high granularity. For instance, we can use a method such as curl’s SWOOSH method to make a more specific kind of PUT:

curl -T localfile -X SWOOSH https://example.com/remove_name -o save

Of course, there are situations where our flexibility needs to not be in the request, but in our environment. One test case might be testing servers internal to our testing environment. If we have a server on 172.0.0.1, we may run into an error with certificates stopping our request and preventing any of our efforts. We can use the great flexibility in curl to instruct the system to expect the certificate mismatch and properly resolve:

curl https://example.com/ --resolve example.com:443:172.0.0.1

In this way, we’ve not only been flexible in our requests, we’ve also been quite flexible in our methodologies. We can even change the understood protocols we are using — for instance, we can set different HTTP versions dependent on our current command request:

curl --http1.0 https://example.com

curl --http2 https://example.com

Leveraging Cookies

Of course, not everything we’d want to do with curl is simple or easily done with basic requests. There are some complex interactions and use cases that we can certainly leverage using curl. As one example, let’s assume we wanted to use specific cookies in a specific request. We could do that using curl’s cookie system! In order to do that, we must first save our cookies to the specified file:

curl -c cookiejar.txt https://example.com

When we make our next request, we can specify that the cookie engine uses that set cookie:

curl -b cookiejar.txt https://example.com

This is a great use case for logging into a resource – we could easily make a request to, say, a login URL, and once we pass the credentials onward and get the logged-in cookies, we could then leverage that saved cookie throughout the rest of our requests.

Compiling and Mimicry

Finally, curl has several great options for compiling our curl content and mimicking curl activities in general. For instance, on Linux, we can use gcc in combination with the sourcecode process in curl to package our curl operants as an application. First, we must export the content the sourcecode:

curl https://example.com --libcurl sourcecode.c

From here, we can use gcc to compile the source code into a proper application:

gcc sourcecode.c -lcurl -o ./myapp

Of course, sometimes we don’t want a full application – in fact, we may simply want to mimic the activity of a known browser. Modern browsers have made this possible by leveraging “copy as curl” – this option is available on Chrome, Firefox, Safari, and Edge, so the chances are very good that you have this built-in to your browser at this moment. You can simply copy a network activity as curl, allowing you to copy that action for perusal, compiling into an application, or even piping into other actions. The sky’s the limit!

Conclusion

curl is a very powerful option for modern web developers as it presents something for pretty much everyone – in common parlance, it’s a veritable Swiss Army Knife, offering a million methods and combinations to do pretty much whatever one might want to. Its simplicity at first blush is somewhat misleading – as we’ve shown today, curl can do complex interactions, handle cookies in interesting and useful ways, and even expose its internal functions with an extreme level of detail.

What do you think about curl? Do you know any functions we didn’t cover here that you feel are exceedingly useful? Let us know in the comments below!