12 TIPS for you to use COMMAND curl as a NINJA

curl is a command/library(libcurl) available on most Unix-based systems.

Feb 12, 2020

Introduction

curl is a command/library(libcurl) available on most Unix-based systems. It is used as an abbreviation for “Client URL”. It is used for several purposes such as: navigation, data insertion via the web, download, data transfer and among others. It supports several protocols.

Installation

To install curl you can use your distro’s package manager, but before installing make sure it is already installed with the command: which curl, examples:

emerge curl # Para Gentoo, Funtoo,...
sudo apt install curl # No Debian, Ubuntu, Linux Mint,...
sudo pacman -S curl # Arch Linux, Manjaro,...
sudo yum install curl # Red Hat, CentOS and Fedora (prefer dnf)
pkg install curl # FreeBSD
pkg_add curl # OpenBSD
pkgin install curl # NetBSD

After installation, check that everything is ok, for example, by looking at the version: curl --version.

Tip 1. Filling out a website form via the command line

curl -F "mail=nobody@nothing.com" -F "msg=Thanks" http://givingsoup.org/
curl -X POST -F "mail=nobody@nothing.com" -F "msg=Thanks" http://givingsoup.org/
# cat data.txt #mail=nothing@all.com&msg=Anything Message
curl --data "mail=iwanna@yoursoap.com&msg=Nice" http://givingsoup.org/
curl -d "@data.txt" -X POST http://givingsoup.org/

Tip 2. Uploading files via the command line

In this case we need to inform the URL of the file path that will be responsible for receiving the file, in this case I know it is upload.php, but it could be at the root of the host, remembering that, as I said, it redirects to ‘m/’without changing the URL.

curl -i -X POST -H "Content-Type: multipart/form-data" -F "data=@TIPS_COMMANDS" http://givingsoup.org/upload.php
curl -F "data=@image.jpg" http://givingsoup.org/upload.php >/dev/null 2>/dev/null

Tip 3. Download Files

Download with the same name as the original file that is on the server: The -O or --remote-name parameter indicates that the source file is binary, so if you don’t pass it, curl will not be able to download it.

curl -O givingsoup.org/downloads/distro.iso

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  841M  100  841M    0     0  46.0M      0  0:00:18  0:00:18 --:--:-- 44.0M

% - Shows the total in PERCENTAGE of the download;
Total - Shows the total in MEGABYTES of the download;
…

If you want it not to show data, use the --silent or -s parameter: curl --silent givingsoup.org/downloads/distro.iso

If you miss the file name, curl will download the 404 file, so it is always good to check the size of the file du -lhs distro.iso, if it is incorrect see what the server’s response was: cat distro.iso (can be it a server error if it is not 404, for example, 301), so the ideal is to HASH the post downloaded file

Note: There is still no parameter to save without overwriting the file if it exists, so you need to create a solution in the shell, example:

[ ! -f data.txt ] && curl -O givingsoup.org/downloads/data.txt || echo File already exists

Tip 4. Download the file saving with a name you defined:

curl givingsoup.org/downloads/distro.iso -o mydistro.iso

Tip 5. Resume an interrupted download

curl -C - -O http://givingsoup.org/downloads/distro.iso

Tip 6. Download multiple files

curl -O http://yoursite.com/mydistro.iso -O http://givingsoup.org/downloads/data.txt

Tip 7. Download URLs from a file

cat urls.txt
givingsoup.org/downloads/distro.iso
givingsoup.org/downloads/data.txt
xargs -n 1 curl -O < urls.txt

Tip 8. Limit download speed

curl --limit-rate 100K http://givingsoup.org/downloads/distro.iso -O

Tip 9. Download with progress bar

curl -# -O givingsoup.org/downloads/distro.iso

Tip 10. Query HTTP headers

curl -I en.terminalroot.com.br

Not all sites allow this query.

Tip 11. Specify a User Agent

curl https://gnu.terminalroot.com.br/ip.php
# Example Windows 10 Edge:
curl http://localhost --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134"

Tip 12. Apache authentication

curl -u user:pass -O http://givingsoup.org/m/secret/my-secret-files.tar.gz

Useful links

curl --help
# And/Or
man curl

Bonus Tip

See all the tips worked in the video below. Remembering that the tutorial is in Portuguese, but you can use the subtitles (which are not fully translated, but the beginning and several other parts are)

The video has English subtitles native to the video itself and with the correct translation. To enable, use the settings in the footer of the video itself, according to the images shown here below.

CLICK ON IMAGES TO VIEW INTO LARGER RESOLUTION.

Swap images by clicking »