Normally, if we want to find the public IP Address we use, we go to google and search “what is my ip address”. But if you use Linux, you can get your public IP Address from the command line using curl command. The curl command with ifconfig.me argument will show your public IP Address. Curl will connect to a server with IPv6 when a host lookup returns an IPv6 address and fall back to IPv4 if the connection fails. The -ipv4 and -ipv6 options can specify which. If the curl version number has -DEV appended to it, it means the version is built straight from a in-development source code and it is not an officially released and 'blessed' version. The rest of this line contains names of third party components this build of curl uses, often with their invidual version number next to it with a slash separator. :) Additional info: Curl appears to be fine with pure-IPv4 hosts (e.g. If I completely disable IPv6 in the kernel, then curl says:. About to connect to fedoraproject.org port 80 (#0). couldn't connect to host. Closing connection #0 curl: (7) couldn't connect to host If I explicitly add '-ipv4', then it works fine.
Overview
In this tutorial, you will learn how to use the curl command in Linux and OSX for making GET and POST requests. You will also learn how to send form data, upload files, and traverse proxies.
The curl command is the prevailing utility for working with the Web from a command-line. The utility provides the same functionality as any web browser, without the UI, that enables you to mimic any type of your request done from a browser. In this tutorial, you will learn ways to use Curl effectively from a command-line to perform common requests.
Data can be downloaded from a website, forms can be posted to, typicaly RESTful commands can be tested against applications. The list of possibilities to endless with this popular tool.
You could effectively use curl in place of Postman, for example. However, there are certain niceties about Postman that we recognize, so it depends on what you need out of the tool.
What’s Covered
The following topics are discussed in this tutorial.
- HTTP Get requests
- HTTP Post requests
- Basic HTTP Authentication
- Setting HTTP Headers with curl
- Send curl requests through a proxy server
Simple Examples
The following are commonly used simple curl commands. Use them to make basic requests to web endpoints and APIs.
Simple GET Request
Following 301 and 302 Redirects with Curl
By default, redirects will not be followed by the tool . In order to instruct it to follow redirects you must use either the -L or --location flags.
Basic HTTP Authentication
curl –basic -u <user> –digest
HTTPS endpoints without verifying TLS
Validation checks will fail when you attempt to curl against a SSLTLS endpoint that is not publicly trusted. Using the --insecure flag curl will ignore certificate validation issues. This most commonly used with self-signed certificates.
Send requests through specific network interfaces
Many servers have separate network interfaces attached them to. When you want to curl from a specific network interface, use the --interface flag with the name of the network interface you want to test with.
Downloading Files or Outputting Results to File
When data needs to be downloaded from a website, such as downloading a script or binary file, the output needs to be outputted to disk. For these use cases, the -o or –output flags are used with the curl command.
Setting TCP/IP Version for Requests
Requests will automatically use IPv4 or IPv6 depending on the system network configuration. However, there are times were you would want to explicitly set which version should be used. Curl provides the –ipv4 and –ipv6 flags for this purpose.
To send requests via IPv4, use the -4 or –ipv4 flags.
curl -4 http://www.serverlab.ca
curl –ipv4 http://www.serverlab.ca
To send requests via IPv6, use the -6 or –ipv6 flags.
curl -6 http://www.serverlab.ca
curl -ipv6 http://www.serverlab.ca
Post Requests
Not every curl command is a GET request. The following examples show you how to perform POSTS to endpoints, and how to set send data with the command. The -X flag is used to set requests to another type, which in the following examples will be POST.
A post without data isn’t very useful. To add data from the command-line the -d option is used. This flag causes curl to pass the content-type value as application/x-www-form-urlencoded.
Curl Ipv6 Invalid Argument
When posting form field data, the data should be url-encoded. This is done using –data-urlencode flag.
To interactively send data to the endpoint, the @ symbol is used with the -d flag.
Data values can be fetched from a file instead, and similar to posting data interactively you use the @ symbol, but with a file name. When using files line breaks are removed, which allows you nicely format the file for better legibility.
Sending Requests through a Proxy
Many organizations use a forward proxy server to handle all external Internet requests. In environments where a proxy isn’t already set with HTTP_PROXY, for example, or where a separate proxy must be used, the -x flag is used.
Simple Proxy Request
Proxy Request with Username and Password
Some proxies require a username and password. This can be done by including the -U or –proxy-user flags to the command.
Conclusion
Curl is a powerful, yet light-weight command-line tool for making requests to web servers and API endpoints. In this tutorial, you have learned how to make simple GET and POST requests, as well as to upload data and send requests through a proxy.
The tool provides a lot more functionality than what is covered in this post. For more information, use the -h flag to display more command and options.
Introduction ¶
The Internet has increasingly become bloated with ads and trackers that slow down your browsing and violate your privacy. I’ve often seen up to 30% of my home network DNS queries be to ad and tracking domains. Ad blockers are becoming quite common on web browsers, but what about devices that don’t support those extensions like smart TVs? Pi-hole is a DNS-level ad and tracker blocker that works on all your devices in your whole home network with nothing else to install or change on them.
While there a are a fair number of tutorials for installing Pi-hole on the internet already, including the official documentation, I personally had some trouble getting it working in Docker with IPv6 enabled, so I wanted to document how I got it working. I’ll also cover adding additional blocklists, unblocking some domains that may be necessary for popular services you use, and extra privacy with DNS over HTTPS.
Don’t want to use IPv6 or Docker?
It’s okay if you don’t have an IPv6 enabled network (or don’t want to set it up), these instructions will also work great for a plain IPv4 setup – just skip the IPv6 steps! Same goes for Docker, I’ve included instructions for setting it up with or without.
Choosing a Device ¶
I’ll be running Pi-hole on a Raspberry Pi 4 with Raspberry Pi OS (formerly Raspbian), but it also works well on other models and most other devices running Linux or that have Docker available. That being said, some low-end devices can get slow if you add a lot of blocklists. I was previously running Pi-hole on a Raspberry Pi Zero and with 700k domains blocked it would often take 5-10 minutes to unblock a domain.
This tutorial assumes you already have a device setup with either direct terminal access or SSH access. Unless otherwise specified, all of the commands should be run in the terminal or over SSH. Check out the official getting started tutorial for the Raspberry Pi to get your device setup if you haven’t yet!
Setting a Static IP Address ¶
Since all of your devices will use Pi-hole as their DNS server, it’s important whatever device you run it on has a static IP address set so you never have to update it in the future.
What Should I use for a Static IPv4 Address? ¶
If you’re not sure what to use as your static IPv4 address, it would be good to know the DHCP range your router uses to assign addresses which can usually be found in your router settings.
Since my router uses the range 192.168.2.10-192.168.2.254, I’m going to choose an IP within the same subnet (192.168.2.x) but outside of that range so there’s no chance of any other device getting assigned the same address. I’ll pick 192.168.2.5 in my case.
If you can’t find the DHCP range in your router, you should usually be fine to choose an IP within your subnet. To find your subnet, you can run ip addr to see your current local IP and replace the last segment with the digits of your choice. Here’s the relevant output on my device (you’re likely looking for either the eth0 or wlan0 section):
We can see that my IPv4 address is 192.168.2.58 so I’ll replace the 58 with whatever I’d like, again I’ll be using 192.168.2.5.
What Should I use for a Static IPv6 Address? ¶
Similarly to IPv4, we can use the ip addr command to see our existing IPv6 address(es) and decide based on that. There should be one that starts with fd00:, this is your Unique local address (ULA) that will be reachable on your private network. Depending on the device this may be static already, but some devices change them regularly for privacy reasons so we’ll make sure it’s static in the next step. Here’s the relevant output on my device showing my ULA:
Configuring Static IPs in the dhcpcd Config ¶
Now that we know what to use for the static IPs, we can configure them on the device. For Raspberry Pi OS (and likely many other Linux distributions), this is as easy as editing the /etc/dhcpcd.conf file. The easiest way to do this is sudo nano /etc/dhcpcd.conf.
Then, find the section with the static IP configuration example, uncomment it by removing the # at the start of each line, and fill in the values you determined. Make sure you enter the same network interface name from ip addr (likely either eth0 or wlan0). You’ll also want to choose an upstream DNS service which Pi-hole will fetch from, I’ll be using Cloudflare’s 1.1.1.1. Here’s my configuration:
Now save that file and restart your device. You should be able to ping it or connect with SSH from another device at the new static IP address. For example:
Installing Pi-hole ¶
Now on to the fun part! You can install Pi-hole with or without Docker. I prefer Docker since it helps keeps services like this isolated from each other, and it’s easy to replicate the exact same setup on different machines. I’ve included instructions for both methods.
Install Pi-hole With Docker ¶
First, if you haven’t already, you’ll need to install Docker.
Install Docker ¶
To start, we need to install some requirements for Docker:
Next, we’ll install Docker from the official script provided:
Finally, we’ll add our user account to the docker group, so we can use the docker command without prefixing it with sudo. You’ll need to log out and back in again after this step for it to take effect.
Setting up Docker With IPv6 Support ¶
Out of the box, Docker doesn’t support IPv6 networking so we’ll have to enable it. Start by editing (or creating) the Docker daemon configuration file at /etc/docker/daemon.json and add these options:
Warning
The official documentation doesn’t mention the fixed-cidr-v6 option but it doesn’t seem to work without this!
Then, restart the Docker daemon to apply the changes:
Next, we’ll setup an iptables route to make sure the docker network interface can receive IPv6 traffic:
Unfortunately these routes won’t persist across reboots out of the box but that’s an easy fix:
Saying yes to the default options when installing those will save your current routes and make sure they apply on boot.
At this point, you should test your connectivity to make sure IPv6 is working within Docker. An easy way to do that is to ping an IPv6 address with a small Docker container:
Setup the Pi-hole Docker Container ¶
Pi-hole has a basic script to run a Docker container that we’ll modify to our needs:
In that file, you’ll first want to change the timezone environment variable (TZ='...') to be your timezone. You can find a list of timezone names here in the TZ database name column.
Next, edit the ServerIP variable to be the static IPv4 address of your device and add a line for the ServerIPv6 variable: (don’t forget the at the end!)
Overall, the docker run part of your script should look something like this:
Running the Docker Container ¶
Now that we have the script configured we can run it! We’ll have make the script executable with chmod +x docker_run.sh and then we can run it with ./docker_run.sh. You should see some output like this to indicate it was successful:
Take note of the password for the next step.
Installing Without Docker ¶
Installing Pi-hole without Docker is quite simply since they provide an automated installer script. Just download and run it with the following command and it will walk you through the whole process!
Make sure to take note of the admin password it generates for the next step.
Testing Pi-hole ¶
Once Pi-hole is installed and running, we can test it from another computer using the dig command. You’ll want to test both the IPv4 and IPv6 address and replace them with your own addresses in these commands:
If you see output similar to mine that means Pi-hole is working and is accessible on your local network! Now you can also navigate to http://<pi-hole ip address>/admin in the browser, log in with the password it generated during installation, and check out the dashboard.
Using Pi-hole on Your Devices ¶
The easiest way to get all of your devices to use Pi-hole is to configure the DNS settings on your router to point to the IP addresses of your Pi-hole device. Most routers will let you change this, however some may have limitations such as not allowing local IP addresses or requiring at least 2 addresses to be entered. If your router requires at least 2 addresses to be entered, you can use one of your upstream DNS servers as the fallback (1.1.1.1 in my case).
If you can’t change the DNS settings at all or your router doesn’t allow local IP addresses, you’ll either need to use Pi-hole as your DHCP server or manually set the DNS settings for each device on your network. Follow the instructions below for setting up the DHCP server, or search online for how to configure the DNS server of each of your devices.
Using Pi-hole as Your DHCP Server ¶
Use Ipv6 Address In Url
If you’re using Docker, you’ll have to change the setup script and recreate your container to give Pi-hole the necessary network permissions. If you’re not using docker, skip to the next step.
Configure Docker Networking for DHCP ¶
Instead of forwarding individual ports to the container, you’ll need to switch to host networking to allow Pi-hole to receive broadcasts from the network by removing the port lines (-p ...) and adding --net=host. You’ll also need to add the NET_ADMIN capability to let Pi-hole run a DHCP server with --cap-add=NET_ADMIN. So the docker run part of your script should look something like this now:
Now you can recreate the docker container with the following commands:
Enabling the DHCP Server ¶
First, navigate to Settings → DHCP in the Pi-hole admin panel. Then, enable the DHCP server, fill in the range of IP addresses to hand out (using the same range as your existing router did), and enter your router’s IP address. If you want IPv6 support make sure to enable that. Next, save the settings and disable DHCP on your router.
Test the DHCP Server ¶
On a different device than you are currently using (so you don’t lose access to the router or Pi-hole), disconnect and reconnect to your network and make sure you still have internet access. Then, reload the Pi-hole DHCP settings page and you should see your device in the DHCP lease table.
Now that the DHCP server is working, as soon as you reconnect your device to your network they will all be using Pi-hole as their DNS server.
Next Steps ¶
Now that you have Pi-hole setup, you should start seeing fewer ads on your devices and some interesting stats on your dashboard about the queries that have been blocked. However, there are a few more things you may still want to do like adding more blocklists, unblocking certain domains, or setting up DNS over HTTPS for additional privacy.
Adding More Blocklists ¶
Pi-hole comes with a few blocklists out of the box, but there’s plenty more available online if you want to block as many ad and tracking servers as possible. There’s an excellent list at firebog.net, broken down by how likely they are to break common websites. If you want to stay on the safe side you can use only the ones with green checkmarks as they are least likely to break everyday browsing, or you can go all out and add whichever ones you’d like. Just be prepared to come back to Pi-hole to whitelist domains if something’s not working! I personally use all of the lists that do not have a strike through them.
To add these lists, first navigate to the Group Management → Adlists page. Then just copy and paste the address in and click Add.
Once you have all the lists you want, you have to update Pi-hole to use the new lists. You can do this by running pihole -g on your Pi-hole device (make sure to run docker exec -it pihole bash first if you’re using Docker to run it in the conainer) or from Tools → Update Gravity in the admin panel.
Unblocking Domains ¶
Curl Ipv6 Invalid Argument
Depending on the blocklists you choose, it’s possible some services you regularly use will now be blocked. There’s a few suggested domains to unblock at the bottom of the firebog.net list, a discussion on the Pi-hole forum, and a set of scripts that will automatically add common domains to your instance.
Unblocking Individual Domains ¶
To unblock individual domains, navigate to the Whitelist page on the Pi-hole admin panel. Then you can either type in a domain or enter a regex pattern. If you want to unblock all subdomains of a domain, you can type in the top-level domain and check the wildcard box and Pi-hole will create a regex rule for you.
Figuring Out What To Unblock ¶
Depending on the website, you can often just unblock the whole domain and that will resolve the issue. However, some websites and apps use other domains that you may not be aware of. In that case, you can navigate to the Query Log in the Pi-hole admin panel and see recent domains that were blocked. There’s also a quick whitelist button right there to automatically unblock it.
Automatically Unblock Common Services ¶
To quickly unblock a lot of common domains we can run anudeepND’s whitelist script. If using Docker, make sure to execute the script in the container by running docker exec -it pihole bash first. It’s as easy as this:
Still Not Working? ¶
If it’s still not working after unblocking some domains or you just need to use this site/app once and can’t be bothered to figure out how to unblock it, you can also temporarily disable Pi-hole completely for a set period of time. Just click on Disable in the Pi-hole admin panel and choose a time, then Pi-hole will let everything through for however long you choose.
Using DNS over HTTPS (DoH) ¶
For better privacy, you can configure Pi-hole to use DNS over HTTPS when fetching DNS results from your upstream provider if it supports it. This way, unlike the regular DNS protocol, all queries from Pi-hole will be encrypted and your ISP will not be able to see them.
While Pi-hole can’t actually use DNS over HTTPS directly, it’s super easy to setup a proxy with Docker by running the following:
If you’re not using Docker, you can also install cloudflared on your device and run it manually with the following:
This will run a DNS proxy on port 5054 that will communicate with Cloudflare’s 1.1.1.1 DNS over HTTPS. It can also be configured to work with other DoH providers. Then, you can navigate to Settings → DNS and configure Pi-hole to use this proxy by entering 127.0.0.1#5054 as a custom upstream DNS server.