Friday, April 5, 2019

How to use wget to download file via proxy in linux?

The wget program allows you to download files from URLs. Although it can do a lot, the simplest
form of the command is: wget [some URL]. Assuming no errors, it will place that file in the current
directory. If you do not specify a filename, by default it will attempt to get the index.html file.

This document descript how to set wget (The non-interactive network downloader) to download file
via proxy.

wget Configuration files
Below are wget configuration files listed by their priorities:

  1. ~/.wgetrc: User startup file.
  2. /etc/wgetrc: Default location of the global startup file.
  3. Set proxy variables in shell for current pseudo-terminal.
  4. ~/.bash_profile: User specific environment.
  5. /etc/profile: System wide environment.

Note: If higher priority configuration is not set, then the very next priority configuration takes
effective. For instance, ~/.wgetrc was not configured with proxy settings but /etc/wgetrc was
configured, then proxy settings in /etc/wgetrc are the working proxys in wget.
Configuring wget proxy

1. Add below line(s) in file ~/.wgetrc or /etc/wgetrc:

http_proxy = http://[Proxy_Server]:[port]
https_proxy = http://[Proxy_Server]:[port]

2. Set proxy variable(s) in a shell manually:

$ export http_proxy=http://[Proxy_Server]:[port]
$ export https_proxy=$http_proxy
$ export ftp_proxy=$http_proxy

Verify the variable values using the “env” command.

$ env | grep proxy

3. Add below line(s) in file ~/.bash_profile or /etc/profile:

# export http_proxy=http://[Proxy_Server]:[port]
# export https_proxy=http://[Proxy_Server]:[port]
# export ftp_proxy=http://[Proxy_Server]:[port]

4. Direct command line
$wget -e use_proxy=yes -e http_proxy=http://[Proxy_Server]:[port]  URL

How to Use pip behind a Proxy ?

On Linux

Set up the proxy through

and then install package:

pip install somepackage

or specify proxy in pip command:

sudo pip install --proxy=https://[username:password@]proxyserver:port somepackage

On Windows

You can manually set up proxy environment variables through right-click on This PC (Windows 10)
or Computer (Widnows 7) –> Proporties –> Advanced system settings –> Environment variables
then add environment variables:

You can also set up the proxy through comand lines:

set http_proxy=http://[username:password@]proxyserver:port
set http_proxy=https://[username:password@]proxyserver:port
After setting up proxies, then you can install packages through running:

pip install somepackage
Alternativelly, you can also specify proxy settings in the pip command:

pip install --proxy=https://[username:password@]proxyserver:port somepackage