W wget2 Project ID: 3163647
81
Clone of the GNU Wget2 repository for collaboration via GitLab
* docs/wget2.md: --no-cache also sets the 'Cache-Control: no-cache' header leon authored 1 week ago
Name Last commit Last update
benchmarks Loading commit data...
ci-test.d
contrib
data
docs
examples
fuzz
gnulib @ 4652c7ba
include
lib
libwget
m4
po
src
tests
unit-tests
.dir-locals.el
.gitignore
.gitlab-ci.yml
.gitmodules
.lgtm.yml
.travis.sh
.travis.yml
.travis_setup.sh
AUTHORS
COPYING
COPYING.LESSER
ChangeLog
Makefile.am
NEWS
README
README.md
bootstrap
bootstrap.conf
cfg.mk
ci-test
configure.ac
libwget.pc.in
pthread_sigmask.c.mingw
todo.txt
README.md
Solaris OpenCSW
GNU Wget2 - Introduction
GNU Wget2 is the successor of GNU Wget, a file and recursive website downloader.
Designed and written from scratch it wraps around libwget, that provides the basic functions needed by a web client.
Wget2 works multi-threaded and uses many features to allow fast operation.
In many cases Wget2 downloads much faster than Wget1.x due to HTTP2, HTTP compression, parallel connections and use of If-Modified-Since HTTP header.
GNU Wget2 is licensed under GPLv3+.
Libwget is licensed under LGPLv3+.
Features
A non-exhaustive list of features
Support for HTTP/1.1 and HTTP/2.0 protocol brotli decompression support (Accept-Encoding: br) zstandard decompression support, RFC8478 (Accept-Encoding: zstd) HPKP - HTTP Public Key Pinning (RFC7469) with persistent database TCP Fast Open for plain text and for HTTPS TLS Session Resumption including persistent session data cache TLS False Start (with GnuTLS >= 3.5.0) HTTP2 support via nghttp2 and GnuTLS ALPN including streaming/pipelining OCSP stapling + OCSP server querying as a fallback (experimental, needs GnuTLS >= 3.3.11) Use libpsl for cookie domain checking (using Public Suffix List) Support link conversion (-k/--convert-links and -K/--backup-converted) Support for RFC 6266 compliant Content-Disposition RFC 6797 HSTS (HTTP Strict Transport Security) Support for bzip2 Content-Encoding / Accept-Encoding compression type New Year 2014 gimmick: added support for XZ Content-Encoding / Accept-Encoding compression type Character encoding of input files may be specified despite from local and remote encoding (--input-encoding) Support scanning RSS 2.0 feeds from local files ( --force-rss -i
Links
Online Docs
Mailing List
Bug Tracker
Development
Code Coverage
Fuzz Code Coverage
Build Requirements
The following packages are needed to build the software
autotools (autoconf, autogen, automake, autopoint, libtool) python (recommended for faster bootstrap) rsync tar makeinfo (part of texinfo) pkg-config >= 0.28 (recommended) doxygen (for creating the documentation) pandoc (for creating the wget2 man page) gettext >= 0.18.2 libiconv (needed for IRI and IDN support) libz >= 1.2.3 (the distribution may call the package zlib*, eg. zlib1g on Debian) liblzma >= 5.1.1alpha (optional, if you want HTTP lzma decompression) libbz2 >= 1.0.6 (optional, if you want HTTP bzip2 decompression) libbrotlidec/libbrotli >= 1.0.0 (optional, if you want HTTP brotli decompression) libzstd >= 1.3.0 (optional, if you want HTTP zstd decompression) libgnutls (3.3, 3.5 or 3.6) libidn2 >= 0.14 (libidn >= 1.25 if you don't have libidn2) flex >= 2.5.35 libpsl >= 0.5.0 libnghttp2 >= 1.3.0 (optional, if you want HTTP/2 support) libmicrohttpd >= 0.9.51 (optional, if you want to run the test suite) lzip (optional, if you want to build distribution tarballs) lcov (optional, for coverage reports) libgpgme >= 0.4.2 (optional, for automatic signature verification) libpcre | libpcre2 (optional, for filtering by PCRE|PCRE2 regex) libhsts (optional, to support HSTS preload lists)
The versions are recommended, but older versions may also work.
Building from git
Download project and prepare sources with
git clone https://gitlab.com/gnuwget/wget2.git cd wget2 ./bootstrap # on shell failure try 'bash ./bootstrap'
Build Wget2 with
./configure make
In Haiku build Wget2 with
setarch x86 ./configure --prefix=/boot/home/config/non-packaged rm /boot/home/config/non-packaged/wget2 && mv /boot/home/config/non-packaged/wget2_noinstall /boot/home/config/non-packaged/wget2
Test the functionality
make check
Install Wget2 and libwget
sudo make install (or su -c "make install")