This project objective consists on recreating some functionalities of [`wget`](https://www.gnu.org/software/wget/manual/wget.html) using **Go**
This project objective consists on recreating some functionalities of [`wget`](https://www.gnu.org/software/wget/manual/wget.html) using **Go**.
These functionalities will include:
These functionalities will consist in:
- The normal usage of `wget`, downloading a file given an URL, example: `wget https://some_url.ogr/file.zip`
- The normal usage of `wget`: downloading a file given an URL, example: `wget https://some_url.ogr/file.zip`
- Downloading a single file and saving it under a different name
- Downloading and saving the file in a specific directory
- Set the download speed, limiting the rate speed of a download
- Continue interrupted downloads
- Continuing interrupted downloads
- Downloading a file in background
- Downloading multiple files at the same time, by reading a file containing multiple download links asynchronously
- Main feature, will be to download an entire website, [mirroring a website](https://en.wikipedia.org/wiki/Mirror_site)
- Main feature will be to download an entire website, [mirroring a website](https://en.wikipedia.org/wiki/Mirror_site)
### Introduction
Wget is a free utility for non-interactive download of files from the Web. It supports HTTP, HTTPS, and FTP protocols, as well as retrieval through HTTP proxies.
To see more about wget you can visit the manual by using the command `man wget`, or you can visit the website [here](https://www.gnu.org/software/wget/manual/wget.html)
To see more about wget you can visit the manual by using the command `man wget`, or you can visit the website [here](https://www.gnu.org/software/wget/manual/wget.html).
The program should be able to give feedback, displaying the:
- Time that the program started, this must include the following format **yyyy-mm-dd hh:mm:ss**
- Status of the request. For the program to proceed to the download it must present a response to the request as status OK (`200 OK`) if not it should say which status it got and finish the operation with an error warning
- Size of the content downloaded, the content length can be presented as raw (bytes) and rounded to Mb or Gb depending on the size of the file downloaded
- Time that the program started: it must have the following format **yyyy-mm-dd hh:mm:ss**
- Status of the request. For the program to proceed to the download, it must present a response to the request as status OK (`200 OK`) if not, it should say which status it got and finish the operation with an error warning
- Size of the content downloaded: the content length can be presented as raw (bytes) and rounded to Mb or Gb depending on the size of the file downloaded
- Name and path of the file that is about to be saved
- A progress bar, having the following:
- A amount of `KiB` or `MiB` (depending on the download size) that was downloaded
@ -96,7 +96,7 @@ student@student$ ls -l ~/Downloads/meme.jpg
---
3. The program should handle speed limit, basically the program can control the speed of the download by using the flag `--rate-limit`. If you download a huge file you can limit the speed of your download, preventing the program from using the full possible bandwidth of your connection, example:
3. The program should handle speed limit. Basically the program can control the speed of the download by using the flag `--rate-limit`. If you download a huge file you can limit the speed of your download, preventing the program from using the full possible bandwidth of your connection, example:
```console
student@student$ go run main.go --rate-limit=400k https://pbs.twimg.com/media/EMtmPFLWkAA8CIS.jpg
@ -106,7 +106,7 @@ This flag should accept different value types, example: k and M. So you can put
---
4. Downloading different files should be possible. For this the program will receive `-i` flag followed by a file name that will contain all links that are to be downloaded. Example:
4. Downloading different files should be possible. For this the program will receive the `-i` flag followed by a file name that will contain all links that are to be downloaded. Example:
```console
student@student$ ls
@ -127,14 +127,14 @@ The Downloads should work asynchronously, it should download both files at the s
---
5. [**Mirror a website**](https://en.wikipedia.org/wiki/Mirror_site), this option should download the entire website being possible to use "part" of the website offline and for other useful [reasons](https://www.quora.com/How-exactly-does-Mirror-Site-works-and-how-it-is-done). For this you will have to download the websites file system and save it into a folder that will have the domain name. Example: `http://www.example.com`, the folder name will be `www.example.com` containing every file from the mirrored website. The flag should be `--mirror`.
5. [**Mirror a website**](https://en.wikipedia.org/wiki/Mirror_site). This option should download the entire website being possible to use "part" of the website offline and for other useful [reasons](https://www.quora.com/How-exactly-does-Mirror-Site-works-and-how-it-is-done). For this you will have to download the website file system and save it into a folder that will have the domain name. Example: `http://www.example.com`, will be stored in a folder with the name `www.example.com` containing every file from the mirrored website. The flag should be `--mirror`.
To mirror a website you will have to implement the following `wget` flags so that the web mirror is complete (you do not need to do the literal flags, but just the theory behind it, so your flag `--mirror` need to behave like the following wget flags combined):
- [`--convert-links`](https://www.gnu.org/software/wget/manual/wget.html), after the download is complete it will convert all links in the document to make them suitable for local viewing
- [`--page-requisites`](https://www.gnu.org/software/wget/manual/wget.html), downloads all files that are necessary to properly display a given HTML page
- [`--no-parent`](https://www.gnu.org/software/wget/manual/wget.html), this will not let the program ascend to the parent directory when retrieving
- [`--convert-links`](https://www.gnu.org/software/wget/manual/wget.html): after the download is complete it will convert all links in the document to make them suitable for local viewing
- [`--page-requisites`](https://www.gnu.org/software/wget/manual/wget.html): downloads all files that are necessary to properly display a given HTML page
- [`--no-parent`](https://www.gnu.org/software/wget/manual/wget.html): this will not let the program ascend to the parent directory when retrieving
##### Try to run the following command "`./wget https://pbs.twimg.com/media/EMtmPFLWkAA8CIS.jpg`"
##### Try to run the following command `"./wget https://pbs.twimg.com/media/EMtmPFLWkAA8CIS.jpg"`
###### Did the program download the file "`EMtmPFLWkAA8CIS.jpg`"?
###### Did the program download the file `"EMtmPFLWkAA8CIS.jpg"`?
##### Try to run the following command with a link at your choice "`./wget <https://link_of_your_choice.com>`"
##### Try to run the following command with a link at your choice `"./wget <https://link_of_your_choice.com>"`
###### Did the program download the expected file?
##### Try to run the following command "`./wget https://golang.org/dl/go1.15.linux-amd64.tar.gz`"
##### Try to run the following command `"./wget https://golang.org/dl/go1.15.linux-amd64.tar.gz"`
###### Did the program download the file "`go1.15.linux-amd64.tar.gz`"?
###### Did the program download the file `"go1.15.linux-amd64.tar.gz"`?
###### Did the program displayed the start time?
@ -24,7 +24,7 @@
###### Did the program displayed the name and path of the file that was saved?
##### Try to download a big file, for example: "`./wget http://ipv4.download.thinkbroadband.com/100MB.zip`"
##### Try to download a big file, for example: `"./wget http://ipv4.download.thinkbroadband.com/100MB.zip"`
###### Did the program download the expected file?
@ -36,27 +36,27 @@
###### While downloading, did the progress bar progressed smoothly (kept up with the time that the download took to finish)?
##### Try to run the following command, "`./wget -O=test_20MB.zip http://ipv4.download.thinkbroadband.com/20MB.zip`"
##### Try to run the following command, `"./wget -O=test_20MB.zip http://ipv4.download.thinkbroadband.com/20MB.zip"`
###### Did the program downloaded the file with the name "`test_20MB.zip`"?
###### Did the program downloaded the file with the name `"test_20MB.zip"`?
##### Try to run the following command, "`./wget -O=test_20MB.zip -P=~/Downloads/ http://ipv4.download.thinkbroadband.com/20MB.zip`", then go to the folder "`~/Downloads/`"
##### Try to run the following command, `"./wget -O=test_20MB.zip -P=~/Downloads/ http://ipv4.download.thinkbroadband.com/20MB.zip"`, then go to the folder `"~/Downloads/"`
###### Can you see the file downloaded?
##### Try to run the following command, "`./wget --rate-limit=300k http://ipv4.download.thinkbroadband.com/20MB.zip`"
##### Try to run the following command, `"./wget --rate-limit=300k http://ipv4.download.thinkbroadband.com/20MB.zip"`
###### Was the download speed always lower than 300KB/s?
##### Try to run the following command, "`./wget --rate-limit=700k http://ipv4.download.thinkbroadband.com/20MB.zip`"
##### Try to run the following command, `"./wget --rate-limit=700k http://ipv4.download.thinkbroadband.com/20MB.zip"`
###### Was the download speed always lower than 700KB/s?
##### Try to run the following command, "`./wget --rate-limit=2M http://ipv4.download.thinkbroadband.com/20MB.zip`"
##### Try to run the following command, `"./wget --rate-limit=2M http://ipv4.download.thinkbroadband.com/20MB.zip"`
###### Was the download speed always lower than 2MB/s?
##### Try to create a text file with the name "`downloads.txt`" and save into it the links below. Then run the command "`./wget -i=downloads.txt`"
##### Try to create a text file with the name `"downloads.txt"` and save into it the links below. Then run the command `"./wget -i=downloads.txt"`
