How can I download weblinks using wget which do not end with .html?

Question

I want to download this webpage using Wget on Windows 7:

http://www.att.com/shop/wireless/devices/smartphones.deviceListView.xhr.flowtype-NEW.deviceGroupType-Cellphone.paymentType-postpaid.packageType-undefined.html?commitmentTerm=24&taxoStyle=SMARTPHONES&showMoreListSize=1000

I am using this command to do this:

wget -E -H -k -K -p -e robots=off -P /Downloads/AT&T_2013-01-29/ http://www.att.com/shop/wireless/devices/smartphones.deviceListView.xhr.flowtype-NEW.deviceGroupType-Cellphone.paymentType-postpaid.packageType-undefined.html?commitmentTerm=24&taxoStyle=SMARTPHONES&showMoreListSize=1000

I am getting "taxostyle not defined", "commitmentterm not defined" or "recognizble method error".

Answer 1

You have two ampersands ( & ) and a question mark in your URL.

I am not sure how your Windows shell handles that. It probably depends on what you are using. (Cmd.exe? PowerShell, Hamilton C-shell for Windows, Cygwin/Bash?)

Regardless, many of those will try to parse those and you will end up with Wget getting a different result than you expect. You would either need to escape those characters (how depends on the shell), or use " marks.

If I use wget -E -H -k -K -p -e robots=off "http://www.att.com/shop/wireless/devices/smartphones.deviceListView.xhr.flowtype-NEW.deviceGroupType-Cellphone.paymentType-postpaid.packageType-undefined.html?commitmentTerm=24&taxoStyle=SMARTPHONES&showMoreListSize=1000" in the Bash 3 shell on BSD (notice the added quotes around the download URL) then the command does seem to work. (It downloads three directories, 2.ecom.attccc.com, 3.ecom.attccc.com, and www.att.com)

---

As an alternative, you can put your URL in a FILE as it is (one in a line) and pass an additional -i FILE parameter to wget instead of URL itself.

Another case to check what is really passed to Wget by your command is echo it, like this echo HERE_IS_URL and if the output is acceptable, you can replace echo with wget or pipe it forward with | wget -i-.