Curl throws error (3) with variable but not with manually written URL

Question

I am communicating with HDFS using curl. Procedure to interact with HDFS via webhdfs is two steps and I receive a url from a first curl command:

create_request=$(curl -i -X PUT "http:/somewhere:50070/webhdfs/v1/${path}?op=CREATE")

destination=$(echo "$create_request" | grep Location | cut -d " " -f 2)

Using the destination variable, I can upload my file with the following:

curl -i -X PUT -T $local_file_path "$destination"

However, the above commands throws :

curl: (3) URL using bad/illegal format or missing URL

Using the exact same command but changing $destination bby the URL it contains (writting it manually) works file.

Why do I have this problem and how to solve this ?

PS: $destination does contain the data: http://datanode:50075/webhdfs/v1/path/test/file.jar?op=CREATE&namenoderpcaddress=cluster&overwrite=false

Edit: Quoting

path="mypath"
local_file_path="untitled.txt"

create_request="$(curl -i -X PUT "http://ip:50070/webhdfs/v1/${path}?op=CREATE")"

destination="$(echo "$create_request" | grep "Location"| cut -d ' ' -f 2)"

curl -i -X PUT -T "$local_file_path" "$destination"

Gives the same error message.

Does this answer your question? [When to wrap quotes around a shell variable?](https://stackoverflow.com/questions/10067266/when-to-wrap-quotes-around-a-shell-variable) — 0stone0, Apr 22 '21 at 14:23
@0stone0 Quoting all the strings does not solve the problem, see edit (why can't we "ping" more than one user at a time in comments... ?) — Itération 122442, Apr 22 '21 at 14:30
@FlorianCastelain Then the destination must be wrong, maybe contains a trailing space? — MichalH, Apr 22 '21 at 14:38
@MichalH A trailing space is probably not it (`cut -d ' ' -f 2` shouldn't leave any spaces at all). — Ted Lyngmo, Apr 22 '21 at 14:42
The result of the diff command is: 1c1 ``< http://url:50075/webhdfs/v1/user/florian_castelain/test/?op=CREATE&namenoderpcaddress=cluster&overwrite=false --- > http://url:50075/webhdfs/v1/user/florian_castelain/test/?op=CREATE&namenoderpcaddress=cluster&overwrite=false`` Url is correct, but I am not able to check what has changed... @MichalH — Itération 122442, Apr 22 '21 at 14:50
@FlorianCastelain Great - that log explains it. I made an answer. — Ted Lyngmo, Apr 22 '21 at 14:54

score 2 · Accepted Answer · answered Apr 22 '21 at 14:52

2

You get a \r (carriage return) back in $destination. You can remove it with tr -d '\r'

destination=$(echo "$create_request" | grep Location | cut -d " " -f 2 | tr -d '\r')

answered Apr 22 '21 at 14:52

Ted Lyngmo

60,763
5
37
77

Wow. Right. Did you understand that from the diff result ? Is carriage return the only possible reason that the diff says there is a difference while not showing it ? – Itération 122442 Apr 22 '21 at 14:56
1

@FlorianCastelain I actually misread the output :-) I thought the diff output was the raw output from my `>$destinationdest – Ted Lyngmo Apr 22 '21 at 14:57

Curl throws error (3) with variable but not with manually written URL

1 Answers1