Why does this url package request receive a different response than the 'equivalent' curl request?

Question

This is for sx.el, a StackExchange client for Emacs. I'm trying to determine whether this issue is a bug in url or a bug in StackExchange's API. If you need the API key, please get in touch with either myself or @Malabarba – we're on the Gitter chatroom for the project pretty regularly.

The following form evaluates to two different responses depending on whether curl is used or whether the url package is used. Since API requests are time-sensitive (and to cut down on your time investment), I've included the responses I get for the two (as parsed by json-read-from-string for readability's sake).

(defconst tmp:access-token
  "YOUR ACCESS TOKEN")

(defconst tmp:key
  "YOUR API KEY")

(defun tmp:api-bug (use-curl access-token key)
  "Post a test answer to the formatting sandbox."
  (let ((random-body-1 (md5 (current-time-string)))
        (random-body-2 (md5 (md5 (current-time-string))))
        (method "https://api.stackexchange.com/2.2/questions/3122/answers/add")
        (args
         (mapconcat
          #'identity
          `(,(format "access_token=%s"
                     (replace-regexp-in-string
                      "%" "%%" (url-hexify-string access-token)))
            ,(format "key=%s"
                     (replace-regexp-in-string
                      "%" "%%" (url-hexify-string key)))
            "site=meta"
            "pagesize=100"
            "filter=%%21GoYr1we0U5inG5G7wBg4JBGpbgX%%29C7LDqpy-%%2AbfwPOujOr4SR4W%%29bLNSyYUpQDdTwTj.XChTFB0gfLaAJq0hv"
            "body=this-is-an-answer-test-for-sx.el--%s")
          "&")))

    (if use-curl
        (shell-command-to-string
         (format
          "curl --silent -X POST --data %S %s | gunzip"
          (format args random-body-2)
          method))
      (let ((url-automatic-caching t)
              (url-inhibit-uncompression t)
              (url-request-data (format args random-body-1))
              (url-request-method "post")
              (url-request-extra-headers
               '(("Content-Type" . "application/x-www-form-urlencoded"))))
          (with-current-buffer
              (url-retrieve-synchronously method)
            (goto-char (point-min))
            (search-forward "\n\n")
            (delete-region (point-min) (point))
            (buffer-string))))))

(require 'json)

(json-read-from-string (tmp:api-bug t tmp:access-token tmp:key))
((total . 1)
 (page_size . 100)
 (page . 1)
 (quota_remaining . 9979)
 (quota_max . 10000)
 (has_more . :json-false)
 (items .
        [((link . "http://meta.stackexchange.com/questions/3122//247295#247295")
          (body_markdown . "this-is-an-answer-test-for-sx.el--e3eeb6228ed9c2c58e5385b73493f0f0")
          (share_link . "http://meta.stackexchange.com/a/247295/188148")
          (answer_id . 247295)
          (creation_date . 1421684370)
          (last_activity_date . 1421684370)
          (score . 0)
          (upvoted . :json-false)
          (downvoted . :json-false)
          (owner
           (display_name . "Sean Allred")
           (reputation . 160)))]))

(json-read-from-string (tmp:api-bug nil tmp:access-token tmp:key))
((total . 1)
 (page_size . 100)
 (page . 1)
 (quota_remaining . 9977)
 (quota_max . 10000)
 (has_more . :json-false)
 (items .
        [((link . "http://meta.stackexchange.com/questions/3122//247297#247297")
          (share_link . "http://meta.stackexchange.com/a/247297/188148")
          (answer_id . 247297)
          (creation_date . 1421684555)
          (last_activity_date . 1421684555)
          (score . 0)
          (owner
           (display_name . "Sean Allred")
           (reputation . 160)))]))

_{Cross-reference: https://stackapps.com/q/6083/11299}

@abo-abo I'm unfortunately unfamiliar with most web stuff. I've very much hacked this together (with a lot of help from @Jonathan), so I'm still learning about all this stuff. There is no mention of user agent in the variables given by url. — Sean Allred, Jan 19 '15 at 16:48
I'm not a pro either, but I know that websites can give different responses, depending on user-agent. Most tools can spoof it. And url-retrieve has a inhibit-cookies option, so at least in some cases this could be relevant. — abo-abo, Jan 19 '15 at 16:50
@abo-abo I just tried, and the cookies option makes no difference. — Malabarba, Jan 20 '15 at 16:55
Did you try url-user-agent and http://www.useragentstring.com/pages/curl/ — abo-abo, Jan 20 '15 at 17:03

score 4 · Accepted Answer · answered Jan 22 '15 at 00:14

4

Annoyingly, the problem was here: (url-request-method "post").

Changing that to (url-request-method "POST") fixes everything.

answered Jan 22 '15 at 00:14

Malabarba

23,148
6
79
164

Why does this url package request receive a different response than the 'equivalent' curl request?

1 Answers1