2

To wrap up the issue I found and need help on,

  • I created a python program that calls a get request from https://bx.in.th/api/pairing/
  • The program works well on my machine (Mac OSX)
  • Once running on a Digital Ocean Ubuntu droplet, it throws HTTP 403 forbidden error.
  • I did a day of research and most of the answers are to modify headers which I tried them all with no light of success.

Some links/references I went through.

Here is the simplified source code that points to the problem :

import urllib.request
import json

url = 'https://bx.in.th/api/pairing/'

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
    'Accept-Encoding': 'none',
    'Accept-Language': 'en-US,en;q=0.5',
    'Connection': 'keep-alive'
}

request = urllib.request.Request(url, headers=headers)

response = urllib.request.urlopen(request)

print(response.read())
print()
print(response.getheaders())

The proper output should be :

b'{"1":{"pairing_id":1,"primary_currency":"THB","secondary_currency":"BTC"},"21":{"pairing_id":21,"primary_currency":"THB","secondary_currency":"ETH"},"22":{"pairing_id":22,"primary_currency":"THB","secondary_currency":"DAS"},"23":{"pairing_id":23,"primary_currency":"THB","secondary_currency":"REP"},"20":{"pairing_id":20,"primary_currency":"BTC","secondary_currency":"ETH"},"4":{"pairing_id":4,"primary_currency":"BTC","secondary_currency":"DOG"},"6":{"pairing_id":6,"primary_currency":"BTC","secondary_currency":"FTC"},"24":{"pairing_id":24,"primary_currency":"THB","secondary_currency":"GNO"},"13":{"pairing_id":13,"primary_currency":"BTC","secondary_currency":"HYP"},"2":{"pairing_id":2,"primary_currency":"BTC","secondary_currency":"LTC"},"3":{"pairing_id":3,"primary_currency":"BTC","secondary_currency":"NMC"},"26":{"pairing_id":26,"primary_currency":"THB","secondary_currency":"OMG"},"14":{"pairing_id":14,"primary_currency":"BTC","secondary_currency":"PND"},"5":{"pairing_id":5,"primary_currency":"BTC","secondary_currency":"PPC"},"19":{"pairing_id":19,"primary_currency":"BTC","secondary_currency":"QRK"},"15":{"pairing_id":15,"primary_currency":"BTC","secondary_currency":"XCN"},"7":{"pairing_id":7,"primary_currency":"BTC","secondary_currency":"XPM"},"17":{"pairing_id":17,"primary_currency":"BTC","secondary_currency":"XPY"},"25":{"pairing_id":25,"primary_currency":"THB","secondary_currency":"XRP"},"8":{"pairing_id":8,"primary_currency":"BTC","secondary_currency":"ZEC"}}'

[('Date', 'Sun, 13 Aug 2017 09:27:02 GMT'), ('Content-Type', 'text/javascript'), ('Content-Length', '1485'), ('Connection', 'close'), ('Set-Cookie', '__cfduid=d51c37ea835bae4a0c892e91f34f7bc131502616422; expires=Mon, 13-Aug-18 09:27:02 GMT; path=/; domain=.bx.in.th; HttpOnly'), ('Cache-Control', 'max-age=86400'), ('Expires', 'Mon, 14 Aug 2017 09:27:02 GMT'), ('Strict-Transport-Security', 'max-age=0'), ('X-Content-Type-Options', 'nosniff'), ('Server', 'cloudflare-nginx'), ('CF-RAY', '38daa2e36e0a836b-BKK')]

The error got from running the source code on the droplet :

raceback (most recent call last):
  File "api-call.py", line 17, in <module>
    response = urllib.request.urlopen(request)
  File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.5/urllib/request.py", line 472, in open
    response = meth(req, response)
  File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.5/urllib/request.py", line 510, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 590, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

Thank you!

Tawan Thampipattanakul
  • 45
  • 3
  • 1
    It may be blocking the digitalocean's ip – t.m.adam Aug 13 '17 at 10:00
  • 1
    Is there a body attached to the response? Catch the exception, you can use `.read()` on it to get the response body. It may be the server included more detail as to why it responded with a 403. – Martijn Pieters Aug 13 '17 at 10:04
  • @MartijnPieters how would you do that? – t.m.adam Aug 13 '17 at 10:11
  • Run `curl https://bx.in.th/api/pairing/` on your server to see if they are blocking your IP range. – Himal Aug 13 '17 at 10:13
  • @t.m.adam: handling exceptions you mean? https://docs.python.org/3/tutorial/errors.html#handling-exceptions – Martijn Pieters Aug 13 '17 at 10:13
  • @MartijnPieters how would you read the response? I'm genuinely curious. – t.m.adam Aug 13 '17 at 10:14
  • @t.m.adam: I already told you; catch the exception; the exception object has a `.read()` method. – Martijn Pieters Aug 13 '17 at 10:18
  • @MartijnPieters thanks for clarifying. I thought you meant `read()` the response. – t.m.adam Aug 13 '17 at 10:22
  • @t.m.adam: the exception object represents the response. – Martijn Pieters Aug 13 '17 at 10:24
  • @MartijnPieters yes, got it now – t.m.adam Aug 13 '17 at 10:25
  • @Himal I tried running the command and then I got HTML page(in text) without error. – Tawan Thampipattanakul Aug 13 '17 at 10:28
  • @TawanThampipattanakul You meant a `text/javascript` response ? You could use `curl -I https://bx.in.th/api/pairing/` to get the headers only so you can see the HTTP status code. – Himal Aug 13 '17 at 12:37
  • I followed your suggestion @MartijnPieters and retrieved HTML bytes from exceptionObject.read(). then I got following page : Please enable cookies. One more step Please complete the security check to access bx.in.th Why do I have to complete a CAPTCHA? Completing the CAPTCHA proves you are a human and gives you temporary access to the web property. ... Cloudflare Ray ID: 38db27a5582c6fcc • Your IP: 128.1XX.XX.XX • Performance & security by Cloudflare – Tawan Thampipattanakul Aug 13 '17 at 14:03
  • @TawanThampipattanakul: so they are using cloudflare to protect their site and digital ocean's IP address is subject to extra checks. – Martijn Pieters Aug 13 '17 at 14:16
  • @MartijnPieters so... is this considered as impossible? – Tawan Thampipattanakul Aug 13 '17 at 14:43
  • @TawanThampipattanakul: Cloudflare certainly tries to make it impossible; perhaps you need to respect the restrictions that the site admins have put in place? – Martijn Pieters Aug 13 '17 at 15:37
  • @MartijnPieters I know the reason. bx.in.th is currently attacked by DDOS frequently, so I believe they setup a restriction against DO droplets' IP since it is low cost and efficient for the attack. – Tawan Thampipattanakul Aug 13 '17 at 15:52
  • Thank you so much for all of your help :) To summarize, the API i want to access uses CloudFlare to block DO IPs for security reasons so I decided to use another cloud provider instead. – Tawan Thampipattanakul Aug 13 '17 at 15:57

1 Answers1

1

You have to use strong proxy like Luminati. I also was getting 403 error status, but it works well with luminati proxy.

jis0324
  • 195
  • 1
  • 15
  • My apology, I somehow missed this answer. Just want to confirm that when I ran into this same problem few years after I opened this question, a friend suggested and I tried luminati proxy which worked pretty well for me. – Tawan Thampipattanakul Jan 16 '22 at 20:10
  • k, sounds good. hope you are going well. thanks. – jis0324 Jan 17 '22 at 18:12