6

I'm trying to fetch a page using python requests with this code in views.py:

s = requests.Session()
r = s.get("https://www.23andme.com/")

I get the error Exceeded 30 redirects. My app is a Google App Engine Django app. I've gotten this code to work in the python console and a django server on pythonanywhere.com, but for some reason it isn't working on google app engine. What could be causing this? Thanks

Edit: It seems like there is another issue with the Requests module in my app. I have this code to add an email to a mailchimp list:

m = mailchimp.Mailchimp(MAILCHIMP_API_KEY)
list_response = m.lists.list()

but if fails with the error HTTPS/SSL is required

Connor
  • 62,633
  • 27
  • 143
  • 140
  • A `requests` session saves cookies too, the site is redirecting you for a different reason. – Martijn Pieters Feb 04 '14 at 22:35
  • Could it be the user agent? How can I make requests simulate a browser? – Connor Feb 04 '14 at 22:45
  • Usually it's one or more of the headers, yes. Add a `headers=..` dictionary and start copying headers your browser sends, see what makes it work. – Martijn Pieters Feb 04 '14 at 22:47
  • I discovered it works while in the python console but not in views.py in django – Connor Feb 04 '14 at 23:20
  • 1
    This code works for me: ``def index(request): import requests s = requests.Session() r = s.get("https://www.23andme.com/") return HttpResponse(r.text)`` When I access the function (Django 1.6.0, python 2.7.3, local server), the page for www.23andme.com is rendered. – xbello Feb 05 '14 at 16:52
  • I was also able to get it to work on a pythonanywhere.com django server but it isn't working on my google apps engine django app. Does GAE have something to do with it? – Connor Feb 05 '14 at 17:11
  • Could be; I've not had problems with `requests` on GAE before though. – Martijn Pieters Feb 07 '14 at 00:13
  • What python version are you using? (also, what happens if you try to do a request to a non `https` url?) – yuvi Feb 08 '14 at 14:39
  • works with request library 2.3.0 – Kevin Lee Dec 12 '15 at 17:38

2 Answers2

5

Try installing urllib3 (and other stuff, see below).

See, historically there were tons of issues for requests with google app engine (and see issue #498). Those were mostly resolved with urllib3 support for GAE that came with v1.3. It came out a long time ago (the current release is 1.7), so its probably not the issue, however when you initially install requests, it includes urllib3 in a folder called packages and maybe it doesn't include the entirey of it.

I also tried searching the source code for requests and found this interesting:

# Attempt to enable urllib3's SNI support, if possible
try:
    from requests.packages.urllib3.contrib import pyopenssl
    pyopenssl.inject_into_urllib3()
except ImportError:
    pass

Going deeper, the contrib package includes a pyopenssl.py script which requires:

SSL with SNI-support for Python 2.

This needs the following packages installed:

  • pyOpenSSL (tested with 0.13)
  • ndg-httpsclient (tested with 0.3.2)
  • pyasn1 (tested with 0.1.6)

So, to sum up:

  1. Install urllib3 and other SSL packages mentioned above, then try running that request you're doing again and see if anything has changed. My guess is that this will (at the very least) help with the mailchimp as it's also complaining about SSL/HTTPS issues.

  2. If that doesn't work, try using urllib3 api instead of requests to do the same task and see if that's working. If so, the problem is specifically with the packaged urllib3 that requests is using, which might need some patching.

    import urllib3
http = urllib3.PoolManager()
r = http.request('GET', 'https://www.23andme.com/')

Sorry this isn't a definite solution, hopefully one of my suggestions will help. Update me as to the progress I'll try and help out as much as I can

yuvi
  • 17,855
  • 7
  • 52
  • 91
2

According to this discussion, the issue is with netrc on Google App Engine. I was able to work around the problem by using an older version of Requests (1.2.3 in my case, but others may work too.)

Edit: According to this answer, Requests 2.1.0 should work as well.

Community
  • 1
  • 1
Connor
  • 62,633
  • 27
  • 143
  • 140
  • 2
    see [this](http://stackoverflow.com/questions/21605328/python-requests-on-google-app-engine-not-working-for-https?rq=1) as well – yuvi Feb 08 '14 at 21:55