Issue itself
During requests get(...).raw we are getting back a raw urllib3.HTTPResponse class. which is fine and intended to have, because of its IO capabilities. Considering that I don't really want to monkey-patch, is there a way to tell to this class that hey, look, I'm doing raw.read() which uses network for every iteration to avoid reading too much data at once from huge files, and so if during that time ECONNRESET comes from the server side, solve the connection and continue where it left?
The server side is external, aka non-manageable by me, only the client side.
I see no any possible way to have retry on this error, even with the urllib3's util.Retry class debugging the used libs themselves locally. Somewhat possibly similar trace-back and issue here: https://github.com/urllib3/urllib3/issues/1331
Code-sample to illustrate
import requests
raw = requests.get('http://httpbin.org/stream/20', stream=True).raw
while not line := raw.read(chunk_size=16):
print line
Traceback:
Traceback (most recent call last):
File "/python3.8/site-packages/urllib3/contrib/pyopenssl.py",
line 317, in recv_into
return self.connection.recv_into(*args, **kwargs)
File "/python3.8/site-packages/OpenSSL/SSL.py", line 1814, in recv_into
self._raise_ssl_error(self._ssl, result)
File "/python3.8/site-packages/OpenSSL/SSL.py", line 1614, in _raise_ssl_error
raise WantReadError()
OpenSSL.SSL.WantReadError
During handling of the above exception, another exception occurred:Traceback (most recent call last):
File "/python3.8/site-packages/urllib3/contrib/pyopenssl.py", line 317, in recv_into
return self.connection.recv_into(*args, **kwargs)
File "/python3.8/site-packages/OpenSSL/SSL.py", line 1814, in recv_into
self._raise_ssl_error(self._ssl, result)
File "/python3.8/site-packages/OpenSSL/SSL.py", line 1614, in _raise_ssl_error
raise WantReadError()
OpenSSL.SSL.WantReadError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/python3.8/site-packages/urllib3/contrib/pyopenssl.py", line 317, in recv_into
return self.connection.recv_into(*args, **kwargs)
File "/python3.8/site-packages/OpenSSL/SSL.py", line 1814, in recv_into
self._raise_ssl_error(self._ssl, result)
File "/python3.8/site-packages/OpenSSL/SSL.py", line 1631, in _raise_ssl_error
raise SysCallError(errno, errorcode.get(errno))
OpenSSL.SSL.SysCallError: (104, 'ECONNRESET')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/python3.8/site-packages/urllib3/response.py", line 438, in _error_catcher
yield
File "/python3.8/site-packages/urllib3/response.py", line 519, in read
data = self._fp.read(amt) if not fp_closed else b""
File "/python3.8/http/client.py", line 455, in read
n = self.readinto(b)
File "/python3.8/http/client.py", line 499, in readinto
n = self.fp.readinto(b)
File "/python3.8/socket.py", line 669, in readinto
return self._sock.recv_into(b)
File "/python3.8/site-packages/urllib3/contrib/pyopenssl.py", line 332, in recv_into
return self.recv_into(*args, **kwargs)
File "/python3.8/site-packages/urllib3/contrib/pyopenssl.py", line 332, in recv_into
return self.recv_into(*args, **kwargs)
File "/python3.8/site-packages/urllib3/contrib/pyopenssl.py", line 322, in recv_into
raise SocketError(str(e))
OSError: (104, 'ECONNRESET')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/My-Code-Calls-With_Fancypath/file.py", line XYZ, in func_name
...
File "/python3.8/site-packages/urllib3/response.py", line 541, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/python3.8/site-packages/urllib3/response.py", line 455, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: OSError("(104, \'ECONNRESET\')")', OSError("(104, 'ECONNRESET')"))
Somewhat similar problem to my question is this one: Python Requests package: lost connection while streaming. The only difference is, that the accepted answer retries the whole download.
Env:
pyOpenSSL==18.0.0 urllib3==1.26.5 requests==2.25.1 python==3.8
Finally
Let's suppose theoretically this way is impossible. Do you have any good suggestions without changing MUCH the behaviour? Like using urllib3.HTTPResponse.iter_content(...) method (although its docs says NOT REENTRANT SAFE :( )?
