8
https://www.sahibinden.com/en

If you open it incognito window and check headers in Fiddler then these are the two main headers you get: enter image description here

When I click the last one and check request headers this is what I get enter image description here

I want to get these headers in Python. Is there any way that I can get these using Selenium? Im a bit clueless here.

user3102085
  • 429
  • 3
  • 8
  • 18

6 Answers6

19

You can use Selenium Wire. It is a Selenium extension which has been developed for this exact purpose.

https://pypi.org/project/selenium-wire/

An example after pip install:

##  Import webdriver from Selenium Wire instead of Selenium
from seleniumwire import webdriver

##  Get the URL
driver = webdriver.Chrome("my/path/to/driver", options=options)
driver.get("https://my.test.url.com")

##  Print request headers
for request in driver.requests:
  print(request.url) # <--------------- Request url
  print(request.headers) # <----------- Request headers
  print(request.response.headers) # <-- Response headers
Konemiees
  • 199
  • 1
  • 6
  • Any chance you might know if something similar exists for ruby? :) – 8bithero Oct 20 '20 at 17:53
  • Unfortunately I've only used selenium with Python :/ – Konemiees Oct 21 '20 at 13:00
  • 1
    what the above package is doing is setting up a mitm proxy on selenium you can do this manually using mitmproxy also you might need to install the certificate of the selenium browser in order for it to function properly – ahmed mani Dec 25 '21 at 02:45
6

You can run JS command like this;

var req = new XMLHttpRequest()
req.open('GET', document.location, false)
req.send(null)
return req.getAllResponseHeaders()

On Python;

driver.get("https://t.me/codeksiyon")
headers = driver.execute_script("var req = new XMLHttpRequest();req.open('GET', document.location, false);req.send(null);return req.getAllResponseHeaders()")

# type(headers) == str

headers = headers.splitlines()
raifpy
  • 87
  • 1
  • 3
2

The bottom line is, No, you can't retrieve the request headers using Selenium.

Details

It had been a long time demand from the Selenium users to add the WebDriver methods to read the HTTP status code and headers from a HTTP response. We have discussed about implementing this feature through Selenium at length within the discussion WebDriver lacks HTTP response header and status code methods.

However, Jason Leyba (Selenium contributor) in his comment straightly mentioned:

We will not be adding this feature to the WebDriver API as it falls outside of our current scope (emulating user actions).

Ashley Leyba further added, attempting to make WebDriver the ideal web testing tool will suffer in overall quality as driver.get(url) blocks until the browser has loaded the page and return the response for the final loaded page. So in case of a login redirects, status codes and headers will always end up with a 200 instead of the 302 you're looking for.

Finally, Simon M Stewart (WebDriver creator) in his comment concluded that:

This feature isn't going to happen. The recommended approach is to either extend the HtmlUnitDriver to access the information you require or to make use of an external proxy that exposes this information such as the BrowserMob Proxy

undetected Selenium
  • 151,581
  • 34
  • 225
  • 281
0

It's not possible to get headers using Selenium. Further information

However, you might use other libraries such as requests, BeautifulSoup to get headers.

Baris
  • 399
  • 5
  • 12
  • These headers are shown only in icognito window when site is first visited. Then cookies are stored and this page is not visited. Would BS be able to capture those headers every time it is run? – user3102085 Jun 08 '20 at 12:47
  • Can you share any resources? – user3102085 Jun 08 '20 at 12:49
  • You can use requests to get HTML content and headers. It doesn't cache by default. (https://stackoverflow.com/questions/20198274/how-do-i-clear-cache-with-python-requests) Then, you can parse this HTML with BeautifulSoup if necessary. – Baris Jun 08 '20 at 13:12
  • I need request headers and not response headers – user3102085 Jun 08 '20 at 13:13
  • Can I intercept requests being made to url? – user3102085 Jun 08 '20 at 13:14
0

Maybe you can use BrowserMob Proxy for this. Here is a example:

import settings

from browsermobproxy import Server
from selenium.webdriver import DesiredCapabilities

config = settings.Config

server = Server(config.BROWSERMOB_PATH)
server.start()
proxy = server.create_proxy()

from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=%s' % proxy.proxy)
chrome_options.add_argument('--headless')

capabilities = DesiredCapabilities.CHROME.copy()
capabilities['acceptSslCerts'] = True
capabilities['acceptInsecureCerts'] = True

driver = webdriver.Chrome(options=chrome_options,
    desired_capabilities=capabilities,
   executable_path=config.CHROME_PATH)

proxy.new_har("sahibinden", options={'captureHeaders': True})
driver.get("https://www.sahibinden.com/en")

entries = proxy.har['log']["entries"]
for entry in entries:
    if 'request' in entry.keys():
        print(entry['request']['url'])
        print(entry['request']['headers'])
        print('\n')

proxy.close()
driver.quit()
Celso Jr
  • 129
  • 9
0
js_headers = '''
    const _xhr = new XMLHttpRequest();
    _xhr.open("HEAD", document.location, false);
    _xhr.send(null);

    const _headers = {};

    _xhr.getAllResponseHeaders().trim().split(/[\\r\\n]+/).map((value) => value.split(/: /)).forEach((keyValue) => {
        _headers[keyValue[0].trim()] = keyValue[1].trim();
    });

    return _headers;
'''

page_headers = driver.execute_script(js_headers)

type(page_headers) # -> dict
keyiflerolsun
  • 36
  • 5
  • Welcome to Stack Overflow, you should see how do I write a good answer which must have some explanation about the code, [answer]. – Sunderam Dubey May 11 '22 at 09:18