通过 Selenium chromedriver 进行 Python 代理身份验证

问题描述

We tried for a few days to setup Proxy Authentication with selenium chromedriver in Python. We couldn't set ip up because Chrome throws a pop-up for authentication. Problem is that selenium can't switch to that window and so, can't type. The only solution that worked for us was using pyautogui which is a bad solution for us because we want to use the headless function.

Here are all the methods we tried:

driver.switch_to_window()

driver.switch_to_active_element()

driver.switch_to_alert()

ActionChains(driver).send_keys("test").perform()

driver.switch_to_alert()

options = webdriver.ChromeOptions()
options.add_argument('--proxy-server=http://user:pass@3.223.68.195:31112')

driver.get("https://user:pass@google.com")

proxy = {'address': '3.209.253.119:31112',
         'username': 'user',
         'password': 'pass'}


capabilities = dict(DesiredCapabilities.CHROME)
capabilities['proxy'] = {'proxyType': 'MANUAL',
                         'httpProxy': proxy['address'],
                         'ftpProxy': proxy['address'],
                         'sslProxy': proxy['address'],
                         'noProxy': '',
                         'class': "org.openqa.selenium.Proxy",
                         'autodetect': False}

capabilities['proxy']['socksUsername'] = proxy['username']
capabilities['proxy']['socksPassword'] = proxy['password']

driver = webdriver.Chrome(executable_path="chromedriver.exe", desired_capabilities=capabilities)
driver.get("http://google.com")

Any help would be really appreciated :)

解决方案

If you need to use a proxy with no authentication (no username or password) with python and Selenium library with chromedriver you usually use the following code:

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=%s' % hostname + ":" + port)
driver = webdriver.Chrome(chrome_options=chrome_options)

It works fine unless proxy requires authentication. if the proxy requires you to log in with a username and password it will not work. In this case, you have to use more tricky solution that is explained below.

To set up proxy authentication we will generate a file and upload it to chromedriver dynamically using the following code below. This effectively create a chrome extension. This code configures selenium with chromedriver to use HTTP proxy that requires authentication with user/password pair.

import os
import zipfile

from selenium import webdriver


def create_chromedriver(PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS, USER_AGENT):
    manifest_json = """
    {
    "version": "1.0.0",
    "manifest_version": 2,
    "name": "Chrome Proxy",
    "permissions": [
        "proxy",
        "tabs",
        "unlimitedStorage",
        "storage",
        "<all_urls>",
        "webRequest",
        "webRequestBlocking"
    ],
    "background": {
        "scripts": ["background.js"]
    },
    "minimum_chrome_version":"22.0.0"
    }
    """

    background_js = """
    var config = {
        mode: "fixed_servers",
        rules: {
        singleProxy: {
            scheme: "http",
            host: "%s",
            port: parseInt(%s)
        },
        bypassList: ["localhost"]
        }
    };

    chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

    function callbackFn(details) {
        return {
            authCredentials: {
            username: "%s",
            password: "%s"
            }
        };
    }

    chrome.webRequest.onAuthRequired.addListener(
            callbackFn,
            {urls: ["<all_urls>"]},
            ['blocking']
    );
    """ % (PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS)


    def get_chromedriver(use_proxy=True, user_agent=USER_AGENT):
        path = os.path.dirname(os.path.abspath(__file__))
        chrome_options = webdriver.ChromeOptions()
        if use_proxy:
            pluginfile = 'proxy_auth_plugin.zip'
            with zipfile.ZipFile(pluginfile, 'w') as zp:
                zp.writestr("manifest.json", manifest_json)
                zp.writestr("background.js", background_js)
            chrome_options.add_extension(pluginfile)
        if user_agent:
            chrome_options.add_argument('--user-agent=%s' % USER_AGENT)
        driver = webdriver.Chrome(
            os.path.join(path, 'chromedriver'),
            chrome_options=chrome_options)
        return driver

    driver = get_chromedriver(use_proxy=True)
    # driver.get('https://www.google.com/search?q=my+ip+address')
    driver.get('https://httpbin.org/ip')


user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36'
create_chromedriver('192.168.3.2', 8080, 'user', 'pass', user_agent)

Function get_chromedriver returns configured selenium web driver that you can use in your application. This code is tested and works just fine.

To implement this into your own code just call the function create_chromedriver with args of host, port, user, pass and user-agent. If you do not want to use proxy or user-agent just edit get_chromedriver accordingly so the args are false.