Selenium webdriver:修改 navigator.webdriver 标志以防止硒检测

2022-01-16 00:00:00 selenium selenium-webdriver webdriver google-chrome-devtools user-agent java webdriver-w3c-spec

我正在尝试使用 selenium 和 chrome 自动执行网站中的一项非常基本的任务，但网站以某种方式检测到 chrome 何时由 selenium 驱动并阻止每个请求.我怀疑该网站依赖于像这样 https://stackoverflow.com/a/41904453/648236

I'm trying to automate a very basic task in a website using selenium and chrome but somehow the website detects when chrome is driven by selenium and blocks every request. I suspect that the website is relying on an exposed DOM variable like this one https://stackoverflow.com/a/41904453/648236 to detect selenium driven browser.

我的问题是，有没有办法可以使 navigator.webdriver 标志为假?我愿意在进行修改后尝试重新编译 selenium 源，但我似乎无法在存储库中的任何地方找到 NavigatorAutomationInformation 源 https://github.com/SeleniumHQ/selenium

My question is, is there a way I can make the navigator.webdriver flag false? I am willing to go so far as to try and recompile the selenium source after making modifications, but I cannot seem to find the NavigatorAutomationInformation source anywhere in the repository https://github.com/SeleniumHQ/selenium

非常感谢任何帮助

PS:我还尝试了 https://w3c.github.io/webdriver/#界面

Object.defineProperty(navigator, 'webdriver', {
    get: () => false,
  });

但它只在初始页面加载后更新属性.我认为网站会在我的脚本执行之前检测到变量.

But it only updates the property after the initial page load. I think the site detects the variable before my script is executed.

推荐答案

先更新1

execute_cdp_cmd():现在有了 execute_cdp_cmd(cmd, cmd_args) 命令，您可以轻松执行 google-chrome-devtools 命令 使用 硒.使用此功能，您可以轻松修改 navigator.webdriver 以防止 Selenium 被检测到.

First the update 1

execute_cdp_cmd(): With the availability of execute_cdp_cmd(cmd, cmd_args) command now you can easily execute google-chrome-devtools commands using Selenium. Using this feature you can modify the navigator.webdriver easily to prevent Selenium from getting detected.

为了防止 Selenium 驱动的 WebDriver 被检测到，一种利基方法将包括以下任一/所有步骤:

To prevent Selenium driven WebDriver getting detected a niche approach would include either/all of the below mentioned steps:

  • 添加参数 --disable-blink-features=AutomationControlled

from selenium import webdriver

options = webdriver.ChromeOptions() 
options.add_argument('--disable-blink-features=AutomationControlled')
driver = webdriver.Chrome(options=options, executable_path=r'C:WebDriverschromedriver.exe')
driver.get("https://www.website.com")

您可以在Selenium 无法打开中找到相关的详细讨论第二页

  • 轮换 user-agent通过execute_cdp_cmd()命令如下:

    #Setting up Chrome/83.0.4103.53 as useragent
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})

  • 将 webdriver 的 navigator 的 property 值更改为 undefined

    driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")

  • 排除enable-automation开关的集合

    options.add_experimental_option("excludeSwitches", ["enable-automation"])

  • 关闭useAutomationExtension

    options.add_experimental_option('useAutomationExtension', False)

    • 合并上述所有步骤和有效的代码块将是:

    Clubbing up all the steps mentioned above and effective code block will be:

    from selenium import webdriver

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:WebDriverschromedriver.exe')
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
print(driver.execute_script("return navigator.userAgent;"))
driver.get('https://www.httpbin.org/headers')

    历史

    根据W3C Editor's Draft，当前的实现严格提到:

    History

    As per the W3C Editor's Draft the current implementation strictly mentions:

    webdriver-active flag 设置为 true 当 user agent 在 remote control 下，最初设置为 false.

    The webdriver-active flag is set to true when the user agent is under remote control which is initially set to false.

    进一步，

    Navigator includes NavigatorAutomationInformation;

    需要注意的是:

    NavigatorAutomationInformation 接口不应暴露在 WorkerNavigator 上.

    NavigatorAutomationInformation 接口定义为:

    interface mixin NavigatorAutomationInformation {
    readonly attribute boolean webdriver;
};

    如果设置了 webdriver-active flag，则返回 true，否则返回 false.

    which returns true if webdriver-active flag is set, false otherwise.

    最后，navigator.webdriver 定义了一种标准方式，用于协作用户代理通知文档它由 WebDriver 控制，以便替代代码路径可以在自动化期间触发.

    Finally, the navigator.webdriver defines a standard way for co-operating user agents to inform the document that it is controlled by WebDriver, so that alternate code paths can be triggered during automation.

    警告:更改/调整上述参数可能会阻塞导航并检测到WebDriver实例.

    Caution: Altering/tweaking the above mentioned parameters may block the navigation and get the WebDriver instance detected.

    更新(2019 年 11 月 6 日)

    就目前的实现而言，在不被检测到的情况下访问网页的理想方法是使用 ChromeOptions() 类添加几个参数:

    • 排除enable-automation开关的集合
    • 关闭useAutomationExtension

    通过一个ChromeOptions的实例如下:

    • Java 示例:

    • Java Example:

    System.setProperty("webdriver.chrome.driver", "C:\Utility\BrowserDrivers\chromedriver.exe");
ChromeOptions options = new ChromeOptions();
options.setExperimentalOption("excludeSwitches", Collections.singletonList("enable-automation"));
options.setExperimentalOption("useAutomationExtension", false);
WebDriver driver =  new ChromeDriver(options);
driver.get("https://www.google.com/");

  • Python 示例

  • Python Example

    from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:path	ochromedriver.exe')
driver.get("https://www.google.com/")

    • 1:仅适用于 Selenium 的 Python 客户端.

    1: Applies to Selenium's Python clients only.

    2:仅适用于 Selenium 的 Python 客户端.

    2: Applies to Selenium's Python clients only.

    3:仅适用于 Selenium 的 Python 客户端.

    3: Applies to Selenium's Python clients only.

相关文章