如何在每个请求上使用 selenium python 轮换各种用户代理

问题描述

我想向 python 客户端.

---

最初回答于 2018 年 11 月 6 日 8:00

否.当您使用 ChromeOptions 配置 ChromeDriver 实例以启动新的 Chrome 浏览器会话 时，ChromeDriver 的配置在 ChromeDriver 的整个生命周期中保持不变并且保持不可编辑.因此，当 WebDriver 实例执行循环发出 10 个请求时，您无法更改 用户代理.

即使您能够提取 ChromeDriver 和 ChromeSession 属性，例如UserAgent、Session ID、Cookies 和已启动的浏览会话中的其他会话属性，您仍然不会能够更改 ChromeDriver 的那些属性.

更简洁的方法是在 tearDown(){} 方法中调用 driver.quit() 以 close 和 destroy 优雅地 ChromeDriver 和 Chrome Browser 实例，然后跨越一组新的 ChromeDriver 和 Chrome 浏览器 具有新配置集的实例.

您可以在此处找到有关 如何用 selenium 重新连接到 webdriver 打开的浏览器?

---

参考

您可以在以下位置找到一些相关的详细讨论:

• 如何更改用户使用 Selenium 和 Python 的代理
• Selenium webdriver:修改导航器.webdriver 标志以防止硒检测

I want to make 10 requests to https://www.google.com/ but with random user agents using selenium and python. I've a loop and inside that loop I'm making 10 requests with random user agents (using fake-user agent). The main problem is for every request web driver is opening a new instance of google chrome and I want to do this in one single instance but with different user agents. How can I make this possible ? 1 google chrome instance and 10 requests with 10 random user agents. Here is my code:

chrome_options = Options()
chrome_options.add_argument('no-sandbox')
chrome_options.add_argument("--start-maximized")
ua = UserAgent()
for i in range(0, 10):
    userAgent = ua.random
    chrome_options.add_argument('--user-agent="' + str(userAgent) + '"')
    driver1 = webdriver.Chrome(chrome_options=chrome_options, 
    executable_path="C:/Python34/chromedriver")
    driver1.get('https://www.google.com/')
    time.sleep(5)

解决方案

First the update ¹

execute_cdp_cmd(): With the availability of execute_cdp_cmd(cmd, cmd_args) command now you can easily execute google-chrome-devtools commands using Selenium. Using this feature you can modify the user-agent easily to prevent Selenium from getting detected.

• Code Block:

  from selenium import webdriver
  
  driver = webdriver.Chrome(executable_path=r'C:WebDriverschromedriver.exe')
  print(driver.execute_script("return navigator.userAgent;"))
  # Setting user agent as Chrome/83.0.4103.97
  driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36'})
  print(driver.execute_script("return navigator.userAgent;"))
  # Setting user agent as Chrome/83.0.4103.53
  driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
  print(driver.execute_script("return navigator.userAgent;"))
  driver.get('https://www.httpbin.org/headers')
  

• Console Output:

  Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36
  Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36
  Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36
  

• Browser Snapshot:

Legend: ¹ - Applicable only to Selenium python clients.

---

Originally answered Nov 6 '18 at 8:00

No. When you configure an instance of a ChromeDriver with ChromeOptions to initiate a new Chrome Browser Session the configuration of the ChromeDriver remains unchanged throughout the lifetime of the ChromeDriver and remains uneditable. So you can't change the user agent when the WebDriver instance is executing the loop making 10 requests.

Even if you are able to extract the ChromeDriver and ChromeSession attributes e.g. UserAgent, Session ID, Cookies and other session attributes from the already initiated Browsing Session still you won't be able to change those attributes of the ChromeDriver.

A cleaner way would be to call driver.quit() within tearDown(){} method to close and destroy the ChromeDriver and Chrome Browser instances gracefully and then span a new set of ChromeDriver and Chrome Browser instance with the new set of configurations.

Here you can find a relevant discussion on How can I reconnect to the browser opened by webdriver with selenium?

---

Reference

You can find a couple of relevant detailed discussions in:

• How to change the User Agent using Selenium and Python
• Selenium webdriver: Modifying navigator.webdriver flag to prevent selenium detection