在 nseindia.com 上单击获取数据按钮以获取每月结算统计信息不会使用 Selenium 和 Python 获取结果

2022-01-15 00:00:00 python selenium selenium-chromedriver selenium-webdriver google-chrome

问题描述

我正在尝试从

解决方案

我对你的代码添加了一些调整并运行测试如下:

  • 代码块:

    从 selenium 导入 webdriver从 selenium.webdriver.common.by 导入从 selenium.webdriver.support.ui 导入 WebDriverWait从 selenium.webdriver.support 导入 expected_conditions 作为 EC选项 = webdriver.ChromeOptions()options.add_argument("开始最大化")options.add_experimental_option("excludeSwitches", ["enable-automation"])options.add_experimental_option('useAutomationExtension', False)driver = webdriver.Chrome(options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')driver.get('https://www1.nseindia.com/products/content/equities/equities/eq_monthly_statistics.htm')Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"select#h_filetype")))).select_by_visible_text("资本市场")Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"select#yearField")))).select_by_visible_text("2019-2020")WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input.getdata-button#get[type='image'][src^='/common/images/btn-get-data.gif']))).点​​击()

<小时>

观察

与您的观察类似，我遇到了同样的障碍，但没有结果如下:

<小时>

深入研究

似乎 Get Data 元素上的 click() 文本确实发生了.但是在检查 webpage 的 驱动 WebDriver 实例，响应被阻塞.

<小时>

参考文献

一些文档:

  • 机器人管理器
  • 机器人经理:基础
<小时>

tl;博士

几个相关的讨论:

  • Selenium webdriver:修改导航器.webdriver 标志以防止硒检测
  • 无法使用 Selenium 自动化 Chase网站登录

I am trying to scrape data from here.

By clicking on the capital market and 2019-20 year. I want to click on Get data.

I have used following code:

driver = webdriver.Chrome(executable_path=chrome_path,options=chrome_options)

driver.get( nse_cash_keystats_page )


 driver.find_element_by_xpath( "//select[@id='h_filetype']/option[text()='Capital Market ']" ).click()

driver.find_element_by_xpath( "//select[@id='yearField']/option[text()='2019-2020']" ).click()

     downloadButton=WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.XPATH,'//input[@type="image"][@src="/common/images/btn-get-data.gif"]')))

driver.execute_script("arguments[0].click();", downloadButton)

By using the above code, I am able to click on Get DATA. But it is not showing output.

Please help me.Thanks in advance.

解决方案

I took your code added a few tweaks and ran the test as follows:

  • Code Block:

    from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions() 
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')
driver.get('https://www1.nseindia.com/products/content/equities/equities/eq_monthly_statistics.htm')
Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"select#h_filetype")))).select_by_visible_text("Capital Market ")
Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"select#yearField")))).select_by_visible_text("2019-2020")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input.getdata-button#get[type='image'][src^='/common/images/btn-get-data.gif']"))).click()

Observation

Similar to your observation, I have hit the same roadblock with no results as follows:

Deep Dive

It seems the click() on the element with text as Get Data does happens. But while inspecting the DOM Tree of the webpage you will find that some of the <script> tag refers to JavaScripts having keyword akam. As an example:

  • <script type="text/javascript" src="https://www1.nseindia.com/akam/11/52349752" defer=""></script>
  • <noscript><img src="https://www1.nseindia.com/akam/11/pixel_52349752?a=dD01ZDZiMTA5OGQ0MDljYTYxN2RjMjc3MzBlN2YwMDQ0NjlkZDNiNTMzJmpzPW9mZg==" style="visibility: hidden; position: absolute; left: -999px; top: -999px;" /></noscript>

Which is a clear indication that the website is protected by Bot Manager an advanced bot detection service provided by Akamai and the response gets blocked.

Bot Manager

As per the article Bot Manager - Foundations:

Conclusion

So it can be concluded that the request for the data is detected as being performed by Selenium driven WebDriver instance and the response is blocked.

References

A couple of documentations:

  • Bot Manager
  • Bot Manager : Foundations

tl; dr

A couple of relevant discussions:

  • Selenium webdriver: Modifying navigator.webdriver flag to prevent selenium detection
  • Unable to use Selenium to automate Chase site login

相关文章