Python urllib2.open 连接由对等错误重置

2022-01-24 00:00:00 python connection screen-scraping reset

问题描述

我正在尝试使用 python 抓取页面

I'm trying to scrape a page using python

问题是，我不断收到对等方重置 Errno54 连接.

The problem is, I keep getting Errno54 Connection reset by peer.

运行此代码时出现错误 -

The error comes when I run this code -

urllib2.urlopen("http://www.bkstr.com/webapp/wcs/stores/servlet/CourseMaterialsResultsView?catalogId=10001&categoryId=9604&storeId=10161&langId=-1&programId=562&termId=100020629&divisionDisplayName=Stanford&departmentDisplayName=ILAC&courseDisplayName=126&sectionDisplayName=01&demoKey=d&purpose=browse")

此页面上的所有网址都会发生这种情况 - 有什么问题?

this happens for all the urls on this pag- what is the issue?

解决方案

$> telnet www.bkstr.com 80 Trying 64.37.224.85... Connected to www.bkstr.com. Escape character is '^]'. GET /webapp/wcs/stores/servlet/CourseMaterialsResultsView?catalogId=10001&categoryId=9604&storeId=10161&langId=-1&programId=562&termId=100020629&divisionDisplayName=Stanford&departmentDisplayName=ILAC&courseDisplayName=126&sectionDisplayName=01&demoKey=d&purpose=browse HTTP/1.0 Connection closed by foreign host.

从 python 或其他任何地方获取该 URL 不会有任何乐趣.如果它在您的浏览器中工作，那么肯定有其他事情发生，例如 cookie 或身份验证等.或者，可能是服务器坏了，或者他们改变了配置.

You're not going to have any joy fetching that URL from python, or anywhere else. If it works in your browser then there must be something else going on, like cookies or authentication or some such. Or, possibly, the server's broken or they've changed their configuration.

尝试在您以前从未访问过该网站的浏览器中打开它以进行检查.然后登录并重试.

Try opening it in a browser that you've never accessed that site in before to check. Then log in and try it again.

毕竟是 cookie:

It was cookies after all:

import cookielib, urllib2 cj = cookielib.CookieJar() opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj)) #Need to set a cookie opener.open("http://www.bkstr.com/") #Now open the page we want data = opener.open("http://www.bkstr.com/webapp/wcs/stores/servlet/CourseMaterialsResultsView?catalogId=10001&categoryId=9604&storeId=10161&langId=-1&programId=562&termId=100020629&divisionDisplayName=Stanford&departmentDisplayName=ILAC&courseDisplayName=126&sectionDisplayName=01&demoKey=d&purpose=browse").read()

输出看起来不错，但您必须检查它是否符合您的要求:)

The output looks ok, but you'll have to check that it does what you want :)

相关文章