XHR 请求 URL 在尝试解析其内容时说不存在

2022-01-15 00:00:00 python python-requests web-scraping xmlhttprequest

问题描述

在我使用 Scrapy 为我的问题构建一个完整的解决方案之前，我发布了一个我想要做的简单版本:

Before I build a full solution to my problem using Scrapy I am posting a simplistic version of what I want to do:

import requests

url = 'http://www.whoscored.com/stageplayerstatfeed/?field=1&isAscending=false&orderBy=Rating&playerId=-1&stageId=9155&teamId=32"'

params = {'d': date.strftime('%Y%m'), 'isAggregate': 'false'}
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36'}

response = requests.get(url, params=params, headers=headers)

fixtures = response.body
#fixtures = literal_eval(response.content)
print fixtures

此代码表示上述 URL 不存在.该 URL 与您从本页主表的总体"切换到主页"选项卡时提交的 XHR 请求相关:

This code is saying that the above URL does not exist. The URL relates to an XHR request that is submitted when you toggle from the 'Overall' to the 'Home' tab of the main table on this page:

http://www.whoscored.com/Teams/32/

如果您在 Google Developer Tools 的控制台中激活 XHR 日志记录，您可以看到 XHR 请求和从服务器以字典形式发送的响应(这是预期的格式).

If you activate XHR logging within the Console of Google Developer Tools you can see both the XHR request and the response sent from the server in the form of a dictionary (which is the expected format).

谁能告诉我为什么上面的代码没有返回我希望看到的数据?

Can anyone tell me why the above code is not returning the data I would expect to see?

谢谢

解决方案

你有几个问题:

  • 网址应为 http://www.whoscored.com/stageplayerstatfeed
  • 错误的GET参数
  • 缺少重要的必填标题
  • 你需要 response.json()，而不是 response.body
  • the url should be http://www.whoscored.com/stageplayerstatfeed
  • wrong GET parameters
  • missing important required headers
  • you need response.json(), not response.body

固定版本:

import requests

url = 'http://www.whoscored.com/stageplayerstatfeed'
params = {
    'field': '1',
    'isAscending': 'false',
    'orderBy': 'Rating',
    'playerId': '-1',
    'stageId': '9155',
    'teamId': '32'
}
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36',
           'X-Requested-With': 'XMLHttpRequest',
           'Host': 'www.whoscored.com',
           'Referer': 'http://www.whoscored.com/Teams/32/'}

response = requests.get(url, params=params, headers=headers)

fixtures = response.json()
print fixtures

打印:

[
    {
        u'AccurateCrosses': 0,
        u'AccurateLongBalls': 10,
        u'AccuratePasses': 89,
        u'AccurateThroughBalls': 0,
        u'AerialLost': 2,
        u'AerialWon': 4,
        ...
    },
    ...
]

相关文章