GCloud Upload httplib2.RedirectMissingLocation: 重定向但响应缺少 Location: 标头

2022-01-25 00:00:00 python file-upload google-cloud-storage gcloud gcloud-python

问题描述

我正在尝试使用一个简单的 python 程序将一个小文件上传到 gcloud

I am attempting to upload a small file to gcloud using a simple python program

client = storage.Client(project=GCLOUD_PROJECT)
bucket = client.get_bucket(GCLOUD_BUCKET)
blob = bucket.blob(GCLOUD_FILE_ON_CLOUD)
blob.upload_from_filename(GCLOUD_FILE_LOCAL)

直到最近它一直在工作，但发生了一些变化.现在，每当我上传大于 5MB 的文件时，都会出现以下错误.小于或等于 5MB 的文件会通过.尺寸不够大，无法实现续传，是吗?

It had been working until recently and something changed. Now, whenever I upload a file greater than 5MB I get the below error. Files less than or equal to 5MB goes through. The size isn't large enough to implement resumable upload, is it?

Traceback (most recent call last):
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/gcloud_upload.py", line 40, in <module>
    blob.upload_from_filename(GCLOUD_FILE_LOCAL)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/storage/blob.py", line 597, in upload_from_filename
    encryption_key=encryption_key, client=client)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/storage/blob.py", line 543, in upload_from_file
    http_response = upload.stream_file(use_chunks=True)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/transfer.py", line 1086, in stream_file
    response = send_func(self.stream.tell())
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/transfer.py", line 1215, in _send_chunk
    return self._send_media_request(request, end)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/transfer.py", line 1125, in _send_media_request
    self.bytes_http, request, retries=self.num_retries)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/http_wrapper.py", line 423, in make_api_request
    check_response_func=check_response_func)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/gcloud/streaming/http_wrapper.py", line 371, in _make_api_request_no_retry
    redirections=redirections, connection_type=connection_type)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/oauth2client/transport.py", line 175, in new_request
    redirections, connection_type)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/oauth2client/transport.py", line 282, in request
    connection_type=connection_type)
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/httplib2/__init__.py", line 1986, in request
    cachekey,
  File "/Users/mmumshad/PycharmProjects/quiz-python-flask-angular/venv36/lib/python3.6/site-packages/httplib2/__init__.py", line 1685, in _request
    content,
httplib2.RedirectMissingLocation: Redirected but the response is missing a Location: header.

当我调试时，我看到以下内容.

When I debug I see the below. 

{
 'content-type': 'text/plain; charset=utf-8', 
 'range': 'bytes=0-1048575', 
 'content-length': '0', 
 'date': 'Sun, 19 Jan 2020 23:52:13 GMT', 
 'server': 'UploadServer', 
 'alt-svc': 'quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000', 
 'status': '308'}

我的点子列表供参考.操作系统是 MAC OSx.我也在Linux上测试过.同样的问题.

My pip list for reference. OS is MAC OSx. I tested on Linux as well. Same issue.

adal                     1.2.2    
bson                     0.5.8    
cachetools               3.1.1    
certifi                  2019.9.11
cffi                     1.13.1   
chardet                  3.0.4    
cryptography             2.8      
dnspython                1.16.0   
gcloud                   0.18.3   
gitdb2                   2.0.6    
GitPython                3.0.5    
google-auth              1.6.3    
googleapis-common-protos 1.51.0   
httplib2                 0.16.0   
idna                     2.7      
itsdangerous             1.1.0    
MarkupSafe               1.1.1    
oauth2client             4.1.3    
oauthlib                 3.1.0    
pip                      19.0.3   
protobuf                 3.11.2   
pyasn1                   0.4.7    
pyasn1-modules           0.2.7    
pycparser                2.19     
PyJWT                    1.7.1    
python-dateutil          2.8.0    
requests                 2.19.1   
requests-oauthlib        1.2.0    
rsa                      4.0      
setuptools               40.8.0   
six                      1.12.0   
smmap2                   2.0.5    
urllib3                  1.23     
websocket-client         0.56.0   
Werkzeug                 0.16.0

这是上周的工作.最近有什么变化吗?

This was working last week. Has something changed recently?

解决方案

解决方案

gcloud 包已被弃用两次，并且与 httplib2>=0.16 不兼容.正确的解决方案是使用 google-cloud-* 软件包系列.

Solutions

gcloud package is deprecated twice and is not compatible with httplib2>=0.16. Proper solution is to use google-cloud-* packages family.

google-api-python-client>=1.7.12 正在使用 redirect_codes API，请升级，它可以正常工作.

google-api-python-client>=1.7.12 is using redirect_codes API, please upgrade, it just works.

httplib2 v0.17.0 刚刚发布，能够修改被视为重定向的响应代码集.如果您可以修改创建 Http 对象的代码，这是最好的选择:

httplib2 v0.17.0 is just released with ability to modify set of response codes treated as redirects. This is the best option if you can modify code that creates Http object:

http = httplib2.Http()
http.redirect_codes = http.redirect_codes - {308}

如果这不可行，请编辑您的 requirements.txt 以固定 httplib2<0.16.0

Iff that's not possible, edit your requirements.txt to pin down httplib2<0.16.0

谷歌云存储服务器使用 HTTP 308对于特殊的 可恢复上传功能，这有点类似于在同一位置重试相同的方法";，但不完全是.

google cloud storage server uses HTTP 308 for special resumable uploads feature which somewhat resembles "retry same method to same location", but not quite.

以上(可能)是 PyPI 包 google-resumable-media 的基本原理它被 gcloud 相关软件包的最新版本使用，并以类似的方式处理 200 和 308，这与通用 HTTP 客户端不同.

Above is (probably) rationale for PyPI package google-resumable-media which is used by more recent incarnations of gcloud related packages and handles 200 and 308 in similar manner, unlike generic HTTP client should.

历史背景:

  • 2016 包 gcloud 被弃用，取而代之的是包 google-cloud
  • 2017 google-cloud-python 从 httplib2 切换 HTTP 传输请求
  • 2018 package google-cloud 再次被弃用，取而代之的是 google-cloud-*包
  • 2020 httplib2 v0.16 支持 308 个重定向，每个 RFC7538
  • 2016 package gcloud was deprecated in favor of package google-cloud
  • 2017 google-cloud-python switches HTTP transport from httplib2 to requests
  • 2018 package google-cloud was deprecated again in favor of google-cloud-* packages
  • 2020 httplib2 v0.16 gained support for 308 redirects per RFC7538

抱歉，有坏消息.作为 HTTP 爱好者，我偏向于 308 支持.如果您有更好的想法如何更优雅地处理这种情况，请联系.

Sorry for bad news. As HTTP enthusiast, I'm biased towards 308 support. Please reach if you have better idea how to handle this situation more gracefully.

相关文章