如何在 Google Cloud ML 上使用 pandas.read_csv?

2022-01-25 00:00:00 python pandas google-cloud-storage google-cloud-ml

问题描述

我正在尝试在 Google Cloud ML 上部署训练脚本.当然，我已将我的数据集(CSV 文件)上传到 GCS 的存储桶中.

I'm trying to deploy a training script on Google Cloud ML. Of course, I've uploaded my datasets (CSV files) in a bucket on GCS.

我曾经使用 read_csv 从 pandas 导入我的数据，但它似乎不适用于 GCS 路径.

I used to import my data with read_csv from pandas, but it doesn't seem to work with a GCS path.

我应该如何继续(我想继续使用 pandas)?

How should I proceed (I would like to keep using pandas) ?

import pandas as pd
data = pd.read_csv("gs://bucket/folder/file.csv")

输出:

ERROR 2018-02-01 18:43:34 +0100 master-replica-0 IOError: File gs://bucket/folder/file.csv does not exist

解决方案

您将需要使用 tensorflow.python.lib.io 中的 file_io 来执行此操作，如图所示下面:

You will require to use file_io from tensorflow.python.lib.io to do that as demonstrated below:

from tensorflow.python.lib.io import file_io
from pandas.compat import StringIO
from pandas import read_csv

# read csv file from google cloud storage
def read_data(gcs_path):     
   file_stream = file_io.FileIO(gcs_path, mode='r')
   csv_data = read_csv(StringIO(file_stream.read()))
   return csv_data

现在调用上面的函数

 gcs_path = 'gs://bucket/folder/file.csv' # change path according to your bucket, folder and path
 df = read_data(gcs_path)
 # print(df.head()) # displays top 5 rows including headers as default

相关文章