如何在 anaconda 中导入 pyspark
问题描述
我正在尝试将 pyspark 与 anaconda 一起导入和使用.
I am trying to import and use pyspark with anaconda.
安装 spark 后，我尝试设置 $SPARK_HOME 变量:
After installing spark, and setting the $SPARK_HOME variable I tried:
$ pip install pyspark
(当然)这行不通，因为我发现我需要通过 tel python 在 $SPARK_HOME/python/ 下查找 pyspark.问题是要做到这一点，我需要设置 $PYTHONPATH 而 anaconda 不使用该环境变量.
This won't work (of course) because I discovered that I need to tel python to look for pyspark under $SPARK_HOME/python/. The problem is that to do that, I need to set the $PYTHONPATH while anaconda don't use that environment variable.
我试图将 $SPARK_HOME/python/ 的内容复制到 ANACONDA_HOME/lib/python2.7/site-packages/ 但它不起作用.
I tried to copy the content of $SPARK_HOME/python/ to ANACONDA_HOME/lib/python2.7/site-packages/ but it won't work.
有没有在anaconda中使用pyspark的解决方案?
Is there any solution to use pyspark in anaconda?
解决方案
这可能只是最近才成为可能，但我使用了以下方法并且效果很好.在此之后，我可以将 pyspark 作为 ps 导入"并毫无问题地使用它.
This may have only become possible recently, but I used the following and it worked perfectly. After this, I am able to 'import pyspark as ps' and use it with no problems.
conda install -c conda-forge pyspark
相关文章