如何使用 call/Popen 调用子进程继承环境变量

2022-01-16 00:00:00 python shell subprocess environment-variables bash

问题描述

首先，对于我对 bash、shell 和子进程的初步了解，我确信很明显很抱歉.

First off, apologies for what I'm sure will be obvious is my rudimentary understanding of bash and shells and subprocesses.

我正在尝试使用 Python 自动调用名为 Freesurfer 的程序(实际上，我正在调用的子程序称为 recon-all.)

I am trying to use Python to automate calls to a program called Freesurfer (actually, the subprogram I'm calling is called recon-all.)

如果我直接在命令行中执行此操作，我将获取"一个名为 mySetUpFreeSurfer.sh 的脚本，该脚本只设置三个环境变量，然后获取"另一个脚本 FreeSurferEnv.sh.FreesurferEnv.sh 在我看来除了设置很多环境变量并将一些内容回显到终端之外什么也没做，但它比其他 bash 脚本更复杂，所以我不确定.

If I were doing this directly at the command line, I'd "source" a script called mySetUpFreeSurfer.sh that does nothing but set three environment variables, and then "source" another script, FreeSurferEnv.sh. FreesurferEnv.sh doesn't seem to me to do anything but set a lot of environment variables and echo some stuff to the terminal, but it's more complicated than the other bash script, so I'm not sure of that.

这是我现在拥有的:

from subprocess import Popen, PIPE, call, check_output
import os

root = "/media/foo/"

#I got this function from another Stack Overflow question.

def source(script, update=1):
    pipe = Popen(". %s; env" % script, stdout=PIPE, shell=True)
    data = pipe.communicate()[0]
    env = dict((line.split("=", 1) for line in data.splitlines()))
    if update:
        os.environ.update(env)
    return env

source('~/scripts/mySetUpFreeSurfer.sh')
source('/usr/local/freesurfer/FreeSurferEnv.sh')

for sub_dir in os.listdir(root):
    sub = "s" + sub_dir[0:4]
    anat_dir = os.path.join(root, sub_dir, "anatomical")
    for directory in os.listdir(anat_dir):
        time_dir = os.path.join(anat_dir, directory)
        for d in os.listdir(time_dir):
            dicoms_dir = os.path.join(time_dir, d, 'dicoms')
            dicom_list = os.listdir(dicoms_dir)
            dicom = dicom_list[0]
            path = os.path.join(dicoms_dir, dicom)
            cmd1 = "recon-all -i " + path + " -subjid " + sub
            check_output(cmd1, shell=True)
            call(cmd1, shell=True)
            cmd2 = "recon-all -all -subjid " + sub,
            call(cmd2, shell=True)

这失败了:

Traceback (most recent call last):
     File "/home/katie/scripts/autoReconSO.py", line 28, in <module>
        check_output(cmd1, shell=True)
      File "/usr/lib/python2.7/subprocess.py", line 544, in check_output
        raise CalledProcessError(retcode, cmd, output=output)
    CalledProcessError: Command 'recon-all -i /media/foo/bar -subjid s1001' returned non-zero exit status 127

我也许明白为什么会这样.我稍后在脚本中的调用"正在引发新的子进程，这些子进程不会从调用 source() 函数引发的进程继承环境变量.我做了很多事情来确认我的理解.一个例子——我写了这些行:

I maybe understand why this is. My "calls" later in the script are raising new subprocesses that do not inherit environment variables from the processes that are raised by invocation of the source() function. I have done a number of things to try to confirm my understanding. One example -- I put these lines:

mkdir ~/testFreeSurferEnv
export TEST_ENV_VAR=~/testFreeSurferEnv

在 FreeSurferEnv.sh 脚本中.该目录制作得很好，但在 Python 脚本中:

in the FreeSurferEnv.sh script. The directory gets made just fine, but in the Python script this:

cmd = 'mkdir $TEST_ENV_VAR/test'
check_output(cmd, shell=True)

这样失败:

File "/usr/lib/python2.7/subprocess.py", line 544, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command 'mkdir $TEST_ENV_VAR/test' returned non-zero exit status 1

问题:

如何让运行recon-all"的子进程继承它需要的环境变量?或者我怎样才能做我需要做的所有事情——运行脚本来设置环境变量，并在同一个过程中调用 recon-all?还是我应该以另一种方式解决问题?还是我可能误解了这个问题?

How can I make the subprocess that runs "recon-all" inherit the environment variables it needs? Or how can I do everything I need to do -- run the scripts to set the environment variables, and call recon-all, in the same process? Or should I approach the problem another way? Or do I likely misunderstand the problem?

解决方案

关于

如果我直接在命令行中执行此操作，我将获取"一个名为 mySetUpFreeSurfer.sh 的脚本，该脚本只设置三个环境变量，然后获取"另一个脚本 FreeSurferEnv.sh.

我认为你最好使用 Python 来自动化编写过程一个shell脚本newscript.sh，然后用one调用这个脚本subprocess.check_output(而不是多次调用 Popen、check_output、调用等):

I think you would be better off using Python to automate the process of writing a shell script newscript.sh, and then calling this script with one call subprocess.check_output (instead of many calls to Popen, check_output, call, etc.):

newscript.sh:

#!/bin/bash
source ~/scripts/mySetUpFreeSurfer.sh
source /usr/local/freesurfer/FreeSurferEnv.sh
recon-all -i /media/foo/bar -subjid s1001
...

然后调用

subprocess.check_output(['newscript.sh'])

<小时>

import subprocess
import tempfile
import os
import stat


with tempfile.NamedTemporaryFile(mode='w', delete=False) as f:
    f.write('''
#!/bin/bash
source ~/scripts/mySetUpFreeSurfer.sh
source /usr/local/freesurfer/FreeSurferEnv.sh
''')
    root = "/media/foo/"
    for sub_dir in os.listdir(root):
        sub = "s" + sub_dir[0:4]
        anat_dir = os.path.join(root, sub_dir, "anatomical")
        for directory in os.listdir(anat_dir):
            time_dir = os.path.join(anat_dir, directory)
            for d in os.listdir(time_dir):
                dicoms_dir = os.path.join(time_dir, d, 'dicoms')
                dicom_list = os.listdir(dicoms_dir)
                dicom = dicom_list[0]
                path = os.path.join(dicoms_dir, dicom)
                cmd1 = "recon-all -i {}  -subjid {}
".format(path, sub)
                f.write(cmd1)
                cmd2 = "recon-all -all -subjid {}
".format(sub)
                f.write(cmd2)

filename = f.name
os.chmod(filename, stat.S_IRUSR | stat.S_IXUSR)
subprocess.call([filename])
os.unlink(filename)

<小时>

顺便说一句，

By the way, 

def source(script, update=1):
    pipe = Popen(". %s; env" % script, stdout=PIPE, shell=True)
    data = pipe.communicate()[0]
    env = dict((line.split("=", 1) for line in data.splitlines()))
    if update:
        os.environ.update(env)
    return env

坏了.例如，如果 script 包含类似

is broken. For example, if script contains something like

VAR=`ls -1`
export VAR

然后

. script; env

可能会返回类似的输出

VAR=file1
file2
file3

这将导致 source(script) 引发 ValueError:

env = dict((line.split("=", 1) for line in data.splitlines()))
ValueError: dictionary update sequence element #21 has length 1; 2 is required

<小时>

有一种方法可以修复 source:让 env 使用零字节而不是模棱两可的换行符分隔环境变量:

There is a way to fix source: have env separate environment variables with a zero byte instead of the ambiguous newline:

def source(script, update=True):
    """
    http://pythonwise.blogspot.fr/2010/04/sourcing-shell-script.html (Miki Tebeka)
    http://stackoverflow.com/questions/3503719/#comment28061110_3505826 (ahal)
    """
    import subprocess
    import os
    proc = subprocess.Popen(
        ['bash', '-c', 'set -a && source {} && env -0'.format(script)], 
        stdout=subprocess.PIPE, shell=False)
    output, err = proc.communicate()
    output = output.decode('utf8')
    env = dict((line.split("=", 1) for line in output.split('x00') if line))
    if update:
        os.environ.update(env)
    return env

不管是否可修复，您最好还是构建一个conglomerate shell 脚本(如上所示)，而不是解析 env 和将 env dicts 传递给 subprocess 调用.

Fixable or not, however, you are still probably better off constructing a conglomerate shell script (as shown above) than you would be parsing env and passing env dicts to subprocess calls.

相关文章