是否可以使用 MPI 将数据从 Fortran 程序发送到 Python?

2022-01-14 00:00:00 python fortran mpi openmpi

问题描述

我正在开发一种工具来模拟波浪能转换器，我需要将两个软件包相互耦合.一个程序是用 Fortran 编写的，另一个是用 C++ 编写的.我需要在每个时间步将信息从 Fortran 程序发送到 C++ 程序.但是，数据首先需要在 Python 中处理，然后才能发送到 C++ 程序.我收到了使用 MPI 在程序之间传输数据的提示.

I am working on a tool to model wave energy converters, where I need to couple two software packages to each other. One program is written in Fortran, the other one in C++. I need to send information from the Fortran program to the C++ program at each time step. However, the data first needs to be processed in Python before it is sent to the C++ program. I have received a tip to use MPI to transfer the data between the programs.

我现在正尝试从 Fortran 代码向 Python 发送一个简单的字符串，但 Python 代码在接收命令处卡住了.

I am now trying to send a simple string from the Fortran code to Python, but the Python code gets stuck at the receive command.

我的 Fortran 代码如下所示:

My Fortran code looks like this:

      USE GlobalVariables
      USE MPI
      IMPLICIT NONE

      CHARACTER(LEN=10):: astring
      INTEGER :: comm, rank, size, mpierr

      ! Initialize MPI on first timestep
      IF(tstep .LT. 2) THEN
        call MPI_INIT(mpierr)
      ENDIF

      ! make string to send to python
      astring = "TEST"

      ! MPI Test
      call MPI_Comm_size(MPI_COMM_WORLD, size, mpierr)
      call MPI_Comm_rank(MPI_COMM_WORLD, rank, mpierr)

      ! Send message to python
      CALL MPI_SEND(astring, len(astring), MPI_CHARACTER, 0, 22, MPI_COMM_WORLD, mpierr)
      print *, 'MPI MESSAGE SENT  ', mpierr

      ! Initialize MPI on first timestep
      IF(tstep .EQ. Nsteps-1) THEN
        call MPI_FINALIZE(mpierr)
        print *, 'MPI FINALIZED!'
      ENDIF

我的 Python 代码如下:

My Python code is the following:

    from mpi4py import MPI
    import numpy as np
    import subprocess as sp
    import os

    # Start OW3D_SPH in the background and send MPI message
    os.chdir('OW3D_run')
    args = ['OceanWave3D_SPH','OW3D.inp']
    pid = sp.Popen(args,shell=False)
    os.chdir('..')

    # Check if MPI is initialized
    comm = MPI.COMM_WORLD
    rank = comm.Get_rank()

    # Receive message from fortran
    test = comm.recv(source=0, tag=22)

    # Let the program end
    output = pid.communicate()

    with open('test.txt','w') as f:
        f.write(test)

Python 代码永远不会通过 MPI 接收命令并且不会完成.Fortran 代码完成并正确打印MPI FINALIZED"消息.

The Python code never gets past the MPI receive command and does not finish. The Fortran code does finish and properly prints the "MPI FINALIZED" message.

我看不出我在哪里做错了，消息从进程 0 发送到进程 0，标签为 22，并在两个代码中使用 MPI_COMM_WORLD.

I don't see where I am doing something wrong, the message gets sent from process 0 to process 0 with a tag 22 and uses MPI_COMM_WORLD in both codes.

解决方案

如果你要在同一个 MPI 作业中同时启动 Fortran 程序和 Python 程序，你必须使用类似的东西:

If you are to start both the Fortran program and the Python one in the same MPI job, you have to use something like:

mpiexec -n 1 fortran_program : -n 1 python main.py

Fortran 程序将成为 MPI 等级 0，Python 程序将成为 MPI 等级 1.您还可以启动多个可执行文件，例如:

The Fortran program will become MPI rank 0 and the Python program will be MPI rank 1. You can also start more than one of each executables, for example:

mpiexec -n 2 fortran_program : -n 4 python main.py

排名 0 和 1 将来自 Fortran 程序，排名 2 到 5 - 来自 Python 程序.

Ranks 0 and 1 will be from the Fortran program, ranks 2 to 5 - from the Python one.

还要注意comm.recv()和mpi4py中其他以小写字母开头的通信方式(comm.send()、comm.irecv() 等)在底层使用 Pickle 并实际使用序列化的 Python 对象进行操作.这与 Fortran 代码发送的字符数组不兼容.您必须使用以大写字母开头的通信方法(comm.Send()、comm.Recv() 等)对 NumPy 数组进行操作并接收显式类型信息.不幸的是，我的 Python fu 很弱，我现在无法提供完整的工作示例，但 MPI 部分应该是这样的(未经验证的代码):

Also note that comm.recv() and the other communication methods in mpi4py that start with small letters (comm.send(), comm.irecv(), etc.) use Pickle under the hood and actually operate with serialised Python objects. This is not compatible with the character array sent by the Fortran code. You have to use the communication methods that start with capital letter (comm.Send(), comm.Recv(), etc.) that operate on NumPy arrays and receive explicit type information. Unfortunately, my Python fu is weak and I cannot provide a complete working example right now, but the MPI part should be something like this (unverified code):

# Create an MPI status object
status = MPI.Status()
# Wait for a message without receiving it
comm.Probe(source=0, tag=22, status=status)
# Check the length of the message
nchars = status.Get_count(MPI.CHARACTER)
# Allocate a big enough data array of characters
data = np.empty(nchars, dtype='S')
# Receive the message
comm.Recv([data, MPI.CHARACTER], source=0, tag=22)
# Construct somehow the string out of the individual chars in "data"

在 Fortran 代码中，您必须将目标等级指定为 1(如果您正在运行一个 Fortran 可执行文件和一个 Python 可执行文件).

In the Fortran code you have to specify a destination rank of 1 (in the case you are running one Fortran executable and one Python one).

相关文章