Cython 代码可以编译成 dll 以便 C++ 应用程序可以调用它吗?

2021-12-25 00:00:00 python dll cython plugins c++

我有一个 C++ 程序，它具有某种插件结构:当程序启动时，它会在插件文件夹中寻找具有某些导出函数签名的 dll，例如:

I have a C++ program and it has sort of plugin structure: when program starts up, it's looking for dll in the plugin folder with certain exported function signatures, such as:

void InitPlugin(FuncTable* funcTable);

然后程序会调用dll中的函数进行初始化并将函数指针传递给dll.从那时起，dll 就可以与程序对话了.

Then the program will call the function in the dll to initialize and pass function pointers to the dll. From that time on, the dll can talk to the program.

我知道 Cython 允许您在 Python 中调用 C 函数，但我不确定我是否可以编写 Cython 代码并将其编译为 dll，以便我的 C++ 程序可以使用它进行初始化.示例代码会很棒.

I know Cython let you call C function in Python, but I'm not sure can I write a Cython code and compile it to a dll so my C++ program can initialize with it. An example code would be great.

推荐答案

在 dll 中使用 cython-module 与使用 cython 没什么不同- 嵌入式 python 解释器中的模块.

Using cython-module in a dll is not unlike using a cython-module in an embeded python interpreter.

第一步是用public标记应该从外部C代码使用的cdef-function，例如:

The first step would be to mark cdef-function which should be used from external C-code with public, for example:

#cyfun.pyx:

#doesn't need python interpreter
cdef public int double_me(int me):
    return 2*me;
        
#needs initialized python interpreter
cdef public void print_me(int me):
    print("I'm", me);

cyfun.c 和 cyfun.h 可以用

cython -3 cyfun.pyx

这些文件将用于构建 dll.

These files will be used for building of the dll.

dll 需要一个函数来初始化 python 解释器，另一个函数来完成它，在使用 double_me 和 print_me 之前应该只调用一次(好吧，double_me 也可以在没有解释器的情况下工作，但这是一个实现细节).注意:初始化/清理也可以放在 DllMain 中 - 请参阅下面的此类版本.

The dll will need one function to initialize the python interpreter and another to finalize it, which should be called only once before double_me and print_me can be used (Ok, double_me would work also without interpreter, but this is an implementation detail). Note: the initialization/clean-up could be put also in DllMain - see such a version further bellow.

dll 的头文件如下所示:

The header-file for the dll would look like following:

//cyfun_dll.h
#ifdef BUILDING_DLL
    #define DLL_PUBLIC __declspec(dllexport) 
#else
    #define DLL_PUBLIC __declspec(dllimport) 
#endif

//return 0 if everything ok
DLL_PUBLIC int cyfun_init();
DLL_PUBLIC void cyfun_finalize();

DLL_PUBLIC int cyfun_double_me(int me);
DLL_PUBLIC void cyfun_print_me(int me);

所以有必要的 init/finalize-functions 和符号是通过 DLL_PUBLIC 导出的(这需要完成见这个 SO-post) 所以它可以在 dll 之外使用.

So there are the necessary init/finalize-functions and the symbols are exported via DLL_PUBLIC (which needs to be done see this SO-post) so it can be used outside of the dll.

cyfun_dll.c-file 中的实现如下:

The implementation follows in cyfun_dll.c-file:

//cyfun_dll.c
#define BUILDING_DLL
#include "cyfun_dll.h"

#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include "cyfun.h"

DLL_PUBLIC int cyfun_init(){
  int status=PyImport_AppendInittab("cyfun", PyInit_cyfun);
  if(status==-1){
    return -1;//error
  } 
  Py_Initialize();
  PyObject *module = PyImport_ImportModule("cyfun");

  if(module==NULL){
     Py_Finalize();
     return -1;//error
  }
  return 0;   
}


DLL_PUBLIC void cyfun_finalize(){
   Py_Finalize();
}

DLL_PUBLIC int cyfun_double_me(int me){
    return double_me(me);
}

DLL_PUBLIC void cyfun_print_me(int me){
    print_me(me);
}

值得注意的细节:

  1. 我们定义了BUILDING_DLL，所以DLL_PUBLIC 变成了__declspec(dllexport).
  2. 我们使用由 cyfun.pyx 中的 cython 生成的 cyfun.h.
  3. cyfun_init 初始化python解释器并导入内置模块cyfun.有点复杂的代码是因为从 Cython-0.29 开始，PEP-489 是默认.更多信息可以在这篇 SO-post 中找到.如果 Python 解释器未初始化或模块 cyfun 未导入，则很有可能从 cyfun.h 调用功能将以分段结束错误.
  4. cyfun_double_me 只是包装了 double_me，所以它在 dll 之外变得可见.
  1. we define BUILDING_DLL so DLL_PUBLIC becomes __declspec(dllexport).
  2. we use cyfun.h generated by cython from cyfun.pyx.
  3. cyfun_init inizializes python interpreter and imports the built-in module cyfun. The somewhat complicated code is because since Cython-0.29, PEP-489 is default. More information can be found in this SO-post. If the Python-interpreter isn't initialized or if the module cyfun is not imported, the chances are high, that calling the functionality from cyfun.h will end in a segmentation fault.
  4. cyfun_double_me just wraps double_me so it becomes visible outside of the dll.

现在我们可以构建 dll！

Now we can build the dll!

:: set up tool chain
call "<path_to_vcvarsall>vcvarsall.bat" x64

:: build cyfun.c generated by cython
cl  /Tccyfun.c /Focyfun.obj /c <other_coptions> -I<path_to_python_include> 

:: build dll-wrapper
cl  /Tccyfun_dll.c /Focyfun_dll.obj /c <other_coptions> -I<path_to_python_include>

:: link both obj-files into a dll
link  cyfun.obj cyfun_dll.obj /OUT:cyfun.dll /IMPLIB:cyfun.lib /DLL <other_loptions> -L<path_to_python_dll>

现在已经构建了 dll，但以下细节值得注意:

The dll is now built, but the following details are noteworthy:

  1. 和 可能因安装而异.查看它们的一种简单方法是运行 cythonize some_file.pyx` 并检查日志.
  2. 我们不需要传递python-dll，因为它会自动链接，但是我们需要设置正确的库路径.
  3. 我们依赖于 python-dll，所以稍后它必须在某个可以找到的地方.
  1. <other_coptions> and <other_loptions> can vary from installation to installation. An easy way is to see them is to run cythonize some_file.pyx` and to inspect the log.
  2. we don't need to pass python-dll, because it will be linked automatically, but we need to set the right library-path.
  3. we have the dependency on the python-dll, so later on it must be somewhere where it can be found.

你从这里开始取决于你的任务，我们用一个简单的 main 测试我们的 dll:

Were you go from here depends on your task, we test our dll with a simple main:

//test.c
#include "cyfun_dll.h"

int main(){
   if(0!=cyfun_init()){
      return -1;
   }
   cyfun_print_me(cyfun_double_me(2));
   cyfun_finalize();
   return 0;
}

可以通过构建

...
:: build main-program
cl  /Tctest.c /Focytest.obj /c <other_coptions> -I<path_to_python_include>

:: link the exe
link test.obj cyfun.lib /OUT:test_prog.exe <other_loptions> -L<path_to_python_dll>

现在调用 test_prog.exe 会导致预期的输出我 4 岁".

And now calling test_prog.exe leads to the expected output "I'm 4".

根据您的安装，必须考虑以下事项:

Depending on your installation, following things must be considered:

  • test_prog.exe 依赖于 pythonX.Y.dll，它应该在路径中的某个地方，以便可以找到它(最简单的方法是将它复制到exe)
  • 嵌入式 python 解释器需要安装，请参阅this 和/或 this SO-posts.
  • test_prog.exe depends on pythonX.Y.dll which should be somewhere in the path so it can be found (the easiest way is to copy it next to the exe)
  • The embeded python interpreter needs an installation, see this and/or this SO-posts.

IIRC，初始化，然后完成，然后再次初始化 Python 解释器不是一个好主意(这可能适用于某些情况，但不是所有情况，请参见例如 this) - 解释器应该只初始化一次并保持活动状态直到程序结束.

IIRC, it is not a great idea to initialize, then to finalize and then to initialize the Python-interpreter again (that might work for some scenarios, but not all , see for example this) - the interpreter should be initialized only once and stay alive until the programs ends.

因此，将初始化/清理代码放入DllMain(并使 cyfun_init() 和 cyfun_finalize() 私有)，例如

Thus, it may make sense to put initialization/clean-up code into DllMain (and make cyfun_init() and cyfun_finalize() private), e.g.

BOOL WINAPI DllMain(
    HINSTANCE hinstDLL,  // handle to DLL module
    DWORD fdwReason,     // reason for calling function
    LPVOID lpReserved )  // reserved
{
    // Perform actions based on the reason for calling.
    switch( fdwReason ) 
    { 
        case DLL_PROCESS_ATTACH:
            return cyfun_init()==0;

        case DLL_PROCESS_DETACH:
            cyfun_finalize();
            break;
        case DLL_THREAD_ATTACH:
         // Do thread-specific initialization.
            break;

        case DLL_THREAD_DETACH:
         // Do thread-specific cleanup.
            break;   
    }
    return TRUE;
}

如果你的 C/C++ 程序已经有一个初始化的 Python 解释器，那么提供一个只导入模块 cyfun 而不初始化 python 解释器的函数是有意义的.在这种情况下，我将定义 CYTHON_PEP489_MULTI_PHASE_INIT=0，因为 PyImport_AppendInittab 必须在 Py_Initialize 之前调用，这在加载 dll 时可能已经太晚了.

If your C/C++-program already has an initialized Python-interpreter it would make sense to offer a function which only imports the module cyfun and doesn't initialize the python-interpreter. In this case I would define CYTHON_PEP489_MULTI_PHASE_INIT=0, because PyImport_AppendInittab must be called before Py_Initialize, which might be already too late when the dll is loaded.

相关文章