Calling Python function with parametrs from C++ project (Visual Studio)

Question

I need to call a pipeline realized as a Python (3.6) function from my C++ project under the Windows platform. Function “function_name” from file "experiment_test.py" takes text string as input parameter and return another text string as the result. I try the code below but it doesn’t work properly – python functions from libraries shutil, codecs, makedirs, etc doesn’t work.

C++ code (reduced):

std::string Text,Result;
PyObject *pName, *pModule, *pDict, *pFunc, *pArgs, *pValue;
Py_Initialize();

pName = PyUnicode_FromString("experiment_test");
pModule = PyImport_Import(pName);    
pDict = PyModule_GetDict(pModule);

pFunc = PyDict_GetItemString(pDict, "function_name");

pArgs = PyTuple_New(1);
pValue = PyUnicode_FromString(Text.c_str());
PyTuple_SetItem(pArgs, 0, pValue);

if (PyCallable_Check(pFunc))
{
    pValue = PyObject_CallObject(pFunc, pArgs);
    if (pValue != NULL)
    {
        Result = PyUnicode_AsUTF8(pValue);    
        Py_DECREF(pValue);
    }
    else return false;    
}
// ...

Py_Finalize();

Python code (reduced):

#!/usr/local/bin/python3
import shutil
import codecs
from os import makedirs
from os import path
from os import unlink
from subprocess import call

def function_name():

    name = 'working_files/current_text'

    if not path.exists('working_files'):
        makedirs('working_files')
    if path.exists('result.txt'):
        unlink('result.txt')
    with codecs.open(name + '.txt', 'w', encoding='utf-8') as f:
        f.write(text)
    # ...
    return result

So no new files will be generated by Python. I tried to import Python modules in C++ by calling PyRun_SimpleString("import shutil"); etc after Py_Initialize(); but it doesn’t help.

What do I do wrong?

Answer 1

I tried replicating the problem with the given intel, but it was impossible, so I created a small example (as close as possible to what's described in the question) - Also referred to as [SO]: How to create a Minimal, Reproducible Example (reprex (mcve)) (that should be included in the question BTW)

So, the problem that I'm illustrating here, is:

• C++
  • Load the Python engine
  • Load a Python module
  • From that module, load a function which:
    • Receives a (string) argument representing a file name
    • Reads the file contents (text) and returns it
    • In case of error, simply returns the file name
  • Call that function
  • Get the function call result

I am using (on Win 10 x64 (10.0.16299.125)):

• Python 3.5.4 x64
• VStudio 2015 Community Edition

The structure consists of:

• VStudio project / solution
  • Source file (main00.cpp (renamed it from main.cpp, but didn't feel like doing all the screenshots (containing it) all over again))
• Python module (experiment_test.py)
• Test file (test_file.txt)

main00.cpp:

#include <string>
#include <iostream>

#if defined(_DEBUG)
#  undef _DEBUG
#  define _DEBUG_UNDEFINED
#endif
#include <Python.h>
#if defined(_DEBUG_UNDEFINED)
#  define _DEBUG
#  undef _DEBUG_UNDEFINED
#endif

#define MOD_NAME "experiment_test"
#define FUNC_NAME "function_name"
#define TEST_FILE_NAME "test_dir\\test_file.txt"

using std::cout;
using std::cin;
using std::endl;
using std::string;


int cleanup(const string &text = string(), int exitCode = 1) {
    Py_Finalize();
    if (!text.empty())
        cout << text << endl;
    cout << "Press ENTER to return...\n";
    cin.get();
    return exitCode;
}


int main() {
    char c;
    string fName = TEST_FILE_NAME, result;
    PyObject *pName = NULL, *pModule = NULL, *pDict = NULL, *pFunc = NULL, *pArgs = NULL, *pValue = NULL, *pResult = NULL;
    Py_Initialize();
    pName = PyUnicode_FromString(MOD_NAME);
    if (pName == NULL) {
        return cleanup("PyUnicode_FromString returned NULL");
    }
    pModule = PyImport_Import(pName);
    Py_DECREF(pName);
    if (pModule == NULL) {
        return cleanup(string("NULL module: '") + MOD_NAME + "'");
    }
    pDict = PyModule_GetDict(pModule);
    if (pDict == NULL) {
        return cleanup("NULL module dict");
    }
    pFunc = PyDict_GetItemString(pDict, FUNC_NAME);
    if (pFunc == NULL) {
        return cleanup(string("module '") + MOD_NAME + "' doesn't export func '" + FUNC_NAME + "'");
    }
    pArgs = PyTuple_New(1);
    if (pArgs == NULL) {
        return cleanup("NULL tuple returned");
    }
    pValue = PyUnicode_FromString(fName.c_str());
    if (pValue == NULL) {
        Py_DECREF(pArgs);
        return cleanup("PyUnicode_FromString(2) returned NULL");
    }
    int setItemResult = PyTuple_SetItem(pArgs, 0, pValue);
    if (setItemResult) {
        Py_DECREF(pValue);
        Py_DECREF(pArgs);
        return cleanup("PyTuple_SetItem returned " + setItemResult);
    }
    pResult = PyObject_CallObject(pFunc, pArgs);
    Py_DECREF(pArgs);
    Py_DECREF(pValue);
    if (pResult == NULL) {
        return cleanup("PyObject_CallObject returned NULL");
    } else {
        int len = ((PyASCIIObject *)(pResult))->length;
        char *res = PyUnicode_AsUTF8(pResult);
        Py_DECREF(pResult);
        if (res == NULL) {
            return cleanup("PyUnicode_AsUTF8 returned NULL");
        } else {
            cout << string("C(++) - Python call: ") << MOD_NAME << "." << FUNC_NAME << "('" << fName << "') returned '" << res << "' (len: " << len << ")" << endl;
        }
    }
    return cleanup("OK", 0);
}

Notes:

• The _DEBUG / _DEBUG_UNDEFINED stuff at the beginning - a (lame) workaround (gainarie) to link against Release Python lib (python35.lib) when building in Debug mode (as opposed to python35_d.lib) - read below

• As I said, tried to simplify the code (got rid of the PyCallable_Check test)

• It's easily noticeable that the code is written in C style, although it uses the C++ compiler

• Since Python API ([Python.Docs]: Embedding Python in Another Application) (both extending / embedding) uses pointers, make sure to test for NULLs, otherwise there's a high chance getting segfault (Access Violation)

• Added the [Python.Docs]: Reference Counting - void Py_DECREF(PyObject *o) statements to avoid memory leaks

• Build (compile / link) / Run options (obviously, you got past these, since you were able to run your program, but I'm going to list them anyway - for sure there are some shortcuts here, when dealing with more than one such project):

  • Check [SO]: LNK2005 Error in CLR Windows Form (@CristiFati's answer) for details building a Win PE
  • To speed-up process when building projects that depend on Python, I created a VStudio User Macro (called e.g. Python35Dir - like in the image below, pointing to my Python 3.5 installation dir)

  Notes:

  • The path ("c:\Install\x64\Python\Python\3.5") points to the installation downloaded from the official site

  • Obviously, for 32bit, the path must be set accordingly (to 32bit Python)

  • This path contains (as expected) a Release version, and this is fine as long as I don't need to get into Python code (and as long as I don't mess around with memory - as (when building my app in Debug mode) I have 2 C runtimes in my .exe - check the links below to see what happens when tampering with MSVC runtimes (UCRTs)):

    • [SO]: When using fstream in a library I get linker errors in the executable (@CristiFati's answer)
    • [SO]: Errors when linking to protobuf 3 on MS Visual C (@CristiFati's answer)
    • For exceptional cases, I have built Python in Debug mode and got the binaries, but that's not my 1^st choice, since it requires settings (paths) changes

  • Compile:

    Let VStudio know about the Python include files location:

  • Link:

    Let VStudio know about the Python lib files location (if only pythonxx*.lib (PYTHONCORE) is required, nothing extra needed, since PYTHONCORE is included by default by Python code; otherwise, all the rest should be specified in the [MS.Docs]: .Lib Files as Linker Input:

  • Run / Debug - let:

    • VStudio know where Python runtime python35.dll (PYTHONCORE) is located (%PATH%)
    • Loaded Python runtime know where additional modules are located (%PYTHONPATH%)

experiment_test.py:

import os
import shutil
import codecs


def function_name(file_name):
    print("Py - arg: '{}'".format(file_name))
    if not os.path.isfile(file_name):
        return file_name
    with open(file_name, "rb") as f:
        content = f.read().decode()
        print("Py - Content len: {}, Content (can spread across multiple lines): '{}'".format(len(content), content))
        return content

Notes:

• An almost dummy module, as specified at the beginning
• Works only with text files (decode will fail for binary files)
• Imports modules that aren't used, to see that they are OK (that's obvious, if one such import statement succeeds, all should)
• Prints some data on stdout (to be matched with what's on the C++ side)
• Located in a path known by Python (%PYTHONPATH% from previous step)
• Has 1 argument (file_name) - crucial difference compared to the one in the question which doesn't have any (don't know whether that's a logical mistake or a typo like one)

test_dir\test_file.txt:

line 0 - dummy
line 1 - gainarie

Output (VStudio console):

Py - arg: 'test_dir\test_file.txt'
Py - Content len: 33, Content (can spread across multiple lines): 'line 0 - dummy
line 1 - gainarie'
C(++) - Python call: experiment_test.function_name('test_dir\test_file.txt') returned 'line 0 - dummy
line 1 - gainarie' (len: 33)
OK
Press ENTER to return...

Final note:

• In this case (the simple one), PYTHONCORE from installation path was used. But in other ones (e.g. shipped in another product), Python's standard library path must be set before Py_Initialize, via Py_SetPythonHome.
  Although it's old, [SO]: What files are required for Py_Initialize to run? (@CristiFati's answer) might contain some useful info