How to create a package from a complex install folder with a cmake, swig setuptools project

Question

I am programming a quite complex library that use C++ code and a Python binding generated with swig. The code is meant to be use in pure c++ or with a python interface. The current deployment process works well but is quite basic: cmake -> visual studio -> install. Then in the python script that use the libraries I have to "append sys.path" with the correct path to the installed libraries and python modules, for instance:

sys.path.insert(0, os.path.abspath(os.path.join('..', 'install', 'install_msvc17_python38_64bits', 'lib', 'python3.8', 'site-packages', 'fms')))

The cmake process itself is quite complex with numerous sub-folders and a specific one for the python wrapping. I don't want to modify it.

I would like to simplify the installation process for the user. Ideally, he would run "python setup.py install" to avoid opening cmake, visual studio and especially adding things to the path.

So, I recently investigated setuptool but to be honest, I don't understand precisely want it does and how. I've found several good tutorials and examples, I have created a setup.py file (see bellow) but it does not exactily what I want.

First I would like to show you the folder created with the cmake building process:

install_folder

• bin
• doc
• include
• lib
  • python3.8
    • site-packages
      • FMS
        • FMS_core.dll
        • FMS_core.py
        • FMS_core.pyd
        • ....
• share (things useful for cmake projects)
• wrapping (a bunch of swig *.i files)

When I install things with setup.py install I get a correct cmake configuration, generation, building, installation but the package causes issues. A copy of this complete organisation is done in a C:\Anaconda3\Lib\site-packages\FMS-0.6.0-py3.8-win-amd64.egg. And of course when I import my library in python I get (as expected...):

>>> import FMS
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'FMS'

A nice organisation would be:

• FMS-0.6.0-py3.8-win-amd64.egg
  • doc
  • FMS
    • FMS_core.dll
    • FMS_core.py
    • FMS_core.pyd

How can I master the folder organisation in the anaconda site-package folder with build_ext ?

Thanks a lot !

things I have already looked at:

• Place pre-compiled extensions in root folder of non-pure Python Wheel package
• https://martinopilia.com/posts/2018/09/15/building-python-extension.html
• Extending setuptools extension to use CMake in setup.py?
• https://docs.python.org/fr/3/install/index.html

# -*- coding: utf-8 -*-
import os
import sys
import subprocess
from subprocess import Popen, PIPE
from sysconfig import get_paths
from pprint import pprint

from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext

# Convert distutils Windows platform specifiers to CMake -A arguments
PLAT_TO_CMAKE = {
    "win32": "Win32",
    "win-amd64": "x64",
    "win-arm32": "ARM",
    "win-arm64": "ARM64",
}

CMAKE_INSTALL_DIR = "C://Program Files//CMake//bin//cmake.exe"

class CMakeExtension(Extension):
    def __init__(self, name, sourcedir=""):
        Extension.__init__(self, name, sources=[])
        self.sourcedir = os.path.abspath(sourcedir)

class CMakeBuild_FMS(build_ext):
    def build_extension(self, ext):
        extdir = os.path.abspath(os.path.dirname(self.get_ext_fullpath(ext.name)))
        #extdir = os.path.join(extdir, 'lib','python3.8','site-packages','fms')

        #  Python
        python_main_paths = get_paths()  # a dictionary of key-paths
        version_python = str(sys.version_info[0]) + str(sys.version_info[1])
        PYTHON_DIR_EXE =    sys.executable
        PYTHON_DIR_INC =    python_main_paths['include']
        PYTHON_DIR_LIBS =   os.path.join(python_main_paths['data'], 'libs', "python" + version_python + ".lib")

        # Utilities (default repositories)
        SWIG_EXE =          os.path.abspath(os.path.join('..','..',"utilities","swigwin-4.0.2","swig.exe"))
        Eigen3_DIR =        os.path.abspath(os.path.join('..','..',"utilities","eigen-3.3.9","install","share","eigen3","cmake"))
        Boost_INCLUDE_DIR = os.path.abspath(os.path.join('..','..',"utilities","boost_1_74_0"))
        FMT_DIR =           os.path.abspath(os.path.join('..','..',"utilities","FMT_V4","install"))

        # RGINE
        RGINE_DIR =         os.path.abspath(os.path.join('..','..',"rgine","install","share","rgine","cmake"))

        # required for auto-detection of auxiliary "native" libs
        if not extdir.endswith(os.path.sep):
            extdir += os.path.sep

        cfg = "Debug" if self.debug else "Release"

        # CMake lets you override the generator - we need to check this.
        # Can be set with Conda-Build, for example.
        cmake_generator = os.environ.get("CMAKE_GENERATOR", "Visual Studio 15 2017 Win64")

        cmake_args = [
            "-DFMS_VERSION_INFO={}".format(self.distribution.get_version()),
            "-DCMAKE_BUILD_TYPE={}".format(cfg),  # not used on MSVC, but no harm
            "-DSWIG_EXECUTABLE={}".format(SWIG_EXE),
            "-DCMAKE_INSTALL_PREFIX={}".format(extdir),
            "-DRGINE_DIR={}".format(RGINE_DIR),
            "-DEigen3_DIR={}".format(Eigen3_DIR),
            "-DBoost_INCLUDE_DIR={}".format(Boost_INCLUDE_DIR),
            "-DFMT_AUTO_DOCUMENTATION={}".format("OFF"),
            "-DFMT_DIR={}".format(FMT_DIR),
            "-DFMT_V4_Mode={}".format("Binaries"),
            "-DPYTHON_EXECUTABLE={}".format(sys.executable),
            "-DPYTHON_INCLUDE_DIR={}".format(PYTHON_DIR_INC),
            "-DPYTHON_LIBRARY={}".format(PYTHON_DIR_LIBS),
            "-DFMS_INSTALL_EXTERNAL_DEPENDENCIES={}".format("TRUE"),
            "-DFMS_AUTO_DOCUMENTATION={}".format("FALSE"),
            "-DFMS_SPHINX_AUTO_DOCUMENTATION={}".format("FALSE"),
            "-DDOXYGEN_SHOW_WARNINGS={}".format("FALSE"),
        ]
        build_args = []

        # Multi-config generators have a different way to specify configs
        cmake_args += [
            "-DCMAKE_LIBRARY_OUTPUT_DIRECTORY_{}={}".format(cfg.upper(), extdir)
        ]
        build_args += ["--config", cfg]

        if not os.path.exists(self.build_temp):
            os.makedirs(self.build_temp)

        print("CONFIGURE / GENERATE PROJET ------------------------------------")
        subprocess.check_call([CMAKE_INSTALL_DIR, "-G", cmake_generator, "-S", ext.sourcedir] + cmake_args, cwd=self.build_temp)

        print("BUILD PROJET ---------------------------------------------------")
        subprocess.check_call([CMAKE_INSTALL_DIR, "--build", "."] + build_args, cwd=self.build_temp)

        print("INSTALL PROJET -------------------------------------------------")
        subprocess.check_call([CMAKE_INSTALL_DIR, "--install", "."], cwd=self.build_temp)

# The information here can also be placed in setup.cfg - better separation of
# logic and declaration, and simpler if you include description/version in a file.
setup(
    name="FMS",
    version="0.6.0",
    author="me",
    author_email="me@somwhere.mars",
    description="FMS",
    long_description="",
    ext_modules=[CMakeExtension("FMS", "")],
    cmdclass={"build_ext": CMakeBuild_FMS},
    zip_safe=False,
)