How to include sql files from a folder to setuptools while packaging as python egg

Question

I have a directory structure as below.

.
├── package
    ├── app
    │   ├── __init__.py
    │   ├── file1.py
    │   └── file2.py
    ├── master_sql_folder
    │   │
    │   │──sql_folder_1
    │   │      ├── world.sql
    │   │      ├── earth.sql
    │   │
    │   └──sql_folder_2
    │          ├── planet.sql
    │          ├── sun.sql
    │         
    └── wrapper_scripts
    │    ├── wrapper.py
    │    └── __init__.py
    ├── setup.py

I am trying to include all the sql files that exist in sub folders of master_sql_folder ( world.sql, earth.sql, planet.sql, sun.sql ) to setuptools while packaging as egg,I cannot use sql_folder_1 and sql_folder_2 in the path as new folders can be added in future under master_sql_folder and I need the code to read them too.I have tried adding the following lines to my setup.py but its not including the sql files in the build.

package_data={'master_sql_folder':['*']}
packages=['app', 'wrapper_scripts']

I appreciate your help in advance.

Possible duplicate of [this question](https://stackoverflow.com/q/32609248/6340496). — S3DEV, Sep 30 '20 at 08:37
The mentioned link did help me to use this data_files=[('master_sql_folder/sql_folder_1', ['master_sql_folder/sql_folder_1/world.sql'])] and this works But I cannot use sql_folder_1 in the path, Is there a way that can pick the files from all the subfolders like data_files=[('master_sql_folder', ['master_sql_folder/*/*.sql'])]. It results in error :can't copy 'master_sql_folder\*\*.sql': doesn't exist or not a regular file — Somu Sinhhaa, Sep 30 '20 at 09:21
If I remember correctly, each file has to be named individually. Or, write a function that collects the files and builds the list of files for you. — S3DEV, Sep 30 '20 at 09:36

score 0 · Accepted Answer · answered Sep 30 '20 at 10:57

Based on @S3DEV 's suggestion and accepted answer here How to add package data recursively in Python setup.py?. I was able to get a solution.

#sql_file_picker.py -- script to feed files to setup.py

import glob
class FilePicker:
    def __init__(self):
        pass
    def sql_file_picker(self):
        sql_files = []
        directories = glob.glob('master_sql_folder\\**\\')
        for directory in directories:
            files = glob.glob(directory + '*.sql')
            if len(files) != 0:
                sql_files.append((directory, files))
        return sql_files

in setup.py

from wrapper_scripts.sql_file_picker import FilePicker
from setuptools import setup
setup(
    name='XXXXXXXXX',
    version='X.X.X',
    packages=['app', 'wrapper_scripts'],
    url='',
    license='',
    author='',
    author_email='',
    description='XXXXXXXXXXXXX',
    data_files=FilePicker().sql_file_picker()
)

How to include sql files from a folder to setuptools while packaging as python egg

1 Answers1

in setup.py