I need clarification about the strengths and limitations of the conda environment and package manager. I need to build better virtual environments for data science applications, which are frequently built from Python and multiple language toolsets like JupyterLab, SQL, HTML, JavaScript, R, and other language packages.
Data science virtual environment needs are complex and cannot easily be met by a single Python-only environment or dependency manager alone (like venv, pyenv, pipenv, poet). I try to use conda as much as possible, because I find the Python-only environment managers messy and confusing because they are fractured and moving in different directions.
Conda works somewhat like the MS-Windows "Registry" database, because it knows about all packages installed into a virtual environment (at least the ones that conda creates). However, conda also does more than the Windows Registry, because it handles package version conflicts and (usually) doesn't get into "DLL Hell" dependency conflicts or revert to some magic like MS-Windows .NET "Assemblies" (which Microsoft overlords have announced they are phasing out.) Conda does configuration and dependency management system, but far less than artificial intelligence. Conda does environment creation, package version control and conflict resolution, package registration/unregistration, environment management, plus other kinds of package and environment bookkeeping. It does all these things for multiple languages and mixtures of language (Python, JavaScript, R, and many others) in the same environment.
My questions are about the limitations of conda, and why other package managers are needed for specific language parts in data science environments.
Q. Why do I have to use node.js and yarn to install JupyterLab extensions and plugins?
Q. Can conda replace something like Bower or yarn for JavaScript environments?
Q. What do yarn or Bower do for JavaScript that conda cannot?
Q. Why can't conda manage these parts of JavaScript packages too for data science project environments?
Q. Is there a better approach for building high-end virtual environments without reverting to something like DockerStacks, or jumping into the operating system containers?
I've started creating standard template environments for projects that makes like easier (like cookiecutter, but for data science environments instead of projects). This has been a big timesaver to start new projects.
I hope these questions are not too vague or wide. I don't want this post to get flagged and re-written by the SO police. I am not the first data science engineer to ponder these questions. I hope some questions can be answered here without having to research them for days or weeks.
