1

I (very) often write python, and I use Vim's python3 backend to test my code. To perform my tests I use:

vnoremap <localleader>p y:<c-r>"<c-b>python3 <cr>

It simply takes my visual selection and runs within Vim's python backend. This is very practical because I can re-run only pieces of code (and I'm careful to remember the state the repl is in).

This works fine even with installed libraries, for example beautifulsoup4 works:

:python3 import bs4

But my issue starts when I try loading libraries that are not fully written in python. For example trying numpy:

:python3 import numpy

I get an error from the backend:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/grochmal/.local/lib/python3.5/site-packages/numpy/__init__.py", line 180, in <module>
    from . import add_newdocs
  File "/home/grochmal/.local/lib/python3.5/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/home/grochmal/.local/lib/python3.5/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/home/grochmal/.local/lib/python3.5/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/home/grochmal/.local/lib/python3.5/site-packages/numpy/core/__init__.py", line 14, in <module>
    from . import multiarray
ImportError: /home/grochmal/.local/lib/python3.5/site-packages/numpy/core/multiarray.cpython-35m-x86_64-linux-gnu.so: undefine
d symbol: PyType_GenericNew

After debugging it a good deal I got to the conclusion that the issue only happens in /home/grochmal/.local and only to libraries that have components compiled from C. In other words, if I install numpy into /usr/lib it works.

The question

The python REPL has no issue with libraries at ~/.local but Vim's python backend does. I tried with and without:

export PYTHONPATH=/home/grochmal/.local/lib/python3.5/site-packages

And the issue persists. Is there a way to use locally installed shared libraries together with Vim's python backend? i.e. Can I tell Vim's python backend to search both places (/usr/lib and ~/.local) for symbols to load?

Extra note: both bs4 and numpy are in ~/.local in the tests above. In other words libraries that are fully written in python do work in ~/.local

grochmal
  • 1,656
  • 14
  • 35
  • PS: I do use makeprg=python3\ %, but that is just too slow when huge libraries are being loaded or a lot of data is read from files. Yes, i'm using Vim like an ide, shame on me :) – grochmal Oct 13 '16 at 19:15

2 Answers2

2

This is because your numpy isn't linked against the Python 3 library (no -lpython3 used). This is fine for most applications that has the library loaded into global space, but Vim uses RTLD_LOCAL so numpy's libraries don't see Python 3's symbols unless it's linked against it.

LD_PRELOAD is fine as long as you don't load Python 2 in the same Vim (or symbols may clash; that's why Vim doesn't use RTLD_GLOBAL). You can also recompile your numpy with -lpython3 (from Arch's PKGBUILD it seems that export LDFLAGS=-shared is enough to do that).

lilydjwg
  • 136
  • 2
  • 1
    Whelp, yeah. I was almost there, but not quite. Tested it (recompiled my numpy with -lpython3) and it works. p2k cannot go away soon enough (I know that people are saying that for 5 years already and will probably not happen in the next 5 either). Many thanks. – grochmal Oct 16 '16 at 18:51
0

Workaround

After a good deal of debugging I found that forcing Vim to have the shared library in memory all the time sorts the problem. i.e.

alias vim='LD_PRELOAD=/usr/lib/libpython3.so vim'

Yes, it is horrible, but works on most systems (tested on CentOS, Debian and Arch). Another problem is that LD_PRELOAD is Linux specific.

After this I can load a numpy library from ~/.local/lib/python3.5/site-packages/ with :python3 import numpy.

Extra info

I'm still baffled at the issue since in Vim code the library should come from:

src/option.h: EXTERN char_u *p_py3dll;  /* 'pythonthreedll' */

Which in turn should come from -DDYNAMIC_PYTHON3_DLL, which should be a static makefile argument. Therefore, I believe that the issue lies somewhere in (src/if_python3.c):

# ifndef WIN3264
#  include <dlfcn.h>
#  define FARPROC void*
#  define HINSTANCE void*
#  if defined(PY_NO_RTLD_GLOBAL) && defined(PY3_NO_RTLD_GLOBAL)
#   define load_dll(n) dlopen((n), RTLD_LAZY)
#  else
#   define load_dll(n) dlopen((n), RTLD_LAZY|RTLD_GLOBAL)
#  endif
...

And that RTLD_LOCAL (i.e. no RTLD_GLOBAL) is possibly used. But I never used dlopen() and may be completely wrong about this.

grochmal
  • 1,656
  • 14
  • 35