List all known NumPy arrays currently in Python interpreter

Question

I want a collection of all numpy arrays currently managed by my Python interpreter. Previous answers suggest that gc.get_objects() should do this for me. However I don't get expected results:

In [1]: import numpy as np

In [2]: import gc

In [3]: x = np.ones(5)

In [4]: any(item is x for item in gc.get_objects())
Out[4]: False

Is there a way to get all of the known NumPy arrays (or objects of any type really) currently instantiated?

For reference, from Jim Crist internal communication at Continuum Analytics: This explains why gc.get_objects doesn't work: https://mail.scipy.org/pipermail/numpy-discussion/2014-September/071186.html Numpy contains a tool for tracking allocations as a context manager. This plus a `WeakSet` could possibly help solve your problem. https://github.com/numpy/numpy/tree/master/tools/allocation_tracking — kalefranz, May 05 '16 at 05:33

score 2 · Answer 1 · edited May 23 '17 at 11:52

2

This isn't a perfect answer, but according to this stackoverflow post you can use locals() to get a dictionary of all of the locally declared variables.

You can then get all of the known NumPy arrays using simple dict comprehension:

import numpy as np
np_arrays = {k:v for k,v in locals().items() if isinstance(v, np.ndarray)}

You can then iterate over the dictionary however you like after that.

edited May 23 '17 at 11:52

Community

1
1

answered May 05 '16 at 04:06

TheF1rstPancake

2,150
17
17

1

I believe that locals only provides local scope, not variables allocated elsewhere. – MRocklin May 05 '16 at 14:54
Indeed, this would not reveal items allocated to class members of objects. – skjerns Mar 06 '19 at 14:57

score 1 · Answer 2 · answered May 05 '16 at 04:01

1

I'm not sure if this addresses the exact problem you're trying to solve, but what about using locals()? That gets you started on at least getting what vars are being kept track of by the interpreter of your current interactive session?

import numpy as np
a = np.array([1, 2, 3])
b = np.array([2, 3, 4])
filter(lambda x : isinstance(x, np.ndarray), locals().values())

answered May 05 '16 at 04:01

Thtu

1,832
14
21

I believe that locals only provides local scope, not variables allocated elsewhere. – MRocklin May 05 '16 at 14:54

score 0 · Answer 3 · answered Mar 06 '19 at 15:12

You can find numpy arrays with gc.get_objects() if you expand the objects recursively, as explained here

# code from https://utcc.utoronto.ca/~cks/space/blog/python/GetAllObjects
import gc
# Recursively expand slist's objects
# into olist, using seen to track
# already processed objects.
def _getr(slist, olist, seen):
  for e in slist:
    if id(e) in seen:
      continue
    seen[id(e)] = None
    olist.append(e)
    tl = gc.get_referents(e)
    if tl:
      _getr(tl, olist, seen)

# The public function.
def get_all_objects():
  """Return a list of all live Python
  objects, not including the list itself."""
  gcl = gc.get_objects()
  olist = []
  seen = {}
  # Just in case:
  seen[id(gcl)] = None
  seen[id(olist)] = None
  seen[id(seen)] = None
  # _getr does the real work.
  _getr(gcl, olist, seen)
  return olist

Now we should be able to find most objects

import numpy as np
import gc

x = np.ones(5)
objects = get_all_objects()
print(any([obj is x for obj in objects]))
# will return True, the np.ndarray is found!

List all known NumPy arrays currently in Python interpreter

3 Answers3