Passing around/Serving Large Model Object in Poltly Dash and using it in multiple callbacks

Question

I have what I believe is a relatively technical question about dash callbacks and enabling user interaction with a large simulation model.

Currently I have in my repository an app directory which contains all things plotly dash related and an src/engine directory which contains all things simluation related.

Within the app I would like now to code some callbacks to allow the user to interact with the simulation models. They can select things like different scenarios, years, quantiles etc. There is a simulation object class that I have coded that creates and object with an API that allows for these types of queries. The object itself it's about 3 GB because it relies on some baseline data (naturally).

So currently in my src/engine there is a file called simulation.py.

# Contains simulation object ...

class Simulator(object):

    def __init__(data1, data2, data3):
        ## Stuff

    ... more stuff helper functions etc ...

    def query(self, q):
        ## Run a simulation for the query q
        return simluation_based_on_q

In my app directory I first need to get a simulator object so I have a models.py file where I can load my multiple models that I need to be interactable with the app.

So in app/models.py I have this:

import sys

sys.path.append('../src/engine/')

from simulation import Simulator
import module_io import DataLoader1, DataLoader2, DataLoader3

data1 = DataLoader1.get()
data2 = DataLoader2.get()
data3 = DataLoader3.get()

sim_object = Simulator(data1, data2, data3)

Now in my app directory I have multiple callbacks that can interact with the model.

So in app/controller/callbacks.py I have something like this.

import dash_bootstrap_components as dbc
from models import sim_object
from server import app

@app.callback(
    Output('years-dropdown', 'children')
    Input('years-dropdown', 'id'), 
)
def years_dropdown_children(id):
    possible_years = simul_object.years
    return [dbc.DropdownMenuItem(year, id=f"{id}-item-{year}") for years in possible_years]

@app.callback(
    Output('quantiles-dropdown', 'children')
    Input('quantiles-dropdown', 'id'), 
)
def quantiles_dropdown_children(id):
    possible_quantiles = simul_object.quantiles
    return [dbc.DropdownMenuItem(quantile, id=f"{id}-item-{year}") for quantile in possible_quantiles]

@app.callback(
    Output('heatmap-graph', 'figure')
    [Input('dropdown-years', 'value'), 
     Input('dropdown-quantiles', 'value'),
)
def heatmap_graph_update(years, quantiles):
    q = [years, quantiles]
    simulation_output = sim_object.query(q)
    ## plot the simulation

... So on and so forth ...

Please note that the above is pseudo-code (that is quite close to the actual code) to demonstrate the logic, the app itself is a bit more complex and not possible to share all code in a Stackoverflow question.

The app was working before creating these callbacks and it even worked when I coded the first callback but as I added more callbacks it crashes when I try to run it. This feels a bit sketchy, but I can't quite put my finger on it, I think its close to "good design" but using the object in multiple callbacks doesn't seem right. I'm thinking that I might need "share" the model object between the callbacks somehow using a redis-cache to cache the model object so each callback is access the same exact version and not somehow creating duplicate objects.

OVERALL QUESTION

What is the best methodology/practices for sharing large objects (2-3 GB) between files in dash especially when using the same object in callbacks to generate graphs.

Passing around/Serving Large Model Object in Poltly Dash and using it in multiple callbacks

0 Answers0