14

I work fairly heavily in mathematical biology/epidemiology, where most of the modeling/computational science work is still dominated by sets of ODEs, admittedly sometimes fairly elaborate sets of them. One of the plusses of these models is that they're rather easy to describe and replicate. A table of parameter values, and the equations themselves and you've given someone everything they need to replicate your research in whatever way they feel like implementing it.

But somewhat more complex models have started to become more popular. Agent-based models, in particular, seem to be both harder to describe in a publication, and harder to replicate, because they aren't necessarily perfectly described by a set of ODEs. Are there any guidelines - or just practical experience - behind describing these models in a way that readers understand what happened, and make them relatively straightforward to replicate?

Robert Cartaino
  • 101
  • 5
Fomite
  • 2,383
  • 1
  • 18
  • 25
  • 1
    My understanding is that a formally described agent-based model is just as deterministic and easy to reproduce as a well-behaved ordinary differential equation. Could you point to some specific examples in the literature? – Aron Ahmadia Dec 06 '11 at 13:31
  • @AronAhmadia Lots of agent-based models rely on non-deterministic components. For instance, the creators of the MASON simulation library thought randomness was important enough as to include their own implementation of a random number generator... – Michael McGowan Dec 06 '11 at 22:33
  • @MichaelMcGowan - I was worried about that. Simulations driven by random number generators should be seedable as part of the reproducibility strategy, but now the scientists must rely on statistics to draw on conclusions. – Aron Ahmadia Dec 07 '11 at 09:47
  • @AronAhmadia Part of the issue is that I've never seen much laid out as to what constitutes a formal description of an ABM. And that leaves aside the question of stochasticity. – Fomite Dec 07 '11 at 13:38
  • a similar issue in economics – Artem Kaznatcheev Jan 10 '12 at 00:20

3 Answers3

4

I don't work in that business but naively I think there are three parts to a complete description

  1. A description of the data landscape they live in. Describe this in terms of the data structure (graph (directed or undirected, weighted or unweighted); tree; array; ...) and the data associated with each node. Make note of special case handling such as periodic boundary conditions or assumed state for neighbors outside the test region. Presumably this has a fairly clear connection with your problem domain.

  2. A description of the internal state of the agent and how it makes decisions. Again, hopefully this has a reasonably clear interpretation.

  3. A description of the relative timing and/or synchronization of action and updates between the agents and the landscape; and between pair or groups of agents.

Pseudo-code (or even real code if it's not too polluted with implementation details) will help.

dmckee --- ex-moderator kitten
  • 1,115
  • 8
  • 17
2

There is something called the ODD (Overview, Design, and Details) protocol, proposed by Volker Grimm and others for describing an agent based model. It consists of a list of elements that are needed for understanding the functioning of an ABM and aims at making descriptions of such models more standardised.

The checklist of what has to be described consists of:

Overview

  1. Purpose
  2. Entities, state variables, and scales
  3. Process overview and scheduling

Design

  1. Basic principles
  2. Emergence
  3. Adaptation
  4. Objectives
  5. Learning
  6. Prediction
  7. Sensing
  8. Interaction
  9. Stochasticity
  10. Collectives
  11. Observation

Details

  1. Initialisation
  2. Input data
  3. Submodels

More details can be found in

Grimm, V., Berger, U., DeAngelis, D. L., Polhill, J. G., Giske, J., & Railsback, S. R. (2010). The ODD Protocol: a Review and First Update. Ecological Modelling, 221, 2760–2768.

Kenji
  • 121
  • 2
1

The best way by far is to include all of your code as supplementary material. If possible, also include files with the relevant random seeds needed to recreate your results. Not only does this allow people to recreate your results (which you might not care about), it also allows them to more easily continue where you left off. This allows for new collaborations and citations to your work. Unfortunately, this comes with the difficulty of forcing you to clean up your code, and make sure its bug free. Hence, it is more an ideal than what is usual in practice. But at the very least, you should archive a version of your code used to produce your results, that way if another researcher asks for code, you can produce it.

In terms of the description in your paper, then I would concentrate on a high-level, implementation independent description of the key novel features of the model (this is the practical part most good paper achieve). Concentrate on the features that will change the result qualitatively if they are tweaked. Most models I work with produce quantitative results, but the specific quantities are usually not of interest, only the qualitative behavior (since the parameters are usually far from ones observable in nature). Thus, I focus on describing the parts of the model, that if changed will change the qualitative behavior of the system. If this mindset forces me to describe every last detail of my model down to the implementation, then I know that my model is not very robust, and thus should be scrapped.

A good way to test if your in-paper description is sufficient, is to ask a friend (or student) who did not work on this project with you to describe how they might implement your model is pseudo-code. If they don't get stuck while trying this (as in they arrive at a sketch of a model which should produce the same qualitative results), then you know you have done a good job of description.

Artem Kaznatcheev
  • 181
  • 6