Ordinal Pattern
Analysis:
A simple method
for assessing simulation-observation fit
CPM Report No.: CPM-12-216
By: Warren Thorngate
and Bruce Edmonds
The thrill of programming a computer simulation of social behaviour can seduce its creators to believe that, if it runs, it must be true. There is something about the intense relationship between simulators and their simulations, especially their agent-based simulations, that blurs the distinction between insight and[1]. Alas, a working simulation is no guarantee of a valid one, so its creators must sooner or later address a sobering question: How well does the simulation mimic the world it was programmed to simulate? A good fit between a simulation and relevant observations may not be sufficient for asserting the simulation’s validity, but it is certainly necessary. In contrast, a bad fit is a simulation’s kiss of death – nature’s way of telling the programmer to give up or to modify the programme and try again. Between good and bad fits is a continuum of so-so, the place where most simulation-observation (S-O) fits in the social sciences are found (see any issue of the Journal of Artificial Societies and Social Simulation).
The purpose of our present effort is to outline a simple method of measuring and evaluating simulation-observation fit, and of using the resulting fitness measures to map domains of validity of competing simulations. We do so in two articles, each emphasizing often-complex simulations and incomplete observations found in the social sciences. In the present article we introduce simple goodness-of-fit indices, based on matches and mismatches of the ordinal properties of predictions and observations, that we believe have many advantages over traditional approaches to evaluating S-O fit. In the sequel we show how to use the resulting indices of S-O fit for detective work needed to explore the situations in which one simulation is more valid than another.
Accessible as:
| WT Home
| BE Home | Other CPM Reports
| CPM home page |