Simulation model analysis by systematic design of simulation experiments

Many still view simulation models as a black box. This perception could change if the systematic design of experiments (DOE) for simulation research was fully realized. DOE can increase (1) the transparency of simulation model behavior and (2) the effectiveness of reporting simulation results. Based on DOE principles, we develop a systematic procedure to guide the analysis of simulation models as well as concrete templates for sharing the results. A major challenge of in particular  agent-based modeling (ABM) is that it addresses complex systems that include a large number of entities, hierarchical levels and processes. As a result, both the validity and credibility of ABMs can be limited. We demonstrate how a cascaded design of simulation experiments (cDOE) may support the validity and efficiency of the analysis of ABMs. Our strategy supports the structural realism of simulation models because both the behavior of their main components and the relationships between these components are explicitly addressed. Overall, the proposed systematic procedure for applying DOE principles complements current initiatives for a more standardized simulation research process.

Publications

Main text

Lorscheid, I., Heine, B. O., & Meyer, M. (2012). Opening the ‘black box’of simulations: increased transparency and effective communication through the systematic design of experiments. Computational & Mathematical Organization Theory, 18(1), 22-62.

Further

Lorscheid, I., & Meyer, M. (2016). Divide and conquer: Configuring submodels for valid and efficient analyses of complex simulation models. Ecological Modelling, 326, 152-161.

Hocke, Sina and Meyer, Matthias and Lorscheid, Iris (2015). Improving simulation model analysis and communication via design of experiment principles: An example from the simulation-based design of cost accounting systems. Journal of Management Control. 26 (2-3), 131-155.

Lee, J. S., Filatova, T., Ligmann-Zielinska, A., Hassani-Mahmooei, B., Stonedahl, F., Lorscheid, I., ... & Parker, D. C. (2015). The complexities of agent-based modeling output analysis. Journal of Artificial Societies and Social Simulation, 18(4), 4.

Lorscheid, I., Meyer, M., Hocke, S.: Simulation model and data analysis: Where are we and where should we go? In: Proceedings of ESSA 2013 Conference, Warsaw, Poland, pp.10–16, September 2013


Simulation model analysis by Structural Equation Modeling (SEM)

Point of Contact: Kai Mertens

Trustable statistical modelling is an emerging challenge in agent-based modeling (ABM). Typically, metamodels support the statistical modeling process. However, typical ABM’s characteristics such as possibly high numbers of entities and parameters, interdependent relations between entities, several layers of effects and emergent social phenomena challenge this process. This might impede the analysis and thus the formulation of trustable conclusions. For this reason, we introduce structural equation modeling (SEM) as a promising statistical modeling method. The suggested method integrates both a priori information from the conceptual model and the simulation data output. Based on this, we estimate and evaluate the core relationships and their predictive capabilities. The resulting structural equation metamodel exposes structures in the behavior of simulation models and allows for their better communication. SEM enables the estimation and evaluation of highly networked systems such as ABMs by explicating interactions between types of agents, measuring emergent phenomena, and identifying robust model behavior. Overall, these contributions foster credibility and trustworthiness of ABMs as well as their communication and understanding.

Publications

Mertens, Kai G. and Lorscheid, Iris and Meyer, Matthias (2016). Structural Equation Modeling for Individual-based Simulation. Proceedings of the 8th International Congress on Environmental Modelling and Software. 4 (1), 492-493.

Mertens, Kai G. (2015). Structural Equation Modeling for Simulation Metamodeling. PhD Colloquium: Winter Simulation Conference 2015.

Mertens, Kai G. and Lorscheid, Iris and Meyer, Matthias (2015). Structural Equation Modeling for Simulation Metamodeling. Proceedings of the 2015 Winter Simulation Conference. 701-712.

Lorscheid, Iris and Meyer, Matthias (2012). How to Effectively Analyze and Communicate Complex Relationships between Variables? A Causal Network Tool based on Structural Equation Modeling for Simulation Model Analysis.