Models

Models, Simulations and Scientific Method: 

Models, Simulations and Scientific Method Caroline Lyon School of Computer Science University of Hertfordshire http://homepages.feis.herts.ac.uk/~comrcml 29.11.2006

Overview: 

Overview What is “scientific method” ? - traditional approach - problem fitting models and simulations into the framework - necessary conditions for scientific simulations Examples to illustrate criteria for evaluation of models and simulations London underground map Nowak’s model of the evolution of words in human language Elman’s simulation of the development of grammar Conclusions: The difference between necessary and sufficient conditions for scientific method.

Scientific method – some historical landmarks: 

Scientific method – some historical landmarks 300BC Aristotle and methods of categorising knowledge c. 1000AD Islamic mathematics and scientific experiments Bacon – controlled experiments, empirical scientific investigations 1637 Descartes - Discourse on Method, rationalism Newton – hypotheses that enabled predictions 1920 Fisher – statistical analysis 1934 Popper – falsifiability as a criterion 1962 Kuhn –The Structure of Scientific Revolutions

What is a model or simulation?: 

What is a model or simulation? Models and simulations contrast with real objects or events Word model used in differing ways. One end of a spectrum: an artifact which aims to reproduce the function of a real biological system as closely as possible. E.g. cochlear implant A contrasting usage: language model, a term which denotes a collection of statistics on word frequencies Simulation typically means a model with dynamic elements IEEE has 2000+ Computer Simulation Standards for scientific processes MDA – Model Driven Architecture

Characteristics of a scientific approach: 

Characteristics of a scientific approach Objective versus subjective Both deductive ( as in mathematics) and empirical knowledge Supporting evidence, repeatable experiments But limitations of an inductive approach e.g. all swans are white Production of hypotheses that can be falsified (cf Karl Popper) Falsification rather than verification Example of Newtonian mechanics Contrast with other forms of belief, e.g. “St Paul’s cathedral is a beautiful building” “It is wrong to eat people”

Problems fitting models into this framework: 

Problems fitting models into this framework Since models and simulations abstract certain features and ignore others they can be falsified by examining other features that were not abstracted. Example: the map of the London underground Purpose: to show passengers which route to take Functional efficiency: excellent But, not to scale. Relative distances between stations do not match actual distances. Models and simulations cannot be assessed using a naïve falsification test Note that underground map is not arbitrary: corresponds topologically to the real system.

Map of London Underground: 

Map of London Underground

Slide8: 

Feature abstraction (1) Picasso – portrait of a young girl

Slide9: 

Feature abstraction (2) Picasso – portrait of a young girl

Criteria for evaluating simulations and models: 

Criteria for evaluating simulations and models Consistency between empirical evidence and declared aims fitness for purpose external validity: a reality check Consistency within the model or simulation internal cohesion Well founded choice of parameters arbitrary choices explicit (caution in use of metaphorical language)

Use of simulations in language acquisition and evolution: 

Use of simulations in language acquisition and evolution Research into language acquisition - in evolutionary time - in historical time - in the lifetime of an individual Simulations needed because There are few ways in which we can find out about events millions of years back. We create a virtual laboratory for experiments They avoid unethical investigations into the functioning of the brain

Example 1“Computational and evolutionary aspects of language” M. A. Nowak et al. (Nature, Vol 417, 6.6.2002): 

Example 1 “Computational and evolutionary aspects of language” M. A. Nowak et al. (Nature, Vol 417, 6.6.2002) Aim: to produce a model, “a theoretical framework explaining how darwinian dynamics lead to fundamental properties of human language” Process includes assumptions: “that a language can be seen as an infinite binary matrix linking phonetic forms to semantic forms” “ambiguity …. is the loss of communicative capacity that arises if individual sounds are linked to more than one meaning” Applying a reality check: this is inconsistent with human language. English and other languages have many ambiguous sounds there / their here / hear one / won two / to / too etc. etc. Disambiguation through context.

Example 1 continued: 

Example 1 continued This model also shows how a limited number of phonemes can be combined to produce an indefinite number of words “the maximum fitness of a language increases exponentially with word length” Plotkin and Nowak, J. of Theoretical Biology, 2000, vol 205, p158 Lacks external validity Not a model for the evolution of human language Could be a model for communication between synthetic agents

Example 2 “Distributed Representations, Simple Recurrent Networks, and Grammatical Structure” Elman, Machine Learning, 1991: 

Example 2 “Distributed Representations, Simple Recurrent Networks, and Grammatical Structure” Elman, Machine Learning, 1991 Purpose of investigation: “How viable are connectionist models for understanding cognition?” (p.220) “The connectionist model can be seen as a mechanism for gaining new theoretical insight” (p. 197) Elman’s model claims to represent long distance dependencies, critical in speech and language. (e.g. pre-planned co-articulation: lip position for “tea” and “two”)

Example 2 (cont): 

Example 2 (cont) Recurrent neural net combines input at time t with previously processed input from time t-n Supervised training, using back prop Prediction task: What word will come next? Is it grammatical? Lexicon of 23 words John feeds dogs. *Boys sees John Boys who see John feed dogs.

Example 2 (cont)Learning Long Term Dependencies with Gradient Descent is Difficult,  Bengio et al.,IEEE Trans. On Neural Networks, 1994: 

Example 2 (cont) Learning Long Term Dependencies with Gradient Descent is Difficult, Bengio et al.,IEEE Trans. On Neural Networks, 1994 It is possible to train a recurrent NN on a particular task Models with short dependencies are trainable Trade off between efficient learning and latching information for longer periods. “gradient descent becomes increasingly inefficient when the temporal span of the dependencies increases” (p. 164)

Necessary conditions for scientific simulations: 

Necessary conditions for scientific simulations External validity Internal cohesion Well founded abstractions Fitness for purpose – but what is the purpose? A simulation might meet these conditions but not be scientific. These conditions are not sufficient. Consider a simulation of bullying among school boys

Bullying scenario (1): 

Bullying scenario (1) Players adopt the persona of Jimmy Hopkins, a 15-year-old thug who has been incarcerated in a boys’ boarding school. Points can be scored by terrorising other pupils with a range of physical and psychological abuse. Players use their on-screen persona to kick and punch other pupils and even to spit in their food. They can use weapons such as baseball bats and catapults. …… The game has angered children’s campaigners. Sunday Times on the game Bully 14.8.2005

Bullying scenario (2)Simulation tool to help victims address bullying problems: 

Bullying scenario (2) Simulation tool to help victims address bullying problems Synthetic agents representing school children populate a virtual playground. They develop relationships as they interact - talking, playing and sometimes bullying. The user identifies with one of the bullied agents and can explore this virtual world, seeing how different reactions might reduce or increase bullying. Scenario 1 is a game. Scenario 2 may be a scientific search for knowledge about ways of teaching

Role of unexpected events in scientific method: 

Role of unexpected events in scientific method Discovery by chance – e.g Galvani and the frogs’ legs, Fleming and penicillin Recent example: Behaviour of agents or robots that was unexpected (but can be explained post hoc) E.g. “Exploiting Physical Constraints: Heap Formation through Behavioural Error in a Group of Robots” Maris and te Boekhorst, IEEE Proc IROS 1996 Experiments with robots on object avoidance failed, but instead delivered results on creation of clusters. Placement of sensors on robots led to blind spots so that objects were sometimes pushed instead of avoided.

Conclusions: 

Conclusions Scientific method has a bundle of characteristics. They must include: A search for knowledge Objective, repeatable experiments Logical deductions They may include Use of models and simulations that meet necessary conditions of external validity, internal cohesion, well founded abstractions Production of falsifiable hypotheses and should include: An open mind, ready to expect the unexpected “Chance favours the prepared mind” Louis Pasteur