B2 01 Haddawy

COMETCollaborative Intelligent Tutoringfor Medical Problem-Based Learning: 

Peter Haddawy Siriwan Suebnukarn CSIM Program School of Dentistry Asian Institute of Technology Thammasat University COMET Collaborative Intelligent Tutoring for Medical Problem-Based Learning

Problem-based Learning: 

Problem-based Learning Respiratory system Circulatory system Nervous system Self-directed study literature, lectures laboratories, specialists, etc. Problem analysis Synthesis and application of newly acquired information New Scenario Scenario Mr. C was involved in a car accident…

Problem-based Learning: 

Should we discuss in more detail the mechanism of the brain moving? S3: Brain moves forward after car accident …… What else is a consequence of the brain moving forward? S1: The brain moving forward might cause diffuse axon injury. …… S2: Then Intracranial pressure would increase. Problem-based Learning Dialogue Example Car accident case Mr. C was involved in a car accident while he was driving home from work. The broken glass window cut deep into his forehead and tore a hole through the frontal bone in his skull. Bleeding and unconscious, he was rushed to the hospital where a surgeon operated on him to stop the bleeding and found out that many parts of his brain had been damaged.

Problem-based Learning: 

Problem-based Learning Hypotheses

Problem-based Learning: 

Problem-based Learning CSCL provides shared learning environments to support students from diverse locations ITS provides one-to-one tutoring with essentially no incremental cost per additional student. Medical students often do not get as much facilitated PBL training as they might like or need. PBL requires the tutor to provide a high degree of personal attention to the students.

COMET: 

COMET COllaborative MEdical Tutor

COMET- Demo: 

COMET- Demo

COMET- System Overview: 

COMET- System Overview

COMET: 

Domain Clinical Reasoning Model Individual and Collaborative Student Modeling Generating Tutoring Hints Evaluation COMET

Domain Clinical Reasoning Model: 

Domain Clinical Reasoning Model Goal Concept Apply Hypothesis Enabling condition Fault Consequence probability of knowing the hypothesis conditioned on whether the parent hypotheses are known, whether the student is able to apply the appropriate knowledge to determine the cause-effect relationship The conditional probability tables were obtained by learning from data obtained from the transcripts of PBL sessions using EM learning algorithm.

Domain Clinical Reasoning Model: 

Domain Clinical Reasoning Model

COMET: 

Domain Clinical Reasoning Model Individual and Collaborative Student Modeling Generating Tutoring Hints Evaluation COMET

Individual and Collaborative Student Modeling: 

Individual and Collaborative Student Modeling The domain clinical reasoning model is instantiated for each student by entering that student’s medical background knowledge as evidence. The evidence is propagated increasing the probabilities of Apply1, Apply2, and Apply3, Scalp Laceration, Skull Fracture and Brain Contusion. Knowledge of Scalp Anatomy, Skull Anatomy, and Cerebral Cortex Anatomy is typically required for creating the hypotheses Laceration, Skull Fracture and Brain Contusion. Individual Expertise

Individual and Collaborative Student Modeling: 

Individual and Collaborative Student Modeling Assume that once a hypothesis in the domain model is created by one student in the group, every student knows that hypothesis, since all students have basic knowledge before they encounter the PBL sessions Integrating group action into each Individual Propagation influences the probabilities of yet to be observed nodes, thus predicting what inferences the student is likely to make.

COMET: 

Domain Clinical Reasoning Model Individual and Collaborative Student Modeling Generating Tutoring Hints Evaluation COMET

Generic Tutoring Strategies: 

Generic Tutoring Strategies 1. focus group discussion 2. promote open discussion 3. deflect uneducated guessing 4. avoid jumping critical steps 5. address incomplete information 6. refer to experts in the group 7. promote collaborative discussion

Focus Group Discussion: 

Focus Group Discussion What is the effect to the cerebral cortex? - Identify the group reasoning path. - Select the hint node as the hypothesis which has not been mentioned and has the highest probability. Think about the cerebro-spinal fluid? Val, can you help the group? Skull fracture Brain damage Unconscious Brain contusion Intracerebral hematoma ICP

Create Open Environment for Discussion: 

Create Open Environment for Discussion Skull fracture Subdural hematoma Can you relate the mechanism of Brain contusion to what we have discussed? Unconscious Brain contusion Determine the degree of divergence against the group reasoning path. A B C Domain A B C D1 D2 D3 B

Avoid Jumping Critical Steps: 

Avoid Jumping Critical Steps Skull fracture Unconscious Can you think of the mechanism underlying why skull fracture causes unconsciousness? Skull fracture  Unconscious Check for the intermediate nodes. The hint node is the highest probability node. A E Domain B C D A E C

Promote Collaborative Discussion: 

Promote Collaborative Discussion Car accident Skull fracture Other members, Do you have any idea? What happens to the structure underlying the skull? Brain damage If there is no input from a student after n hypotheses (a group of n students) have been mentioned, the system should point to that student. To avoid having one student dominate the discussion, each student has a chance to suggest 3 hypotheses consecutively.

Evaluation: 

Quality of the Hints Accuracy of the Student Models Student Clinical Reasoning Gains Evaluation

Evaluation: Accuracy of the Model: 

Evaluation: Accuracy of the Model Compare the probabilities of hypotheses and causal links from the student model with student actions (gold standard). Receiver Operating Characteristic (ROC) curve plots sensitivity (TP) vs 1-specificity (FP) of the posterior probabilities of the nodes in the BN model. The area under the curve (AUC) represents an overall measurement of performance of the student model. 1.0 represents a perfect test. 0.5 represents a model with no discriminating capacity. Diagnostic Model Receiver Operating Characteristic (ROC) curve

ROC Curve Analysis: 

ROC Curve Analysis The models are highly accurate in predicting individual student actions. 570 hypotheses 1140 links

Evaluation: 

Quality of the Hints Accuracy of the Student Models Student Clinical Reasoning Gains Evaluation

Evaluation: Clinical Reasoning Gain: 

Evaluation: Clinical Reasoning Gain COMET Human Tutor

Clinical Reasoning Problems: 

Clinical Reasoning Problems Case 1. 1

CRP Mean Score: 

CRP Mean Score Cronbach’s alpha reliability coefficient Pre-test questions = 0.901 Post-test questions = 0.921 Pre-test Post-test

Clinical Reasoning Gain: 

Clinical Reasoning Gain Significant improvement in both groups (Wilcoxon, p = 0.000). Student clinical reasoning gains from COMET are significantly higher than those obtained from human tutored sessions (Mann-Whitney, p = 0.011).

Conclusions: 

Conclusions COMET is the first general domain-independent framework for group intelligent medical tutoring. COMET uses Bayesian networks to model individual and collaborative problem solving for medical problem-based learning. Generic domain-independent tutoring algorithms use the models to generate tutoring hints. ROC curve analysis indicates that the models are highly accurate in predicting student actions. Student clinical reasoning gains from COMET are significantly higher than those obtained from human tutored sessions.

Ongoing and Future Work: 

Ongoing and Future Work A collaborative case authoring tool that employs the UMLS ontology to assist in creation of case scenarios. Completed evaluation and preparing for deployment

Ongoing and Future Work: 

Ongoing and Future Work Using UMLS to expand the set of solutions COMET can accept. Chat tracking tool to enable the tutor to follow and comment on the discussion. Intelligent tutoring for clinical skill acquisition using simulated environments with haptic devices. Anatomical sketch understanding.

More Information (www.cs.ait.ac.th/~haddawy): 

More Information (www.cs.ait.ac.th/~haddawy) Siriwan Suebnukarn and P. Haddawy, A Bayesian Approach to Generating Tutorial Hints in a Collaborative Medical Problem-Based Learning System. In Artificial Intelligence in Medicine, 2006. Siriwan Suebnukarn and P. Haddawy, Modeling Individual and Collaborative Problem-Solving in Medical Problem-Based Learning.  In User Modeling and User-Adapted Interaction, 2006. P. Haddawy, et al, Anatomical Sketch Understanding: Recognizing Explicit and Implicit Structure.  In Artificial Intelligence in Medicine, 2006. Siriwan Suebnukarn and P. Haddawy, Clinical-Reasoning Skill Acquisition through Intelligent Group Tutoring, In Proc. Int’l Joint Conference on Artificial Intelligence (IJCAI-05), July 2005.