Intelligent sensorand learning challengesfor context aware appliances: Intelligent sensor and learning challenges for context aware appliances Intelligent sensor and learning challenges for context aware appliances andgt;andgt; Stéphane Canu
scanu@insa-rouen.fr
asi.insa-rouen.fr/~scanu
INSA Rouen, France - EU
Laboratoire PSI
1984: La souris et leMacintoch: 1984: La souris et leMacintoch 200X : la nouvelle rupture 'break through'
La technologie d'aujourd'hui: La technologie d'aujourd'hui Loi de Moore
Communication 'sans fil'
L'ère des données
Quelles applications ?
Wearable: Wearable
IHM: IHM Olympus Optical Co., Ltd. is pleased to announce its new
wearable user interface technologies.
Employing gestures and other hand movements for input, the system
is an ideal match for new wearable PCs.
Wearable: Wearable http://www.redwoodhouse.com/wearable/index.html
http://wearables.cs.bris.ac.uk/public/wearables/esleeve.htm
http://www.ices.cmu.edu/design/streetware/
Reasearch on wearable: Reasearch on wearable
Wearable: Wearable
context aware appliances: context aware appliances Phone by night http://mediacup.teco.edu/overview/engl/m_what.html The mediacup
(calm version of the active badge)
General Motors and CMU: General Motors and CMU The car
- drives together
- informs you
- in a parking… GM/CMU Companion driver interface system
Oops! Where is my car?: Oops! Where is my car? Old fashion software design: process
Match the sentence
Send the query to the satellite
Satellite send query to the car on its own frequency
Car answers…
Tell the computer what to do (where is the switch)
Distributed software design: interaction
Software agents talk together
Future way: Programming by Example
Show the computer what to do
Today's solution: Louis my 3 years old son Disappearing computer andgt;andgt; Your Wish is My Command: Programming by Example Henry Lieberman, editor, Published by Morgan Kaufmann, 2001.
Calm technology: Calm technology Ubiquitous computing
One people - many computer
Technology at our service
Reactive to what user do
Proactive - Prepare what to do next
Situated – sharing context
(Hans Gellersen, Sensing in Ubiquitous Computing)
Adapted to our needs
New functionalities and new behaviors
New way of communicating
Learn to adapt Machines have to know their context andgt;andgt; M. Weiser 'The Computer for the 21st Century.' Scientific American, September 1991
What is the context?: user
activity (available/meeting)
location,
identity, profile
environment monitoring
time, day/night, temperature, weather,
resources (networks, services…)
appliance - proprioception
usage - functionalities
maintenance
resources (energy…) What is the context? Abstract representation of the situation
Knowledge?
How to find it from data? + history… Adapted From Henry Lieberman and Ted Selker, Out of Context:
Computer Systems That Adapt To, and Learn From, Context,
IBM Systems Journal 39, 2000.
Sensing context from the environmentpresentation roadmap: Sensing context from the environment presentation roadmap andgt;andgt; Kristof Van Laerhoven, Kofi Aidoo: Teaching Context to Applications
In Personal and Ubiquitous Computing, Volume 5 Issue 1 (2001) pp 46-49 Data
Representation
Information retrieval
Context evolution
User interaction
Context from data: Context from data Unbelievable capacity
Moore’s law
New sensors
Artificial nose
Bio sensor
'Personal' data
humor: affective computing
Data Era! http://www-stat.stanford.edu/~donoho/lectures.html Data
Representation
Information retrieval
Context evolution andgt;andgt;
Biological sensors: Biological sensors http://www.teco.edu/tea/sensors.html How are you? Data
Representation
Information retrieval
Context evolution andgt;andgt;
Expression recognition: http://markov.ucsd.edu/~movellan/mplab/index.html Machine Perception Lab Face Detection and Expression Recognition Expression recognition Data
Representation
Information retrieval
Context evolution andgt;andgt;
Too much informationkills information: Too much information kills information Critic of the 'Data Era'
Data smog
Non measurable things
Ethical consequences
the Orwellian future
Filter data! Data
Representation
Information retrieval
Context evolution andgt;andgt; 'We are drowning in information and starving for knowledge.'
- Rutherford D. Roger
Intelligent sensors: Intelligent sensors Requirements:
Data
Accuracy and confidence
Self diagnostic
Self calibration
How to do it?
Uncertainty management
Learning ability
Network + database
Adaptation ability
Fault detection mechanism Associated software sensors Data
Representation
Information retrieval
Context evolution andgt;andgt; andgt;andgt;S. Canu et al., 'Black-box Software Sensor Design for Environmental Monitoring' , in International Conference on
Artificial Neural Networks , Skovde, Sweden. Sep 2-4, 1998 (and related work on data validation within the EM2S project)
Data validation: Data validation Mono sensor validation
Static validation
Mean, variance
Dynamic validation
Cusum (control charts)
Trend analysis
Multisensor validation
Residual analysis
Fusion: Joint probability estimation
Prior knowledge: Balanced relations
Hierarchical validation
Multisensor perception Interactive matrix of smart sensors andgt;andgt; http://www.accenture.com/xd/xd.asp?it=enWebandamp;xd=services\technology\research\tech_sensor_matrix.xml
andgt;andgt; K. Van Laerhoven, A. Schmidt and H.-W. Gellersen. 'Multi-Sensor Context-Aware Clothing'. In Proceedings
of the 6th International Symposium on Wearable Computers, 2002 Data
Representation
Information retrieval
Context evolution andgt;andgt;
Software sensor: Software sensor Value + confidence interval + validity domain
How to build it ?
From a model: tracking = Kalman filter
When no model is available: learn it! Raw data v(t) Raw data x(t) Raw data y(t) Raw data z(t) learning = Black box modeling Data
Representation
Information retrieval
Context evolution andgt;andgt; environment
Towards proprioceptors: Towards proprioceptors Learn
How to learn?
Gaussian mixture + EM
Include prior: Bayesian networks
Deal with uncertainty: Evidence framework
Use to:
Detect non nominal situations
Replace missing data d = Curse of dimensionality (Belman) Data
Representation
Information retrieval
Context evolution andgt;andgt; andgt;andgt; E. Petriu et al., 'Sensor based information appliances',
What is data?: What is data? Individuals or measurements
Associated variables
Data set (matrix)
line = measurements
column = variable
Data: point clouds
Data exploration: recognize patterns
too many data: SUMARIZE Data
Representation
Information retrieval
Context evolution andgt;andgt;
Summarize data: Summarize data Non linear components analysis
Feature space: kernel (PCA or ICA)
Local linear
Quantisation (SOM)
Relevant distance
Select features
Local adapted representation
Feature selection
Select relevant situations
Sparse learning
Kernel learning Data
Representation
Information retrieval
Context evolution andgt;andgt; andgt;andgt; J. Mäntyjärvi, J. Himberg, P. Korpipää, H. .Mannila, 'Extracting the Context of a Mobile Device User',
8th Symposium on Human-Machine Systems-HMS,Kassel, Germany, 2001. Kernel representation
Kernel representationDistance maps: Example in 2 dimension of the influence map of the 'black circle'. Red color denotes a high influence while the low influence zones are in blue. Kernel representation Distance maps Analyze data proximity through the kernel map i j andgt;andgt; B. Scholkopf and A. Smola, 'Leaning with Kernels', MIT Press, 2001
Example of kernel map: Example of kernel map Data clouds in two dimensions Associated kernel map Class 2 Class 1 Class 2 Even in d dimensions you can visualize andgt;andgt; S. Canu and al., 'Functionnal learning through kernels', invited lecture at the NATO institute in Leuven, 2002
Looking for hiden shapes: andgt;andgt; Balázs Kégl
http://www.iro.umontreal.ca/~kegl/research/pcurves/ Looking for hiden shapes Data point = information + noise
Principal curve
Non linear PCA
Independent curve
Non linear ICA Data
Representation
Information retrieval
Context evolution andgt;andgt; Kernel representation + linear analysis
Navigatein high dimensional space: andgt;andgt; J. B. Tenenbaum, V. de Silva and J. C. Langford
http://isomap.stanford.edu/handfig.html Navigate in high dimensional space Data
Representation
Information retrieval
Context evolution andgt;andgt;
Information retrieval: Information retrieval What for
User profiling
User identification
Battery discharge rate
Sequence induction…
Classification problem
Decision theory
Example based programming
Learning machine Select relevant cases Data
Representation
Information retrieval
Context evolution andgt;andgt;
A brief historical perspectiveof machine learning: A brief historical perspective of machine learning Before machines
Statistics: PCA, DA, regression, CART, kNN
70's - Learning is logic
Grammatical inference in expert systems
80's - Learning is human
Neural networks: backprop
90's - Learning is a problem: COLT
Kernel machines: SVM
Mixture of experts: adaboost What is the learning problem? Data
Representation
Information retrieval
Context evolution andgt;andgt; andgt;andgt; T. Hastie, R. Tibshirani and J. Friedman, 'The elements of statistical learning', Springer, 2001
What is learning?: Data
Training set
Test point looking for such that
Learning is balancing
Hypothesis set (Neural networks, Kernels)
Fitting criterion (least square, absolute value)
Compression criterion (penalization, Margin)
Balancing mechanism (cross validation, generalization) What is learning? Learning is summarizing
Linear discriminationseparable case: Linear discrimination separable case + + + + + + + wx+ b=0
(w,b) ??? Use hyperplane Data
Representation
Information retrieval
Context evolution andgt;andgt; How to
correctly
classify
all points? Occam
Razor's
Linear discriminationseparable case: Linear discrimination separable case + + + + + + + wx+ b=0 Be sparse Data
Representation
Information retrieval
Context evolution andgt;andgt; How to
correctly
classify
all points?
The classifier Margin: The classifier Margin wx+ b=0 Margin Margin Be sparse Data
Representation
Information retrieval
Context evolution andgt;andgt; How to
correctly
classify
all points?
Maximize the marginBe sparse: Maximize the margin Be sparse wx+ b=0 wx+ b=-1 wx+ b=1 Margin Margin Support Vector Machines: SVM Data
Representation
Information retrieval
Context evolution andgt;andgt; How to
correctly
classify
all points? andgt;andgt; V. N. Vapnik, 'The nature of statistical learning theory', Springer-Verlag, 1995
What is learning?: What is learning? Learning is summarizing SVM Data
Training set
Test point looking for such that
Learning is balancing
Hypothesis set (Neural networks, Kernels)
Fitting criterion (least square, absolute value)
Compression criterion (penalization, Margin)
Balancing mechanism (cross validation, generalization) andgt;andgt; S. Canu, A. Rakotomamonjy, Ozone peak and pollution forecasting using Support Vectors, IFAC workshop, Yokohama, 2001.
Summarize Inputadaptive scaling: Summarize Input adaptive scaling Enumerate all combination
…and score
Preprocessing
Information theory
Statistical test
Wrapper
Use a relevance index
Learn and select together
Global formulation Example of relevance index for a toy problem
with 2 relevant features and 50 irrelevants Data
Representation
Information retrieval
Context evolution andgt;andgt; andgt;andgt; Y. Gandvalet and S. Canu, 'Adaptive Scaling for Feature Selection in SVMs', accepted for publication at NIPS 2002
Summarize patterns: Dimension reduction by
andgt;andgt; multi-resolution analysis
(just like in your eyes…)
Learn at the relevant scale
andgt;andgt; multi scale representation
Efficient implementation
ridgelets, curvelets
wavelets’ kernel Data
Representation
Information retrieval
Context evolution andgt;andgt; Summarize patterns 'Kernelize' wavelets andgt;andgt; A. Rakotomamonjy and S. Canu, 'Frame, Reproducing Kernel, Regularization and Learning', accepted in JMLR 2002
Learning machines challenges: Learning machines challenges Hypothesis set
Multi scale data representation: wavelets
Use context: mixture of experts
Fitting criterion
Sparse distance criterion
Select relevant input (adaptive scaling)
Relevant distance: adapt the kernel
Compression criterion
Information issues
Global optimization
Balancing mechanism
Efficient direct algorithm (one shot learning) Towards Context based learning Data
Representation
Information retrieval
Context evolution andgt;andgt;
Context assessment: Context assessment Deal with uncertainty
plausibility / credibility
unknown states / ability to evolve
data fusion: evidence theory
Take into account prior knowledge: transitions
temporal representation
uncertain transitions
learn probabilities or possibilities
Learn the model
don't start from scratch
create and delete contexts
Adapt context determination to user
from a global imprecise context to specific context How to implement context? Data
Representation
Information retrieval
Context evolution andgt;andgt;
Context implementation: Context implementation Context = state
List of variables
Petri's nets
State = stochastic
Markov model
Bayesian networks
Identify = decision theory (data fusion)
Information retrieval
Learn context
Knowledge discovery
Create / delete
Context hierarchy (time granularity) Context is a language
How to retrieve the context? Henry Lieberman: http://web.media.mit.edu/~lieber/
New idea to deal with context: New idea to deal with context Current context: working memory
Prior knowledge: transition law
Available information: evidence
Data fusion
Learn context
Transition law
Context retrieval from data
Context is a language
Speech recognition
Markovian model
Evidence
Language + previous state
Locator's adaptation Adapt speech recognition ideas to context http://htk.eng.cam.ac.uk/ Data
Representation
Information retrieval
Context evolution andgt;andgt;
Context: Research chalenges: Context: Research chalenges Inputs
Deal with uncertainty (and missing data)
Representation
Data fusion (multimedia fusion)
Context
Define a language
Represent previous state
Learn transition
Feed Back to inputs
Adapt transition to the user
Loop the user: reinforcement
Control mechanism (stability/plasticity dilemma) Challenging research issues http://cslu.cse.ogi.edu/tutordemos/nnet_training/tutorial.html Data
Representation
Information retrieval
Context evolution andgt;andgt;
Break through: Break through What is information?
Computer science
Coding
Signal
Mathematics
Statistics andamp; computer science
Pattern recognition
Functional analysis
?????? andgt;andgt; L. Devroye, L. Györfi and G. Lugosi, 'A Probabilistic Theory of Pattern Recognition', Springer-Verlag 1996. …remember Albert and relativity Theoretical models are essentials
(Mark Weiser, Computer Science Challenges for the Next Ten Years)
My long bet: My long bet Before 2050
We will faced a scientific revolution
regarding information definition
Comparable with the one induced in physics
by the relativity theory $ 500
To greenpeace Long bet fundation at San Francisco http://www.longbets.org/
Research challenges: Research challenges create context
how to define prior contexts: user’s needs
how to represent contexts: stochastic automaton
learn from data: modify, create and destroy context
decide context
validate data software sensors
select relevant inputs representation + distance
select relevant patterns wavelets
select relevant situations SVM and kernel
make decision using data fusion Dempster-Shafer + EM
loop with the user
reinforcement learning
user’s needs Bayesian
networks Integrate: create relevant learning architecture
Questions?: Questions? Asia
Scurry™, Wearable andamp; Virtual Keyboard - Samsung,
K. Doya for reinforcement
America
Context Aware Computing group - Media lab MIT
CMU, Stanford
Georgia tech: Future Computing Environments
Smart Matter Integrated Systems (Xerox PARC)
Montreal – learning lab
Australia
ANU for learning
University of South Australia - wearable computer lab
Europe
Telecooperation Office (TecO) at the University of Karlsruhe
The disappearing computer, a EU-funded proactive initiative
The Smart-Its project
Equator project focuses on the integration of physical and digital interaction
Perceptual Computing in general and Computer Vision in ETH Zurich
IDIAP for machine learning and speech recognition
PSI, France for learning Some context aware references
Slide48:
From macroscopic…: From macroscopic… Data
Representation
Information retrieval
Context evolution andgt;andgt;
…to Microscopic data: http://www.mcell.cnl.salk.edu/ MCell Simulation of miniature endplate
current generation at the neuromuscular junction.
Image rendered with Pixar Photorealistic RenderMan. …to Microscopic data Data
Representation
Information retrieval
Context evolution andgt;andgt;
Emotion detection >> E-Motions: Emotion detection andgt;andgt; E-Motions Data
Representation
Information retrieval
Context evolution andgt;andgt; andgt;andgt; R. Picard, Affective Computing, MIT Press, 1997
http://graphics.usc.edu/~dfidaleo/Emotion/
http://www.mis.atr.co.jp/~mlyons/facial_expression.html Towards affective computing
Learning accuracy: Learning accuracy Find (a,b) such that
Model the model
The sandwich estimator, (Tibshirani, 1996)
Likelihood Based on the Hessian matrix
Confidence machine (Gammerman RHC, 1999)
Confidence: 73.11% - Credibility: 51.37%
Sample the models
Bootstrap (Heskes 1997)
Learn the error
Train using absolute error How to compute error bars? andgt;andgt; R. Tibshirani, 'A comparison of some error estimates for neural network models,' Neural Computation, 8, 152-163, 1996.
andgt;andgt; Tom Heskes, 'Practical confidence and prediction intervals', Advances in Neural Information Processing 9,
eds. Mozer, M., Jordan, M. and Petsche. T., pp. 176-182, 1997.
http://nostradamus.cs.rhul.ac.uk/~leo/pCoMa/ Data
Representation
Information retrieval
Context evolution andgt;andgt; Accuracy confidence
Looking for hiden shapes: Locally linear
representations Looking for hiden shapes andgt;andgt; Sam T. Roweis andamp; Lawrence K. Saul
http://www.cs.toronto.edu/~roweis/lle/ Data
Representation
Information retrieval
Context evolution andgt;andgt;
Movie synthesis from text: Movie synthesis from text Turing proof andgt;andgt; Tony Ezzat and Tomaso Poggio
http://cuneus.ai.mit.edu:8000/research/mary101/mary101.html …from text to movie Data
Representation
Information retrieval
Context evolution andgt;andgt;
From one expression to another: From one expression to another Data
Representation
Information retrieval
Context evolution andgt;andgt;
Non euclidian metrics: Non euclidian metrics
http://cs.unm.edu/~joel/NonEuclid/ Data
Representation
Information retrieval
Context evolution
andgt;andgt;
Hyperbolic Self-Organizing Map: Hyperbolic Self-Organizing Map Data
Representation
Information retrieval
Context evolution
andgt;andgt; What is the 'distance' between two objects
Example on movies - HSOM: http://www.techfak.uni-bielefeld.de/ags/ni/projects/hsom/hsom.html 650 documents
from 16000 reviews
Internet Movie Database Disney's animation
Movies are closed Data
Representation
Information retrieval
Context evolution andgt;andgt; Example on movies - HSOM
Example on movies: Example on movies Data
Representation
Information retrieval
Context evolution andgt;andgt;
Curvlets: Data
Representation
Information retrieval
Context evolution andgt;andgt; Curvlets Deal with high dimensional space http://www-stat.stanford.edu/~jstarck/
Curvlets: Data
Representation
Information retrieval
Context evolution andgt;andgt; Curvlets Deal with high dimensional space The original image 64,536 coefficients. http://www-stat.stanford.edu/~jstarck/
Select relevant situations: Select relevant situations Relevant representation
'Invent' features
Select features
Relevant 'distance'
map
Use kernel
Summarize the examples
Define a relevant global criterion to be minimized
Support vector machines (SVM) Be sparse Data
Representation
Information retrieval
Context evolution andgt;andgt;
Learning architecture: Learning architecture Agent - Data base - Communication
Metadata
Context language
Adaptability: control mechanism
Pre programming: anticipation
Open – modular – distributed
The Ektara Architecture (MIT for wearable)
Nexus - A Platform for Context-Aware Systems
The Context-Toolkit (Geargia Tech) How to debug such software? http://web.media.mit.edu/~rich/ Data
Representation
Information retrieval
Context evolution
Future appliances?: Future appliances? Deal with the context
Recognize
Adapt
Create
Inference, Learning, discovery,
Represent
Decide
Deal with time
From user interface to user interaction
Reinforcement learning
Human factors
How to know what we need? Human factors: cool technology is at our service