2001 EL Public talk

EasyLiving: 

<![CDATA[ EasyLiving Steve Shafer et al. Ubiquitous Computing group http://easyliving r]]>

EasyLiving Team: 

<![CDATA[ EasyLiving Team John Krumm Steve Harris Brian Meyers Barry Brumitt Steve Shafer Greg Smith Andy Wilson]]>

EasyLiving: 

<![CDATA[ EasyLiving Is developing an architecture and technologies for Ubiquitous Computing ]]>

Ubiquitous Computing: 

<![CDATA[ Ubiquitous Computing ]]>

Beyond Mobile Computing: 

<![CDATA[ Beyond Mobile Computing Processor Data Store UI Device Bus Highly Integrated Computing Desktop Computing Mobile Computing]]>

Services At Multiple Scales: 

<![CDATA[ Services At Multiple Scales Campus: GPS / Cell phone 10m precision -- “locate building” Building: Badge 1m precision “locate printer” Rooms: Camera 10cm precision “select UI device” City / Zip Code -- deliver pizza]]>

The EasyLiving Video: 

<![CDATA[ The EasyLiving Video Progress report for January 2001]]>

Slide8: 

<![CDATA[]]>

How EasyLiving Works: 

<![CDATA[ How EasyLiving Works Applications World Model UI Services Sensors UI Devices]]>

The EasyLiving System: 

<![CDATA[ The EasyLiving System Rules Engine Person Tracker Person Detector Person Detector Seat Sensors PC Logon Fingerprint Logon Room Lights A/V Media Systems Terminal Server Room Control UI KB/Mouse Redirect Desktop Manager World Model Agent Lookup Future … person tracking world model room control authentication]]>

EasyLiving Summary: 

<![CDATA[ EasyLiving Summary World Model database Agent = process + properties Geometry built into database Current developments: Room modeling without cameras Integration with .NET and GEO Navigation and Visitor applications Intelligent Room (perception for UI)]]>

Elements of EasyLiving: 

<![CDATA[ Elements of EasyLiving Research in Sensing and World Modeling Distributed System Architecture UI and Applications of Computing in the Physical World http://www.research.microsoft.com/easyliving]]>

Ubiquitous Computing Research: 

<![CDATA[ Ubiquitous Computing Research Mobility Communication Programming Tools & Models Location & Context Data Applications Speech UI Vision-Based UI Dialog & MMUI Conforming GUIs Semantic UI Specs User Studies Most “ubiquitous” computing is very parochial Need for standardization]]>

Revenge of the Knowledge Representation: 

<![CDATA[ Revenge of the Knowledge Representation KR was traditionally an AI specialty Procedural v. Declarative KR Widgets / Agents v. Databases / Blackboards Procedural is easier to engineer & create Declarative is easier to integrate with Uncertainty Knowledge Engineering]]>

Community Building: 

<![CDATA[ Community Building UBICOMP conference Georgia Tech, Sept. 29-Oct. 2 2001 Gregory Abowd & me www.ubicomp.org IEEE Pervasive Computing magazine Satya, CMU – Sept. 2001 planning mtg. Possible proposal for ACM-SIGUBI I’m spearheading To be discussed at UBICOMP]]>

Slide16: 

<![CDATA[]]>

Person Tracking in EasyLiving: 

<![CDATA[ Person Tracking in EasyLiving]]>

Person TrackingSystem: 

<![CDATA[ Person Tracking System ]]>

Person Detection: 

<![CDATA[ Person Detection Stereo processing with commercial software color depth Reports sent to central person tracker about 7 Hz patches people Background subtraction and person detection]]>

Person Tracking: 

<![CDATA[ Person Tracking Predicted location Past locations New sensor measurement “Person creation zone” Process each new report from a sensor Timeouts for unsupported tracks Nearby tracks “repel”]]>

Seat / Floor Mat Sensors: 

<![CDATA[ Seat / Floor Mat Sensors Additional information for person-tracking Distinguish stand v. sit Less invasive than cameras]]>

EasyLiving Person Tracking: 

<![CDATA[ EasyLiving Person Tracking none maintain recognize identity 0% 50% 100% occlusion stand stand & sit stand, sit, lay postures one image video tape always on show canned sequence live demos naïve user real time number of people in view number of cameras 1 2 3 n 1 2 3 n ]]>

Interactive Camera Calibration: 

<![CDATA[ Interactive Camera Calibration]]>

New Person Tracker (Future): 

<![CDATA[ New Person Tracker (Future) Multiple sensors of different modalities Explicit reasoning about occlusion No more expensive multi-camera heads One vision computer per room Voxel representation of 3D space in room ]]>

Active Badges (Future): 

<![CDATA[ Active Badges (Future) Badges for person tracking: Larger area, less precision Fusion via generic person tracker Ultrasound badges (AT&T) in lab use Diffuse-IR system from Arial Systems Signal strength from Wireless LAN?? Bluetooth beacons?]]>

Keyboard Detection: 

<![CDATA[ Keyboard Detection Match color and approximate shape 1 2 3]]>

Slide27: 

<![CDATA[]]>

World Model in EasyLiving: 

<![CDATA[ World Model in EasyLiving]]>

World Model: 

<![CDATA[ World Model What is described? Computing devices What they are, where they are, their status, the proxy agent, the service area People: Who they are, where they are, their preferences, what they are doing Services Rooms and doorways Inert things in the world The world model has many parts]]>

InConcert Distributed Agent System: 

<![CDATA[ InConcert Distributed Agent System Process Properties Agents represent all interesting things in the world An Agent has both a Process and a set of Properties Agent]]>

InConcert Distributed Agent System: 

<![CDATA[ InConcert Distributed Agent System Process Properties Process Process Properties Properties SQL Database Differs from pure-object models]]>

InConcert Distributed Agent System: 

<![CDATA[ InConcert Distributed Agent System Process Properties Process Process Properties Properties SQL Database Activation Service Model Service Messaging Service Application]]>

How Agents Find Each Other: 

<![CDATA[ How Agents Find Each Other Specific knowledge: Direct pointers from other agents Searching in real time: Attribute matching Geometric indexing]]>

The Geometric Model: 

<![CDATA[ The Geometric Model Polygon objects and their relationships Display Service region of display Joe]]>

Geometry in the Real World: 

<![CDATA[ Geometry in the Real World Features: Uncertainty representation All measurements are relative Stored in SQL Server Camera 1 John camera person display badge beacon room Pocket PC person Camera 3 viewing area ]]>

Using the Geometric Model: 

<![CDATA[ Using the Geometric Model Queries Commands Measurements Applications/UIs ]]>

The Data Chain: 

<![CDATA[ The Data Chain perception behavior World Model service application 100 Hz 1 sec 10 sec Dataflow Blackboard object 10 Hz]]>

Why SQL for World Model?: 

<![CDATA[ Why SQL for World Model? Commercial database is supported Robust and stable Very fast (throughput) Lots of features Scalable But: Slow (latency) Awkward knowledge representation Requires client & server software]]>

Slide39: 

<![CDATA[]]>

Room Control in EasyLiving: 

<![CDATA[ Room Control in EasyLiving]]>

Let There Be Light: 

<![CDATA[ Let There Be Light Flip a switch Touch screen on wall computer Use pocket computer Say “Turn on the lights” Make a funny gesture It’s dark outside Ubiquitous computing gives new options]]>

Turning On The Lights: 

<![CDATA[ Turning On The Lights Flip a switch GUI dialog box Click on a GUI map “Turn on the light by the sofa” “Turn on that light” “I want to read a book” Make a funny gesture Walk into the room]]>

Turning On The Lights – How?: 

<![CDATA[ Turning On The Lights – How? Flip a switch – hardwired GUI dialog box – list (GM) Click on a GUI map – GM “Turn on the light by the sofa” – speech “Turn on that light” – multimodal “I want to read a book” – infer action Make a funny gesture – vision Walk into the room – automatic ]]>

Device Control: 

<![CDATA[ Device Control Device Proxy Agent Device UI Agent XML Application Other Agent Type = Display Size = 20 inches Address = 17284 Web Page UI]]>

Context-Awareness: 

<![CDATA[ Context-Awareness What is “context”? Location, Activity, History, … Origin Services collect or measure data and expose it Effects Change what is displayed to you Change your options Change effect of commands you issue Predict what is most likely to come next]]>

EasyLiving Media Player: 

<![CDATA[ EasyLiving Media Player Speakers Speaker Proxy Media Player Agent Type = Speakers Address = 17284 Display Proxy Type = Display Address = 9375 Owner = Susan Person Proxy Type = Person Name = Susan Playlist = … Room Proxy Type = Room Playlist = … ]]>

Automatic Behaviors: 

<![CDATA[ Automatic Behaviors Example: Turn on the lights when you enter the room Condition  Action (program) Behaviors of the room Behaviors of the device Behaviors of the person Automatic behaviors are part of the UI]]>

Slide48: 

<![CDATA[]]>

Authentication in EasyLiving: 

<![CDATA[ Authentication in EasyLiving]]>

Authenticated or Not?: 

<![CDATA[ Authenticated or Not? When you walk into the room, you are an unknown person Toolbar appears on the screen Limited privileges – lights, browse web You can authenticate to become known as a specific individual Greater privileges – desktop Personal data available – music list]]>

Authentication in EasyLiving: 

<![CDATA[ Authentication in EasyLiving Fred Barney Wilma Person 17 Person 23 Person Tracker Thumb Scanner Playlist = … database ID = ? Wilma ? Keyboard login Badge system Biometric ID Geometry provides the link]]>

Personal Desktop: 

<![CDATA[ Personal Desktop Display Desktop Manager T.S. Client T.S. Server Wilma’s Desktop Wilma Desktop = … Person 23 ID = Wilma Proxy Owner = Person 23 Click the “Desktop” button in the toolbar Needs info to pass to T.S. Client Wilma gets her own desktop no matter where she goes]]>

Disaggregated Computing: 

<![CDATA[ Disaggregated Computing Proxy Display Proxy Keyboard Proxy Mouse Desktop Manager Room Control UI Automatic Behavior Laptop or PDA devices can also be proxied]]>

Slide54: 

<![CDATA[]]>

Future Directions for EasyLiving: 

<![CDATA[ Future Directions for EasyLiving]]>

Project Milestones: 

<![CDATA[ Project Milestones July 1997 Project conceived]]>

Project Milestones: 

<![CDATA[ Project Milestones July 1997 Project conceived July 1998 First logo]]>

Project Milestones: 

<![CDATA[ Project Milestones July 1997 Project conceived July 1998 First logo February 1999 Second logo]]>

Project Milestones: 

<![CDATA[ Project Milestones July 1997 Project conceived July 1998 First logo February 1999 Second logo May 1999 First shirt]]>

Project Milestones: 

<![CDATA[ Project Milestones July 1997 Project conceived July 1998 First logo February 1999 Second logo May 1999 First shirt January 2001 Spinning logo]]>

Connected Computing: 

<![CDATA[ Connected Computing Both local and global Connections Pairwise activities Unstructured list of devices & services ]]>

Ubiquitous Computing: 

<![CDATA[ Ubiquitous Computing EasyLiving Semantic unity through central model Multi-device services Structured directory and search ]]>

Models of ID Privacy: 

<![CDATA[ Models of ID Privacy Invoke application on trusted device ID stored on device Trusted application requests your ID ID stored in application Automatic behavior personalized ID stored in world model (EL demo) World Model Application ]]>

The Future ofEasyLiving: 

<![CDATA[ The Future of EasyLiving Room List – who/what is where All built on .NET (SOAP) and SQLServer CAD Model – navigation Vision System – dynamic & detailed]]>

Context-Adaptive Apps: 

<![CDATA[ Context-Adaptive Apps Database Agents Display so everyone can see Person-specific custom show Use multiple display screens Media player Message delivery MSR-TV Example: PowerPoint that uses available context]]>

EasyLiving + TV: 

<![CDATA[ EasyLiving + TV Follow-Me TV Who’s Using the Remote? Automatic Pause Control ]]>

Geometric Path Planner: 

<![CDATA[ Geometric Path Planner 1. Start Here 2. Visit Printer 3. Destination “I need to print a handout on the way to my next meeting.” (Not yet integrated with geometric model)]]>

Vision-Based Interaction: 

<![CDATA[ Vision-Based Interaction Head tracking Gaze tracking Lip reading Face recognition Facial expression Hand tracking Hand gestures Arm gestures Presence: Is anyone there? Location: Where are they? Identity: Who are they? Activity: What are they doing?]]>

Speech and Vision Interaction: 

<![CDATA[ Speech and Vision Interaction Pointing at things: “Put my desktop on that display” Paper as mouse Paper keyboard Device does not need to be a real object]]>

World Model Building: 

<![CDATA[ World Model Building Build geometric model using drawing program Use cameras to help build model]]>

“Sidewalk Warrior”: 

<![CDATA[ “Sidewalk Warrior” Novel low-power devices (Turner W.) Watch w/ display Pen w/ accelerometers Communicate via BodyNet Automatic discovery of room system Associate devices with this person Advertise services provided by devices Integrate with services in room]]>

Contact Anyone Anywhere: 

<![CDATA[ Contact Anyone Anywhere ]]>

Long-Term Issues: 

<![CDATA[ Long-Term Issues Privacy Security Complexity Extensibility & Standardization EasyLiving is a testbed for exploration of these issues EasyLiving is an “advance scout” for Microsoft in ubiquitous computing]]>

Slide74: 

<![CDATA[]]>

What is Ubiquitous Computing?: 

<![CDATA[ What is Ubiquitous Computing?]]>

Ubiquitous Computing: 

<![CDATA[ Ubiquitous Computing Ubiquitous Computing means getting computing “off the desktop” Connect with what else people do “Pervading” the physical world = “Pervasive”, “Ambient”, “Situated” Many ways to break the traditional mainframe/PC model Many CS communities working hard Many different definitions]]>

Desktop Computing: 

<![CDATA[ Desktop Computing Processor Data Store UI Device Network Bus Bus provides speed, connection, ID, trust, synchrony Processor assumes minimum suite of devices Key feature: User has to sit in chair ]]>

Mobile Computing: 

<![CDATA[ Mobile Computing Differences: Everything is smaller Power is limited Network connection is wireless Person is standing or moving! Network Includes Wearable Computing]]>

Wireless Connectivity : 

<![CDATA[ Wireless Connectivity People mean different things by “wireless”: Wireless WAN is cell-phone-based ISP Wireless LAN gives access to network resources Ad Hoc (Proximity) LAN gives access to nearby resources (e.g. Bluetooth) Wireless LAN Ad Hoc LAN Wireless WAN ]]>

Hands-Free Computing: 

<![CDATA[ Hands-Free Computing Processor Data Store UI Device Bus Change UI devices to work at a distance No need to carry anything! Mobility very limited Network speak move look point gesture]]>

Intelligent Environment: 

<![CDATA[ Intelligent Environment Processor Data Store UI Device Bus Hands-free with greater area of coverage Need connectivity for UI devices Need to dynamically select UI devices Network UI Device UI Device ?]]>

Disaggregated Computing: 

<![CDATA[ Disaggregated Computing Processor Data Store UI Device Ensemble of devices is dynamic Each device actually has processor & proxy agent Application processor has environment proxy agent Can incorporate mobile devices also Network UI Device UI Device UI Device ]]>

Non-Traditional Devices: 

<![CDATA[ Non-Traditional Devices Why be limited to traditional computing devices? More pervasive, more subtle, more informative Enable novel applications – “immobile robot” ]]>

Location-Sensitive Computing: 

<![CDATA[ Location-Sensitive Computing Computing experience changes as you move - Migrating (“Follow-Me”) UI - Location-Specific Data Need device to know where you are Cell phone, GPS, active badge, camera, … ]]>

Object-Sensitive Computing: 

<![CDATA[ Object-Sensitive Computing Object tag  Mobile device  Database ( URL) Technologies: IR, Bar code, RF, … Services: e-Commerce, Location, Device control, Ad hoc networking, … ]]>

Invisible Computing: 

<![CDATA[ Invisible Computing Everything in the world is electronically active Your attention is on your task Of course, there are computers present “Calm” or “ambient” computing Related: Personal memory devices; Phicons ]]>

Non-Interface Interfaces: 

<![CDATA[ Non-Interface Interfaces Phicons – Physical Icons Endowing ordinary objects with meaning Input, storage, output Static v. dynamic association What does this salt shaker mean? Automatic Behaviors The world “does the right thing” What is the right thing?]]>

Wearable Computing: 

<![CDATA[ Wearable Computing Variation on mobile computer Novel UI devices enable novel applications Stand-alone VR (Virtual Reality) With location, AR (Augmented Reality) ]]>

Ubiquitous Computing: 

<![CDATA[ Ubiquitous Computing The Ubiquitous Computing Equation: Ubiquitous Computing = Mobile Computing + Intelligent Environment]]>

Our Quest: 

<![CDATA[ Our Quest Desirability]]>

Long-Term Issues: 

<![CDATA[ Long-Term Issues Attention Complexity Privacy Security Extensibility]]>

Attention: 

<![CDATA[ Attention How to signal a person about an event? Nature and parameters of the event Available devices in environment Policies governing environment Preferences of person Person’s current activity and attention Challenge: How not to bother people “Ambient” information]]>

Complexity (to users): 

<![CDATA[ Complexity (to users) Lots of decisions for system to make Lots of devices to manage Need some degree of automation “Behavior Rules” Where do they come from? How do you manage them? “What will happen if I sit in that chair?”]]>

Privacy: 

<![CDATA[ Privacy System is expected to make lots of decisions for / about you System needs lots of information “Who” gets that information? Maybe you give explicit permission Maybe you post information Good intentions, lousy practices]]>

Privacy Policies: 

<![CDATA[ Privacy Policies No “passive” authentication Right of revocation of identity (not yet) No transmission of live images Images only for videoconferencing (?) Cover sensitive areas (how?) Personal control of information Various levels of sensitivity of data Various radii of distribution]]>

Security: 

<![CDATA[ Security This is a story about delivery of services on the network How do you know who you’re talking to? Ubiquitous computing requires … … very dynamic connectivity … exchange of lots of personal data … lots of exposure to fraudulent access]]>

Extensibility: 

<![CDATA[ Extensibility Interoperability All domains are one domain Need for standardization Configuration Tools for creating intelligent spaces Tools for managing devices & processes Debugging Oy, oy, oy!]]>

How Will Issues Be Addressed?: 

<![CDATA[ How Will Issues Be Addressed? Industry is moving forward Some mechanisms are emerging … though sometimes controversial Trial and error Slow growth of capabilities gives time to develop solutions Unified v. diversified]]>

How We Steer Technology: 

<![CDATA[ How We Steer Technology Introspection Demo and Discussion User Studies Business Case Marketplace Literature Standardization Legislation ?]]>

Situation-Aware Computing: 

<![CDATA[ Situation-Aware Computing Casual Access to Computing You can always signal the computer The computer can always signal you You can go get better I/O if you want Interactions sensitive to your context Migrating User Interface When you move, interaction moves with you Interaction decoupled from computation]]>

Contact Anyone Anywhere: 

<![CDATA[ Contact Anyone Anywhere Caller initiates transaction with computer Computer locates recipient Computer signals recipient Recipient decides whether to proceed During call, (1) caller changes UI device, (2) caller moves, (3) UI on mobile device moves This will be our initial goal scenario for EasyLiving !]]>

Child Care Assistant: 

<![CDATA[ Child Care Assistant Computer as enhanced baby monitor: UI adapts to parent Record and interpretation of events Dangerous interactions with environment, other people, pets Access to off-limits spaces or information Privileges for parents, children, babysitters, visitors How to program all this? Can be simple, or arbitrarily deep]]>

Disability Applications: 

<![CDATA[ Disability Applications Do everyday things with custom UI Vision Impairment Navigate via PDA or speakers Where Is It? What Is It? Who’s There? Hearing Impairment Noises in environment TTYD Everywhere Mobility Impairment Telepresence to reduce travel Monitor you for emergencies]]>

Intelligent Appliances: 

<![CDATA[ Intelligent Appliances The intelligent oven: Control by knobs or by computer Program by time or by food readiness Is it time for self-cleaning? Schedule+ it! Is something burning or boiling over? Was something spilled that needs wiping? Are the pot handles turned inwards? Is this person authorized to operate the oven? What would it take for you to trust your oven? Your home?]]>

A Safer World: 

<![CDATA[ A Safer World Notifying about unsafe conditions, preventing unsafe actions, emergency notification How hard to achieve these goals? How intrusive? How easily fooled? How configurable and self-configurable? How dependent will people become?]]>

Other Promising Scenarios: 

<![CDATA[ Other Promising Scenarios Vision-Based Home Automation Intelligent recording / playback of events Where did I put my keys? Energy control, security, … Autonomous Vacuum Cleaner General mobility platform Sharing information with environment Intelligent planning and execution Futuristic: Deciding when to vacuum]]>

Ten Dimensions: 

<![CDATA[ Ten Dimensions Meaningfulness World Modeling User Modeling Distribution Accessibility Extensibility Heterogeneity Automation Usability Ubiquity]]>

Slide108: 

<![CDATA[]]>