XML Mgmt Update: XML Mgmt Update Dr. Glenda Hayes MITRE/DII-COE SHADE


Agenda: Agenda Market-Driven Data Strategy Electronic Marketplace For XML DII-COE XML Registry XML Coordination & Guidance


Data Management Challenges: Data Management Challenges No Single Standard can be imposed! Various DoD communities will adopt multiple “standards:” Government (message, database, symbology), Commercial, International, de facto/legacy etc. Expect Heterogeneity!


XML Management Challenges One Language, Many Vocabularies: XML Management Challenges One Language, Many Vocabularies 30N 30 N 30 These 3 XML fragments are: Equally valid ways to express the same data in XML Well-formed per W3C Specification Different terms and XML structures NOT INTEROPERABLE!


Management OptionsContrasting Styles: Management Options Contrasting Styles Top-down, “Command” versus Market LOOSE TIGHT SPECTRUM OF CONTROL What will work for Defense? Recommended Approach: Market with Some Controls


Principles from President Bush: Principles from President Bush Citizen-centered Results-oriented Market-based Mar-15-01


Key Management Mechanism : Key Management Mechanism Market Visibility Developers and Warriors Asking for Help!! What data is available? Who has it? How do I get it? Which data is better?


Market Players: Market Players Developers Re-use available data components and/or Register new ones they have created Community Data Managers Use Market visibility services for configuration control (e.g., current version distro, version change notification etc.) Defense Acquisition Policy Makers Use Market metrics for acquisition oversight (e.g., reflecting Program participation, specific data component re-use etc.)


Market Features: Market Features Low barriers to entering the process Electronic Marketplace as Visibility mechanism Components as unit of exchange (commodities) Communities of Interest (COIs) as arenas for trade Program Engineers as primary “traders” Must engage “hands on” Developers who build important Data Resources!


Electronic Market DISA’s Data Emporium : Electronic Market DISA’s Data Emporium “One Stop” Publish & Subscribe for Defense Metadata Data Component Registration COI Creation & Management New Version Jan 01 Public Access via WWW plus Password protected and classified instances http://diides.ncr.disa.mil/shade Purpose: visibility and re-use, not standardization through mandate!


Marketplace Rules: Marketplace Rules Data Component Registration Consult Emporium before creating new components and reuse existing data where practical Indicate planned use of components by formally subscribing to them Register additional components or recommended mods Communities of Interest (COIs) Formation Created “as required” when someone will agree to manage Requirements for new COIs staffed with: Existing COI Managers Senior Service/Agency engineers Flag Level Review Board


Market Organization: Market Organization Configuration Control features: Distributed across DISA Engineering Staff and “Hands On” Managers from multiple COIs Capitalizes on existing configuration control bodies COI “Managers” govern data in multiple physical forms Coordination Venues: Individual COI “boards” COI Managers’ Forum Data Technical Working Group (Any voice heard) Ad hoc contacts with other Government, Allied, and Commercial activities } Configuration Control Organizations


Electronic Market Place XML Registry : Electronic Market Place XML Registry IOC May 99 Public Access via WWW (FOUO) Password protected Improved Version Jan 01 10K elements and schema SIPRNET version early CY01 Handles XML Registration Namespace Creation & Mgt CIO EB Expanding Scope XML Registration Policy and Plan http://diides.ncr.disa.mil/xmlreg Purpose is management visibility and re-use, not standardization through mandate!


XML Market Place Rules: XML Market Place Rules XML Registration Requirement Developers using XML for public interfaces should: Consult XML Registry before creating new XML components and reuse existing XML where practical Indicate planned use of Registered XML by formally subscribing to it Submit (where required) additional XML components (with amplifying information) or recommended modifications to existing components Creation of XML Namespaces Formed “as required” when someone will agree to manage Requirements for new Namespaces staffed with: Existing Namespace Managers Senior Service/Agency engineers Flag Level Review Board CIO EB kept apprised


Namespaces/Managers: COE/DISA Ground Operations/Army General Military Intelligence/DIA Aerospace Operations/USAF Messages/DISA Tracks & Reports/Navy Combat Support/DISA Geospatial & Imagery/NIMA METOC/Navy Personnel/DIMHRS Finance and Accounting/DFAS TBD/DISA Enterprise/DISA Namespaces/Managers Other Proposed Logistics MASINT NBC Transport Training Trade/Export


DII-COE XML Registry Update: DII-COE XML Registry Update Features Structure/Schematic Submission Process Submission Assistance 3 case studies Namespace Population Status


DII-COE XML Registry Features: DII-COE XML Registry Features Search/Browse Filters Across or within namespaces By Information Resource Type By Substring in Name, Definition, Comment By Submitter By Status By Version Subscription On-Line Submission Pkg Verification & Submission On-Line Registry Administrative Features


DII-COE XML Registry Resources and Relationships…: DII-COE XML Registry Resources and Relationships… XML schema ATO confirmation Information Resources and Relationships


DII-COE XML Registry Information Resource Types: DII-COE XML Registry Information Resource Types Manifest an XML document IAW http://diides.ncr.disa.mil/xmlreg/DTD/registry.dtd Schema e.g., Bookshop.dtd XML Element (atomic and container) Atomic: Title Container: Book XML Attribute e.g., Genre, in_stock Domain Document an XML document IAW http://diides.ncr.disa.mil/xmlreg/DTD/registry_domain_values.dtd e.g., con_genre.xml, con_yesno.xml Stylesheet e.g., Bookshop.xsl XML Example e.g., Bookshop.xml, BookshopStyled.xml Amplifying Document e.g., bookshop.gif


DII-COE XML Registry XML Component Submission Process: DII-COE XML Registry XML Component Submission Process 1) User submits package (wo/write cap) 2) Verifier emails to DISA 3) DISA screens for embarrassing material 4) DISA submits package (w/write cap) Registry DB Submission Script DII-COE XML Registry Registry Submission Package Verifier BookshopSubmission.zip email 4 1 2 http://diides.ncr.disa.mil/xmlreg/XMLUserSubmitForm.cfm 3


DII-COE XML Registry Submission Package Example: DII-COE XML Registry Submission Package Example Package Contains 1 manifest (XML) 0+ docs 0+ domain docs (XML) 0+ schemas Manifest.xml Submission.zip con_genre.xml bookshop.gif Bookshop.dtd con_yesno.xml Bookshop.xml Bookshop.xsl BookshopStyled.xml


DII-COE XML Registry Leveraging Existing Metadata Assets: DII-COE XML Registry Leveraging Existing Metadata Assets Domain Values Domain-Child Info Child Info Parent Info Parent-Child Assoc SAT1* Submission Manifest Domain Values Registry Verifier COE XML Registry Registry Ingestor Winzip xml Domain Values Domain Values Domain Values Erwin Rational SAT = Submission Assistance Tool Exports from Modeling Tools + MITRE Prototype Submission Assistance Tool XML Schema (DTD, Schema) SAT2*


Leveraging Existing Metadata Assets Case 1 - USMTF: Leveraging Existing Metadata Assets Case 1 - USMTF USMTF CCB Maintains Data Dictionary in relational database Algorithm to transform USMTF-to-xml-mtf Stylesheet to transform xml-mtf-to-USMTF Challenges Mismatch in Data Types with DII-COE Registry Lack of Name for Groups, Segments SHADE Strategy Enter FUDs (atomic elements) Enter Domains, link to FUDs, store xml files of domains Enter groups, sets, segments, messages Registration Status Messages Namespace 6165 XML Elements Registered (2/28/2000)


USMTF & xml-mtf: MSGID/TACREP/CTF 124// MAROP/011800Z/1/US/SUB/CL:WASHINGTON/NAME:SEAROVER /LM:4040N01100E// OPSUP/ACTTYP:ASW// AIROP/020200Z/6/US/FTR/F15/TN:401/LM:4130N01000E/CRS:180/SPD:600KPH /ALT:12000FT// OPSUP/ACTTYP:DCA// USMTF & xml-mtf USMTF = United States Message Transmission Format MIL-STD 6040 USMTF


xml-mtf Elements Registered: xml-mtf Elements Registered


Leveraging Existing Metadata Assets Case 2 – Data Models: Leveraging Existing Metadata Assets Case 2 – Data Models MIDB Erwin Physical Model 7 add’l domain documents DIA-supplied Long Names Challenges Relational v. Hierarchical Models Erx errors Multiple techniques for storing domains SHADE Strategy Harvest metadata from erx Register Columns & Attributes as “atomic” XML Elements Register Tables & Entities as “container” XML Elements Register Enumerated Domains as Domain Documents Develop Viewer Register Better Definitions for Domain Documents Register Logical-to-Physical Mappings Register Parent-Child Relationships as “contain” relationships Register algorithmic constraints using XML Schema Definition Registration Status GMI Namespace 2000 XML Elements 228 Domain Docs 1 XML Mapping Doc


MIDB Elements Registered: MIDB Elements Registered


MIDB Domain Values Registered: MIDB Domain Values Registered


Leveraging Existing Metadata Assets Case 3 – XML Schema Definition: Leveraging Existing Metadata Assets Case 3 – XML Schema Definition Convert DTDs to XML Schema Definition Tool support is available Augment elements/attributes with Annotation Data Type Constraints Patterns Enumerated Values SHADE Strategy: Construct Submission Manifest Shell from .xsd No progress to report at this time


DII-COE XML Registry Namespace Population Progress: DII-COE XML Registry Namespace Population Progress Messages 6165 XML elements Domain docs Container (complex) elements Aerospace Operations 1 DTD, 8 XML elements Ground Operations Technical coordination required Tracks and Reports 452 XML elements 1 schema 8 domain docs Intelligence 2000 XML elements 228 domain docs 1 mapping doc MDITDS request Geospatial & Imagery Baseline GML and IML Finance & Accounting Leveraging existing metadata METOC Leveraging existing metadata


XML Coordination and Guidance: XML Coordination and Guidance


XML Management “at-a-glance”: XML Management “at-a-glance” Participates in Participates in Consults & Submits to/Downloads from XML Registry Supports Governs Operates, Maintains Participate in Hosts


XML Convergence Process: XML Convergence Process Namespaces & managers chartered “as required” Overlap among Namespaces is inevitable Namespaces act as “Buckets” for in-use XML Managers run Working Groups (collaborative venues) Entry points for XML Registry submissions & Developer requirements Status mechanism (developmental, operational, deprecated) provides ability to express COI preference Enterprise Namespace holds DoD “common” XML Identifies and Registers “Common” XML from COI Namespaces Governed by Namespace Managers’ Forum Registry provides XML “Market” visibility Includes system and developer usage information


DII-COE XML Guidance XML Data Compliance: DII-COE XML Guidance XML Data Compliance Integration and Runtime Specification DII-COE Compliance Level for XML Reconciled Differences with Standards Used “Production” Components from COI Submitted Mapping of Logical Model to Physical Schema Submitted XML Components to Registry


XML CoordinationDII-COE XML Forum: XML Coordination DII-COE XML Forum Sponsor (DII-COE DATATWG) Defense Information Infrastructure (DII) Common Operating Environment (COE), Data Access Services Technical Working Group (DATATWG), Semi-Structured Data and Metadata Subpanel (SSD-MD) Objectives Develop specifications and/or DTDs Select metadata standards and tools Create DTD repository / distribution mechanism / versioning management Provide guidance for tag terminology Develop “enhanced” XML editors for coded XML docs Develop application interpreters for XML Reference implementations


XML CoordinationDII-COE XML Forum (cont’d): Meetings Chartered by DII-COE Architectural Oversight Group (AOG): Nov 1998 Distribution List = 400 Names, Attendance = 70+ Bi-monthly meetings, 11 meetings since Jan 1999 Topics XML Activities Survey XML Vendor Briefings and Demos DII-COE XML Requirements DII-COE XML Registry Debut XML Technical Discussions XML Policy Coordination SOAP XML Schema XML Coordination DII-COE XML Forum (cont’d) Next Meeting: 27 April


XML Coordination XML Namespace Managers Forum: XML Coordination XML Namespace Managers Forum Draft Forum Objectives: Propose, review, and implement DoD XML policy.  Develop and promote best practices in XML.  Seek opportunities for convergence.  Oversee the operation of the Registry.  Determine what metrics to use, analyze and make recommendations as inputs to DoD and other policies (I&RTS, JTA)  Define, review, update the procedures for the Namespace Mgrs Forum.  Review proposals for additional namespaces to make recommendation to DII-COE AOG.  Participate, Respect, and influence international and coalitions metadata standards.         Next Meeting: 10 May


DoD XML Registration Policy Draft Implementation Plan Coordination: DoD XML Registration Policy Draft Implementation Plan Coordination CRCB directs Chief Engineer to propose DII-COE XML process for DoD Chief Engineer briefs DoD CIO EB 29 Aug CIO EB asks for Draft registration Policy memo Draft implementation plan Resource assessment Staff through ASD C3I, USD AT&L, etc. Info Brief to CIO EB for status May


Contact Information: Contact Information Mr. Pete Pasek - pasekp@ncr.disa.mil Mr. Jim Pipher - pipherj@ncr.disa.mil Mr. Stan Davis - davis2s@ncr.disa.mil Ms. Toni Weir - weirt@ncr.disa.mil Ms. Alesia Jones-Harewood – harewooa@ncr.disa.mil Ms. Ellen Minderman (FGM) - minderma@fgm.com Dr. Glenda Hayes (MITRE) - ghayes@mitre.org Mr. Chuck Heazel (MITRE) - cmheazel@mitre.org SHADE Data Emporium http://diides.ncr.disa.mil/shade


Backup: Backup


DII-COE Guidance : DII-COE Guidance Establishes TWG Priorities Selects or Mods Tech & Mgt Options CIO EB – DoD CIO Executive Board AOG – Architectural Oversight Group CRCB – Configuration Review Control Board TWG – Technical Working Group Establishes/Advises on DOD Policy Namespace Mgrs Forum SSD-MD (XML) Subpanel


Registration is Easy!: Registration is Easy! Developer identifies or creates XML not currently registered. Developer creates Registry package for new XML and proposed mods. Developer submits Registry package via on-line capability, specifying an existing namespace. http://diides.ncr.disa.mil/xmlreg/XMLUserSubmitForm.cfm Registry Ops reviews submittal for propriety and registers package as “Developmental.” Receiving Namespace Mgr reviews submittal and accepts or rejects as properly belonging to another namespace. If rejected, Namespace Mgrs Forum determines whether another namespace will accept it. Upon developers' request, entries rejected by all namespaces are retained in TBD namespace. Namespace Mgrs may review accepted XML to change status from “Developmental” to “Operational.”


Viewing the Intermediate ResultsEnumerated Domain: Viewing the Intermediate Results Enumerated Domain


MIDB BaselineSubmission Package: MIDB Baseline Submission Package


Registered XML Status: Registered XML Status Developmental = “Registered” Operational = Employed by Community Deprecated = Legacy Operational version (still “in use”) Retired = No longer in use


Analyzing the Market: Analyzing the Market Registry support for Market Analysis: ID of duplicate terms XML Component “page” visits, downloads and subscriptions ID of users COE Compliance AOP GOP TAR GEO ENT XML Registry GMI MET PER FIN Lat Long Lat Long Lat Long Lat Long Lat Long Lat Long ? Exploit Observed Commonality!