2003 Jan OpenForum Defense Hayes

DoD XML Management: 

DoD XML Management Dr. Glenda Hayes, MITRE ghayes@mitre.org Open Forum on Metadata Registries – January 23, 2003

Agenda: 

Agenda Background Market-Driven Data Strategy Visibility Coordination Guidance Revised DoD Data Strategy Improve Interoperability While Reducing Development Time and Cost

Theory v. Practice: 

Theory v. Practice

Management OptionsContrasting Styles: 

Management Options Contrasting Styles Top-down, “Command” versus “Market” LOOSE TIGHT SPECTRUM OF CONTROL What will work for the DoD? Recommended Approach: Market with Some Controls DoD has: C2 Intelligence Finance Personnel Weather Geospatial Transportation Records Mgmt Training Security Legal Medical Veterans Postal And more…

Key Management Mechanism : 

Key Management Mechanism Market Visibility & Accessibility Developers and Warriors Asking for Help!! What data is available? Who has it? How do I get it? Which data is better?

Data Management Requirements: 

Minimum: “Available” Data must be made visible* and accessible Then: Avoid Conflicting Data. Ensure uniqueness (at least!) Facilitate Data component re-use (Interoperability and savings) Publish relationships Provide for open, participatory Community Data configuration control Achieve convergence where possible *”You can’t manage what you can’t see!” Data Management Requirements

Slide7: 

Visibility

DoD Data EmporiumBuild Time Metadata Discovery Service : 

DoD Data Emporium Build Time Metadata Discovery Service “One Stop” Publish & Subscribe for: Build Time Metadata Registration Re-usable Data Components Information Community Management Vending: Database Segments Reference Sets Transformations XML Data Tools More Products soon (e.g., ontologies, translations) http://xml.dod.mil Purpose: visibility and re-use, not standardization through mandate!

DoD XML Registry Features: 

DoD XML Registry Features IOC: 19 May 1999; Current Version: 3.1 Information Resource Visibility, Navigation, and Registration Search/Browse Filters for Discovery & Reuse On-Line Submission Processing Subscription Services Context Identification REST-based API (using http, uri, xml) Synchronized Components on Multiple Networks Community of Interest (COI) Registration Multiple Feedback Mechanisms http://xml.dod.mil/xmlreg/user/feedback.cfm Public XML Venue Namespace Managers’ Forum On-Line Registry Administrative Features DoD XML Registration Policy signed by ASD(C3I) & USD(AT&L) http://xml.dod.mil

DoD XML RegistrySupporting Discovery & Reuse of Components: 

DoD XML Registry Supporting Discovery & Reuse of Components Information Resources Document Domain Value Set XML Attribute XML Element XML Sample Schema Schema Data Type Stylesheet Source Code Associations Contains IsNewerVersionOf IsConstrainedByDomain DescribedBy IsXMLSpecFor IsQualifiedByAttribute IsDerivedFrom Example: messages.xsd (Schema) IsXMLSpecFor air_to_air_refueling_combined_task_message (XML Element)

Sample of RegisteredNodes and Arcs: 

Sample of Registered Nodes and Arcs messages.xsd (schema) composites.xsd (schema) sets.xsd (schema) fields.xsd (schema) DescribedBy Contains IsXMLSpecFor DescribedBy air_to_air_refueling_combined_task_message (element) DescribedBy imagery_intelligence (element) abbreviated_security_classification (element) Contains IsConstrainedByDomain abbreviated.security.classification.901.2 (simpleType) IsDerivedFrom abbreviated.security.classification.901.2 (Domain Value Set - XML) DescribedBy schema simpleType element XML Legend air_to_air_refueling_combined_task_message.xml (sample XML) DescribedBy DescribedBy

DoD XML Registry “Posts” XML to every seat on the GIG: 

DoD XML Registry “Posts” XML to every seat on the GIG Unclassified (Open) NIPRNET w/Internet Access http://xml.dod.mil/xmlreg Unclassified (Sensitive) Open Registry contents plus “password protected” components NIPRNET with Internet Access http://xml.dod.mil/xmlreg Secret Unclassified (Sensitive) content plus secret components SIPRNET http://diides.ncr.disa.smil.mil Top Secret (SCI) Secret content plus SCI components JWICS Available Dec 02 XML building blocks available at every keyboard Unclassified (Sensitive) Secret Top Secret (SCI) Unclassified (Open) NEW! Synchronized Components on Multiple Networks Users Discover and Pull XML they are cleared for -

DoD XML RegistryCommunities of Interest: 

DoD XML Registry Communities of Interest Other Proposed MASINT Document Training Health Affairs Formal Process to Establish New COIs

DoD XML Registry Current Inventory: 

DoD XML Registry Current Inventory Includes Password-protected Components (USMTF, etc.) Current As of 20 Jan 2003

Namespace Search Filter: 

Namespace Search Filter Name, Definition, Comment, All Submitters to this COI Status relevant to this COI IRs registered in this COI Developmental, Operational… Effective Dates in Submissions Responding to Customer Feedback

Search Results: 

Search Results Name COI Date Version Definition Responding to Customer Feedback

DoD XML Registry’s REST-ful API: 

DoD XML Registry’s REST-ful API Representational State Transfer (REST) Web service via HTTP (Get, Post, Submit), URI, & XML Included as 1 of 3 WSDL v1.2 Bindings (HTTP/1.1 GET/POST) DoD XML Registry REST-ful service Search HTTP Method: GET URI: described in http://xml.dod.mil/xmlreg/user/ws_rest_based_search.doc XML return: ws_result.xml

DoD XML Registry’s REST-ful APIReference Implementations: 

DoD XML Registry’s REST-ful API Reference Implementations Demo Reference Implementation Download N Schemas from Registry MS Access 2000 + MS XML Core Services Service Pack 4 Disk Space: 1.6M (app) Download: 391K (zipped) Supplementary Registry Tool Updateable XML Registry Snapshot MS Access 2000 + MS XML Core Services Service Pack 4 Command-Line winzip (wzcline.exe) Disk Space: 41M (app) + 135M (file space) Download: 14M (zipped app) To be available from the DoD XML Registry

Slide19: 

Coordination

DoD XML RegistryEngineering-level Processes: 

DoD XML Registry Engineering-level Processes DOD Developer Namespace Managers & WGs DoD XML Registry Registry Ops Staff Controls/Displays on Web COI/Namespace Managers’ Forum Participates in Participates in Consults & Submits to/Downloads from XML Registry Namespace Managers & WGs Namespace Managers & WGs DISA Engineering Staff Supports Governs/Coordinates Acts as Registry CM Board Operates, Participate in DATATWG SSD-MD (Public XML Forum) Maintains Hosts

DISA XML Coordination: 

DISA XML Coordination Venues Public XML Venue (SSD-MD) http://diicoe.disa.mil/coe/aog_twg/twg/ssdmd/ssd-md_page.html Namespace Mgrs Forum (NSMF) https://portal.mitre.org (Private Portal hosted by MITRE) Federal CIO XMLWG http://xml.gov http://xmlregistry.nist.gov/xml-gov/ NIST Namespace Management Workshop http://xw2k.sdct.itl.nist.gov/namespaceWS/index.htm Email and Newsgroup dataemp_support@fgm.com http://diides.ncr.disa.mil/shade/feedback.cfm news://coenews.ncr.disa.mil/coenews.data And LOTS of Briefings…

Public XML VenueSemi-Structured Data & Metadata Subpanel (SSD-MD): 

Public XML Venue Semi-Structured Data & Metadata Subpanel (SSD-MD) Objectives Develop specifications and/or schema Select metadata standards and tools Evolve schema repository / visibility mechanisms / versioning management Provide guidance for tag terminology Develop “enhanced” XML editors for coded XML docs Develop application interpreters for XML Explore Reference implementations Keep up on X-technologies See DOD XML Registry “Links” NCES XML Coordination Done Here!

DoD XML Registry Coordination Namespace Managers Forum: 

DoD XML Registry Coordination Namespace Managers Forum NSMF Objectives: Propose, review, and implement DoD XML policy  Develop and promote best practices in XML  Seek opportunities for convergence  Define Registry requirements, oversee development and operation   Determine what metrics to use, analyze and make recommendations to DoD and other policies (I&RTS, JTA)  Review proposals for additional namespaces and make recommendations to AOG.  Participate, Respect, and Influence industry, international and coalition metadata (XML) standards        

Convergence Task Forces Addressing Specific Joint Data Issues : 

Convergence Task Forces Addressing Specific Joint Data Issues TBD Namespace Consolidation – DISA & NSMF Identify redundancies in TBD Delivery Address Pilot – DFAS, LOG & USPS “Address” Schemas Data Transformation Pilot – DISA & NIMA FIPS 10-4 & ISO 3166 Country Codes: Representation and Cross-reference XML Conventions Proposal – DISA & Service PEOs Naming Conventions Transport Guidance Enterprise Discovery Metadata – C3I, DISA & IC* on NCES C2&I Pilot Critical Path, see Walker Presentation (Friday morning)

Slide25: 

Guidance

Responsible XML Development: 

Responsible XML Development Use the DoD XML Registry to: Identify reusable components Subscribe to change notifications Register new components Participate in XML venues Coordinate with namespace managers Do NOT Develop XML in a Vacuum!

DoD Policy for XML RegistrationASD(C3I) & USD(AT&L) signed 22 Apr 2002: 

DoD Policy for XML Registration ASD(C3I) & USD(AT&L) signed 22 Apr 2002 XML Component Registration Consult DoD Registry before creating new components and reuse existing XML where practical Indicate planned use of XML components by formally subscribing to them Consult other Registries before inventing new XML Register any additional XML or recommend modifications Community of Interest Namespace Formation Formed “as required” when someone will agree to manage Requirements for new Namespaces staffed with: Existing Namespace Managers Senior Service/Agency engineers Flag Level Review Board http://xml.dod.mil/xmlreg/user/Documents/XML.PDF

FIPS10-4CountryCodeType Domain Source: DOD Data Emporium RefSet: 

FIPS10-4CountryCodeType Domain Source: DOD Data Emporium RefSet documentation XML Elements: UpperCamelCase XML Attributes: lowerCamelCase Types: UpperCamelCase + Type Data Type, Data Size

Slide29: 

Data Strategy

Emerging DoD Data Strategy ASD (C3I) Principles: 

Emerging DoD Data Strategy ASD (C3I) Principles “Post” Data as early and as widely as possible Empower Users to pull whatever they want Clearly identify Data maintenance authorities Decompose Data Management Communities of Interest (COIs) Build-Time and Run-Time Exploit Market Forces Visibility (Allows data producers and consumers to find each other) Supply and Demand (“Incentivizes” timely, high quality production) Market Metrics (Allows adjustments for investment to follow value) TASK - POST - PROCESS - USE (TPPU)…..

Data Transformation Strategy Towards Net-Centric DoD Data Policy: 

Data Transformation Strategy Towards Net-Centric DoD Data Policy DOD XML Policy pilots new Strategy ASD(C3I) & USD(AT&L) 22 April Memo requires XML Registration and defines rules XML Registry and Governance being extended to cover other data forms (e.g., data elements, message formats, data base structures, etc) Revision of 8320 guidance ASD(C3I) 3 Jan Memo “Planned Changes to DoD Data Administration and Cancellation of Associated Procedures” Capitalizes on new technology and lessons learned Covers full range of data regimes (Run-time as well as Build-time) Leverages commercial standards Governed through Communities of Interest (COI) Emphasizes registration for visibility over top down standardization Integrates DDDS content into Comprehensive DoD Metadata Services (DoD Data Emporium + Net Centric Enterprise Services) Selects and Cleanses DDDS data worth salvaging Associates Data Standards and Engineering Components Legacy 8320 capability in Maintenance Mode Maintaining DDDS operations in parallel

Lessons-Learned: 

Lessons-Learned Registration (posting) and Access (Pulling) must be simple and efficient – capability v. simplicity hard to balance Developers also want Run-Time access for applications Robust Discovery services are required Convergence process needs to be codified Implementation deadlines are key drivers Better metrics required Make subscription mandatory? Anticipated Benefits: Cost, Schedule and Interoperability improvements Requirements, convergence opportunities and players more visible Unanticipated Benefits: Metadata Quality Improvements Clearinghouse more active than expected

Questions?: 

Questions?

What’s the benefit?: 

What’s the benefit? For the Program Manager Re-use registered components Faster, Cheaper, Lower Risk For Management Metrics for Acquisition Oversight For the end-user in Run Time Better data understanding & consistency

Reference Implementation: 

Reference Implementation

Pseudocode: 

Pseudocode Query Registry for IRs where IR type is Schema http://xml.dod.mil/xmlreg/user/ws_search.cfm?type=XMLSchemaDocument Results are returned in an XML document Name IRFetchString Process each result in XML document Use MSXML2.XMLHTTP40 (methods: open, send; property: responseText) HttpReq.open "GET", IRFetchString, False HttpReq.send HttpReq.responseText Create new directory for each item (to ensure uniqueness) Write “responseText” to file

Structure of REST APIQuery Results: 

Structure of REST API Query Results

Output from Reference Implementation: 

Output from Reference Implementation

DoD XML RegistrySchema: 

DoD XML Registry Schema Displaying Registry Core Only!

XML Registry Snapshot DB Schema: 

XML Registry Snapshot DB Schema DB Schema enforces Identity & Referential Integrity Import Services enforces Add’l Registry Business Rules Logical Keys Namespace Name Version Information Resource Type Physical DB Deviations String Size > 255 = Memo VarChar(2000) = Memo

XML Registry SnapshotAdd New Packages: 

XML Registry Snapshot Add New Packages

XML Registry SnapshotLocally Stores Submission Pkgs: 

XML Registry Snapshot Locally Stores Submission Pkgs

XML Registry SnapshotBrowse Packages: 

XML Registry Snapshot Browse Packages

XML Registry SnapshotAd Hoc Queries: 

XML Registry Snapshot Ad Hoc Queries

Data Engineering Proposal: 

Data Engineering Proposal Briefed to DoD XML Registry Namespace Mgrs’ Forum Guidelines for NEW XML Revised per Comments (e.g., use of Dublin Core for enumeration meaning) FIPS and ISO Schemas + XML Mapping Doc Under Review by GEO COI Mgr (Pam Stephens, NIMA) for registration in GEO To be considered as candidate for “Enterprise” Comments Welcome!! ghayes@mitre.org

Public XML VenueSemi-Structured Data & Metadata Subpanel (SSD-MD): 

Public XML Venue Semi-Structured Data & Metadata Subpanel (SSD-MD) Meetings Bi-monthly at MITRE, with VTC support 20 Meetings since Jan 1999 Topics DoD XML Efforts XDBI, XIS, ASAS, MLDB, XMLMTF, Space XML, WAWF & DoDAAC, DLIS, DAML, PDML, ngJBI, DiMeS, XML Component Engineering, XML Convergence Task Force Registry Enhancements & COI Status Briefs Specifications & Architectural Styles XMI, CWM, SOAP, XML Schema, ebXML Registry, Topic Maps, REST XML Conventions/Guidelines Navy, AF, COE Proposed Consortium Efforts OASIS UBL, NIST, OpenGIS, OMG, ISO 11404 Vendor Briefings

DoD XML RegistryCustomer Service Transactions: 

DoD XML Registry Customer Service Transactions ’01 Emporium “hits” = approx 24K (100/day) XML Registry = 50-plus/day Other Emporium pages = approx 50/day XML Component Re-use Army Intelligence Data Integration GCCS-I3 XML Target Folders JMPS to re-engineer w/GMI vocabulary Ground Moving Target Indicator Clearinghouse Function 25 - 30 Q&A transactions/mo Initial Convergence Task Forces Logistics/DFAS lead: Alcon to agree on “Postal Address” Proposed Conventions and Normative Schemas

What’s the benefit?: 

What’s the benefit? For the Program Manager Re-use registered components Faster, Cheaper, Lower Risk For Management Metrics for Acquisition Oversight For the end-user Pragmatic Approach for Understanding & Consistency

Process for Creating a New Namespace: 

Process for Creating a New Namespace Contact DISA Team Jim Pipher (703-882-1367), pipherj@ncr.disa.mil Stan Davis (703-882-1373), davis2s@ncr.disa.mil Toni Weir (703-882-1366), weirt@ncr.disa.mil Glenda Hayes (703-883-7175), ghayes@mitre.org Present case informally Describe need and scope Pre-Brief case to Namespace Mgrs Forum Discussion and feedback Brief case to senior DoD engineers NCES PM notifies proposer & DoD CIO

COI Proposal ContentSlides and/or Voice Track: 

COI Proposal Content Slides and/or Voice Track What COI is being proposed? What is the scope of the proposed COI? Provide sample XML. How will the proposed COI relate to other existing COIs? How will the COI impact relate to key DoD systems? What organization will manage the COI? Why is the organization being proposed as COI manager the most appropriate group to serve this role? (what is their authority? Is there an alternative to this org?) What data does the COI manager produce and/or manage? What larger bodies does the org participate in? (e.g., Personnel namespace mgr serves on both OMG and ISO bodies) How does the COI mgr intend to provide the “coordination function” within his COI? Is there an existing coordination body that can be leveraged? Has the proposed COI mgr POC briefed this proposal to the Namespace Mgrs Forum? Is the proposed COI mgr POC prepared to participate in the SSD-MD and Namespace Managers Forum?

COI Management TasksCM for Collections of XML Components : 

COI Management Tasks CM for Collections of XML Components Coordinate Stakeholders Establish and Run Coordination Venue(s) Resolve Issues within Namespace Coordination across Namespaces (bilateral, multilateral and/or via Namespace Mgrs Forum) Oversee Development of XML Documents, DTDs, XSL style sheets Submit Information Resource Packages to XML Registry Vet Information Resources and Publish Status Perform Versioning Participate in Namespace Mgrs Forum and SSD-MD (public XML venue)

Each COI Determines Its Management Process: 

Each COI Determines Its Management Process Courtesy of METOC, Elizabeth Warner <xsd:element name = "JMO_surfaceObs"> <xsd:complexType> <xsd:attribute name = "airTemperature“ type=“xsd:decimal“ /> <xsd:attribute name = "dewPointTemperature“ type=“xsd:decimal“ /> <xsd:attribute name = "windDirection“ <xsd:restriction base = "xsd:integer"> <xsd:maxInclusive value = "360"/> <xsd:minInclusive value = "1"/> </xsd:restriction> </xsd:attribute>

Working in a World of Codes: 

Working in a World of Codes Many communities use codes to serve as proxies for enumerated domains The codes become the effective standard Use of codes present challenges for training new users But, what if… XML in a browser

With Common Encoding of Code “Meanings”: 

With Common Encoding of Code “Meanings” Tools to leverage additional semantics Browsers via “Mouseover event” (proposed to Microsoft & Mozilla) Exploit via XSL for display Reduced End-User Training, Improved Readability Relevant for Military Messaging and EDI Tomorrow’s XML browser?

Mapping Standard Domains FIPS and ISO Country Codes: 

Mapping Standard Domains FIPS and ISO Country Codes Both standards exist, neither can be ignored Challenge for interoperability Transforms needed IQ IZ US US GZ FX TP UM WE Qty w/Same code 93 Qty w/Different code 143 Qty not in ISO 26 Qty not in FIPS 3 FIPS Qty: 262 ISO Qty: 239 IRAQ

Domain Mapping Challenge: 

Domain Mapping Challenge Reuse of simpleTypes and complexTypes is less contentious than reuse of elements Lack of common standard for neutral mapping of simpleTypes No xpath representation for simpleTypes Using FIPS Using ISO <LivingIn> <LivingIn> <CC> <Allegiance> <Country> <COUNTRY> <country> <CountryOfResidence> <XYZ> <Nationality>

Neutral Mapping of simpleTypesfor machine translation: 

Neutral Mapping of simpleTypes for machine translation IRAQ Ireland Israel Proposed Representation

Using Mapping Representation With Translator: 

Your Translator Using Mapping Representation With Translator From MyEnemies.xml <LivingIn>IZ</LivingIn> From MyEnemies.xsd: <xsd:element name=“LivingIn" type="FIPS10-4CountryCodeType"> From FIPS-ISOMapping.xml: <Set> <Term type="FIPS10-4CountryCodeType" value="IZ" /> <Term type="ISO3166-2charCountryCodeType" value="IQ" /> <Term type="ISO3166-3charCountryCodeType" value="IRQ" /> <Term type="ISO3166-3numCountryCodeType" value="368" /> </Set> From BadBloke.xsd: <xsd:element name=“Residence" type="ISO3166-2charCountryCodes"> From BadBloke.xml <Residence>IQ</Residence> FIPS ISO

Proposed DISA Conventions for New* XML: 

Proposed DISA Conventions for New* XML Preference for W3C XML Schema Recommendation Use annotation/documentation for elements, attributes, types to provide documentation Use UpperCamelCase for elements, lowerCamelCase for attributes Use annotation/appinfo <dc:title> to provide meaning for enumeration values Use explicit Data Types rather than inline definitions Use UpperCamelCase for types and append “Type” to differentiate types from elements For aggregation tags, append “List” to element name Include xml:lang tag within schema and xml instance where needed. Limit the use of “generic” tags (e.g., no “item” tag) Express tie-back to XML Registry * Not intended for evaluating suitability for reuse

From Build-Time to Run-TimeComposing NCES-based Architectures: 

“Custom” Components From Builders Discovery & Meditation NCES Discovery Catalog Other NCES Network Check-in Registration Process GIG Build-Time Run-Time Plug-in/unplug Data Emporium From Build-Time to Run-Time Composing NCES-based Architectures Finished Capbility Plug-in Metadata (XML) Router Firewall Security Registry Transformation Service Other Capabilities COTS Components From Vendors Data/Metadata Components Install & physical connection Development Vocabulary Management