Virtuoso Product Family :1 © 2008 OpenLink Software, All rights reserved. Virtuoso Product Family Orri Erling - Program Manager, Virtuoso


Virtuoso Product Categories :2 © 2008 OpenLink Software, All rights reserved. Virtuoso Product Categories Virtual Database Engine Native Data Management (multi-model covering: SQL, RDF, XML, and Free Text) Discussion Platform Mail Proxy Services Client Connectivity Kit Virtuoso Universal Server 2


Virtual Database Engine :3 © 2008 OpenLink Software, All rights reserved. Virtual Database Engine External ODBC or JDBC accessible SQL Data Sources External XML based Data Sources External SOAP or RESTful Web Services External RDF Data (e.g. Oracle) Custom Data Sources via Server Extensions API 3 RDF, XML, SQL Conceptual Views over:


Virtual Database Engine Contd. :4 © 2008 OpenLink Software, All rights reserved. Virtual Database Engine Contd. SQL Queries over Remote SQL, RDF, XML, and Web Services based Data Sources SPARQL Queries over Remote SQL, RDF, XML, and Web Services based Data Sources XQuery/XPath Queries over Remote RDF, SQL, and XML based Data Sources Web Services based access to Remote RDF, SQL, XML, and other Web Services based Data Sources 4


Virtual Database Engine Contd. :5 © 2008 OpenLink Software, All rights reserved. Virtual Database Engine Contd. Distributed Query Optimization Locality Sensitive Query Cost Optimization (Collocated Joins, Pass-Through Queries, and Array Parameters) Deductively Abstracts SQL Dialect Differences (via ODBC and JDBC metadata call exploitation) Message Latency Factored into Cost Model Hash Joins Used When Appropriate, Replacing Multiple Remote Lookups with Single Sequential Read 2-Phase Commit for Distributed Transactions MS DTC for Windows Tuxedo on Unix 5


Virtual Database Engine Contd. :6 © 2008 OpenLink Software, All rights reserved. Virtual Database Engine Contd. ATTACH TABLE Statement incorporates Remote Table, Indexes and Statistics into Local Virtuoso Schema Allows Incorporation of SQL Functions and Stored Procedures from Remote Relational Database Engines Support for Remote XML, Full Text Indexing for Oracle, Microsoft SQL Server 6


Native Data Management – Relational (RDBMS) :7 © 2008 OpenLink Software, All rights reserved. Native Data Management – Relational (RDBMS) Native SQL 92/2K Engine Rich Procedure Language (PSM-95 based) Database Engine Optimized for SMP Performance Native Full Text Indexing 7


RDBMS Features - Transactions :8 © 2008 OpenLink Software, All rights reserved. RDBMS Features - Transactions Full ACID Properties Checkpoint + Roll Forward Log, Optional Archiving of Logs Uncommitted/Read Committed/Repeatable/Serializable Isolations Non-blocking Read Committed Shows Latest Committed Versions of Uncommitted Updated Rows Can Work as XA/MS DTC Resource Manager 8


RDBMS Features - SQL :9 © 2008 OpenLink Software, All rights reserved. RDBMS Features - SQL Full SQL 92 with many 2K Features SQLX, XPATH, XSLT, Xquery SQL 2K Objects, Implementation in SQL/Java/.net Transparent Mixing of Local and Remote Tables 9


RDBMS Features – Query Optimization :10 © 2008 OpenLink Software, All rights reserved. RDBMS Features – Query Optimization Cost Based Optimization On The Fly Sampling of Table/Column/Literal Key Cardinalities Fixed Statistics for Deterministic Query Plans Loop/Hash/Merge Join SQL Options for Explicitly Specifying Query Plan 10


RDBMS Features - Storage Engine :11 © 2008 OpenLink Software, All rights reserved. RDBMS Features - Storage Engine Rows Stored At Leaves of Primary Key Index Tree Non PK Indexes Refer to Row By Value of PK Bitmap Index Full Text Index Striping Across Disks, No Separate Files Per Table/Key Incremental Online backup 11


RDBMS Features - Run Time Hosting :12 © 2008 OpenLink Software, All rights reserved. RDBMS Features - Run Time Hosting User Defined Type via Java or .NET Objects Hosted in Process User Defined Types Persisted in LOB Columns Java/.NET Methods Called Transparently From SQL ‘C” based Plugin Mechanism for adding SQL Functions 12


RDBMS Features - Security :13 © 2008 OpenLink Software, All rights reserved. RDBMS Features - Security SQL Role Based Security, Column/Table/View/Procedure Level Row Level Security With Policy Functions A Policy Function Can Add Extra Conditions to Queries/Updates Depending on User, Time, Other Considerations 13


Data Center Features - Clustering :14 © 2008 OpenLink Software, All rights reserved. Data Center Features - Clustering Combine Multiple Servers for Massive Scale and Parallelism All Servers Show the Same SQL/RDF Data and Application Logic, A SQL or Web Client Can Connect to Any for the Same Service Data Partitioning Specifiable Index by Index Optional Replicated Storage of Partitions for More Load Balancing, Fault Tolerance Shared Nothing Architecture, Works With Commodity Hardware and Networks 14


Data Center Features - Query Penalization :15 © 2008 OpenLink Software, All rights reserved. Data Center Features - Query Penalization Latency: One Message Round Trip is 20 Single Row Random Lookups Virtuoso Divides Queries into Collocated Fragments, Ships All Filtering, Aggregation, Joining to Where the Data Is. Sends Arrays of Hundreds of Operations at a Time, Whenever Possible 15


Data Center Features - Transactions :16 © 2008 OpenLink Software, All rights reserved. Data Center Features - Transactions Full ACID Properties Two Phase Commit with Single Phase Optimization Detection of Distributed Deadlocks Without Timing Out Each Cluster Node Keeps Own Transaction Log No External Monitor, Virtuoso Handles Distributed Recovery Cycle By Itself Transactions/Logging Can BE Disabled for Bulk Load etc. 16


Data Center Features - Parallel SQL :17 © 2008 OpenLink Software, All rights reserved. Data Center Features - Parallel SQL Transparent Map-Reduce Style Execution of Specified Partitioned SQL Functions/Procedures PL Extensions for Async Remote Execution of SQL Code, With and Without Transactional Semantics 17


Data Center Features - Futures :18 © 2008 OpenLink Software, All rights reserved. Data Center Features - Futures Dynamic Deployment, Adding and Removing Cluster Nodes Without Interruption of Service Keeping Data in Small, Self-Contained, Easily Relocatable Mini-Partitions 18


SQL Client Connectivity - Data Access Drivers :19 © 2008 OpenLink Software, All rights reserved. SQL Client Connectivity - Data Access Drivers Cross Platform ODBC 3.0 Drivers JDBC 2.0 Drivers OLE-DB Provider ADO.NET Provider XMLA Provider 19


Native Data Management - XML :20 © 2008 OpenLink Software, All rights reserved. Native Data Management - XML Native XML Data Type SQLX + Oracle Compatible XML Functions in SQL Document Centric Persistence of XML with Special Support in Text Index XSLT XQuery XML Views – XML Mapping Schema based Views of SQL Data Sources 20


Native RDF Data Management :21 © 2008 OpenLink Software, All rights reserved. Native RDF Data Management Native RDF Quad Storage (Physical Quads) SQL Enhanced With RDF IRI and Typed/Language Tagged Data Bitmap Indices and Key Compression for Compact Storage Selectable Index Scheme, Optionally Allows Queries Against Union of All Graphs Optional Full Text Index of Literals Reuses SQL Cost Model and Execution Engine With RDF Tailored Statistics 21


RDF Data Services – Client Connectivity :22 © 2008 OpenLink Software, All rights reserved. RDF Data Services – Client Connectivity SPARQL Protocol Jena Storage Provider Sesame Storage Provider Redland Storage Provider Linq2Rdf Storage Provider SPASQL SPARQL execution within SQL Processor Plethora of Built-In Functions, Stored Procedures, Web Services 22


RDF Data Services – SPARQL :23 © 2008 OpenLink Software, All rights reserved. RDF Data Services – SPARQL Full SPARQL, Language and Protocol Support Jena Compatible SPARUL for Create Graph, Insert, Update, and Delete Extensions for Aggregates & Grouping Nested Queries, SQL-Like Existence and Value Subqueries Expressions in Result Sets Path Expressions for Compact Notation, Also in Expressions Full Text & XPath Magic Predicate Extensions 23


RDF Data Services – Inference :24 © 2008 OpenLink Software, All rights reserved. RDF Data Services – Inference 24 Backward Chaining Inference Support, No Materialization of Entailed Triples needed for: Subclass and Subproperty Hierarchies OWL sameAs for Instances, Classes and Properties OWL equivalentClass and equivalentProperty Inference Enabled at Query or Individual Triple Pattern Level


Linked Data Services - RDF-ization Middleware :25 © 2008 OpenLink Software, All rights reserved. Linked Data Services - RDF-ization Middleware Declarative RDF Views (or Covers) over SQL Data In-Built RDF Middleware (Sponger) for RDF-ization of Harvested Web Content (bulk ingest or “on the fly”) Extended SPARQL Against Mapped and Stored RDF RDF-ization Cartridges for 30+ non RDF data sources Used by SPARQL Processor Used by in-built Content Crawler Cache Invalidation based on HTTP Caching Rules Configurable URI dereferencing via pragmas for node selection and path traversal 25


Linked Data Services - Deployment :26 © 2008 OpenLink Software, All rights reserved. Linked Data Services - Deployment URL Rewrite Rules combined with SPARQL for flexible association of URIs and RDF Data Sets Proxy (or wrapper) URIs construction for materializing Linked Data “on the fly” from existing Web information resources REST or SOAP based Web Services that expose functionality to Web Clients such as OpenLink Data Explorer, Marbles, Zitgist Data Explorer, DISCO, Tabulator etc. 26


RDF Data Services – RDF Views over SQL Data Sources :27 © 2008 OpenLink Software, All rights reserved. RDF Data Services – RDF Views over SQL Data Sources SPARQL Data Definition Statements for RDB Mapping Declare Correspondences Between Graph/Triple Patterns and SQL Objects Specify Mapping Between URI's and Keys , Supporting All Data Types, Multipart Keys Not Restricted to Table per Class and Column per Property Use Arbitrary Joins, SQL Functions and Search Conditions Automatically Generate Basic Class per Table, Property per Column Mapping of Given SQL Schema 27


RDF Data Services - RDF Views Contd. :28 © 2008 OpenLink Software, All rights reserved. RDF Data Services - RDF Views Contd. Evaluate Arbitrary SPARQL Against an RDF View In One Query, Some Graphs May Come from Views, Others From Stored RDF RDF Views Generate a Single SQL Statement, The IRI Generation and IRI Parsing is Only in Selection and Constant Expressions SQL Has Full Optimization Possibilities and the Generated SQL Does not Depend on Virtuoso Specifics Hence, RDF Views Are Efficient for Querying Remote, non-Virtuoso SQL Data 28


RDF Data Services - Clustering :29 © 2008 OpenLink Software, All rights reserved. RDF Data Services - Clustering Cluster-Optimized RDF Loader and SPARUL RDF-Aware Data Partitioning Automatic Statistics Sampling Across Cluster for Best Query Plan 29


RDF Benchmarks :30 © 2008 OpenLink Software, All rights reserved. RDF Benchmarks TPC H With SPARQL Extensions and RDF Views LUBM Berlin SPARQL Benchmark with Triples and with RDF Views 30 Bundled With:


Web Services Platform – HTTP Services :31 © 2008 OpenLink Software, All rights reserved. Web Services Platform – HTTP Services HTTP/1.1 and HTTPS Server for Static and Dynamic Content Dynamic Web Pages in PHP, Virtuoso SQL Procedures, ASP .net, Others SOAP and Rest Web Services in Virtuoso PL, Java, .NET DAV 31


Web Services Platform - WebDAV :32 © 2008 OpenLink Software, All rights reserved. Web Services Platform - WebDAV Documents Stored in Virtuoso Database ACL Based plus Unix Style Security, SQL User Accounts and Roles Own Documents and Collections Automatic RDF Metadata Extraction Optional Full Text Indexing and Versioning Dynamic Collections for Alternate Views of Directory Hierarchy 32


Web Services Platform – SOAP & REST :33 © 2008 OpenLink Software, All rights reserved. Web Services Platform – SOAP & REST SOAP 1.1/1.2 End Points Exposing SQL Procedures in All SOAP Styles Automatic WSDL Generation SQL Extensions for Declaring Full XML Schema Signatures for End Points Exposing Java and .net via SOAP Dynamic Web Pages and XML Functions for REST Services XMLA for SQL Access over SOAP 33


Web Services Platform - Dynamic Server Pages :34 © 2008 OpenLink Software, All rights reserved. Web Services Platform - Dynamic Server Pages Configure a Virtual Directory as Executable Publish Dynamic Web Pages in PHP, Virtuoso PL, Ruby, PERL, ASP .net Without Using External Web Server 34


Administration Services :35 © 2008 OpenLink Software, All rights reserved. Administration Services Web Interface for Setup of Web End Points, SQL, XML, RDF Functions SQL Functions for Full Programmatic Admin Access Simple Tuning, Only Specify File Layout and Amount of Threads and Memory to Use 35


Virtuoso RDF Applications :36 © 2008 OpenLink Software, All rights reserved. Virtuoso RDF Applications Dbpedia BIO2RDF Neurocommons Zitgist, Pingthesemanticweb, Musicbrainz 36


Product :37 © 2008 OpenLink Software, All rights reserved. Product Open Source and Closed Source Versions, Closed Source AddsVirtual Database and Clustering All Code, Applications, Samples, Docs in Single Download Minimal Installation Consists of Single Executable + Config File Web Admin Interface and Bundled ODS Collaborative Apps Suite Available for All Linux, Unix, Windows, 32 and 64 bit Available Preinstalled on Amazon EC2, With Optional Preloaded Dbpedia, BIO2RDF, Other RDF Data Sets 37