Rally Lookback API Hack-a-thon Raleigh October 2012

Views:
 
     
 

Presentation Description

No description available.

Comments

Presentation Transcript

The Rally Lookback API and Temporal Data Model:

Larry Maccherone (Twitter: @LMaccherone) 1 The Rally Lookback API and Temporal Data Model

PowerPoint Presentation:

LARRY MACCHERONE Twitter: @ LMaccherone

The ODIM Framework:

The ODIM Framework

Begin with the end in mind:

Begin with the end in mind Larry Maccherone (Twitter: @LMaccherone) 4

Kanban Time Machine video:

Kanban Time Machine video Larry Maccherone (Twitter: @LMaccherone) 5

Cumulative Flow Diagram demo:

Cumulative Flow Diagram demo Larry Maccherone (Twitter: @LMaccherone) 6

Time in Process (TIP) chart demo:

Time in Process (TIP) chart demo Larry Maccherone (Twitter: @LMaccherone) 7

Concepts:

Concepts Temporal data model Query language Basics Hierarchy Larry Maccherone (Twitter: @LMaccherone) 8

A Temporal Data Model consists of1:

A Temporal Data Model consists of 1 Data Structures Constraints Operations Larry Maccherone (Twitter: @LMaccherone) 9

The Data Structures:

The Data Structures Start with this "snapshot" (with valid-time information) { _id: B2E..., / / Unique identifier for this document (A GUID) _ ValidFrom : "2011-01-01T12:34:56.789Z”, // Start of time interval for the valid time for this document _ ValidTo : "9999-01-01T00:00:00.000Z",     // End of time interval = Infinity = ongoing ObjectID : 777, / / Unique identifier for the “Object” that will change over time Name: "Footer disappears when using new menu”, // One attribute making up the logical “state” of the object ScheduleState : " Submitted”, // Another attribute / / Other fields not shown } Change the ScheduleState field at 2011-01-02T02:46:13.578Z { _id: B2E... _ ValidFrom : "2011-01-01T12:34:56.789Z”, _ ValidTo : ”2011-01-02T02:46:13.578Z",     // Updated to the moment the change came in ObjectID : 777, Name: "Footer disappears when using new menu”, ScheduleState : "Submitted”, // Other fields not shown } { _id: A37...     // A new document so it gets a new GUID _ ValidFrom : ”2011-01-02T02:46:13.578Z",     // Updated to the moment the change came in _ ValidTo : "9999-01-01T00:00:00.000Z",     // New latest so it is infinity ObjectID : 777, Name: "Footer disappears when using new menu”, ScheduleState : ”Open”, _ PreviousValues : { ScheduleState : "Submitted" }, // Other fields not shown }

The Constraints Overlaps: Never! Gaps & Ends: Allowed.:

T 1 T 2 T 3 T 4 T 5 T 6 T 7 T 8 T 9 T 10 ∞ The Constraints Overlaps: Never! Gaps & Ends: Allowed. Larry Maccherone (Twitter: @LMaccherone) 11 ObjectID _ ValidFrom _ ValidTo 777 1 2 777 2 4 777 4 10 777 10 ∞ 888 1 3 888 3 4 888 4 6 888 8 9 888 9 ∞ 999 3 6

_ _At: T5:

T 1 T 2 T 3 T 4 T 5 T 6 T 7 T 8 T 9 T 10 ∞ _ _At: T 5 Larry Maccherone (Twitter: @LMaccherone) 12 ObjectID _ ValidFrom _ ValidTo 777 1 2 777 2 4 777 4 10 777 10 ∞ 888 1 3 888 3 4 888 4 6 888 8 9 888 9 ∞ 999 3 6

_ _At: T5 is just syntactic sugar for _ValidTo: {$gt: T5}, _ValidFrom: {$lte: T5}:

T 1 T 2 T 3 T 4 T 5 T 6 T 7 T 8 T 9 T 10 ∞ _ _At: T 5 is just syntactic sugar for _ ValidTo : {$ gt : T 5 } , _ ValidFrom : {$ lte : T 5 } Larry Maccherone (Twitter: @LMaccherone) 13 ObjectID _ ValidFrom _ ValidTo 777 1 2 777 2 4 777 4 10 777 10 ∞ 888 1 3 888 3 4 888 4 6 888 8 9 888 9 ∞ 999 3 6

_ _At: T8 (beginning is included/closed):

T 1 T 2 T 3 T 4 T 5 T 6 T 7 T 8 T 9 T 10 ∞ _ _At: T 8 (beginning is included/closed) Larry Maccherone (Twitter: @LMaccherone) 14 ObjectID _ ValidFrom _ ValidTo 777 1 2 777 2 4 777 4 10 777 10 ∞ 888 1 3 888 3 4 888 4 6 888 8 9 888 9 ∞ 999 3 6

_ _At: T6 (end is excluded/open):

T 1 T 2 T 3 T 4 T 5 T 6 T 7 T 8 T 9 T 10 ∞ _ _At: T 6 (end is excluded/open) Larry Maccherone (Twitter: @LMaccherone) 15 ObjectID _ ValidFrom _ ValidTo 777 1 2 777 2 4 777 4 10 777 10 ∞ 888 1 3 888 3 4 888 4 6 888 8 9 888 9 ∞ 999 3 6

Reasons to exclude the end:

Reasons to exclude the end It’s easier to say from 2011-02 up to but EXCLUDING 2011-03 than from “first day in February” to and INCLUDING “last day in February” . The later requires you to know how many days are in February, which is not easy. It allows the timestamps in the database to be millisecond granularity but the queries to at a courser granularity. Larry Maccherone (Twitter: @LMaccherone) 16

Scope:

Scope Queries typically include: A point in the timeline (already discussed) "Scope" limiting clauses Larry Maccherone (Twitter: @LMaccherone) 17

ObjectID: {$in: [777, 999]}, _ _At: "current" :

1 2 3 4 5 6 7 8 9 10 ∞ ObjectID : {$in: [777, 999] } , _ _At: "current" Larry Maccherone (Twitter: @LMaccherone) 18 ObjectID _ ValidFrom _ ValidTo 777 1 2 777 2 4 777 4 10 777 10 ∞ 888 1 3 888 3 4 888 4 6 888 8 9 888 9 ∞ 999 3 6

Other fields to determine scope:

Other fields to determine scope _Type or _ TypeHierarchy (defect, story, task, test case, etc.) _ ProjectHierarchy (think teams and sub-teams) _ ItemHierarchy (think work breakdown structure) Tag Priority (P1, P2, etc.) Severity (S1, S2, etc.) Owner (user) Leaf ScheduleState , KanbanState , State, etc. Any custom field Larry Maccherone (Twitter: @LMaccherone) 19

The Operations:

The Operations {a: 10} - docs where a is 10 or an array containing the value 10 {a: 10, b: "hello"} - docs where a is 10 and b is "hello" {a: {$ gt : 10}} - docs where a > 10, also $ lt , $ gte , and $ lte {a: {$ne: 10}} - docs where a != 10 {a: {$in: [10, "hello"]}} - docs where a is either 10 or "hello" {a: {$exists: true}} - docs containing an "a" field {a: {$exists: false}} - docs not containing an "a" field {a: {$type: 2}} - docs where a is a string (see bsonspec.org for more types) {a: /foo.*bar/} - docs where a matches the regular expression "foo.*bar" {" a.b ": 10} - docs where a is an embedded document where b is 10 {$or: [{a: 1}, {b: 2}]} - docs where a is 1 or b is 2 {$and: [{a: 1}, {b: 2}]} - docs where a is 1 and b is 2 Also _ _At: - syntactic sugar converted to appropriate clauses on _ ValidFrom and _ ValidTo "current" - syntactic sugar used anywhere a timestamp is expected. Means "now"

Demo queries (http://try.mongodb.org):

Demo queries (http:// try.mongodb.org ) db.rally2.save({name:' Westley ', nature:'good ', weight:150}) db.rally2.save({ name:'Roberts ', nature:'bad ', weight:150}) db.rally2.save({ name:'Buttercup ', nature:'good ', weight:120}) db.rally2.save({name:' Fezzik ', nature:'good ', weight:340}) db.rally2.save({ name:'Humperdinck ', nature:'bad ', weight:185}) db.rally2.find(null, {_id:0}) // shows all documents db.rally2.find({ nature:'bad '}, ['name']) // Roberts and Humperdinck db.rally2.find({weight:{$gte:185}}, {_id:0}) // Fezzik and Humperdinck db.rally2.find({ nature:'good ', weight:{$gte:130, $lt:300}}, {_id:0}) // Westley db.rally2.find({$or: [{ nature:'good '}, { name:'Roberts '}]}, {_id:0}) // Westley , Roberts, Buttercup, and Fezzik db.rally2.find({weight:{$ gte : 130}, weight:{$ lt : 300}}, {_id:0}) // BAD - DON'T USE A KEY MORE THAN ONCE IN THE SAME MAP Larry Maccherone (Twitter: @LMaccherone) 21

Demo hierarchy:

Demo hierarchy db.rally.save ({ ObjectID : 1, _ ItemHierarchy :[1]}) db.rally.save ({ ObjectID : 2, _ ItemHierarchy :[1, 2]}) db.rally.save ({ ObjectID : 3, _ ItemHierarchy :[1, 2, 3]}) db.rally.save ({ ObjectID : 4, _ ItemHierarchy :[1, 4]}) db.rally.save ({ ObjectID : 5, _ ItemHierarchy :[5]}) db.rally.find (null, {_id:0}) // shows all documents db.rally.find ({_ItemHierarchy:1}, {_id:0}) // shows documents 1-4 db.rally.find ({_ItemHierarchy:2}, {_id:0}) // shows documents 2, and 3 Larry Maccherone (Twitter: @LMaccherone) 22

Hierarchy in Rally:

If you have this: Story 111 Story 222 Portfolio Item 333 Portfolio Item 444 Story 555 Story 666 Task 12 Defect 777 (via the Requirement field) Task 13 The document for Story 666 would look like this: { ObjectID : 666, _Type: " HierarchicalRequirement ", Parent: 555, _ ItemHierarchy : [333, 444, 555, 666], ... } To get all Stories that descend from 333 (includes 333, 444, 555, and 666 but not Defect 777): { _ ItemHierarchy : 333, _Type: " HierarchicalRequirement " } Larry Maccherone (Twitter: @LMaccherone) 23 Hierarchy in Rally

Leaf nodes only:

Leaf nodes only for just Stories : Children : null for Stories or Portfolio Items : $ or: [ { _Type : " HierarchicalRequirement ", Children : null } , { _Type :" PortfolioItem ", Children : null, UserStories : null } ] Larry Maccherone (Twitter: @LMaccherone) 24

Transitions, not just states:

Transitions, not just states By storing the _ PreviousValues , each snapshot is not only a representation of the state of the object for a period of time but a representation of the transition that it undertook to get to that state. Example usages: Find all transitions where items moved backwards in the workflow Find all transitions into the Released workflow state in a given quarter/month/week Larry Maccherone (Twitter: @LMaccherone) 25

A Monthly Throughput Calculation:

A Monthly Throughput Calculation { _ ProjectHierarchy : "The A-Team", State: {$ gte : "Completed"}, _ PreviousValues.State : {$ lt : "Completed"}, _ ValidFrom : { $ gte : <first day of some month>, $ lt : <first day of next month> } } Larry Maccherone (Twitter: @LMaccherone) 26

But wait, there’s more…:

But wait, there’s more… Lumenize Larry Maccherone (Twitter: @LMaccherone) 27

Timelines in Lumenize:

Timelines in Lumenize { start:'2009- 01- 02T00', pastEnd:'2009- 01-07T00 ', // could also specify count granularity: 'hour', // defaulted from above workdays: ['Monday', 'Tuesday', ...], // default M-F holidays: [ { month: 1, day: 1}, // Every year, a Thursday in 2009 "2009-01-02" // Also got Friday off in 2009 ], startWorkTime : {hour: 9, minute: 0}, pastEndWorkTime : {hour: 17, minute: 0 }, timezone : 'America/ New_York ' } Larry Maccherone (Twitter: @LMaccherone) 28

Precision matters:

Precision matters Larry Maccherone (Twitter: @LMaccherone) 29

Prototype replacement burn chart:

Prototype replacement burn chart Larry Maccherone (Twitter: @LMaccherone) 30

So, what were we trying to do?:

So, what were we trying to do? I remember now, charts and such Larry Maccherone (Twitter: @LMaccherone) 31

A visualization includes:

A visualization includes ( optional ) Configuration Data from user/environment to configure the visualization Snapshot Data A scope specification (Project X, User A, etc.) A timeline specification Aggregation C alculations Highcharts Specification Larry Maccherone (Twitter: @LMaccherone) 32

PowerPoint Presentation:

Scope (Direct LBAPI) {_ ProjectHierarchy : 'The A-Team'} Timeline ( Lumenize ) { pastEnd : "current", granularity: "day", count: 120, timeZone : "America/ New_York " } Aggregation ( Lumenize ) { AggregationType : " GroupBy ", GroupByField : " KanbanState ", Function: "Count" } Visualization (next page) Each visualization needs spec for: Scope Timeline Aggregation Visualization Larry Maccherone (Twitter: @LMaccherone) 33

PowerPoint Presentation:

Larry Maccherone (Twitter: @LMaccherone) 34 Visualization ( HighCharts ) { chart: { defaultSeriesType : 'area', zoomType : ' xy ' }, title : {text : 'Cumulative Flow Diagram '} , subtitle: { text: 'State field: ' + config.groupByField }, xAxis : { categories: cfdCalculation.categories , tickInterval : Math.floor ( cfdCalculation.categories .length / 8), title: {enabled : false} }, yAxis : { title: {text : config.aggregationLabel } } , tooltip: { this.x + '< br />' + this.series.name + ': ' + this.y ;}, allowPointSelect : true, series : cfdCalculation.series }

Summary of Lumenize capability:

Summary of Lumenize capability Declarative specification of Precise T imelines Granularity of Year, Quarter, Month, Week, Day, Hour, etc. Custom granularities of Release, Iteration, etc. Knock out Weekends Holidays Non-work hours (for hour or lower granularity) Timezone support. Shift the boundaries of any sub-range in your timeline to any timezone Declarative specification of Advanced Aggregations Time in State GroupBy Standard aggregations (Count, Sum, Mean, Median, Quartiles, any percentile) Derived fields Inject your own aggregations with a function https ://github.com/lmaccherone/ Lumenize Larry Maccherone (Twitter: @LMaccherone) 35

Lumenize end-user usage Rally Customer Hack-a-thon May 2012:

Larry Maccherone (Twitter: @LMaccherone) 36 Lumenize end-user usage Rally Customer Hack-a-thon May 2012

PowerPoint Presentation:

Larry Maccherone (Twitter: @LMaccherone) 37 "It's been really nice using some of the charts that some of the other people have written so far and we've just been molding them to how we want to display our data."

Common patterns:

Common patterns Aggregation (sum, count, average, std-dev , etc.) vary over time visualized with column, line, or box-whisker series (mix and match simultaneously) GroupBy over t ime (sum, count) vary over time visualized with a stacked area or stacked column chart GroupBy summary ( sum, count, average) bar or column chart showing the summary of items in each group Transition counts (or sums) vary over time visualized with column chart Time in state visualized with a scatter chart and histogram Larry Maccherone (Twitter: @LMaccherone) 38

App SDK vs. Direct Lookback API:

App SDK vs. Direct Lookback API App SDK and wrapped Lumenize Examples: Kanban Time Machine Defects by Priority (Mark Smith) Better UI integration Many Rally developers for pairing Direct Lookback API and full Lumenize Examples: Cumulative Flow Diagram Time in state Advanced/precise timelines M ore advanced aggregations Larry Maccherone (Twitter: @LMaccherone) 39

References:

References A. Steiner. A Generalisation Approach to Temporal Data Models and their Implementations. PhD thesis , Swiss Federal Institute of Technology, 1998 R. Snodgrass. The Temporal Query Language TQuel . ACM Transactions on Database Systems, 12, 247-298, 1987 R. Snodgrass et al. A TSQL2 Tutorial. SIGMOD Record , 23(3), September, 1994 Larry Maccherone (Twitter: @LMaccherone) 40

authorStream Live Help