logging in or signing up square pegs and round holes Donato Download Post to : URL : Related Presentations : Share Add to Flag Embed Email Send to Blogs and Networks Add to Channel Uploaded from authorPOINTLite Insert YouTube videos in PowerPont slides with aS Desktop Copy embed code: (To copy code, click on the text box) Embed: URL: Thumbnail: WordPress Embed Customize Embed The presentation is successfully added In Your Favorites. Views: 87 Category: Education License: All Rights Reserved Like it (0) Dislike it (0) Added: February 07, 2008 This Presentation is Public Favorites: 0 Presentation Description No description available. Comments Posting comment... Premium member Presentation Transcript Square pegs and round holes: http://popcorn.cx/ Square pegs and round holes A reflection on Class::DBI Who am I?: http://popcorn.cx/ Who am I? Developer at Monash University Applications in and related to the staff and student portal Built upon open source Perl, HTML::Mason, Apache Class::DBI: http://popcorn.cx/ Class::DBI Database abstraction layer for Perl Allows mapping of classes to database tables Each row of the table is an instance of the class The square peg: http://popcorn.cx/ The square peg Initial development: http://popcorn.cx/ Initial development Bulk written in Perl, remainder PL/SQL and Java Chose Class::DBI as it gave us a leg up due to time constraint But not as much as we expected Clash of standards: http://popcorn.cx/ Clash of standards Music::CD->has_a( artist => 'Music::Artist’ ); print $cd->artist->name; Clash of standards: http://popcorn.cx/ Clash of standards Music::CD->has_a( artist_id => 'Music::Artist’ ); print $cd->artist_id->name; Clash of standards: http://popcorn.cx/ Clash of standards Music::CD->has_a( artist_id => 'Music::Artist’ ); *artist = \&artist_id; print $cd->artist->name; Clash of standards: http://popcorn.cx/ Clash of standards Music::CD->has_a( artist_id => 'Music::Artist’ ); *get_artist = \&get_artist_id; *set_artist = \&set_artist_id; print $cd->get_artist->name; Playing nice with friends: http://popcorn.cx/ Playing nice with friends Our expectation was for error at runtime If remote service is not available: Module loads Only when used is error indicated By return value or an exception Playing nice with friends: http://popcorn.cx/ Playing nice with friends Class::DBI connects to database, and errors, at module load Which mean an error at load paradigm If database is unavailable Module will not load Thus stopping remaining code from loading We needed to be more defensive than usual and wrap the use in an eval I canna do it captain!: http://popcorn.cx/ I canna do it captain! Takes XML extract of all user accounts - upwards of 90,000 Compares against data in database Updates accordingly First attempt: http://popcorn.cx/ First attempt Run through extract and call retrieve() for each record 90,000 database queries Database bottleneck Runtime of over an hour Second attempt: http://popcorn.cx/ Second attempt Load all objects into memory via retrieve_all() Single database query no longer the bottleneck Now encounter memory allocation issues Still a runtime of over an hour Third attempt: http://popcorn.cx/ Third attempt Avoid Class::DBI and use DBI directly Single database query gave all records in an array of hash references Then we only call retrieve() if a change is required Runtime of less than ten minutes Fourth attempt: http://popcorn.cx/ Fourth attempt Did not try as third attempt gave satisfactory results Would have probably loaded entire extract and then used iterator from retrieve_all() Overview of initial development: http://popcorn.cx/ Overview of initial development A success Built working system in a few weeks Class::DBI gave significant head start. Although we needed to: learn how to use it bend to our standards avoid in some cases for performance Maintenance and enhancement: http://popcorn.cx/ Maintenance and enhancement Now been running for over two years Huge increase in usage Second largest installation in the world Also expansion of functionality Can at times be frustrating Where is the object?: http://popcorn.cx/ Where is the object? print $section->get_course(); Many hours wasted A number is printed Not the expected blessed referent Due to stringification feature of Class::DBI That we had not mentioned in our documentation Unforeseen implications: http://popcorn.cx/ Unforeseen implications Extended update(), create() and delete() to require a username for an audit trail $object->delete($username) Worked well Until we tried to use features such as cascade delete Unforeseen implications: http://popcorn.cx/ Unforeseen implications Class::DBI follows ‘has many’ relationships and calls delete() Does not know to pass our username Which makes it fail Our solution to a business need limited the features we could use Better knowledge of Class::DBI internals should provide a better solution Building a round peg: http://popcorn.cx/ Building a round peg Development: http://popcorn.cx/ Development All Perl solution Except for Oracle database Perl CGI for public website Perl under HTML::Mason for administration interface Perl for batch load process Class::DBI or not Class::DBI: http://popcorn.cx/ Class::DBI or not Class::DBI Our data model led to a matching set of classes and database schema But Class::DBI was not available on the target environment Recent issues (at the time) soured our opinion of Class::DBI Decided to build our classes from scratch It lives: http://popcorn.cx/ It lives Quickly became apparent that some form of framework was needed. So we built our own: Basic setup: http://popcorn.cx/ Basic setup use base 'Base'; __PACKAGE__->_set_table_name( ’table' ); __PACKAGE__->_setup_attribtes( qw{ code name ... }, ); ‘has_a’ relationship: http://popcorn.cx/ ‘has_a’ relationship __PACKAGE__->_setup_has_a_relationship( 'class' => 'Faculty', 'get_method' => 'get_faculty', 'set_method' => 'set_faculty', ); ‘has_many’ relationship: http://popcorn.cx/ ‘has_many’ relationship __PACKAGE__->_setup_has_many_relationship( 'class' => 'OtherClass', 'table' => 'other_table', 'get_method' => 'get_others', 'add_method' => 'add_other', 'remove_method' => 'remove_other', ); ‘has_many’ relationship: http://popcorn.cx/ ‘has_many’ relationship __PACKAGE__->_setup_has_many_relationship( 'class' => 'OtherClass', 'table' => 'other_table', 'get_method' => 'get_others', ); Compensating for ‘issues’: http://popcorn.cx/ Compensating for ‘issues’ Built to expect our standards Follows error at runtime paradigm Still need to bypass in some cases Only built what we needed/wanted So no stringification to primary key Natively requires username for audit trail Was it worth it?: http://popcorn.cx/ Was it worth it? Course Finder project was a success Framework allowed later changes to be implemented quickly We knew exactly how our framework worked Would we do it again?: http://popcorn.cx/ Would we do it again? No. Because we can reuse this framework Which we have done in one other project and fed enhancements back Another peg: http://popcorn.cx/ Another peg There are other database abstraction frameworks: DBIx::Class is biggest alternative Either didn’t exist or appear on our radar at the time They are not excluded for future projects Conclusion: http://popcorn.cx/ Conclusion I don’t really have one… Conclusion: http://popcorn.cx/ Conclusion Any framework is beneficial But you need one that does what you need And you will need to bypass it Questions?: http://popcorn.cx/ Questions? Slide 37: http://popcorn.cx/ Blackboard Learning System Vista Enterprise License Release 3 Patch something or other. Potato! This slide left intentionally blank: http://popcorn.cx/ This slide left intentionally blank You do not have the permission to view this presentation. In order to view it, please contact the author of the presentation.
square pegs and round holes Donato Download Post to : URL : Related Presentations : Share Add to Flag Embed Email Send to Blogs and Networks Add to Channel Uploaded from authorPOINTLite Insert YouTube videos in PowerPont slides with aS Desktop Copy embed code: (To copy code, click on the text box) Embed: URL: Thumbnail: WordPress Embed Customize Embed The presentation is successfully added In Your Favorites. Views: 87 Category: Education License: All Rights Reserved Like it (0) Dislike it (0) Added: February 07, 2008 This Presentation is Public Favorites: 0 Presentation Description No description available. Comments Posting comment... Premium member Presentation Transcript Square pegs and round holes: http://popcorn.cx/ Square pegs and round holes A reflection on Class::DBI Who am I?: http://popcorn.cx/ Who am I? Developer at Monash University Applications in and related to the staff and student portal Built upon open source Perl, HTML::Mason, Apache Class::DBI: http://popcorn.cx/ Class::DBI Database abstraction layer for Perl Allows mapping of classes to database tables Each row of the table is an instance of the class The square peg: http://popcorn.cx/ The square peg Initial development: http://popcorn.cx/ Initial development Bulk written in Perl, remainder PL/SQL and Java Chose Class::DBI as it gave us a leg up due to time constraint But not as much as we expected Clash of standards: http://popcorn.cx/ Clash of standards Music::CD->has_a( artist => 'Music::Artist’ ); print $cd->artist->name; Clash of standards: http://popcorn.cx/ Clash of standards Music::CD->has_a( artist_id => 'Music::Artist’ ); print $cd->artist_id->name; Clash of standards: http://popcorn.cx/ Clash of standards Music::CD->has_a( artist_id => 'Music::Artist’ ); *artist = \&artist_id; print $cd->artist->name; Clash of standards: http://popcorn.cx/ Clash of standards Music::CD->has_a( artist_id => 'Music::Artist’ ); *get_artist = \&get_artist_id; *set_artist = \&set_artist_id; print $cd->get_artist->name; Playing nice with friends: http://popcorn.cx/ Playing nice with friends Our expectation was for error at runtime If remote service is not available: Module loads Only when used is error indicated By return value or an exception Playing nice with friends: http://popcorn.cx/ Playing nice with friends Class::DBI connects to database, and errors, at module load Which mean an error at load paradigm If database is unavailable Module will not load Thus stopping remaining code from loading We needed to be more defensive than usual and wrap the use in an eval I canna do it captain!: http://popcorn.cx/ I canna do it captain! Takes XML extract of all user accounts - upwards of 90,000 Compares against data in database Updates accordingly First attempt: http://popcorn.cx/ First attempt Run through extract and call retrieve() for each record 90,000 database queries Database bottleneck Runtime of over an hour Second attempt: http://popcorn.cx/ Second attempt Load all objects into memory via retrieve_all() Single database query no longer the bottleneck Now encounter memory allocation issues Still a runtime of over an hour Third attempt: http://popcorn.cx/ Third attempt Avoid Class::DBI and use DBI directly Single database query gave all records in an array of hash references Then we only call retrieve() if a change is required Runtime of less than ten minutes Fourth attempt: http://popcorn.cx/ Fourth attempt Did not try as third attempt gave satisfactory results Would have probably loaded entire extract and then used iterator from retrieve_all() Overview of initial development: http://popcorn.cx/ Overview of initial development A success Built working system in a few weeks Class::DBI gave significant head start. Although we needed to: learn how to use it bend to our standards avoid in some cases for performance Maintenance and enhancement: http://popcorn.cx/ Maintenance and enhancement Now been running for over two years Huge increase in usage Second largest installation in the world Also expansion of functionality Can at times be frustrating Where is the object?: http://popcorn.cx/ Where is the object? print $section->get_course(); Many hours wasted A number is printed Not the expected blessed referent Due to stringification feature of Class::DBI That we had not mentioned in our documentation Unforeseen implications: http://popcorn.cx/ Unforeseen implications Extended update(), create() and delete() to require a username for an audit trail $object->delete($username) Worked well Until we tried to use features such as cascade delete Unforeseen implications: http://popcorn.cx/ Unforeseen implications Class::DBI follows ‘has many’ relationships and calls delete() Does not know to pass our username Which makes it fail Our solution to a business need limited the features we could use Better knowledge of Class::DBI internals should provide a better solution Building a round peg: http://popcorn.cx/ Building a round peg Development: http://popcorn.cx/ Development All Perl solution Except for Oracle database Perl CGI for public website Perl under HTML::Mason for administration interface Perl for batch load process Class::DBI or not Class::DBI: http://popcorn.cx/ Class::DBI or not Class::DBI Our data model led to a matching set of classes and database schema But Class::DBI was not available on the target environment Recent issues (at the time) soured our opinion of Class::DBI Decided to build our classes from scratch It lives: http://popcorn.cx/ It lives Quickly became apparent that some form of framework was needed. So we built our own: Basic setup: http://popcorn.cx/ Basic setup use base 'Base'; __PACKAGE__->_set_table_name( ’table' ); __PACKAGE__->_setup_attribtes( qw{ code name ... }, ); ‘has_a’ relationship: http://popcorn.cx/ ‘has_a’ relationship __PACKAGE__->_setup_has_a_relationship( 'class' => 'Faculty', 'get_method' => 'get_faculty', 'set_method' => 'set_faculty', ); ‘has_many’ relationship: http://popcorn.cx/ ‘has_many’ relationship __PACKAGE__->_setup_has_many_relationship( 'class' => 'OtherClass', 'table' => 'other_table', 'get_method' => 'get_others', 'add_method' => 'add_other', 'remove_method' => 'remove_other', ); ‘has_many’ relationship: http://popcorn.cx/ ‘has_many’ relationship __PACKAGE__->_setup_has_many_relationship( 'class' => 'OtherClass', 'table' => 'other_table', 'get_method' => 'get_others', ); Compensating for ‘issues’: http://popcorn.cx/ Compensating for ‘issues’ Built to expect our standards Follows error at runtime paradigm Still need to bypass in some cases Only built what we needed/wanted So no stringification to primary key Natively requires username for audit trail Was it worth it?: http://popcorn.cx/ Was it worth it? Course Finder project was a success Framework allowed later changes to be implemented quickly We knew exactly how our framework worked Would we do it again?: http://popcorn.cx/ Would we do it again? No. Because we can reuse this framework Which we have done in one other project and fed enhancements back Another peg: http://popcorn.cx/ Another peg There are other database abstraction frameworks: DBIx::Class is biggest alternative Either didn’t exist or appear on our radar at the time They are not excluded for future projects Conclusion: http://popcorn.cx/ Conclusion I don’t really have one… Conclusion: http://popcorn.cx/ Conclusion Any framework is beneficial But you need one that does what you need And you will need to bypass it Questions?: http://popcorn.cx/ Questions? Slide 37: http://popcorn.cx/ Blackboard Learning System Vista Enterprise License Release 3 Patch something or other. Potato! This slide left intentionally blank: http://popcorn.cx/ This slide left intentionally blank