Open-source LMS for mission-critical, enterprise-level application

An Open Source LMS for a Mission Critical Enterprise-Level Application: Are we there yet? : 

An Open Source LMS for a Mission Critical Enterprise-Level Application: Are we there yet? Kevin Kelly, Wen Hao Chuang Academic Technology, San Francisco State University Copyright © [Wen Hao Chuang and Kevin Kelly] [2008]. This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the authors. To disseminate otherwise or to republish requires written permission from the authors.

Session topics (outline) : 

Session topics (outline) Background: Learning Management System (LMS) landscape at SF State Lessons learned about making an open source LMS enterprise-ready Development / Systems Perspective Teaching and Learning Perspective Organizational Administration Perspective

Introduction: Review of LMS at SF State : 

Introduction: Review of LMS at SF State F 05 S 05 F 04 S 04 F 03 S 03 F 02 S 02 Reasonably steady state with Bb 1997 to 2002 Period of problems with Bb Jan 2003 to present Faculty focus groups to determine need to investigate alt. LMS Dec 2003 Investigation of LMS alternatives Spring 2004 iLearn alpha test (5 instructors, 400 students) Fall 2004 iLearn beta test (25 instructors, 1500 students) Spring 2005 iLearn large scale deployment test (150+ instructors, 9,000+ students) Fall 2005 to Spring 2006 S 06

Usage statistics : 

Usage statistics

Usage statistics : 

Usage statistics

Looking at the LMS from 3 perspectives : 

Looking at the LMS from 3 perspectives

Looking at the LMS from 3 perspectives : 

Looking at the LMS from 3 perspectives

Technology Management (outline) : 

Technology Management (outline) System Architecture Development & QA Lessons learned and Recommendations

System architecture (before) : 

System architecture (before)

System architecture (now) : 

System architecture (now)

System setup (now) : 

System setup (now) 5 Sun Fire X4100 Servers 2 dual-core 64-bit AMD processors (for each server) 1 Load Balancer (running squid) 2 Application Servers 1 Database Server (running MySQL) 1 Admin and Backup Server LAMP (Linux, Apache, MySQL, PHP) Stack Red Hat Enterprise Linux 4 Apache 2.2.x MySQL 5.0.22 PHP 5 Others Zend Optimizer (PHP accelerator) -> eAccelerator Squid, clamav, MRTG, awstats, Nagios, Splunk(x), Ganglia, etc.

Data and storage solutions : 

Data and storage solutions Backup solutions For a large database like ours, the moodle default site-wide backup codes do not work. MySQL dump and nightly rsync to disk storage, then to tape. Database storage engines MyISAM (default) InnoDB Moodle’s “course-centered” design Solution: Diva? (diva.sfsu.edu)

Problems with current architecture : 

Problems with current architecture During mid-term and final exam periods, the two application servers get really loaded Also related to how teachers are using it Database MySQL vs. Postgres vs. Oracle MyISAM vs. InnoDB (storage engines) Other possible optimization solutions Squid vs. varnish Apache vs. lighttpd LAMP tuning and optimization

Problems with Data and Storage : 

Problems with Data and Storage You probably need a DBA! (for larger sites) Some tuning tips: http://day32.com/MySQL/ Backup issues MyISAM vs. InnoDB Database issues: Current database size (MySQL dump) over 11G! Avg 200 queries per second In 30 days avg 151,675,735 queries Peak connections between 400-500 concurrent connections Table index Optimization and table maintenance

Future hardware plans (1) : 

Future hardware plans (1) Pillar NAS and SAN storage unit Switch from xRaid to dedicated terabyte NAS system. Higher efficiency for storing data. Allows for snapshotting of data. Introduce BAD (Bad Ass Database) High-end 64 gigabyte database server Repurpose current database system as another application server (3 APP servers behind squid).

Future hardware plans (2) : 

Future hardware plans (2) Virtualization Moving to VMWare ESX (possibly this summer) Dynamic allocation of server resources Enhance the flexibility and scalability of enterprise level application Reduce possible downtime (VMotion)

Programming & Development : 

Programming & Development Small development team: 2 full-time developers (including project management) Version control system SVN: (internal) CVS: (external with Moodle core team) Bug tracking system: Trac (open source) For bug reports For project management purposes Build-in Wiki (for internal documentation)

Typical Build Process : 

Typical Build Process Developers’ codes got committed into their personal branch (SVN commit) “Buildmaster” merges all the codes before code freeze date Deploy a testing build on ilearntest (DEMO) If tested without any problem, code releases are “svn switched” to a “deployed” branch by system admin team (avoid conflict of interest).

Development tools : 

Development tools PHP NuSphere PhpED (great debugger/IDE) Eclipse (with PHPeclipse plugin) (free) PSPad (Windows) and vi/vim (Linux) (free) Other tools JMeter (simulating traffic) Perl (script for performance profiling) XML Editors (BB to moodle conversion tool)

QA Testing : 

QA Testing Small QA testing team 3 part-time student assistants Supervised by DEV staff members Tools that we are using Selenium IDE / RC TestNG / JUnit JMeter (stress testing) Tools that we are evaluating QA Wizard Pro (Seapine software inc.) Quick Test Pro (HP) Silktest (Borland)

Problems with QA : 

Problems with QA Training student assistants Time, resource, and testing scope After all, it’s open-source! Budget, select the right tool, and manual testing To automate or not to automate?

Lessons learned and recommendations : 

Lessons learned and recommendations Make a good projection about your LMS traffic and usage, plan your server architecture accordingly. Plan your workflow / processes in advance. Start with a fresh DB every year Schedule regular down time to repair/maintain tables QA testing! Knowledge: * [Promote] HSU “Introduction to Moodle Programming” course

Looking at the LMS from 3 perspectives : 

Looking at the LMS from 3 perspectives

Teaching and Learning:Engaging the Community : 

Teaching and Learning:Engaging the Community Outreach Training and support team visits faculty department meetings Conduct needs assessment and formative evaluation focus groups Advertise training and support options Input Advisory groups help inform decisions Exemplary instructors share best practices

Teaching and Learning:Engaging the Community - Best Practices : 

Teaching and Learning:Engaging the Community - Best Practices

Work Flow:Support for Faculty, Staff, & Students : 

Work Flow:Support for Faculty, Staff, & Students

Work Flow:Support for Faculty, Staff, & Students : 

Work Flow:Support for Faculty, Staff, & Students

Teaching and Learning: Training and support : 

Teaching and Learning: Training and support Constant evolution of training options Held frequent workshops throughout Fall and Spring semesters with extra offerings and open labs at the beginning Added week-long summer and winter workshops (with lunch) Modularized week-long workshops Big change for support Standardized format for "QuickGuides" Switch to database-driven QuickGuides Dynamic approach will update multiple QuickGuides at once

Looking at the LMS from 3 perspectives : 

Looking at the LMS from 3 perspectives

Organizational Administration:Policy issues : 

Organizational Administration:Policy issues Academic policies (Academic Senate)  Intellectual property  Faculty workload  Recognition for Retention/Tenure/Promotion purposes  Copyright/TEACH Act  Accessibility  Articulation  Privacy  Contact hours for students  Assessment and Teacher Evaluations  Quality of online courses  Minimum requirements for online/hybrid courses  Communication with students (virtual office hours) Student code of conduct Academic integrity

Organizational Administration:Policy issues : 

Organizational Administration:Policy issues Operational policies (Service Level Agreement) Technology Management Storage limits Archive practices Organizational Administration Incentives for training Also see article: Waterhouse, S. & Rogers, R.O. (2004). The importance of policies in e-learning instruction. Educause Quarterly 3, 28-39.

Closing : 

Closing Q & A Contact information Wen Hao Chuang wchuang@sfsu.edu 415.405.3543 Kevin Kelly kkelly@sfsu.edu 415.338.6455