2006cihass Stephen Beck MusicAppsForCI

CyberInfrastructure Applications in Music & Digital Audio: 

CyberInfrastructure Applications in Music & Digital Audio Stephen David Beck, Ph.D. Director, CCT Laboratory for Creative Arts & Technologies

Outline: 

Outline History of Computation & Music CI requirements for digital media Current Applications Emerging Applications

Computation & Music: 

Computation & Music Music & Digital Audio have been at the forefront of CS research from the beginnings of the Information Age

Hiller & Isaacson - UIUC: 

Hiller & Isaacson - UIUC Explored Markov chains to model music composition Computer-assisted composition Early efforts in Machine Learning Illiac Suite (1955) for string quartet Illiac computer

Research @ Bell Labs: 

Research @ Bell Labs Digital Audio experimentation in 1950s & 1960s lead to fundamental paradigm for DSP and audio synthesis Max Mathews @ Bell Labs Developed music/sound synthesis language (Music I - V) Paradigm of “orchestra/score”

Research @ Bell Labs: 

Research @ Bell Labs Bicycle Built For Two (1961) Music & Voice synthesized First “singing” computer Referenced in 2001 A Space Odyssey

Computer Music: 

Computer Music 1970s GROOVE - real-time digital control of analog synthesizers 1980s FM Synthesis - a computationally inexpensive synthesis system Yamaha DX7 - FM synth, most successful synthesizer of all-time MIDI protocol developed

Computer Music: 

Computer Music Early 1990s DSP-based synthesis and signal-processing on specialized audio cards (ProTools, ISPW, Kyma, Creative) Late 1990s CPU-based real-time synthesis/signal processing programming environments (Max/MSP)

Computer Music: 

Computer Music 2000-pres. Real-time Virtual Instruments Modeling of classic analog synthesizers Virtual physical models Real-time convolution (reverberation) Interactive Performance Environments Moore’s law has effectively “solved” the audio problem Or has it?

The Next Steps: 

The Next Steps Use distributed and grid technologies to leverage underutilized CPUs for audio production and analysis Use high-speed networks for distributed performances Use network communication tools (AccessGrid, Portals) to enhance and develop educational opportunities

CI Requirements: 

CI Requirements

Current Applications: 

Current Applications Access Grid for musical performances In Common: TIME UF Digital Worlds Institute, James Oliverio Created network-metronome to synchronize performances with network latency Distributed Signal Processing Logic Audio Compute nodes are established on idle machines over a 1GB switched network Uses additional CPUs to generate real-time audio and signal processing First (and only) commercial product to do this

Current Apps - DART: 

Current Apps - DART Distributed Audio Rendering using Triana Triana is a Java-based graphic programming environment Program objects are linked together Objects include a range of signal processing algorithms Objects can access web services, or distribute computing across a peer-to-peer network Originally designed for use in Gravitational Wave Astronomy

Triana: 

Triana

Schroeder Reverb: 

Schroeder Reverb

Voice Cancel: 

Voice Cancel

Wave Viewer: 

Wave Viewer

DART Applications: 

DART Applications Complex FFT-based Signal Processing Large FFT-size analysis High window count FFT analysis Multipass FFT analysis Bulk processing of large file sets Using high-speed networks to play sound files on remote computers

Music Info Retrieval: 

Music Info Retrieval A “Grand Challenge” for Music Automatic metadata creation and music analysis for matching and searching algorithms

Music Info Retrieval: 

Music Info Retrieval Goals: Searchable databases using musical descriptors and parameters Query by Humming Beat & Tempo detection Melody extraction Harmonic Analysis Pitch Histograms Pattern matching for analysis of style and content Style Identification & Classification Automatic metadata generation

MIR Technologies: 

MIR Technologies Based upon analysis of audio signals Fast Fourier Transforms Translates signal from time-domain to frequency-domain (spectral analysis) Critical parameters and measurements Zero-crossings Spectral Centroids Base Frequency Tracking Spectral Harmonicity

MIR Examples: 

MIR Examples Manual Analysis Pandora PlayProduction Automatic Computer Analysis M2K Marsyas DART & Alchemist

Music Genome Project: 

Music Genome Project A private venture to establish a taxonomy of “musical” information 400 “parameters” of musical characteristics are identified Human analysis of music creates a meta-data representation of these 400 parameters It takes 20-30 minutes per song to analyze Music from the past 60 years only Six years of work Why?

Pandora: 

Pandora Personalized Internet Radio

How does it work?: 

How does it work? You select a song you like It searches its database for other songs “like” it Those songs whose metadata is similar to the one you want It creates a radio station from this new list of songs Amazon for Music

Why Pandora?: 

Why Pandora? Money!!! Subscription model Free model, with advertising Competition? Satellite Radio Online Music Stores (iTunes) Distinction from competition? Intelligent(?) selection process “Smart Muzak”

Spin-offs: 

Spin-offs

EMI PlayProduction: 

EMI PlayProduction Search tool for EMI catalog Designed for music production in: Advertising & Television

Analysis-based MIR: 

Analysis-based MIR Computer-based analysis and MIR is the “grand challenge” of music Application environments: Marsyas Music To Knowledge (M2K) DART! Testbed Framework (IMIRSEL) International Music Information Retrieval Systems Evaluation Laboratory

Analysis-based MIR: 

Analysis-based MIR Successes Style matching Query by Humming Beat & tempo detection Note detection Score following Left to do? Classical music! Formal analysis Transcription Instrument Identification And so much more…

Compositional Uses?: 

Compositional Uses? Spectral Re-animation or Substitution Create a library of FFT frames Take a source file, generate FFT frames, and then replace each frame with a close match from the library Inverse FFT recreates the original sound but with different FFT frames

Alchemist: 

Alchemist A proposed multimedia analysis framework based on Triana Audio and Video based analysis tools Super-peer-to-peer network distribution Motivation is to use audio analysis to assist with audio/video analysis and extraction of salient meta-data Application areas Audio and Video searching (MVIR)?

Why CI for MIR?: 

Why CI for MIR? Massive data stores required Audio libraries Analysis files are larger than the audio files themselves Distributed Analysis Large libraries of source audio Copyright Issues Grids bring computing to the data, not data to the computing

Contact: 

Contact Stephen David Beck, Ph.D. Center for Computation & Technology Louisiana State University Baton Rouge, LA 70803 225-578-2594 sdbeck@lsu.edu