Use of AR/VR Expanding in School Music Programs

Creating music has always been both a personal and social activity. From the first banging on a hollow log to the centuries of development of wind, string and percussion instruments and now the technological marvels that allow a single performer to sound like an entire orchestra, humans have made music individually and collectively.

Purists often denigrate the use of technology and computers to enhance or alter the creation of music but like all things in life, some like it and some do not. The ongoing improvement in computers and the ability of Artificial Intelligence (AI) to further expand what is possible has led to the expansion of the use of Augmented Reality (AR) and Virtual Reality (VR) in musical preparation and performance. And with that, coming soon will be the possible use of AR/VR in school music programs to provide additional or supplemental opportunities for students. And like all things technology- driven, the costs decrease and the capabilities expand moving forward on a daily basis.

Augmented Reality is where technology enhances the user’s actual environment, considered partially immersive. Virtual Reality is fully immersive where the user’s environment is created totally by technology. The use of VR in music has already begun to provide total performance experiences, allowing the user to experience attending a live or recorded concert, or be a part of a group’s latest video.

To provide that full-immersion experience requires quite a bit of equipment including headphones or headsets, goggles and other controllers. Since the hardware is basically the same for all VR experiences, a school’s investment in that gear would benefit more than just one program or department as VR experiences can be valuable tools in many academic fields and are used widely in athletes. Being able to tour the Pantheon, face a big league pitcher, or watch the development of anti-matter in a particle accelerator at CERN are examples of how VR could be used in a school setting, shared by different departments, programs and educators.

The creation and performance of music lends itself more to AR technology where the equipment needs are simpler (as small as a smartphone) and can be done in a classroom or practice room, individually, in a group or as a shared experience. AR’s use in general music or music appreciation classes would give educators the chance to bring the concert hall or rehearsal space to their classroom.

The use of AR in live performances will not only allow the students to interact with the performers, but also connect with students in other classrooms across the country who are viewing the same musical activity. Some AR experiences go well beyond the basic senses of hearing and sight, adding touch and sensory to the experience. Music is meant to be shared and technology, in all its forms, can be used to broaden those chances.

AR can be used to provide instruction as well as access. Through an AR experience, students can experiment with different instruments that may not be available through the school. Beginning musicians can “try out” different instruments to see what they would be like to hold and play. Student practice sessions could be enhanced by using AR to perform duets or trios even though no one else plays their instrument at their school. And entire rehearsals could be done in a single practice room where AR provides the rest of the ensemble and the conductor. The technology would allow for multiple sessions and the ability for the student to not only run through the program several times but to stop, pause or repeat a section to improve their learning and performance.

Music Theory, the study of the concepts in music, can also be supported in an AR environment. The technology would allow the student to experience and view sounds in terms of the music staff, but also using colors, shapes and vibrations. With AR, students could see what each instrument looks like, what it takes to produce the sound/timber it makes and how they can blend or clash when played together. The examination of musical concepts can be enhanced with AR being able to present those ideas in a variety of different ways, adjusted to match how the student conceptualizes and understands.

AR could add a new dimension to the creation or composition of music, giving composers the opportunity to experiment with instruments they do not have access to, combine them in different ways, and have changes to their scores performed immediately afterwards. Being able to hear what was just written, performed by the instruments intended, would provide composers with the feedback needed to have their compositions match their vision. Melodic lines could be represented with colors or shapes and allow the user to compose based on painting with sounds. Not everyone works in the same manner.

While some of these AR/VR possibilities have been developed and are available for school use, the majority of those opportunities lie ahead, waiting to be developed. They, and dozens that we cannot conceive of at this point, will become part of music education in the future. Will AR/VR replace live performances, students dreading the practice room, and parents applauding their 10-year-old’s first notes on the sax? People still bang on hollow logs to hear how they will sound, so there is anticipation about the enhancements technology will bring to music knowing that actual musical performance will always be there.