Pulao, Qiuniu, Diting

Interactive Music Performance System

This is an interactive music performance system that consists of a controller panel, a visual display, a processor, and an audience submission web. The submission web can collect sound waves from the audience to enhance the involvement. In this way, the system improves the overall experience of a music performance.

Live
Performance

My first live performance with this system is in 3/10/2019. Below is the recording.

Background

Low
Involvement

Although many musicians acknowledge the important role the audience plays in a concert, most composers do not involve listeners in the composition. In most cases, the audience can only sit and listen.

Challenge

As a result, in spite of how great the show is, a large portion of the audience will feel bored or even fall asleep at the very beginning, especially those who sit far away from the stage.

Journey Map

Based on a series of interviews with more than 20 people, I made this journey map about the audience's feeling during a musical concert.

Interview

This is three typical answers in the whole interviewed sample.

“It was a orchestra. My seat was on the 3rd floor. I can barely see the players' face. Although the music was good, It was not much better than listening a record."

Simone Woods

"I was in a classic chorus show. They are great singers but it was a little boring. The lyrics was in Latin, which made it worse. I even fell asleep several times."

Nicole Yang

“I participated a music festival last month. A singer who I like was on the stage. It was exciting at first, but all she did was singing three songs. I think it will be more satifying if she can talk more with fans.”

Calvin Smith

Solution

One solution is to involve the audience in the show. Allowing listeners to manipulate the outcome can attract more attention and make uninterested songs to be interesting.

Overview

Concepts

Besides the will to enhance the involvement, I also want to involve cultural elements to the show. As an international student from China in the US, this is the way I highlight my identity. Considering there are many mythical creatures in Chinese tradition and features I employed, I designed the topic to be an ensemble of mythical creatures.

Screenshot 2023-12-27 at 10.19.24 PM.png

Design Sketch

Plans of how the audience can be involved:

#1 Visual Background

Let the audience control the background light and visual effects

Good:

Visually dramatic
Immediate feedback

Bad:

Chaotic visual patterns
Not relate to the music

#2 Instruments

Assign sensors to the audience to control different instruments

Good:

Interact in various ways
The audience can participate in the whole concert

Bad:

Melodic quality can not be guaranteed
Only a few listeners can be involved

#3 Timbre

Let the audience design the timbre of the instrument which generates music in real time.

Good:

The audience can heavily affect the sound while the melody can be controlled
As many listeners can be involved as needed

Bad:

The feedback is not immediate
Sometimes the difference between timbres is subtle

Choice

Considering the #3 "Timbre" plan avoids most disadvantages of the other plan while making the whole performance controllable, I took this as my final decision.

User Flow

Audience
Submission
Web

Acoustics Background

Envelope

One sound characteristics that influences the timbre of a sound is the envelope of a waveform, which can be regarded to be a smooth curve outlining its extremes in amplitude. In sound synthesis, the envelope of a signal to be generated is often described by a model called ADSR, which consists of an attack (A), decay (D), sustain (S), and release (R) phase. The relative durations and the amplitudes of the four phases have a significant impact on how the synthesized tone will sound. The following figure shows an idealized ADSR model, as well as envelopes of a piano and violin sound (playing a single note C4).

Timbre

The term “timbre” or tone quality or tone color is the quality that distinguishes different types of tone production, such as the difference between a flute and a trumpet playing the same note.

The timbre of a sound is greatly affected by its envelope. The physical characteristics of sound that determine the perception of timbre include both the frequency spectrum and envelope. The perception of timbre is closely related to the physical phenomena of the unfolding of the frequency spectrum over the duration of the envelope. This can be called the spectral envelope.

Submission
UI/UX

With the slide bars below, the audience can change the lengths of attack (A), decay (D), sustain (S), and release (R) phases. In this way, people can manipulate the sound wave. The submit button will push the data to the processor.