Overture
Exploring a novel form of music composition
Project Overview
Overture is a first-person music narrative in which the player makes decisions on a journey which writes a piece of music as you progress. The aim of Overture is to provide an inspirational tool for potential music composers; to allow the player to experience the impact of various compositional structures and the emotional connections created by their choices, as well as showing the effect of different musical layouts. It is intended for people aged 11 and up who are interested in taking up music composition but not exactly sure where to start from a structural standpoint, and also for musicians and composers who are experiencing ‘writer’s block’ and need an inspirational tool to kickstart some ideas.
This undertaking was a term project for Stony Brook University’s CSE 323 - Human Computer Interaction course in the Spring of 2017.
My Role
On this project, I took on the role of product designer and developer. My responsibilities included game design, level design and Unity developer.
Challenge
Create a proof of concept for a virtual world whose surroundings vary based on the user’s choices with corresponding and progressively enriched localised music and sound effects, has an intuitive structure and interface, and is both emotionally and artistically engaging to play.
Conceptualization
Inspiration for Overture came initially from one of our group members who is a musician and composer. Highlighting some pain points of his own when getting started in music composition and his more recent struggles with creative block.
When digging further and doing some competitive analysis in the world of currently available music composition software, we saw an underserved niche for tools which inspire rather than instruct, and can be educational in drawing connections between what music sounds like and what it connotes to a listener, again assisting in creative decision making.
Most of the current music production software or applications (for both pop music and classical music) are either implemented using a staff or by copying the physical instrument’s interface onto the computer. For example, many software products copy the interface of a dj mixer to use as a model. There are few alternative applications trying to think outside the box and presenting their ideas in a different way.
The interfaces that music composition software or apps usually use:
Here are two examples that are different from above, and could be considered competitors in the space:
Happy Notes Music Composer:
Pros:
- The interface is easy to manipulate
- Children can play without too much instruction
Cons:
- Less variety of the timbre/tone
- The design of the interface lacks polish
Ninja Jamm:
Pros:
- A variety of features
- The interface is simple but creative
- The users can show their music talents without having too many limitations
Cons:
- It takes some time to learn how to use the product and navigate the interface
- A bit difficult to understand for children
Overture focuses particularly on the connection between the audible and the visual, and the way in which certain sounds connote certain mental imagery and emotions. Those who are unsure of how music can create these emotional responses can use the game to learn a few basic compositional techniques to convey this.
With this vision in mind, we started to sketch out what this would look like. Arguably the most appropriate and intuitive approach was to take the form of a branching tree. Below is one of the initial concept sketches for the project, which showcase the core mechanic.
As the user navigates down a certain path, they are faced with branching paths in which to travel. These paths differ in their appearance and surroundings, which in turn reflect the kind of music that is being composed. From the first divergence onwards, the user is composing their own piece, first deciding between electronic and acoustic, then major or minor, fast or slow tempo, instrumentation and so on.
Concept sketch of branching composition development for acoustic paths
With our target audience and basic idea starting to be fleshed out, we created some user personas to highlight the goals and potential concerns of example user archetypes:
Prototyping & Development
While one of the team members started producing the in-house music resources and samples in Logic X, I started prototyping the levels with branching paths in Unity. To create each level/scene, for the electronic side of our branching tree I utilized Unity’s basic shapes, colors, and textures. On the acoustic side, I opted for a more natural environment and utilized some free models from the Unity Asset Store. I created each scene in Unity as a level to hold all of the objects placed, along with a corresponding scene to transition between levels in the form of hallways or path segments. Adjacent scenes were then arranged so that they could be walked between from beginning to end.
Prototyped stages in Unity showcasing branches paths for the proof of concept
By creating a Prefab with a first person controller I was able to set up our user fairly simply. After adding the terrain and ensuring the collider of the camera was above the terrain’s collider, I had a blank canvas upon which to craft all the environments.
For scripting, C# was used as the preferred language of Unity. To handle scene loading, OnTriggerEvent was overwritten such that when the “player” game object with RigidBody component and collider enters a block without a mesh (such that it is transparent) placed in front of a door or path, the next scene/level would be loaded. A bit further along the path would be another transparent block which when hit would destroy the previous scene. It was possible to travel back to previous levels using the same method, adding a check to see if the other side is loaded. This was also used in conjunction with a simple audio script to play the audio files associated with each scene, where the new audio clip would be passed in, checked against the current playing track, played and set as the current track. This was managed by a script that always ran in the background and was used to store all variables while moving between scenes.
To create the menu interfaces, the Unity Samples UI asset package was used to render the buttons and panels on the menu’s canvas, in conjunction with scripts to load the scene (our starting area) on click of the “Start”, “Resume”, or “New” buttons, quit the application on click of the “Exit” button, and surface a pop up modal for “About” and “Controls” options. The load and save feature was not implemented for this prototype.
Main menu, controls, pause and end screens from Overture
As a team we faced some challenges when merging local files together that were created on different versions of Unity and across both mac and pc versions, resulting in lighting not being baked into objects properly in some cases, and some objects being misaligned in some of the more intricate electronic scenes. This required some additional tweaks and debugging to get in a working state for our class demo.
Solution
At the start of the user’s journey, they are presented with a blank room with two doorways. One leads to a gravel path (acoustic), the other a pure white corridor (electronic).
The next additional musical element introduced is a choice between major or minor keys - reflected as seasons for acoustic levels - winter (minor) or summer (major), and in colors for electronic levels - darker greys and black for minor and brighter colors like yellow for major.
Third in choosing between a fast and slow tempo, we restrict or increase the user’s movement speed and begin to open up the world in either case as we progress through our composition.
Next in deciding on instrumentals, for electronic music colours again can determine instrumentation and more varied shapes start to appear. If choosing an instrumental that is a more pleasant, smoother choice, the world is populated with more colourful, rounder objects. For a drum and bass instrumentals option, shapes become spiky, and more aggressively coloured. In the case of acoustic music, the smoother choice is a forest, while a more percussive choice leads to a rocky mountain crop.
Finally for the ending choice, the piece can either crescendo or decrescendo. In the case of crescendoing music, the electronic world becomes further involved, with floating objects and shapes flying. The acoustic world sees the mountain getting bigger and a grand view being revealed, with the forest becoming more luscious and warmer in colour (in the case of summer), or more festive and christmassy in the case of the winter in choice 2. For the music decrescendoing, in the electronic music world everything gradually reduces down to the corridor again, and in the acoustic world, the pathway becomes smaller and more enclosed, though both retain the colours and objects that have been gained throughout the journey.
The composition ends by walking through a portal which brings the user back to where they started, presented with an end menu screen.
Future Considerations & Impact
As an additional feature for a follow-up phase, I feel the addition of VR headset support would greatly enhance the experience and immersiveness of exploring in Overture. Another additional feature which would be a strong value add would be for functionality to save created tracks for the user, especially with the addition of further branching paths, more music and variety.
I would also like to make some improvements to the usability and the in game experience such that it is more clear to the user where and what decision they are making with branching paths which are more apparent, perhaps in the form of the choices being labeled on the ground and highlighting the paths where the fork is, and/or providing some in-engine signage. Additionally the beginning room should be laid out in a way that lets the user see down both initial paths before having to choose one. This was an issue that came about when trying to merge all the unity files from different team members into one project, in addition to some of the misaligned objects in the electronic stages. After these initial changes are made we go through a few rounds of user testing, iterating on the prototype using feedback from these sessions.
We also saw some opportunity to use our product to aid in the treatment of emotional illnesses, given the time and resources to research this and perform usability testing. By bridging the two mediums of music and video games, we felt Overture could combine the therapeutic effects of music and use of video games as a method of treatment or rehabilitation in providing users with an immersive and emotional journey through music. The music composition mechanic would allow Overture to be utilized as a platform for users to express themselves by selecting musical components based on their mood or preferences.
Project Retrospective
As a first foray into Human Computer Interaction, Unity and even Music Composition, this project taught me many different skills, got my feet wet in human centered design principles, and inspired me to further pursue this field as a whole.