Build a song by finding the stems

Unlock new music by playing a game in AR

# spatial-computing

Project overview

Technologies

  • Augmented reality
  • iOS

Tools

  • 3D directional audio
  • OpenUSD
  • Xcode - SwiftUI, RealityKit
  • Reality Composer

Key features

  • Hunt for song stems in your environment
  • Increase brand awareness with engaged users

Contributors

Why did we make this

  1. Create a fun AR experience that can help promote new music and brand partners
  2. Creating a great AR experience on iOS and how we could build from it with features like directional audio

Methodology

Hunting the sounds

First we obtained the USD files from TurboSquid and used Reality Composer to give them properties such as 3D directional audio so that the user can hear which way the sound is coming from and know where to start looking.

The instruments are randomly placed in the environment and it is up to the user to find and tap on them to collect the individual stems that make up the song.

Once the user found the sound they could then win a bonus prize by answering a quiz question.

Building the song

As the user finds more and more song stems they can listen to the what they have so far to piece together what the song is. We used LALAL.AI to split the song audio to get the stems to use for each instrument in the game.

Discovering new music

Once the user has found all the stems the song is complete and is revealed to them. They can play the full track from the UI and claim all their bonus rewards for answering the quiz questions.

The prototype

Hunting for more

Tourism and City Exploration

Historical tours: Users explore cities and historical sites, collecting virtual artefacts or interacting with historical figures.

Cultural landmarks: Discover cultural landmarks with AR-enhanced information, offering immersive experiences and detailed histories.

Local guides: Gamified exploration of local attractions, restaurants, and shops, encouraging users to discover hidden gems.

Education

Science trails: AR experiences in parks or nature reserves to learn about local flora, fauna, and geology.

Museum experiences: Interactive AR exhibits in museums, bringing static displays to life with historical context and multimedia.

Language learning: Gamified language practice by finding and interacting with virtual characters that teach new vocabulary.

Retail and Marketing

Virtual shopping: Locate and collect virtual items or coupons from local stores, enhancing the shopping experience.

Brand engagement: Interactive AR campaigns where users can interact with virtual mascots or advertisements in real-world locations.

Product hunts: Scavenger hunts for limited edition or promotional items, driving foot traffic to stores.