The Challenge
The challenge assigned by the course Advance Programming of Interactive System was to use cutting-edge technologies to create innovative user interactions within an application in a 4 week timeframe. We needed to create something unexplored and innovative.
The Solution
We created a gesture-controlled music production platform that revolutionizes how people interact with music creation tools.
Key Features
- Web App: Chromium browsers with webcam and microphone support for universal accessibility.
- Smartwatch App: Heartbeat communication and tactile feedback for enhanced user experience.
- Gesture Recognition: MediaPipe integration for accurate hand gesture interpretation in real-time.
- Voice Commands: Touch-free music control with voice recognition for hands-busy scenarios.
Key Innovations
Tune Crafter brings together multiple interaction modalities to create an intuitive music creation experience that's accessible to everyone, regardless of musical background.
- Accessibility First: Designed for musicians and non-musicians alike with an intuitive interface.
- Real-time Processing: Ultra-low latency between gesture detection and sound output.
My Role
As the Lead UI Designer & Developer, I spearheaded the design and implementation process.
- UX/UI Leadership: Led sketching sessions, user flows, and wireframes for multiple interaction scenarios.
- Technical Architecture: Benchmarked and selected tech stack optimized for complex gesture tracking.
- Feature Development: Implemented gesture recognition system and state machine for multi-modal interactions.
Design Process
User Research & Ideation
Initial user interviews identified key pain points in music creation. We conducted 3 rapid ideation sessions, producing 20+ concept sketches to address these needs.
Interface Exploration
Tested two interface approaches with users: a familiar DJ console layout and an innovative gesture-first design. The hybrid approach yielded 85% higher user engagement.
Iterative Prototyping
Created 3 low-fidelity prototypes with increasing complexity. Each iteration was user-tested, resulting in a 40% improvement in gesture recognition accuracy.
Gesture Mapping
We designed an intuitive gesture system that transforms natural hand movements into musical controls.
UX Design Approach
- Used cognitive mapping principles to match gestures to expected sound outputs.
- Conducted usability testing with 12 users to refine gesture recognition thresholds.
- Implemented visual feedback cues to confirm gesture recognition.
Results & Impact
- Growth: 60 new monthly users through word-of-mouth from the app's "wow effect".
- Tech Implementation: Integrated MediaPipe hand tracking and web-smartwatch communication via sockets.
- Visibility: Live demos at Université Paris-Saclay and featured interview on Good Soup Media.