A three-session user study was conducted with university students, most of whom had no prior experience with mixed reality or mallet percussion. Participants demonstrated adoption of foundational techniques, high levels of accuracy during physical performance assessments, and successful transfer of skills between virtual and physical performance contexts. Participants also highlighted the value of visual learning aids, guided instruction, and opportunities for independent exploration within the environment.

The findings provide empirical evidence supporting the feasibility of skill transfer within Musical Metaverse instrument-learning experiences and demonstrate the potential of pedagogically framed digital twin environments to support early-stage musical skill development. More broadly, this work contributes to ongoing discussions surrounding music learning, representation, and embodied interaction within emerging Musical Metaverse environments.

Keywords: Musical Metaverse; Mixed Reality; Mallet Percussion Pedagogy; Digital Twin; Skill Transfer

At the same time, the emergence of extended reality and the Musical Metaverse introduces new possibilities for music performance and pedagogy. As a virtual space where users interact through avatars, the Musical Metaverse can reshape how learners engage with instruments, instruction, and one another. Embodied presence in these environments may strengthen motivation and engagement, while also enabling more accessible and inclusive forms of musical participation. Beyond individual practice, such systems also point toward future possibilities in networked collaboration and shared musical spaces.

However, while musical XR systems continue to advance technically, many approaches prioritize system capability over pedagogical understanding. In educational XR environments, before addressing challenges such as networking and multi-user interaction, it is necessary to first understand how learners engage with virtual instruments within the Musical Metaverse and whether these experiences translate beyond. This paper argues that instruments within the Musical Metaverse, designed as embodied digital twins paired with pedagogical framing, expand access to musical training while supporting meaningful skill transfer to real-world performance. To investigate this claim, I present Mallets in Reality, a mixed reality environment for mallet percussion performance and guided learning experiences.

Comparatively less attention has been given to how these systems support musicians as supplementary resources. Understanding the importance of representational figures, how users engage with virtual instruments, and whether these interactions translate to physical performance, remains underexplored. This gap is especially relevant for spatial and embodied instruments such as mallet percussion.

Embodied interaction plays a key role in musical learning in extended reality environments, where performance can emerge through coordinated physical movement, spatial awareness, and real-time feedback. Unlike traditional digital interfaces, XR systems allow users to engage musical material through full-body interaction, integrating perception, movement, and feedback. Prior work in virtual conducting demonstrates how simulated environments can reinforce gesture clarity, focus, and interpretive control through repeated embodied practice [@orman_effect_2016; @orman_feasibility_2017]. Similarly, research in virtual percussion systems highlights the role of movement and timing in shaping interaction, where auditory, visual, and haptic feedback collectively support skill development and engagement [@willemsen_digidrum_2020]. Pinkl and Cohen explored virtual co-embodiment in VR drumming pedagogy, where learners shared avatar control with an autonomous exemplar [@pinkl_vr_2023]. Hackl and Anthes [@hackl_holokeys_2017], Fernandez et al. [@fernandez_piano_2016], and Guo et al. [@guo_hand-by-hand_2021] further explored overlaying spatial and visual feedback onto physical instruments to support teaching and learning, demonstrating the pedagogical value of blending physical and virtual layers. Together, these studies reinforce that musical learning in immersive environments is sensory and embodied.

Mallet percussion instruction is highly visual and embodied, relying on nuanced demonstrations of posture, grip, stroke type, and spatial navigation across the instrument. Traditional instructional media on flat screens often limit this visibility, particularly in ways that obscure full-body technique or fail to adapt to individual learners. In contrast, embodied avatars in virtual environments enable instruction to be recorded, replayed, and spatially aligned with the learner's own instrument configuration, supporting more accessible and repeatable learning experiences. To support this, the instructional system consists of a full-body avatar animation recorder, object transform recorder, and voice recorder unified into a single component, enabling performance gestures and verbal guidance to be captured as cohesive, reusable assets. An Animation Player component sequences these recordings using state-based logic and dynamically adapts them to each user's instrument height and scale, ensuring consistency across personalized configurations.

Equally important is the role of avatars in expanding representation within the field. Public education in the United States remains demographically imbalanced. National data from a 2020-2021 study indicate that roughly 80% of educators identify as White, while student populations have become significantly more diverse, highlighting a persistent representation gap [@schaeffer_key_2024]. Within this context, mallet percussion instruction often reflects a similarly narrow set of visible models in both educational materials and performance spaces [@bettelheim_racialethnic_2023]. This system intentionally positions avatars as a flexible representational layer, allowing for the inclusion of diverse identities and instructional voices. In doing so, this work responds to broader issues of access and belonging by enabling learners to see themselves reflected within instructional content.

A one-to-one dimensional mapping between the physical and virtual instrument (Fig. 1{reference-type="ref" reference="fig:one_to_one_dimensions"}) preserves spatial continuity and supports accurate motor transfer between environments. The system is modeled as a digital twin of a full-sized 5-octave marimba [@la_favre_marimba_2007]. While resonators are acoustically significant in physical instruments, they are omitted[^1] in the virtual environment.

To approximate acoustic responsiveness, three articulations are defined: normal, dampened, and node, corresponding to open resonance, immediate damping, and nodal striking techniques [@ren_tabletop_2012]. These interactions are implemented using Unity's collider system. To address limitations in Unity's predictive collision handling, particularly during fast rhythmic passages, the collider width of each bar is reduced. This minimizes unintended double-triggering during rapid lateral movement while preserving reliable strike detection.

Unlike other percussive XR systems, mallets in this environment are not pre-positioned. Users can freely grab mallets, after which the system constrains hand positioning to reflect recommended grip technique. This approach supports guided skill development while maintaining user agency. Future iterations will allow these constraints to be relaxed to support accessibility and alternative playing styles.

Haptic feedback is implemented using controller vibration mapped to stroke velocity at the moment of impact. Rather than relying on audio-driven haptics [@meta_meta_2023], this method directly reflects collision intensity, providing responsive feedback that reinforces dynamic control [@gao_enhancing_2025; @willemsen_haptic_2020]. To further support dynamic awareness, each mallet includes a visual meter indicating strike intensity (Fig. 2{reference-type="ref" reference="fig:dynamics_visual"}).

The calibrated dynamic range encourages appropriate playing force, promoting technique development while discouraging excessive impact that could transfer poorly to physical instruments.

Sonic realism is achieved using a SoundFont-based system integrated through a FluidSynth backend [@peter_hanappe_fluidsynth_2000]. The marimba patch from [@daindune_malletssectiongm_2023] is triggered using a MidiStreamPlayer component [@paxsteller_maestro_2024]. Running FluidSynth on a separate processing thread supports stable low-latency performance, a critical requirement for interactive musical systems [@pai_neuraldrum_2020; @turchet_networked_2022; @schlagowski_wish_2023]. The SoundFont format also enables compact file sizes and scalable instrument libraries without requiring additional mapping.

The system supports both seated and standing use, thumbstick locomotion, and global instrument scaling to accommodate different user needs and physical contexts. Instrument height is estimated from controller position and can be adjusted dynamically, with all interface elements updating accordingly. These features support autonomous learning by enabling users to explore the instrument at their own pace and within their own physical constraints, aligning with the system's goals of accessibility and learner-centered design within the Musical Metaverse.

A user study, approved under IRB protocol number: \\\\\\\, was conducted to investigate skill development and transfer within Mallets in Reality*. The study examined how musical and motor skills acquired through interaction with a mixed reality marimba translated to performance on a physical instrument. Particular attention was given to technique adoption, scale performance, learner engagement, and participants' experiences navigating between virtual and physical learning contexts. Through three instructional sessions, the study explored how embodied interaction within a Musical Metaverse environment can support early-stage mallet percussion learning and skill transfer between virtual and physical performance contexts [@zhong_literature_2023; @juntunen_embodied_2020].

The first session introduced participants to the study environment and foundational mallet percussion concepts within the Musical Metaverse. As shown in Fig. 4{reference-type="ref" reference="fig:piano_marimba"}, participants were guided by a virtual avatar through an overview of common mallet instruments, their sounds, and the keyboard layout of the marimba in relation to the piano.

Participants then completed an introduction to virtual controls, mallet interaction, and instrument height adjustment. Visual feedback mechanisms were used to reinforce proper hand orientation and playing position.

The avatar subsequently led participants through a call-and-response instructional sequence covering fundamental striking mechanics, alternating strokes, and performance of the C-major scale. Rote instruction was selected to support accessibility across varying levels of musical experience [@grey_rote_2020].

Following guided instruction, participants were given time to freely explore the virtual instrument and experiment with mallet interaction, sound production, and spatial navigation across the marimba. This exploration period was intentionally included to support learner agency and autonomous engagement within the Musical Metaverse environment.

After exploration, participants transitioned to the physical marimba and were asked to demonstrate the techniques introduced in mixed reality. Researcher observations focused on technique adoption, including hand orientation, wrist motion, and alternating strokes, as well as participants' ability to transfer scale patterns from the mixed reality environment to the physical instrument. The session additionally examined the effectiveness of visual feedback mechanisms for reinforcing proper technique and monitored participant comfort while using the headset over time.

The second session expanded upon the foundational skills introduced in Session 1 by focusing on rhythmic performance, chromatic navigation, and movement across the instrument. Participants began with an interactive song activity using vertically moving visual cues to represent rhythm and pitch. Each pitch was color-coded using the Boomwhackers Tuned Percussion Tubes chromatic color scheme [@rhythmbandinstrumentsllc_boomwhackers_2009], consistent with the color system applied to the virtual marimba bars.

As shown in Fig. 5{reference-type="ref" reference="fig:stretching"}, participants then completed a brief stretching routine presented through a pre-recorded instructional video in MR to encourage healthy playing habits and reduce injury risk [@workman_percussionists_2006].

Participants then transitioned to the physical marimba, where instruction focused on performance of the chromatic scale across the full 5-octave instrument, including note grouping, hand shifting between manuals, and foot positioning for efficient movement across the keyboard.

Following physical instruction, participants returned to the mixed reality environment for five minutes of self-guided practice. The session concluded with a physical assessment of chromatic-scale performance.

Researcher observations focused on participants' use of visual learning aids, chromatic-scale navigation, hand-shifting strategies, and the transfer of movement patterns between virtual and physical performance contexts.

The third session emphasized the integration of virtual and physical learning environments. Participants performed on the physical marimba while receiving instruction through the mixed reality system. The MR headset feed was displayed on a monitor positioned in front of the instrument, and audio was routed through room speakers, allowing the Musical Metaverse environment to function as a live instructional medium alongside the physical instrument. Participants received instruction on the B-flat and E-flat major scales, as shown in Fig. 6{reference-type="ref" reference="fig:mixed_setup"}.

Following this instruction, participants were introduced to the assistive scale tools available within the mixed reality environment, shown in Fig. 7{reference-type="ref" reference="fig:scale_ui"}. Two visualization modes were presented:

Normal: All bars within a selectable range are illuminated to indicate scale notes. Striking a highlighted note returns the bar to its unlit state.

Halo: All bars within a selectable range are outlined with an illuminated border to indicate scale notes. The halo remains visible after a note is struck.

Participants were then given five minutes of self-guided practice using the scale interface as a visual aid. The session concluded with a physical assessment of the B-flat and E-flat major scales.

Researcher observations focused on participants' use of the visual scale aids, integration of virtual and physical instruction, and performance of the major scale patterns across both learning environments.

Three technique-related measures were coded from Session 1 observations: hand orientation, wrist motion, and alternating strokes. The proportion of participants demonstrating each technique correctly is shown in Table 1{reference-type="ref" reference="tab:s1technique"}. Most participants successfully adopted the foundational performance techniques introduced during instruction, with full adoption observed for wrist motion and high levels of adoption observed for hand orientation and alternating strokes.

Participant reflections further supported these observations. When asked what they learned during Session 1, participants frequently referenced proper mallet grip, wrist rotation, hand position, and striking technique. One participant reported learning "the correct way to hold the mallets; how to control the virtual instrument" (P2, Q3.6), while another highlighted "playing position of the hands" and "rotating the wrists" as meaningful takeaways (P3, Q3.6). A third participant specifically noted that the visual reminder system reinforced correct technique, stating "Just reminding me to play palms down!" (P5, Q3.6). Several participants also referenced learning proper sticking patterns and mallet control. For example, one participant described learning "the correct sticking for scales" (P5, Q44), while another reported learning "whether to start with left or right hand depending on how many octaves to play" (P2, Q44).

Participant reflections frequently referenced the transition between virtual and physical performance contexts. One participant noted that after removing the headset and playing a physical marimba, "it felt like I understood the instrument and the only calibration I really had to do was the force I applied to the wooden keys as opposed to the VR keys" (P6, Q3.6). The same participant later reported that "practicing in VR definitely improved my real-world playing" (P6, Q43).

Participants also highlighted skills related to movement and navigation across the instrument. One participant described learning "the action of shifting hands when playing chromatic scale; the stepping needed to reach further notes" (P2, Q43), while another reported learning "switching the positions of the hands when going up and down the chromatic scale" (P3, Q43).

Participants frequently referenced visual learning aids when describing their experiences with the mixed reality environment. Commonly cited features included color-coded bars, note labels, falling-note visualizations, scale highlighting, and halo indicators. These visual supports were introduced throughout Sessions 2 and 3 to assist with note identification, scale navigation, and rhythm performance.

Several participants described the color-coding and highlighting systems as particularly helpful for understanding note relationships and scale patterns. One participant identified "the halos and color-coding of some of the mallet keys to show scales" because "it allowed me to follow the visuals and play right away" (P3, Q4.5). Another participant cited "AR features such as alphabetic labels placed on keys, highlighting of keys, \[and the\] virtual instructor" as major strengths of the application (P2, Q7.3).

Participants also discussed the falling-note visualizations introduced during Session 2. One participant reported that "falling notes felt like playing a rhythm game, and it was exciting" (P4, Q43). Another identified "Session 2 falling notes" as one of the most engaging aspects of the experience (P4, Q4.5). Several participants further referenced the scale visualization tools introduced during Session 3, noting that repeated practice with color-coded scale notes supported learning and recall of major scale patterns. For example, one participant stated that "seeing the colors greatly helped in playing on the real marimba" (P6, Q44).

Following guided instruction in Session 1, participants were given the opportunity to freely explore the mixed reality instrument. Exploration times varied considerably, with a mean of 576.4 seconds (SD = 304.8), ranging from 150 seconds to the maximum allotted duration of 900 seconds. Several participants continued interacting with the instrument until the session time limit was reached.

Participants also highlighted opportunities for autonomous exploration and experimentation as positive aspects of the experience. One participant identified "the haptic feedback, color coding of notes, ability to experiment and play freely" as particularly engaging (P6, Q4.5), while another stated that "I enjoyed the free time to just play on the instrument" (P3, Q7.5). Consistent with these comments, participants frequently cited the ability to independently explore the virtual instrument, visual aids, and practice activities as strengths of the system.

Several participants additionally reported increased interest in learning mallet percussion. One participant stated that the application "made me feel very confident about playing the marimba in the future" (P6, Q7.5), while another reported that the experience "made me very curious to take on the marimba as an instrument that I could actually learn" (P10, Q7.5).

Participants also described the system as a useful supplement when physical instruments were unavailable. One participant reported that "the VR application provides the accessibility and sound quality to physical mallet percussion instruments" while also providing educational resources directly related to performance practice (P6, Q6.5). Another participant emphasized that "the marimba is such a big instrument (and expensive too)" and viewed the application as a way of increasing access to an instrument that is often difficult to obtain (P10, Q6.5).

Participants generally described the virtual instrument as realistic while identifying areas for improvement related to physical interaction. Several participants specifically referenced mallet weight, impact sensation, and tactile feedback. One participant suggested "maybe more weight to the mallets?" (P6, Q6.6), while another noted that "the lack of weight balance in my hands … was a little distracting" (P8, Q2.6). Participants additionally referenced accidental strikes, collision sensitivity, and note-registration issues during performance. For example, one participant reported occasionally striking unintended adjacent bars (P3, Q2.6), while another noted that visual mallet contact did not always align with note activation (P5, Q4.6).

Despite these observations, participants frequently characterized the virtual instrument as realistic in relation to its educational purpose. One participant described the experience as "almost as effective as practicing in reality" for developing the movements associated with mallet percussion performance (P2, Q6.5), while another stated that the application "does a great job of providing access" to an otherwise difficult-to-access instrument (P10, Q6.5).

Visual supports emerged as a recurring theme in participants' learning experiences. Across sessions, participants frequently referenced technique helpers, color-coded notes, falling-note visualizations, scale highlighting, and halo indicators as useful learning aids. Participant responses suggest that visual cues were most effective when paired with guided instruction, rather than as standalone additions, with several participants also highlighting the virtual instructor as a useful component of the learning experience. Visual scaffolding appeared particularly valuable for helping participants interpret note relationships, navigate unfamiliar instrument layouts, and connect abstract musical concepts to physical actions.

Participants frequently referenced instrument cost, physical size, availability, and portability when discussing potential uses for the system. Most expressed that the system was a practical way to engage with mallet percussion when physical instruments were unavailable.

Participants also described the experience as enjoyable, engaging, and self-directed. The substantial variation in exploration times, combined with participant reflections regarding experimentation and free practice, suggests that learners engaged with the environment in ways that extended beyond required instructional activities and valued opportunities for independent exploration.

Additionally, participants completed all activities in a controlled, partially researcher-guided setting and did not engage in extended independent practice outside the study environment. Future work may examine longer-term adoption patterns, retention of musical skills, and the effectiveness of self-guided learning experiences over extended periods of time. Future studies could also investigate the experience with percussionists exclusively, incorporating more challenging performance tasks and notated music scores.

Networked multi-user environments could allow a remote instructor and learner to interact within the same virtual performance space, expanding possibilities for remote music education. Additionally, the digital twin pipeline introduced in this study may be extended to other mallet percussion instruments. WebXR implementations could increase accessibility across a broader range of hardware platforms. Finally, adapting many of the visual support systems introduced here to experiences for AR/MR Smart Glasses, alongside use with physical mallet instruments, may provide additional insight into how pedagogical cues function across hybrid performance environments.

This study examined whether Mallets in Reality, a mixed reality environment designed for mallet percussion, could support early skill development and transfer between virtual and physical performance contexts. Across three instructional sessions, participants demonstrated adoption of foundational techniques, high levels of accuracy during physical performance assessments, and sustained engagement with the mixed reality environment. Participant responses also highlighted the value of visual learning aids, guided instruction, and opportunities for independent exploration.

[^1]: Resonators are excluded in the virtual environment to improve visibility in pass-through mode and support safe physical navigation, prioritizing interaction clarity while preserving spatial relationships.