The look of music

When it comes to classical music competitions, one could be forgiven for thinking how well someone plays is the single most important factor.

New research conducted by a recent Harvard graduate, however, suggests otherwise.

In a study by Chia-Jung Tsay, who last year earned a Ph.D. in organizational behavior with a secondary Ph.D. field in music, nearly all participants — including highly trained musicians — were better able to identify the winners of competitions by watching silent video clips than by listening to audio recordings. The work was described in a paper published this month in the Proceedings of the National Academy of Sciences.

“It’s a very counterintuitive finding — there have been some interesting reactions from musicians,” Tsay said. “What this suggests is that there may be a way that visual information is prioritized over information from other modalities. In this case, it suggests that the visual trumps the audio, even in a setting where audio information should matter much more.”

Tsay herself has performed at Carnegie Hall and Lincoln Center, and said it was her experience in classical music competitions that piqued her interest in visual vs. audio information.

“For the last two decades, I’ve taken part in various competitions,” she said. “Through this experience, I found that depending on what type of evaluations were used — whether it was live rounds or audio recordings that had to be submitted — the results might vary widely. My intuition was that there was a much more sophisticated role for visual information.”

Though she ultimately performed seven experiments as part of the study, Tsay’s research began close to home — with videos of her own performances from youth competitions.

“I was curious, if I gave people just video recordings or just audio recordings, or video with audio, which would allow them to identify the actual winners?” she said. “Even from the first experiments, it seemed clear visual information can have a lot of impact.”

Tsay was surprised to find that the effect held up even in high-level international competitions, which often feature not only top performers, but also highly trained musicians as judges.

To conduct the study, Tsay recruited volunteers who received either video clips without sound, audio clips, or video clips that included sound. After viewing the short clips, participants were asked to identify the winners.

“What I found was that people had a lower chance of identifying the eventual winner if they only listened to the sound,” Tsay said. “People who just had the video — even without the sound — had surprisingly high rates of selecting the actual winner. Even with professional musicians, who are trained to use sound, and who have both expertise and experience, it appeared that the visual information was overriding the sound.”

Because musical differences between two top performers are often slight, viewers can more easily pick up on visual cues they associate with high-quality performance, Tsay believes. Factors such as a performer’s engagement, passion, and energy resonate.

“Those aspects are more closely tied to performance, or what we think of as performance, that allow the judges to distinguish between two performances,” Tsay said. “I wouldn’t expect musical novices to be able to use auditory information the same way a trained musician with 20 years of experience would, but when I ran the studies with professional musicians — people who perform as part of orchestras, or who teach at music conservatories — and I saw the same result, that was when I realized that regardless of the amount of experience, people still seem to rely on visual information.”

While orchestras have in recent years turned to blind auditions in an effort to blunt factors such as gender and race, it’s doubtful the impact of visual information could ever be entirely eliminated from the process, Tsay said.

“Even if we implement more objective evaluations, a lot of music is consumed live, so you can’t take out the visual,” she said. “Unless we decide we only want to experience music through auditory means, there will always be a visual element to these performances.”