The Sight of Music

Which is more important you are evaluating the quality of a piece of music, the sound of the music, or the visual information you obtain from watching the performance? If you're like most people, you say the sound. In a new study by Chia-Jung Tsay, 83.3% of participants said the sound mattered more. But maybe we're mistaken.

In a series of studies, participants were presented with six-second clips of the three finalists from prestigious classical music competitions and asked to guess the winner. Some were given sound only, others video only, and the rest both sound and video. Both novices and expert participants (professional musicians) showed the same results. When given the video only clip, from 46% to 52% of them correctly identified the winner, which was significantly better than chance (33%, since there were three contestants). Those given sound only performed worse than chance, while those given both sound and video performed at about chance level. Thus, not only was sound of the music not helpful in predicting the winner, it's inclusion on the sound plus video clip distracted the participants from the visual information that was more useful in predicting the winer.

Two of Tsay's studies may help identify the mechanisms responsible for the effect. In one of them, participants were shown silent videos of black-and-white moving outlines of the performers. They also predicted the winners at better than chance. This suggests that participants are using movement and gestural information to evaluate performance, rather than static information such as race, gender and physical attractiveness, which were not visible on these clips.

All rights reserved by Vell Ojala

In another study, participants were given either sound-only or video-only recordings and asked to choose the performers who were most confident, creative, involved, motivated, passionate and unique. In the video condition, the performer chosen as the most passionate was the winner 60% of the time. This was true, but to a lesser extent, in the audio condition as well. Those chosen as most creative, involved, motivated and unique in the video condition were also more likely to have won.  Only confidence didn't help to identify the best performer.

These results remind me of a social-psychological study by Ambady and Rosenthal in which participants were shown brief video clips of college teachers lecturing without sound. Participants' evaluations of the instructors from these “thin slices” of behavior closely matched evaluations from the students in their classes at the end of the semester. Short observations of therapists in action also predict patient outcomes in psychotherapy.  Such studies show that nonverbal cues are important sources of accurate information in impression formation.

Tsay's results suggest that when we watch a musical performance, visual cues overwhelm auditory cues. We judge musical performance by watching cues such as posture, gesture and movement, especially when they are interpreted as indicators of passion. Both expert and novice judges were surprised by these results, which suggest that we are less affected by “content,” i.e., musical notes or a professor's words, and more by self-presentational style than we think.

These results have several limitations. Since the performers were contest finalists, there may not have been large differences in sound quality for the observers to detect. They were obtained with classical music performances, and it's not clear whether they would generalize to popular music or other genres. Of course, you might argue that the visual mode is even more important in popular music. The clips were unusually short, although from another perspective, the accuracy that was obtained with such short selections makes the results even more impressive. Although one of Tsay's studies suggests that physical attractiveness information is not necessary to produce accurate judgments, the studies do not rule it out as an influence, especially in popular music, where we have reason to think it's important.

The studies imply that the original judges in these competitions were heavily influenced by visual information, even though this was almost certainly not their intention. If we want to judge musical performance primarily by sound, the data suggest that we avoid music videos and live performances, since our evaluations are likely to be contaminated by what we probably consider irrelevant information. Of course, that would remove some of the enjoyment from the experience, which may be the whole point.

This article is cross-posted from my music blog, The Blues and the Abstract Truth.