Onomatopiea
One particular form of written language, onomatopoeia, could be a form of written language that could be used for the expression of audio-vision. According to Hugh Bredin, this class of words is characterised by the “relation between signifier and signified in which the signifier is motivated, in part, by its sound” (1996, p. 562). Onomatopoeia, such as “boom” or “bang”, is written in the way the referent word sounds and, as such, provides the reader with an enactment of the sound in question (ibid. p. 568), while becoming a way to experience the world described in words through sensory modalities. While onomatopoeia commonly features in both literature and poetry, going as far back as Plato who discussed it in Kratylos (Gonda, 1975), a recent usage can be found in comics. Here,complex and graphically stylized onomatopoeia is a common method for expressing and describing audio both in terms of natural sounds and sound effects, even language and dialect. According to Scott McCloud, onomatopoeia in comics attempt to capture the “essence of sound” (1994, p. 134), and therefore implies a synaesthetic relationship between the components. Within speech research, it is established knowledge that slight, universal forms of synaesthesia contribute to speech structures, and that fully to understand the onomatopoeic structure means including cross-modal relationships (Ramachandran et. al., 2001, Assaneo et. al. 2011).
Even though comics have no audio per se, it can be argued that onomatopoeia provides them with a functional method that, through the direct involvement of the reader and the audio-visual process, works by “bringing the full power of our own experiences to bear on the world our eyes report” (McCloud, 1994, p. 136), or, as Winifred Nowottny claims “enact the senses” (1969, p. 116). One can therefore say that audio is present, the ears are appealed to, but only through the one sense of sight, engaged and animated by the written word.
Things that go /B/ump or /Th/ump
Listening to the audiobook version of Roald Dahl’s Danny the Champion of the World, I was struck by the use of the onomatopoeic thump to illustrate ripe apples falling on the roof of the gypsy caravan in which Danny lives. Hearing it, I thought: “shouldn’t that be Bump and not THump” feeling that the hard (Voiced) /b/ sound would better illustrate an apple falling on a hard surface (the roof), while the soft (Unvoiced) /th/ump would be more suited for an apple landing on grassy ground. Having read the book many times, I was certain that the reader in the audiobook had made a mistake. So, I looked it up, and, I was wrong. Roald Dahl used the soft /th/:
From Danny the Champion of the World by Roald Dahl (p. 7, 2010).
But, what was interesting was that I never really thought about this, until I heard the words being read. It wasn’t until now, when heard the /th/ being pronounced and voiced that I felt it should have been a hard /b/ instead. As I am writing this, I stopped to drop an apple onto the wooden table where I am sitting, and it really does go /b/ump and not /th/ump.
This is important because onomatopoeia is not only a relation of sound to sound, of similarity or instantiation. According to Bredin, onomatopoeia is also a form of “sound symbolism” that describes the relation of sounds to “semantic fields such as brightness and darkness”, making it also a relation of association (1996, p. 568). This kind of understanding of sound, and speech sound in particular, was a significant field of research outlined by Eichenbaum (1965, p. 115), given serious thought by Osip Brik (1978), and explored by Jakobson, throughout his career (see Jakobson & Waugh, 1987). These conceptions also point to the possible creation of meaning through word sounds, what Jan-Olof Svantesson calls “ideophones” (2017, p. 2), for example connecting the letter i- to something small, and insignificant – “tiny”, or the unit gl- to words that share a meaning component – “light” to “gleam” and “glitter” (ibid.), or consider the “impressions” of softness/hardness given by different classes of English consonants (Leech, 1969, p. 98). [1]
In this way, language can be used to describe cross-modal sensory relationships. Another process for cross-modal combination of sound and image, is montage. As Mehring proposes: “Just as the juxtaposition of the meaning of shot A with the meaning of shot B creates a new third meaning C, so does the simultaneous combining of pictures and sounds create new ideas that would not exist if heard and viewed separately” (1990, p. 184). This process for the combination of sound and image, and the use of sound to illustrate images, was a major avenue of both thinking and experimentation by Sergei Eisenstein through his method of audiovisual montage: the synaesthetic combination of image with sound and vice versa [2], and should surely be a way of illustrating the audio when writing a film’s audiovision.
[1]See Jan Gonda (1975) for a review of sound-symbolism and onomatopoeic words.
[2]See Eisenstein (1968).
Sources
Assaneo, M. F., Nichols, J. I., & Trevisan, M. A. (2001). The Anatomy of Onomatopoeia. PLoS One,6 (12). Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3237459/
Bredin, H. (1996). Onomatopoeia as a Figure and a Linguistic Principle. New Literary History,26(3), 555-569. Retrieved from https://muse.jhu.edu/article/24304.
Brik, O.M. (1978). Contributions to the Study of Verse Language.In L. Matejka & K. Pomorska (Ed.), Readings in Russian Poetics: Formalist and Structuralist Views.(pp. 117-125). Ann Arbor, MI: University of Michigan Press.
Dahl, R. (2010). Danny the Champion of the World. UK: Puffin Books.
Eichenbaum, B. (1965). The Theory of the Formal Method. In L. T. Lemon & M. J. Reis (Trans. & Ed.), Russian Formalist Criticism: Four essays. (pp. 99-139). Lincoln: University of Nebraska Press.
Eisenstein, S.M. (1968). Synchronization of the Senses. In J. Leyda, (Trans. Ed.). The Film Sense. (pp. 3-65). New York: Harcourt Brace Jovanovich.
Gonda, J. (1975). Some remarks on onomatopoeia, sound-symbolism and wordformation a propos of the theories of C. N. Maxwell. Selected Studies Vol. V: Indonesian Linguistics.(pp. 166-243). Leiden: Brill.
Jakobson, R., & Waugh, L. R. (1987). The Sound Shape of Language. Berlin: Mouton de Gruyter.
Leech, G. N. (1969). Patterns of Sound. In A Linguistic Guide to English Poetry. (pp. 89-102). London: Longman.
McCloud, S. (1994). Understanding Comics. New York: Harper Perennial.
Mehring, M. (1990). The Screenplay: A Blend of Film Form and Content. London: Focal Press.
Nowottny, W. (1965). The Language Poets Use.London: Athlone Press, University of London.
Ramachandran V., & Hubbard E. (2001). Synaesthesia: A window into Perception, Thought and Language. Journal of Consciousness Studies, 8, 3–34.
Svantesson, J. (2017). Sound symbolism: the role of word sound in meaning. WIREs Cognitive Science, 8, 1-12. Retrieved July 23, 2018 from https://onlinelibrary.wiley.com/doi/pdf/10.1002/wcs.1441.