Generating Functional Multimodal Sign Systems

Stefan Schreiboeck explores the role and the importance of multimodality to communication in a social and societal context. What is a mode? What is a medium? What is the difference between the two? Can a medium be turned into a mode? An understanding of how modes interact is crucial to an understanding of modern communication.The author offers an overview of the social semiotic approach to communication and complements this approach by adding a cognitive perspective on communication. Thus, the adapted Hallidayan model is extended to take into account the visual mode, music and many others more. Schreiboeck promotes a sign system on the basis of which modes other than the verbal cannot only be scientifically defined but also analysed. The author also offers practical tools for those who want to explore meaning making across modes, illustrated by a corpus consisting of the advertising campaign for the VW Polo in four languages.