聲音藝術與展演的科技疆界 Technology of Sound Art

聲音藝術與展演的科技疆界-「新音樂表達介面」研討會作品選談
Technology of sound art and performance——discussion about the works at NIME


「電子樂、細微聲響、極低頻、電子原聲、電腦音樂、idm、類比音樂、後數位音樂、Glitch、聲音生態學、噪音、聲音藝術...。在科技協助下,人們對音樂迫切提出種種疑問;透過科技,我們得以用各種不同的方式來聆聽、探索音樂,科技的使用不再限定於少數風格。對此,我感到非常欣慰,因為這代表著,人們透過聆聽而創造音樂。」

- 凱瑟琳·諾曼 Katharine Norman (1960- ): 聲響藝術 Sounding Art (2004)


「新音樂表達介面國際研討會(The International Conference on New Interfaces for Musical Expression,以下簡稱NIME)」是一年一度的聲音藝術領域盛事。自2001年起每年由不同學術單位承辦,會中聚集了來自世界各地的音樂家、研究者與藝術家,針對「音樂介面(Musical Interface)」的開發與應用,交流彼此的經驗、知識與成果。近年NIME的方向,隨著個人化電子產品迅速發展,從聲音合成與控制,逐漸轉變為針對人-電腦介面(human-computer interface)與現場展演上的互動。與會者的背景從電腦音樂、演奏、作曲、多媒體,到產品設計、人機介面(HCI)、互動設計、自動控制等皆有所涵蓋,因此對話的基礎,便是建立在對「音樂表達」的想像,以及音樂性與科技性的均衡上。而會中所展示的成果,也引領著科技藝術跨界應用上的思維、語彙與工具。

The International Conference on New Interfaces for Musical Expression (aka NIME) is an annual event of field of sound art. It has been hosted by different academic units since 2001, gathering musicians, researchers and artists around the world. People share their experiences and knowledge, focusing on the development and application of musical interface. In recent years, with the rapid growth of personal electronic product, NIME centers on human-computer interface and interaction on live performance. The background of participants ranges from electronic music, composition, multi-media to product design, HCI, interactive design and automatic control. The result that presented in the conference is a leading thought and tool in the tech-based art field. 




2014年在倫敦大學金匠學院舉辦的NIME研討會,為期一週的活動中,隨著作品與計畫的呈現,以及展覽、演出跟工作坊等周邊活動,帶出當前與「音樂介面(Musical Interface)」相關議題中受到關注的面向,例如現場性(Liveness)、機器學習(Machine Learning)、多聲道配置(Ambisonic)、介面中心下的自體運作(Self-organization)...等等。近年感應技術(Sensing technology)的發展,以及智慧型手機、電玩控制器等行動裝置的普及,許多音樂家開始有能力利用較低技術門檻的工具來開發工具與軟體。電腦基礎(Computer-based)的音樂展演,不再受限於特定的風格與群眾,反而成為一種廣泛且多元的音樂型態。互動音樂科技的成功,也使得相對於科技的美學論述需要被提出與審視。

In the 2014 NIME conference, hosted by University of London Goldsmiths for a week, the event brought out controversial topics related to musical interface, which included liveness, machine learning, ambisonic, self-organization and so on. Recently, with the development of sensing technology and the spread of mobile devices, many musicians are capable of developing softwares, using tools that have lower threshold. The computer-based musical performance is no longer limited by specific styles and audience but becoming general and diverse. The success of interactive music technology also makes it necessary to have more discussion on aesthetics of technology.




本文將透過三件在2014年NIME研討會中展示的作品,從介面延伸觀眾對於聲音藝術的想像。進行討論前,我們必須先瞭解什麼是「音樂介面(Musical Interface)」。概括來說,「介面」意指使用者與某種機制進行溝通的橋樑。各種樂器都是一種特定的音樂介面,如鋼琴、小提琴等原音樂器,都是因應不同的聲音需求而被設計出的工具。使用者透過特定操作方式,去影響介面能控制的變數,進而使聲音照演奏者的意圖去改變。當然音樂介面並不只限於可被演奏的樂器,發出聲響的裝置、取樣及再現的裝置,以及處理類比與數位訊號轉換的裝置,都是音樂介面所可以指涉的對象。當代電子音樂與合成器的發展,我們對音樂介面的想像也不再受限於原音樂器,各種新型態的樂器與控制器被發明,而演奏方式與對聲音的美學思維,也在此風潮下有了更多元的樣態。

This article will talk about three display of NIME in 2014, extending audience’s imagination of sound art from interface. In general, interface refers to the bridge connecting users and some kind of mechanism. All kinds of musical instruments are interfaces, such as pianos, violins and other acoustic musical instruments, and they are designed for different demands. Of course music interface is not merely a performable instrument, but also an installation that could produce sound, take a sample and deal with digital signal, which covers from acoustic instruments to new types of instruments and controllers. Thanks to them, we have a more diverse performance and sound.





新型態的樂器跟互動方式,可以說是NIME的核心。樂器和作曲結合,創造極具吸引力的現場表演手段。聲音藝術在形式與內容上的創新,其實是圍繞著音樂表現的介面在發展。在留聲技術發明前,音樂僅能在現場被聽到。觀眾聚集在特定的時間與空間,欣賞表演者現場操作器樂。此種「現場性(Liveness)」也成為聲音做為一種藝術形式的典範。而若我們採用菲利普・亞斯蘭德(Philip Auslander)的觀點,其實從現場演出到展場中的聲音裝置,聲響都是發生於現場,來自物理碰撞、數位類比轉換或是預錄聲響的再現。無可避免地,在科技與電腦躍升音樂展演要角後,我們必須處理與真實性及經驗相關的問題,以及媒介化後表演者/觀眾在動態上被削弱的現象。

New type of instruments and interaction is the core of NIME. The combination of instrument and composition creates an appealing performance and actually the innovation of sound art is all about the development of interface. Before the invention of photography, concert could only be heard live. Audience gather in the specific space at the specific moment enjoying performance. This kind of “liveness” becomes a modal of sound art. However, from the perspective of Philip Auslander, sound happens at the scene and it comes from physical collision, Digital-to-analog conversion or the recurrence of prerecorded sound. Unavoidably, with an increasingly importance of technology and computer in the music performance, we have to deal with issues of authenticity and experience.




形象化再現(Figurative Representations)與感知- 「鏡像賦格(MirrorFugue)」

MirrorFugue ——figurative representations and senses



來自麻省理工媒體實驗室(MIT media lab)的藝術家蕭瀟,其作品「鏡像賦格(MirrorFugue)」,即對「現場性」及經由介面的「再現」做了深入探討。此作品同時也是藝術家的博士研究計劃,歷經許多不同版本,本次於倫敦大學金匠學院展出的形式,為一具有自動演奏功能的平台鋼琴,演奏者的手部及臉部動作會與樂音連動,分別投影於琴鍵與譜架上,如實呈現演奏當下的動態與情緒。參與作品材料錄製的演奏者,除了蕭瀟本人外,還包含多納爾·福克斯(Donal Fox)、坂本龍一(Ryuichi Sakamoto)、賀比·漢考克(Herbie Hancock)、馬文·閔斯基(Marvin Minsky)等知名演奏家與藝術家。

Xiao Xiao, an artist from MIT media lab, expand on the idea of reoccurrence of liveness with her work MirrorFugue. MirrorFugue is a set of interfaces for the piano to visualize the gesture of a performance. Based on the idea that the visibility of gesture contributes to learning and synchronization, MirrorFugue displays the hand and body movements of piano playing using metaphors from the physical world to connect musicians from disparate spaces and times. In addition to Xiao Xiao, Donal Fox, Ryuichi Sakamoto, Herbie Hancock, Marvin Minsky and other famous artists also participated with this performance.




「鏡像賦格」重構了現場演出的感受,但實際上卻無人在場。演奏者無非是光線與鏡面的錯覺、是當下在場也不在場的鬼魂。蕭瀟本人在研討會中表示,「鏡像賦格」可用來與超越距離跟時間的內容進行互動,並將音樂結構可視化。觀眾可以遠遠地看著鋼琴自動彈奏,或是坐下來與鏡像中的演奏者合奏。雖然作品本身應用的技術並非特別新穎,但透過研究卻能闡明這樣的介面如何增進觀眾的體驗。

MirrorFugue reconstructs the sentiment of live performance, yet there is no one at the scene. Performers are merely illusions of light and mirror. Xiao Xiao told us at the conference that MirrorFugue could interact beyond space and time and visualize the structure of music. MirrorFugue can be used in remote lessons to enable teachers and students to see each other's playing. It could also be used to playback recordings of the self and others. Even though from a technical side, the work is not too innovative, it still explains how to improve audience’s experience with this kind of interface.




通過身體演奏音樂時,同時傳遞著多層次的訊息;從複雜的器樂操作到生動的情緒表達,都經由表演者身體與音樂介面互動而被觀眾所接收。但是,當我們將音樂保留下來,跨越距離與時間重播,表演者的身體便常受到侷限甚至完全缺席。藝術家試圖深入了解「介面」是如何介入觀眾本能的感知,特別是不同視覺回饋(移動的琴鍵、手和身體的投影)對認知建構的影響,以及不同排列下如何影響觀眾的體驗。

From the control of instrument to facial expression, performers use their bodies to interact with interfaces, which delivers multiple information to an audience. However, if we replay the same music again in another age and space, performers might not show up. Artist tries to find out how interface gets involved with audience’s instinct, especially how different visual feedback (like the movement of keyboards and the projection of hands and body) affect their cognition.




卡農(Canons)和賦格(Fugues)是典型的巴洛克音樂,也是專為鍵盤樂器設計的經典作品。
在複音織體開始流行後,大部分曲作都由多層次線條構成。而在鋼琴獨奏時,每一旋律線條都由同一演奏者執行,因此較難完整傳遞和聲的織度與細膩。作品名為「鏡像賦格」,即意指在身臨其境的場景中,使觀眾獲得更強烈的感官體驗,耳朵聽到的聲音,與眼前落下的琴鍵、演奏者的肢體仿若融為一體,近乎本能地感受虛擬鋼琴家的存在。

Canon and Fugues are classical baroque music and they are designed for  keyboard instruments. While polyphony texture becomes a smash hit, most of works are constructed with multiple concurrent vocal lines. However, in piano solo, each vocal line is played by the same one performer, which makes it hard to deliver harmony. By displaying the body and the hands at full scale on the surface of the instrument, MirrorFugue evokes the presence of a pianist with a sense of intimate emotional expression.




材質的發聲與介面的錯置- 「我不是在拉小提琴(I am not playing Violin)」

I am not playing Violin —— the voice of material and displacement of interface



「我不是在拉小提琴」是一件動力聲音裝置(Kinetic sound installation) ,由曾於台灣失聲祭活動演出的香港聲音藝術家馮俊彥製作。展出現場,許多懸掛的廢棄小提琴,透過自動控制的伺服馬達牽引並相互碰撞,呈現出有別於傳統小提琴演奏的自動音樂(automatic music)。這樣的呈現方式,同時處理到「音樂介面」上對操作性(Operability)與功能性(Functionality)的討論。小提琴在裝置中的操作方式,由一般習慣的拉弓磨擦琴弦,變成由琴身相互碰撞敲擊木質琴身,觀眾由錯置的介面操作中,重新建構對既有物件的理解。而不同琴身彼此碰撞造成的震動,透過接觸式麥克風(Contact mic)被捕捉下來,琴身的功能即由共鳴箱轉變為敲擊樂器,音樂便在此時從破損琴身中發生。

Created by Jasper Feng, a Hon Kong sound artist that has performed in Lacking Sound Fest. in Taiwan, his work “ I am not playing Violin “ is a kinetic sound installation, which makes use of a series of abandoned violins to compose automatic music. This kind of display deals with operability and functionality of music interface. In this work, programmed servomotors provide oscillations to each violin. From gentle to destructive crashing moments, contact microphones attached to each violin capture these moments and amplify them through the speakers. By this method, the music is not only generated by the string, but also from the crashing body of the violins. The audience is able to experience both a specific way of listening while the work creates a multidimensional visual impact. 




日常生活中,我們接受音樂或錄音作品的方式,通常是透過已規格化的形式與播放工具,連帶影響了當代音樂產製的標準。當多數演奏者都致力於如何「真實」再現曲譜,並依循著一定典範來追求音色跟技法上的精確時,部分音樂家卻透過相同器材提供了另類思考方向。 在聲音藝術的歷史上,如路易吉‧盧梭羅(Luigi Rossolo)以及法國國家視聽研究院電子音樂研究中心(GRM)都曾利用不規則的噪音聲響,或跳脫原音樂器思維的技法,來做為音樂創作的素材。

In our daily life, we often accept music that is produced with standardized form and player, which rigidifies contemporary music production. When most of performers devoted themselves in how to reproduce music and pursue precision of timbre and technique, some of the musicians use same instruments to make a breakthrough. In the history of sound art, Luigi Rossolo and Groupe de Recherches Musicales (GRM) has utilized irregular noise as material to produce music, thinking outside the box.




而「我不是在拉小提琴」以噪音與機率做為樂曲基本元素,破壞既有結構並質疑已被規則化的音樂形式,似乎與1920年代未來主義藝術家的目標之一-反古典主義和唯美主義-相呼應,並能從中察覺藝術家對古典音樂框架解構的意圖。小提琴做為一種象徵古典音樂的符號,在再加強了此件作品帶給觀眾的刺激。無論在聽覺與視覺上,樂器自體作為介面與聲源,打破物件原本給予觀眾的既有印象,並創建了結合視覺反饋的特殊聆聽經驗。

I am not playing piano use noise and probability as basic elements of the song, breaking the standardized rigid form of music, which echoes with anti-Classism and Aestheticism. Violins, as a symbol of Classical music, reinforce the stimulus that the work brings to audience. From both visual and auditory perspective, instrument, as interface and sound source, break its stereotype and create a special listening experience combining visual feedback. 




在此補充一個「混合共振集合」概念,重新思考在為了音樂表達而設計的新介面下,樂器、觸摸跟演出這幾件事。以大衛·都鐸(David Tudor)的「雨林(Rainforest)」系列作品為例,他自1968年起,試圖透過一系列創作去探究共振的「揚聲物件(Speaker objects)」。在這系列作品中,震動傳感器與接觸式麥克風會同時被貼在選定的物件或材質上,使其自體產生共鳴。以此概念去呼應「我不是在拉小提琴」,藝術家皆是企圖去挖掘被材質外表覆蓋的本質聲響,一種鏡像(mirrored)的聲音。利用材質本身的特性,透過外在施力去映照出某種物理性回饋。藉由樂器發聲方式的錯置,呈現出一種既熟稔又陌生的奇異感受。

On top of that, I want to introduce the concept of hybrid resonance gathering. Take David Tudor as an example. Since 1968, he has explored resonant ‘speaker objects’ in his “Rainforest” series. In his work, vibration sensors and contact microphones are stick to picked objects and produce resonance with itself, which echoes with the work of I am not playing piano.




轉化日常物件的聲音回饋- Mogees

Mogees ——Transform the sound of everyday object 



最後以做為樂器形式的音樂介面Mogees,與聲音裝置做一對照。英國知名的跨界音樂人伊莫金·荷普(Imogen Heap)去年也以其特製的手套控制器參與了NIME研討會,她在會上表示,一個好的音樂表達介面,在演出上應該要能兼顧技術性、音樂性與現場性,而Mogees即在此定義之下,創造了令人耳目一新的均衡。

Mogees, combines a revolutionary app and vibration sensor to transform any object into a unique musical instrument. The famous musician Imogen Heap participated NIME conference last year with her musical gloves. She mentioned that a good music interface should balance technicality, musicality and liveness.




Mogees團隊專注於研究與日常物件透過手勢進行互動的方法,藉由創造新型態的介面,發掘音樂表達上的應用。為了打破數位和現實世界之間的障礙,並將我們身邊潛藏的聲音釋放出來,負責人布魯諾·蘭布林(Bruno Zamborlin)從巴黎龐畢度中心的IRCAM到倫敦大學金匠學院,歷時五年研究開發,Mogees成為他集大成的作品。產品曾於TED論壇分享,並在Kickstarter募資平台上獲得熱烈迴響,目前已經正式量產販售。

Mogees team focus on methods for gestural interaction with everyday objects, discovering the application of music expression through new type of interface. In order to overcome the obstacle between digital world and reality and release people’s inner voice, Bruno Zamborlin spent 5 years developing Mogees. He has shared his product in TED talk and made a splash on Kickstarter. Now Mogees is officially in massive production.




動畫先驅奧斯卡·費欽格(Oskar Fischinger)曾對約翰·凱吉(John Cage)說,世界上每個物體的內在,都藏有靈魂;想解放這些靈魂,我們只需要拂過物體的表面,勾勒出他們的聲音......。而後凱吉自述,自此多年來,他始終保持著用觸摸來認識物體的習慣。他說:「我觸碰物體,始之發聲、再發聲、探索其所能製造的聲響。無論我到哪兒,我總是在傾聽物體。」以此與Mogees作為音樂介面的功能上相呼應,在探索時,仍舊是延續著前人的腳步,凸顯出聲音藝術在觀念與技術上的與時俱進。

Oskar Fischinger has once told John Cage, “Everything in the world has a spirit which is released by its sound.” and years later, John Cage said he always understand object through touching. He touches object to make it produce sound and explores it. Mogees, as a music interface, has same concept with John Cage, which shows that sound art keep breast of technology and new ideas.



技術上,Mogees是透過接觸式麥克風(Contact Mic)去進行手勢判斷(Gesture Recognition)。當肢體與不同材質的平面接觸,接觸方式與手勢的差異,會製造不同的震動。這些細微震動,或許因為頻率或振幅過低而不被人耳所接收,但其特製的麥克風卻能抓取這些震動,並重新合成創造新的音色。這些音色並非單純取樣(Sampling),而是經由物理模擬(Physical modeling)與取樣合成(resampling synthesis)來進行,使用者及觀眾也因此能更直接地感受到動作與聲響的連動性。

Technically Mogees use contact microphone to do gesture recognition. When body touches platforms that is composed with distinct materials, the difference of gesture will produce different vibration. Some of these vibration will not be perceived by human ears due to its low frequency but its special microphone could capture those vibration and recompose a new timbre. There timbres are not merely sampling, but process with physical modeling and resampling synthesis. Hence, users and audience can feel the interrelation of movement and sound more directly. 




Mogees讓每個人不管有無音樂或技術背景,都可以輕易將身邊的物品變成嶄新、獨特的樂器。其特色在於轉化我們對既有材質存在的音色認知。最令人感到興奮地,即是其直覺性地透過接觸物體來發聲,然而此樂音卻不被原本既有材質所囿,跳脫框架卻又仍然能在視覺回饋的接收下,使人理解眼前發生的演奏。

Mogees allows everyone, no matter have music background or not, transform objects into a unique instrument. What surprises me the most is that the timbre of the object does not belong to material itself, it is beyond imagination and yet satisfied. 




正由於藝術家選用的多是日常物件,我們通常就難以想像它們還具有扮演另一 個角色、發出另一種聲響的可能。與其說Mogees為平凡的日常物件創造新的聲音,不如說是透過這樣的介面,引導我們發現那些其實一直存在,只是被人們在匆忙與庸碌中所忽視的聲響可能。

"This project is about not telling people what to do,"  Bruno Zamborlin says, "There are no rules with this technology. I want people to just look around and find their own inspiration. All I'm going to do is try to make their experience better."





對科技主導的音樂表達介面反思

Reflection on tech-dominated music interface


藉由各種新型態的音樂介面,我們可觀察到音樂家們不停在重新思考關於樂器的定義。而,在各種新型態的音樂介面被提出後,我們仍面臨到同一問題,即是這些樂器是否真的具有不可取代的價值?如果以演奏功能而言,相較於規格化的電子音樂控制器或原音樂器,這些新介面未必在操作或音色上具有優越性。在觀眾聆聽時的接收上,是否真能創造新的語彙、又是否非得透過這些介面才能達成,進而創造可被再現的聆聽經驗?

With each new types of music interface, we can observe that musicians keep redefining about instruments. Even though many new types of music interface have been raised, we still encounter same problem, that is if these instruments are irreplaceable. If we talk about functionality of performance, these new interfaces are not likely compatible with standardized electronic music controller or acoustic instruments.




回到前面介紹的三件作品,雖然取向與方法有所差異,但仍舊依賴著「現場性」才得以成立。假使只聽見作品的錄音,我們並無法單從聲響中獲得太多趣味。那些細微而慧詰的巧思,似乎無法脫離介面本身被重現。以一個使用者能實際操作的介面來說,我們通常可以通過觸摸或不同手勢來傳達訊息,並引導這些訊息轉換為聲音,在身體、介面和聲音的張力之間,構成特殊的表達方式。在先前的討論中,我們可以發現許多介面是建立於選擇(choice-based)上。可能是選單、標籤、表單乃至手勢解讀等形式。經由使用者的選擇,在預先設定的機制下產生內容。換句話說,操作者被限制在一定框架中,會傾向於保持在「分析」介面的狀態,這也是為何許多當前的電腦音樂展演,往往會使人感到犧牲了表演者的身體性。

Above three works use different techniques but they still rely on “liveness” to perform. Suppose one could merely hear record of the work, he or she might lose interest in it. The ingenuity of work has to go with interface. We usually deliver information through gestures or touch, and convert these information into sound, which creates a special expression among body, interface and tension. In the previous discussion, we could find that many interfaces are choice-based, like menu, tag, form and gesture recognition. The instrument will produce sound by preset system once the user make a decision. In another word, user is restricted in some way and tends to analysis the interface.




創新具有外部性(Externality),對跨領域創作及科技藝術而言,技術與觀念上的發展,更是支撐美學想像的骨架。因為在NIME中,科技創新無所不在,使得「技術」的優越性消失於討論中,迫使人們更專注於音樂性上。在追求創新時的嘗試,與一般大眾對音樂接受程度上的落差,也因此在討論中被凸顯。或許,有關音樂介面的討論,不能只限於創作者與藝術家關注的產製層面,有關聆聽方式(Modes of listening)以及閱聽人對「音樂」的看法,都需要更廣泛的對話。而NIME在將經驗傳承給學生、音樂家乃至業餘愛好者,以及建立可供參考資料的努力上,便可成為讓大眾獲得更豐富的音樂體驗的契機之一。

Innovation has externality and to interdisciplinary creations and technology art technique and idea are the bone of it. NIME has nothing but technology innovation, which makes people focus more on musical side, rather than technical one.The discussion about music interface should not be limited to producers and artists and we should dig more into the modes of listening and audience's perspective. NIME aims to pass down knowledge and experience to students, musicians and amateurs  and collect useful information in order to improve better experience to the public. 



[原文刊登於藝術收藏+設計雜誌2015年4月號]

NIME, Musical Interface, Synth, Instrument, Sound Arts, New Media