Experimental – Sound and Music Production

towards automation for sound effects

MIT/CSAIL researchers add realistic sounds to silent videos, a step toward automating sound effects for movies?

MIT researchers have developed a computer system that independently adds realistic sounds to silent videos. Although the technology is nascent, it’s a step toward automating sound effects for movies.

“From the gentle blowing of the wind to the buzzing of laptops, at any given moment there are so many ambient sounds that aren’t related to what we’re actually looking at,” says MITPhD student Andrew Owens . “What would be really exciting is to somehow simulate sound that is less directly associated to the visuals.”

The notion of artificial sound generation has been around for sometime now, with concepts such as procedural audio, and in many ways its long overdue that the same amount of attention and computing power that is afforded to visual effects, be directed towards sound generation. CSAIL is directed by Tim Berners-Lee and is the largest research laboratory at MIT and one of the world’s most important centres of information technology research. I have found several articles which discuss this new development and have selected sections of them here :

see demonstration video here

the following is a selection from articles:

“Researchers envision future versions of similar algorithms being used to automatically produce sound effects for movies and TV shows, as well as to help robots better understand objects’ properties.

“When you run your finger across a wine glass, the sound it makes reflects how much liquid is in it,” says CSAIL PhD student Andrew Owens, who was lead author on an upcoming paper describing the work. “An algorithm that simulates such sounds can reveal key information about objects’ shapes and material types, as well as the force and motion of their interactions with the world.”

The team used techniques from the field of “deep learning,” which involves teaching computers to sift through huge amounts of data to find patterns on their own. Deep learning approaches are especially useful because they free computer scientists from having to hand-design algorithms and supervise their progress.

The paper’s co-authors include recent PhD graduate Phillip Isola and MIT professors Edward Adelson, Bill Freeman, Josh McDermott, and Antonio Torralba. The paper will be presented later this month at the annual conference on Computer Vision and Pattern Recognition (CVPR) in Las Vegas.

In a series of videos of drumsticks striking things — including sidewalks, grass and metal surfaces — the computer learned to pair a fitting sound effect, such as the sound of a drumstick hitting a piece of wood or of rustling leaves.

The findings are an example of the power of deep learning, a type of artificial intelligence whose application is trendy in tech circles. With deep learning, a computer system learns to recognize patterns in huge piles of data and applies what it learns in useful ways.

In this case, the researchers at MIT’s Computer Science and Artificial Intelligence Lab recorded about 1,000 videos of a drumstick scraping and hitting real-world objects. These videos were fed to the computer system, which learns what sounds are associated with various actions and surfaces. The sound of the drumstick hitting a piece of wood is different than when it disrupts a pile of leaves.

Once the computer system had all these examples, the researchers gave it silent videos of the same drumstick hitting other surfaces, and they instructed the computer system to pair an appropriate sound with the video.
To do this, the computer selects a pitch and loudness that fits what it sees in the video, and it finds an appropriate sound clip in its database to play with the video.

To demonstrate their accomplishment, the researcher then played half-second video clips for test subjects, who struggled to tell apart whether the clips included an authentic sound or one that a computer system had added artificially.
But the technology is not perfect, as MIT PhD candidate Andrew Owens, the lead author on the research, acknowledged. When the team tried longer video clips, the computer system would sometimes misfire and play a sound when the drumstick was not striking anything. Test subjects immediately knew the audio was not real.

And the researchers were able to get the computer to produce fitting sounds only when they used videos with a drumstick. Creating a computer that automatically provides the best sound effect for any video — the kind of development that could disrupt the sound-effects industry — remains out of reach for now.

Although the technology world has seen significant strides of late in artificial intelligence, there are still big differences in how humans and machines learn. Owens wants to push computer systems to learn more similarly to the way an infant learns about the world, by physically poking and prodding its environment. He sees potential for other researchers to use sound recordings and interactions with materials such as sidewalk cement as a step toward machines’ better understanding our physical world.

taken from this article
and this webpage

The Computer Science and Artificial Intelligence Laboratory – known as CSAIL – is the largest research laboratory at MIT and one of the world’s most important centers of information technology research.

CSAIL and its members have played a key role in the computer revolution. The Lab’s researchers have been key movers in developments like time-sharing, massively parallel computers, public key encryption, the mass commercialization of robots, and much of the technology underlying the ARPANet, Internet and the World Wide Web.

CSAIL members (former and current) have launched more than 100 companies, including 3Com, Lotus Development Corporation, RSA Data Security, Akamai, iRobot, Meraki, ITA Software, and Vertica. The Lab is home to the World Wide Web Consortium (W3C), directed by Tim Berners-Lee, inventor of the Web and a CSAIL member.

June 16, 2016 Audio Post Production, Experimental, field recordings, film sound, game sound, Research, sound design, Technology 1 Comment

BBC radiophonic Workshop: Tape Loops & Tape Replay Setups

Elizabeth Parker and Paddy Kingsland from the BBC Radiophonic Workshop in 1979 demonstrate the use of tape loops and tape-replay setups. We hear Elizabeth Parker’s “bubble music” and Paddy Kingsland on the electric guitar with twin Studer tape recorders.

This excerpt is from the BBC documentary The New Sound of Music produced in 1979.

Tape Loops BBC Radiophonic Workshop

Paddy Kingsland demonstrates twin Studer recorders in a delay-replay setup that some might refer to as “Frippertronics’ – named after Robert Fripp I believe. Fripp may have used twin Revox machines in a similar way for some of his compositions. It is an interesting setup, possibly described in some Workshop writings from the 1960s.

The BBC Radiophonic Workshop, one of the sound effects units of the BBC,
was created in 1958 to produce effects and new music for radio.

It was closed in March 1998, although much of its traditional work had already been outsourced by 1995.

The original Radiophonic Workshop was based in the BBC’s Maida Vale Studios
in Delaware Road, London.

We have more on the Radiophonic workshop elsewhere in this blog –
e.g.
free-thinking-bbc-radiophonic-workshop/

doctor-who-how-norfolk-man-created-dalek-and-tardis-sounds/

The techniques initially used by the Radiophonic Workshop were closely related to those used in musique concrète; new sounds for programs were created by using recordings of everyday sounds such as voices, bells or gravel as raw material for “radiophonic” manipulations. In these manipulations, audio tape could be played back at different speeds (altering a sound’s pitch), reversed, cut and joined, or processed using reverb or equalisation. The most famous of the Workshop’s creations using ‘radiophonic’ techniques include the Doctor Who theme music, which Delia Derbyshire created using a plucked string, 12 oscillators and a lot of tape manipulation; and the sound of the TARDIS (the Doctor’s time machine) materialising and dematerialising, which was created by Brian Hodgson running his keys along the rusty bass strings of a broken piano, with the recording slowed down to make an even lower sound.

Much of the equipment used by the Workshop in the earlier years of its operation in the late 1950s was semi-professional and was passed down from other departments, though two giant professional tape-recorders (which appeared to lose all sound above 10 kHz) made an early centrepiece. Reverberation was obtained using an echo chamber, a basement room with bare painted walls empty except for loudspeakers and microphones. Due to the considerable technical challenges faced by the Workshop and BBC traditions, staff initially worked in pairs with one person assigned to the technical aspects of the work and the other to the artistic direction.
[source https://en.wikipedia.org/wiki/BBC_Radiophonic_Workshop]

March 6, 2016 Experimental, Industry, Radiophonic Workshop, sound history, Technology, TV, Uncategorized No Comments

Lincoln Soundscape

Blog post by Alex O’Brien – Level 3 Audio Production.

For one of my final year projects I created a soundscape of Lincoln, which was showcased in The Little Red Gallery in the Bailgate. The concept behind this soundscape was to construct a sonic interpretation of 24 hours in Lincoln.
The piece is made up of 3 sections, Cathedral, Town and Evenings. Each section presents the listener with a different part of Lincoln. The recording process was very enjoyable but incredibly time consuming. For the Evening recordings I would often find myself sat in the cold recording traffic and such for hours on end or I would be in the Cathedral for hours at a time, listening closely for interesting sounds as they echoed around the Cathedral walls. I wanted to make sure I had every recording I needed to piece this 25 minute project together so I could layer these sounds up and paint an audio picture of Lincoln.
Having my piece showcased in a gallery was fantastic. Seeing people walk around the space, listening to my piece made me feel as if I had really accomplished something. But also working for a client was a great experience and it has definitely given me confidence in knowing that I can go out and get my work showcased.
Overall, the project process was a great learning curve and I’d love to do another sometime soon.

Listen to Alex’s Soundscape here:

September 3, 2014 Alumni, Experimental, field recordings, Soundscape, Students No Comments

Free Thinking – BBC Radiophonic Workshop

The BBC Radiophonic workshop was founded in 1958 by Desmond Briscoe and Daphne Oram. This group of experimental composers, sound engineers and musical innovators provided music for programmes including The Body in Question, Horizon, Quatermass, Newsround, The Hitchhiker’s Guide to the Galaxy, Chronicle and Delia Derbyshire’s iconic Doctor Who Theme before being shut down by Director General John Birt in 1998.

In an edition recorded just as the Workshop prepare to release a new album, and tour the UK, Matthew Sweet brings together Radiophonic Workshop members Dick Mills, Paddy Kingsland, Roger Limb, Peter Howells, and Mark Ayres to reflect on the days and nights they spent in the workshop, coaxing ageing machines into otherworldly life, and pioneering electronic music. Also in the programme, The Prodigy’s Kieron Pepper and Vile Electrodes on the influence the Radiophonic Workshop had on them.

Listen to the BBC Radio 3 programme here:

Taking part in the programme:
Dick Mills
Mark Ayres
Roger Limb
Peter Howell
Paddy Kingsland
Matthew Howden
Kieron Pepper
Vile Electrodes
Steven Price

May 1, 2014 electronic music, Experimental, featured posts, film sound, Radio 3, Radiophonic Workshop, sound design, sound history 1 Comment

Pauline Oliveros – Pioneer of Electronic Classical Music

Pauline Oliveros is an American improvisor, accordionist and composer who is considered a pioneer of electronic classical music in 20th century America. At 81 years old, her career spans some fifty years of boundary breaking music-making, and she has been the recipient of numerous awards. On April 1st 2014, she performs a real-time improvised performance linking musicians in Stanford (California), Troy (New York) and Montreal. This is the first performance of this kind in the UK at the Birmingham Conservatoire.

The performance forms part of Frontiers: Extraordinary Music from Downtown New York & Birmingham, a major festival of music presented by Birmingham Conservatoire and Third Ear.

Listen to Pauline being interviewed for BBC Radio 4’s Woman’s Hour here:

April 1, 2014 Composition, electronic music, Experimental, Radio 4 No Comments

The Art Of The Loop

Most current pop music is created not with live instruments, but from pre-formed, off the shelf chunks of music known as loops. Musician Matthew Herbert explores the art of the loop and the million-dollar industry that has grown up around it, and asks whether it is setting music makers free from the constraints of traditional instruments or killing creativity.

Loops are pre-recorded performances, typically of a solo instrument, and typically 1 or 2 bars long. Looping isn’t new – it started soon after the advent of tape recorders. But recent advances in computer technology and software mean that effects which once needed a full-scale studio costing thousands of pounds can be created for little or no cost on a laptop or even a mobile phone. A CD of loops costing £10 can be used to make a million-selling international hit, but who is the real composer?

Matthew once made an entire album from the sounds of a single pig’s life, so he’s no stranger to the benefits of loops and sampling. He talks to producers, musicians and loop-creators and experiments with technology ancient and modern; he hears from looping’s defenders and detractors and looks into a musical future which he finds fascinating but many find terrifying.

And, along the way, he builds a dance track out of a Radio 4 Continuity announcer.

Produced by Micky Curling
A Folded Wing production for BBC Radio 4

March 1, 2014 electronic music, Experimental, featured posts, Radio 4, Radiophonic Workshop No Comments

Who Killed Classical Music?

The Composer Gabriel Prokofiev (grandson of Sergei Prokofiev) looks at the increasing disconnection between classical music and its audience. He investigates the argument that composers such as Schoenberg killed off 20th century classical music for all but a small elite audience.

Until the early 20th century, each composer of classical music developed his own style built on the traditions of previous composers. Then Arnold Schoenberg changed all this, by devising ‘Serialism’ where melodies were no longer allowed.

In the 1950s, composers such as Pierre Boulez created ‘Total Serialism’. Every aspect of a piece of music – rhythms and loudness as well as notes – was rigidly controlled by a fixed formula.

And the sense of composers being remote from their audience was exacerbated by the elevation of musical performance to a kind of ritual.

But even at a time when Serialism gripped major parts of the classical music establishment, music that was overtly emotional was still being written by composers such as Shostakovich and Prokofiev in Russia. Ironically, in these countries, the State continued to support classical music, whereas in more liberal regimes in Europe it retreated to the intellectual margins.

Now the Serialist experiment has been largely abandoned and a whole new generation of composers – including Gabriel himself – is embracing popular culture, just as composers used to in the past when folk music or dance music were a major source of inspiration.

So has the death of classical music been exaggerated? Will it find new homes and new means of expression to attract the audiences of the future?

With contributions from Arnold Whittall, Stephen Johnson, Alexander Goehr, David Matthews, Ivan Hewett and Tansy Davies

Listen to the BBC Radio 4 programme here:

January 25, 2014 Composition, Experimental, featured posts, Radio 4 No Comments

LSM Research Seminar – Marie Thompson

Blog Post by Dr Dean Lockwood:

Staff and students from the Lincoln School of Media (including Audio Production) welcomed Marie Thompson for the LSM Research Seminar series which took place Wednesday, 30th October.

Marie is an artist and researcher based in Newcastle upon Tyne. She is currently a PhD candidate at Newcastle University, based in the International Centre for Music Studies. Her thesis uses a Spinozist notion of affect to critically rethink the correlation between noise, ‘unwantedness’ and ‘badness’, so to more fully allow for the use of noise as a musical resource. She is the co-editor of the collection, Sound, Music, Affect: Theorizing Sonic Experience (New York: Bloomsbury, 2013). Marie is also regularly audible as a noisemaker and improviser. She plays individually as Tragic Cabaret and in the band, Beauty Pageant. Here is Marie’s abstract for her talk for our research seminar:

‘Rethinking noise, rethinking noise music: Affect, relationality and the poetics of transgression’:

‘In this paper, I outline a relational, ethico-affective approach to noise that works to disrupt the definitive correlation between noise, ‘unwantedness’ and ‘badness’. Rather than defining noise as a type of sound, or a subjective judgement of sound, noise is posited as a productive, transformative force and a necessary component of material relations. This approach to noise, I argue, is advantageous: firstly, because it allows for the noise that occurs out of (human) earshot, insofar as it no longer relies upon a constitutive listening subject; and secondly, because it allows for noise’s capacity to be good as well as bad, generative as well as destructive. A greater space is thus made for noise’s positively productive capacity, which has been readily explored within the arts.

In the second half of this paper, I discuss how a relational, ethico-affective approach to noise provides a means of (re)conceptualising noise music that moves away from the language of failure, taboo and contradiction. Rather than approaching noise music in terms of transgression, which is underlined by a dualistic conceptualisation of the relationship between (wanted, ‘good’) music and (unwanted, ‘bad’) noise, I suggest that noise music can be understood as an act of exposure, in that it foregrounds the presence of noise that is always already within the technical-musical system.’

October 30, 2013 Experimental, featured posts, Noise, Research, staff No Comments

Guest Lecture – Mike Harding – Touch

Blog post by Senior Lecturer Dr Dean Lockwood.

On Friday 18th October, Mike Harding, founder and supremo of Touch, came to LSM to talk to Audio Production students taking the critical studies module, Auditory Culture. Given that some of the key concerns of the module are debates around the concepts of noise and the soundscape, it was a great opportunity to talk to someone intimately involved with a label which has specialized in promoting artists exploring precisely these areas. As quickly became clear, Touch has a philosophical orientation which propels it way beyond the narrow exigencies of the music industry. Touch has always been conceived as an art project rather than simply a label. Because of its obsessively experimental ethos, it has survived pretty much on its own terms and has never fit well with the complacent mainstream and its genre categories. As Mike explained, Touch was established in the early eighties in the wake of punk. Capitalizing on the energies generated by the so-called ‘New Wave’ independent scene, Touch was a key post-punk project, its first releases heavily involved in early cassette culture and the ‘mixtape’ phenomenon. With cassette magazines such as Feature Mist and Ritual: Magnetic North, Touch presented sophisticated cut-ups and powerful work by bands such as New Order, Einstürzende Neubauten and Cabaret Voltaire, as well as musics from around the world (before such a thing as ‘World Music’ existed). ‘No one ever said no’, which stands as a great testament to the label’s reputation and integrity. Mike took us, in the first part of his talk, through the early history of Touch, spicing things up with personal anecdotes, and in the second part addressed Touch’s present concerns. It is the home of artists such as Christian Fennesz, Bruce Gilbert, Ryoji Ikeda and Chris Watson. Mike played us a good selection of pieces which some of these artists have put out on Touch. These artists have in common, I would suggest, what we might term an ecological sensibility, a particular attention to the relations which comprise acoustic space, sometimes through glitch aesthetics, sometimes through field recordings or other means.

On Saturday 19th, Touch presented two world premieres at Lincoln Cathedral as part of the Frequency festival. The evening, after Mike’s introduction, commenced with Anna Von Hausswolff’s performance of an austere, resonant new score for the organ, titled Källan. Chris Watson and Hildur Guðnadóttir then presented a stunning new collaborative multi-channel sound work, titled Sönghellir (The Cave of Song), which I think captivated everyone present. Touch’s website describes the work as ‘a sound journey from under the waters of Faxafloi, Iceland, alongside some of the largest animals on the planet. Up, onto the lava beach, across the lava fields and reindeer moss to the foot of the snow mountain, Snaefellsnes. The journey continues up and then into the mountain, ending inside Sönghellir, the song cave…’ It was a perfect example of the art of acoustic space that Touch releases exemplify.

October 22, 2013 Acoustics, culture, Experimental, featured posts, field recordings, Guest lecture, Noise, Soundscape No Comments

Björk Tackles The Grid

When using a DAW for electronic music production, it is often frustrating to be shackled to the grid as it often dictates the feel in the rhythmic elements of a composition.

In the songs and apps for Biophilia, Björk tackles this problem of the rigidity of the grid in her exploration of nature, music and technology.

In the very interesting clip below, she describes a much more fluid approach made possible by today’s sophisticated technology and its ability to behave in a much more organic way.

Listen to the short clip from the BBC 6 Music programme ‘The First Time’ here:

August 2, 2013 6Music, Composition, electronic music, Experimental No Comments

1 2 Next Page »

Content

MIT/CSAIL researchers add realistic sounds to silent videos, a step toward automating sound effects for movies?

see demonstration video here