towards automation for sound effects

MIT/CSAIL researchers add realistic sounds to silent videos, a step toward automating sound effects for movies?

Screen Shot 2016-06-16 at 08.35.26MIT researchers have developed a computer system that independently adds realistic sounds to silent videos. Although the technology is nascent, it’s a step toward automating sound effects for movies.

“From the gentle blowing of the wind to the buzzing of laptops, at any given moment there are so many ambient sounds that aren’t related to what we’re actually looking at,” says MITPhD student Andrew Owens . “What would be really exciting is to somehow simulate sound that is less directly associated to the visuals.”

The notion of artificial sound generation has been around for sometime now, with concepts such as procedural audio, and in many ways its long overdue that the same amount of attention and computing power that is afforded to visual effects, be directed towards sound generation. CSAIL is directed by Tim Berners-Lee and is the largest research laboratory at MIT and one of the world’s most important centres of information technology research. I have found several articles which discuss this new development and have selected sections of them here :

see demonstration video here 

the following is a selection from articles:

“Researchers envision future versions of similar algorithms being used to automatically produce sound effects for movies and TV shows, as well as to help robots better understand objects’ properties.

“When you run your finger across a wine glass, the sound it makes reflects how much liquid is in it,” says CSAIL PhD student Andrew Owens, who was lead author on an upcoming paper describing the work. “An algorithm that simulates such sounds can reveal key information about objects’ shapes and material types, as well as the force and motion of their interactions with the world.”

Screen Shot 2016-06-16 at 08.34.20

The team used techniques from the field of “deep learning,” which involves teaching computers to sift through huge amounts of data to find patterns on their own. Deep learning approaches are especially useful because they free computer scientists from having to hand-design algorithms and supervise their progress.

The paper’s co-authors include recent PhD graduate Phillip Isola and MIT professors Edward Adelson, Bill Freeman, Josh McDermott, and Antonio Torralba. The paper will be presented later this month at the annual conference on Computer Vision and Pattern Recognition (CVPR) in Las Vegas.

In a series of videos of drumsticks striking things — including sidewalks, grass and metal surfaces — the computer learned to pair a fitting sound effect, such as the sound of a drumstick hitting a piece of wood or of rustling leaves.

The findings are an example of the power of deep learning, a type of artificial intelligence whose application is trendy in tech circles. With deep learning, a computer system learns to recognize patterns in huge piles of data and applies what it learns in useful ways.

In this case, the researchers at MIT’s Computer Science and Artificial Intelligence Lab recorded about 1,000 videos of a drumstick scraping and hitting real-world objects. These videos were fed to the computer system, which learns what sounds are associated with various actions and surfaces. The sound of the drumstick hitting a piece of wood is different than when it disrupts a pile of leaves.
Screen Shot 2016-06-16 at 08.34.52

Once the computer system had all these examples, the researchers gave it silent videos of the same drumstick hitting other surfaces, and they instructed the computer system to pair an appropriate sound with the video.
To do this, the computer selects a pitch and loudness that fits what it sees in the video, and it finds an appropriate sound clip in its database to play with the video.

To demonstrate their accomplishment, the researcher then played half-second video clips for test subjects, who struggled to tell apart whether the clips included an authentic sound or one that a computer system had added artificially.
But the technology is not perfect, as MIT PhD candidate Andrew Owens, the lead author on the research, acknowledged. When the team tried longer video clips, the computer system would sometimes misfire and play a sound when the drumstick was not striking anything. Test subjects immediately knew the audio was not real.

And the researchers were able to get the computer to produce fitting sounds only when they used videos with a drumstick. Creating a computer that automatically provides the best sound effect for any video — the kind of development that could disrupt the sound-effects industry — remains out of reach for now.

Although the technology world has seen significant strides of late in artificial intelligence, there are still big differences in how humans and machines learn. Owens wants to push computer systems to learn more similarly to the way an infant learns about the world, by physically poking and prodding its environment. He sees potential for other researchers to use sound recordings and interactions with materials such as sidewalk cement as a step toward machines’ better understanding our physical world.

 

taken from this article
and this webpage

csail_logo

The Computer Science and Artificial Intelligence Laboratory – known as CSAIL ­– is the largest research laboratory at MIT and one of the world’s most important centers of information technology research.
CSAIL and its members have played a key role in the computer revolution. The Lab’s researchers have been key movers in developments like time-sharing, massively parallel computers, public key encryption, the mass commercialization of robots, and much of the technology underlying the ARPANet, Internet and the World Wide Web.  
CSAIL members (former and current) have launched more than 100 companies, including 3Com, Lotus Development Corporation, RSA Data Security, Akamai, iRobot, Meraki, ITA Software, and Vertica. The Lab is home to the World Wide Web Consortium (W3C), directed by Tim Berners-Lee, inventor of the Web and a CSAIL member.

 

BBC radiophonic Workshop: Tape Loops & Tape Replay Setups

Elizabeth Parker and Paddy Kingsland from the BBC Radiophonic Workshop in 1979 demonstrate the use of tape loops and tape-replay setups. We hear Elizabeth Parker’s “bubble music” and Paddy Kingsland on the electric guitar with twin Studer tape recorders.

This excerpt is from the BBC documentary The New Sound of Music produced in 1979.

Tape Loops BBC Radiophonic Workshop

Paddy Kingsland demonstrates twin Studer recorders in a delay-replay setup that some might refer to as “Frippertronics’ – named after Robert Fripp I believe. Fripp may have used twin Revox machines in a similar way for some of his compositions. It is an interesting setup, possibly described in some Workshop writings from the 1960s.

BBC radiophonic workshop The BBC Radiophonic Workshop, one of the sound effects units of the BBC,
was created in 1958 to produce effects and new music for radio.

It was closed in March 1998, although much of its  traditional work had already been outsourced by 1995.

The original Radiophonic Workshop was  based in the BBC’s Maida Vale Studios
in Delaware Road, London.

 

Screen Shot 2016-03-06 at 10.46.17

 

We have more on the Radiophonic workshop elsewhere in this blog –
e.g.
free-thinking-bbc-radiophonic-workshop/

doctor-who-how-norfolk-man-created-dalek-and-tardis-sounds/

 

 

The techniques initially used by the Radiophonic Workshop were closely related to those used in musique concrète; new sounds for programs were created by using recordings of everyday sounds such as voices, bells or gravel as raw material for “radiophonic” manipulations. In these manipulations, audio tape could be played back at different speeds (altering a sound’s pitch), reversed, cut and joined, or processed using reverb or equalisation. The most famous of the Workshop’s creations using ‘radiophonic’ techniques include the Doctor Who theme music, which Delia Derbyshire created using a plucked string, 12 oscillators and a lot of tape manipulation; and the sound of the TARDIS (the Doctor’s time machine) materialising and dematerialising, which was created by Brian Hodgson running his keys along the rusty bass strings of a broken piano, with the recording slowed down to make an even lower sound.

Much of the equipment used by the Workshop in the earlier years of its operation in the late 1950s was semi-professional and was passed down from other departments, though two giant professional tape-recorders (which appeared to lose all sound above 10 kHz) made an early centrepiece. Reverberation was obtained using an echo chamber, a basement room with bare painted walls empty except for loudspeakers and microphones. Due to the considerable technical challenges faced by the Workshop and BBC traditions, staff initially worked in pairs with one person assigned to the technical aspects of the work and the other to the artistic direction.
[source https://en.wikipedia.org/wiki/BBC_Radiophonic_Workshop]

Screen Shot 2016-03-06 at 10.46.02

All You Need Is Lab: How Technology Inspired Innovation in Music

Screen Shot 2014-08-05 at 13.08.15

Musician and songwriter Midge Ure looks at the many ways scientific and technological innovation have stimulated creativity in pop music.

From the invention of the steel guitar string, through the tape recorder and the synthesiser, to the drum machine and Autotune, musicians have always embraced the latest ideas and adapted or distorted them to produce new sounds.

Musicians Anne Dudley (Art of Noise) and Thomas Dolby join music journalist David Hepworth and blues researcher Tom Attah, exploring how the laboratory has informed and inspired the studio.

Midge demonstrates what you can achieve with just a laptop these days – but laments the passing of an age of invention in popular music.

Featured music includes The Beatles, Chopin, Thomas Dolby, Robert Johnson, Muddy Waters, Little Walter, Charlie Christian, Les Paul and Mary Ford, The Tornados, The Small Faces, Queen, The Sweet, Stevie Wonder, Band Aid, Art of Noise, Donna Summer, Fat Boy Slim, Cher, Daft Punk and Nick Clegg.

Producer: Trevor Dann
A Trevor Dann production for BBC Radio 4

The 45 @ 65

Screen Shot 2014-03-28 at 11.14.44

Sixty-five years ago RCA Victor launched a small, round, plastic green disc on to the listening public. Journalist David Quantick charts people’s love affair with the 45 rpm single and examines one of the most important revolutions in the modern music business.

The single was always aimed at the younger generation, while the LP originally catered for a middle-aged, middle-class, well-heeled audience. The cheaper 45 took on the poorer, cooler youth market and spun with it. The vinyl single launched rock and roll, pop and the teenager on the world and provided a lynchpin for Western popular culture. It has defined the popular music of last 60 years and shows no signs of dying.

In the first programme, David looks at the war of the speeds and the early, glory days of the vinyl single, which pitted stars like Judy Garland up against Frank Sinatra and then brought audiences Elvis and Bill Haley. All this, set against a brave new world of cheap ‘portable’ record players, exotic new vinyl jukeboxes and the birth of the singles charts.

David also examines the early days of the charts, the effect the single had on that new phenomenon, the teenager, the power the TV Music shows had on the single and the cultural power of the 45, from the revolution of rock and roll to teeny and weeny boppers and glam rock’s children of the revolution.

The series features contributions from Tom Jones, actor Martin Freeman, Myleene Klass, songwriter Diane Warren, musician Soweto Kinch, Bob Stanley from Saint Etienne, Michael Bradley from The Undertones, the Reverend Run, DJ Cosmo, Pete Shelley from The Buzzcocks, Pete Waterman, as well as DJs including Mike Read, David Jensen, Johnnie Walker, Bob Harris, and Bill Brewster Neil Fox amongst others.

Presenter/ David Quantick, Producer/ Anna Harrison and Frank Stirling for Unique Broadcasting

Episode 1

Episode 2

New Facilities

Screen Shot 2014-01-09 at 15.02.04

In addition to further updates to the Lincoln Sound Theatre, and investment in a new stock of Zoom H4N audio recorders for radio students, LSM created a new Radio Drama Studio last summer, making creative use of space that was largely for storage, and giving much better opportunities for high class productions to be produced in collaboration with staff and students in Performing Arts.

Moreover, in October ’13, we had news that we had been successful in our bid to the University to get funding to replace and update, quite substantially, our multitrack studio. It is now equipped with an Audient ASP8024 mixing console, a range of new studio microphones (including the SE2200A and Audix drum mics) and outboard processing devices (such as the Chameleon Labs 7720 compressor).

This new kit, along with the Pro Tools HD2 system and Waves plug ins, provides students with industry standard recording and mixing facilities as recommended by our prestigious accreditation body JAMES (Joint Audio Media Education Support).

Barlow & Sons?

The increasing use of folk instrumentation and driving bass drum beats has obviously come to the notice of the Ivor Novello winning Gary Barlow.

Recently Barlow has released a new song “Let me go”, and almost as quickly people started to make comparisons with Mumford and Sons, largely due to the instrumentation (banjos and bass drum pounding)
Of course they both share a similarity in musical style to Johnny Cash

I also thought the similarity with Mumford’s “I will wait” was uncanny – thought I’d put it to the test.
It’s true that neither of them are the first to use the 1,4,5 chord sequence (80% of all pop songs use this), but the combination of rhythm, percussion and build to the chorus are uncanny.
They’re both 4/4, with the bass drum on the 1 and 3 beat., around 126 bpm.

Gary’s song is exactly the same tempo as “I will wait”, but it’s in a higher key so it needs pitching down by a tone and a half – other than that its fairly convincing.

Click here for my side by side comparisson

According to Digital Spy, Gary has said:

“I’ve always liked folky, acoustic music but I’ve never fully explored it. I turned back time and was listening to Johnny Cash and early Elton John before I wrote ‘Let Me Go’,” Barlow told the Sun. “I’m 42, I don’t want to do urban or dance music.”
On claims his new single sounds similar to Mumford & Sons, he added:
“I love Mumford & Sons – it’s good, English music, but let’s be honest, they got it off Johnny Cash too.”

Since I Saw You Last also features a duet with Sir Elton John titled ‘Face to Face’ and is out on November 25, followed by a tour next year.

GARY BARLOWUnknown-1

 

 

 

 

 

 

MUMFORD AND SONS Unknown

 

Other bass drum songs out at the moment might include Aveci’s “Wake me up” in the chorus.. Coldplay’s “Viva la Vida”, etc
Can you think of any more?

The Guardian had this to say:

“Disappointingly constructed using the Mumford & Sons’ formula of dusty “woah, woah, woah” backing vocals, plucked banjo and copious amounts of thigh slapping, it’s a far cry from the glorious Back For Good, or Rule The World or even Sing. Perhaps the worst part of it all is imagining the inevitable performance of it on The X Factor, complete with backing dancers dressed as Marcus Mumford.

Here’s my comparison uploaded to YouTube – it will be interesting to see if by pitch changing Gary’s track, and using a live version of Mumfod and Sons’ song, will the automatic copyright checker miss this.
And hopefully Dave McSherry won’t take this post down – ’cause I’m guessing he’s not a big Barlow fan 😉

Food For Thought From Eno

Screen Shot 2013-05-10 at 10.43.28

Brian Eno is always worth listening to. He is a very engaging speaker and his thoughts about art and music are extremely thought provoking and inspiring for me.
He speaks about many things in his Red Bull 2013 lecture, but two in particular caught my attention.

In the first he discusses the creative problems in music composition that come with an abundance of options in the modern DAW. Listen to the clip here

In the second he discusses his approach to film scoring and how he avoids the cliches of many Hollywood style film scores. Listen to the clip here

Both these clips should be of interest to Audio Production students who specialise in music composition.

PS. Look out for Kieran Hebden (Four Tet) in the Q&A at the end.

The Most Significant Beat

A comparative study of changes in bass drum sounds from 70s disco to electronic dance music of the 1980s and 1990s.

This Paper was presented at the Art of Record Production conference at the University of Edinburgh, September 8th -10th 2006

The title for this paper; The Most Significant Beat, refers to the beat fulfilled as a bass drum sound. The material in focus contains the so-called four-to-the-floor bass drum pattern,where the author believes the bass drum is functionally crucial in initiating kinetic patterns amongst its receivers.

the Sound of Skyfall

The sound of Skyfall

The SoundWorks Collection dives into the latest installment in the long-running saga based on Ian Fleming’s James Bond character.

Director Sam Mendes (American Beauty and Jarhead) brings the audience an entirely new storyline to the Bond character in “Skyfall”.

Exploring the sound and music of the film we talk with Scott Millan (Sound Re-recording Mixer), Greg Russell (Sound Re-recording Mixer),
Karen Baker Landers (Supervising Sound Editor), and Per Hallberg (Supervising Sound Editor).

see also

http://soundworkscollection.com/skyfall