towards automation for sound effects

MIT/CSAIL researchers add realistic sounds to silent videos, a step toward automating sound effects for movies?

Screen Shot 2016-06-16 at 08.35.26MIT researchers have developed a computer system that independently adds realistic sounds to silent videos. Although the technology is nascent, it’s a step toward automating sound effects for movies.

“From the gentle blowing of the wind to the buzzing of laptops, at any given moment there are so many ambient sounds that aren’t related to what we’re actually looking at,” says MITPhD student Andrew Owens . “What would be really exciting is to somehow simulate sound that is less directly associated to the visuals.”

The notion of artificial sound generation has been around for sometime now, with concepts such as procedural audio, and in many ways its long overdue that the same amount of attention and computing power that is afforded to visual effects, be directed towards sound generation. CSAIL is directed by Tim Berners-Lee and is the largest research laboratory at MIT and one of the world’s most important centres of information technology research. I have found several articles which discuss this new development and have selected sections of them here :

see demonstration video here 

the following is a selection from articles:

“Researchers envision future versions of similar algorithms being used to automatically produce sound effects for movies and TV shows, as well as to help robots better understand objects’ properties.

“When you run your finger across a wine glass, the sound it makes reflects how much liquid is in it,” says CSAIL PhD student Andrew Owens, who was lead author on an upcoming paper describing the work. “An algorithm that simulates such sounds can reveal key information about objects’ shapes and material types, as well as the force and motion of their interactions with the world.”

Screen Shot 2016-06-16 at 08.34.20

The team used techniques from the field of “deep learning,” which involves teaching computers to sift through huge amounts of data to find patterns on their own. Deep learning approaches are especially useful because they free computer scientists from having to hand-design algorithms and supervise their progress.

The paper’s co-authors include recent PhD graduate Phillip Isola and MIT professors Edward Adelson, Bill Freeman, Josh McDermott, and Antonio Torralba. The paper will be presented later this month at the annual conference on Computer Vision and Pattern Recognition (CVPR) in Las Vegas.

In a series of videos of drumsticks striking things — including sidewalks, grass and metal surfaces — the computer learned to pair a fitting sound effect, such as the sound of a drumstick hitting a piece of wood or of rustling leaves.

The findings are an example of the power of deep learning, a type of artificial intelligence whose application is trendy in tech circles. With deep learning, a computer system learns to recognize patterns in huge piles of data and applies what it learns in useful ways.

In this case, the researchers at MIT’s Computer Science and Artificial Intelligence Lab recorded about 1,000 videos of a drumstick scraping and hitting real-world objects. These videos were fed to the computer system, which learns what sounds are associated with various actions and surfaces. The sound of the drumstick hitting a piece of wood is different than when it disrupts a pile of leaves.
Screen Shot 2016-06-16 at 08.34.52

Once the computer system had all these examples, the researchers gave it silent videos of the same drumstick hitting other surfaces, and they instructed the computer system to pair an appropriate sound with the video.
To do this, the computer selects a pitch and loudness that fits what it sees in the video, and it finds an appropriate sound clip in its database to play with the video.

To demonstrate their accomplishment, the researcher then played half-second video clips for test subjects, who struggled to tell apart whether the clips included an authentic sound or one that a computer system had added artificially.
But the technology is not perfect, as MIT PhD candidate Andrew Owens, the lead author on the research, acknowledged. When the team tried longer video clips, the computer system would sometimes misfire and play a sound when the drumstick was not striking anything. Test subjects immediately knew the audio was not real.

And the researchers were able to get the computer to produce fitting sounds only when they used videos with a drumstick. Creating a computer that automatically provides the best sound effect for any video — the kind of development that could disrupt the sound-effects industry — remains out of reach for now.

Although the technology world has seen significant strides of late in artificial intelligence, there are still big differences in how humans and machines learn. Owens wants to push computer systems to learn more similarly to the way an infant learns about the world, by physically poking and prodding its environment. He sees potential for other researchers to use sound recordings and interactions with materials such as sidewalk cement as a step toward machines’ better understanding our physical world.

 

taken from this article
and this webpage

csail_logo

The Computer Science and Artificial Intelligence Laboratory – known as CSAIL ­– is the largest research laboratory at MIT and one of the world’s most important centers of information technology research.
CSAIL and its members have played a key role in the computer revolution. The Lab’s researchers have been key movers in developments like time-sharing, massively parallel computers, public key encryption, the mass commercialization of robots, and much of the technology underlying the ARPANet, Internet and the World Wide Web.  
CSAIL members (former and current) have launched more than 100 companies, including 3Com, Lotus Development Corporation, RSA Data Security, Akamai, iRobot, Meraki, ITA Software, and Vertica. The Lab is home to the World Wide Web Consortium (W3C), directed by Tim Berners-Lee, inventor of the Web and a CSAIL member.

 

The SOUND OF CAPITAL

THE SOUND OF CAPITAL

BBC new Drama Capital is edited by an ex-colleague of mine Philip Kloss and the Dialogue editor is by co-incidence the man who designed our Sound Effect Software we use here at LSFM.
This recent article is a fascinating insight into some of the sound post challenges.

To create an authentic soundscape for BBC drama Capital, dubbing mixer Howard Bargroff took a trip to the part of south London in which it is set,
writes George Bevir [from an article first published in Broadcast Online]

1230371_capital-bts

 

 

 

 

 

 

CREDITS

TX 9pm, Wednesdays, from 24 November, BBC1
Length 3 x 60 minutes
Dubbing mixer/ FX editor Howard Bargroff
Foley editor Stuart Bagshaw
Dialogue editor Peter Gates (two episodes); Michele Woods (one episode)
ADR supervisor Kallis Shamaris
FX editor Mike Wabro (one episode)
Picture post Technicolor
Director Euros Lyn
Writer John Lanchester

capital-bts-_395

 

 

 

 

 

 

Lesley Sharp (Mary)

Stepping out of the studio to capture authentic audio is not always without peril, no matter how inconspicuous the individual or discreet the recording device. A few years ago, dubbing mixer Howard Bargroff needed some crowd noise for a rap album he was working on but a trip to a pub at closing time to record the sound of a pack of people nearly resulted in him being “lynched” by the suspicious boozers.

“You have to be careful you don’t look like a psychopath, but people can still be suspicious,” he says.

Fortunately for Bargroff, his trip to Clapham to capture the sounds of south London for BBC drama Capital passed by with little more than a few sideways glances.

The three-parter, which is based on John Lanchester’s novel, is a portrait of a road in Clapham that is transformed by rising property prices and then rocked by an anonymous hate campaign. Bargroff ’s brief was to give London a presence so that the city becomes a character in its own right and “leaks” in to every scene.

 

“It’s a contemporary piece about the gentrification of London, so I went to south London and made a bunch of recordings,” says Bargroff. “As I moved around Clapham, I saw microcosms of the plot – people interacting with builders, posh mums coming out of buildings, and so on. It was as though the book was coming alive and I was recording it.”

Bargroff says his recordings, made using a Zoom H5, became the “sonic backbone” of the series, comprising around 50% of the background sound.

“We used lots of bridging sounds, such as sirens, between cuts. At first I thought the recordings were a luxury, but they soon became a necessity; those recordings of Clapham High Street, of a park, of close and distant traffic, planes passing overhead and sirens became themes throughout the episodes. It helped to create the feeling that in London you are never more than a few streets away from a busy high street.”

capital-bts-_395 (3)

Exiting London
Bargroff, who worked at De Lane Lea, Future Post, Videosonics and Pepper before going freelance, waved goodbye to the city a couple of years ago when he moved from Battery Studios in Willesden to a studio attached to his home in Woburn Sands. Since then, through his company Sonorous, he has mixed both series of Broadchurch (ITV) and From There To Here (BBC1) from home, and as a freelancer completed the pre-mix for Fortitude (Sky Atlantic) and Luther (BBC1) in his home studio.

Bargroff ’s standard approach for an hour of drama is to spend three days premixing at home, followed by two client-attended days at a dry-hire facility in London. That means he needs to keep his home set-up as up-to-date as possible so that it is compatible with other facilities (see box) and he can quickly pick up where he left off. For Capital, he completed the final mix with director Euros Lyn at the “excellent” Hackenbacker, which also provided the ADR and Foley.

 

“Dru Masters created a fantastic score and delivered it quite early so I had time to weave it and the music treatments in. That meant when I turned up with Euros at Hackenbacker on the first day, we could play the whole episode, so we had quite a lot of review time. I like the three-day premix because it means you can turn up with something cohesive. I try to protect that 3:2 approach; most jobs fit that template and clients are usually happy to accommodate it.”

CAPITAL KEY KIT

Bargroff’s home studio is equipped with Avid Pro Tools HDX2, an Icon D-Command 16-fader desk and PMC twotwo active monitors. Plug-ins are “industry standard”, including Audio Ease Altiverb, Waves WNS, iZotope RX, ReVibe and Speakerphone.

Icon-D-Command-16-fader-des_360

Projects are transferred between facilities using portable drives, with Cronosynch software to synchronise work completed at home with a transfer drive and a local drive in a dry-hire facility. “I keep the transfer drive synched to both ends so at any point during the job I have a mirror of the media in two locations, which is great for back-up. At the end of the job, everything is backed up to a Raid system for archiving.”

Bargroff also uses the Soundminer librarian program for managing his library of work. “It can scan multiple terabytes in a few hours and give you complete breakdown, or you can do a keyword search. All my libraries are well organised, but without that software I wouldn’t be able to find a thing.”

(The copyright to this content lies with Broadcast Online and is reproduced here under educational licence)
Original article is here http://www.broadcastnow.co.uk/techfacils/capital-bbc1/5097498.article

Adrian Bell – Feature Film and TV sound recordist

ADRIAN BELL

We get all the best visitors to Lincoln School of Film and Media.

This is Adrian Bell, a Film and TV recordist, just back from filming the new DaVinci Code film ‘Inferno’ directed by Ron Howard.
Check out his CV at www.adrianbell.net  He won a BAFTA for best sound in 2014 for Dancing on the Edge
adrian bell

He’s in town to be interviewed by BBC Radio Lincs tomorrow about the feature film ‘Everest’ he worked on earlier in 2014 Directed by Baltasar Kormakur, and yes I tool the oportunity of giving him a tour and collared him to come back and talk to students some time soon. He’s on BBC Lincolnshire sometime after 11.00am tomorrow

Adrian lives in London but is originally from Lincolnshire. He has a wealth of experience and is keen to come and contribute to the Audio Production course content if he can.

here’s a short photo slideshow

Lincoln Soundscape

Screen Shot 2014-09-03 at 11.16.30

Blog post by Alex O’Brien – Level 3 Audio Production.

For one of my final year projects I created a soundscape of Lincoln, which was showcased in The Little Red Gallery in the Bailgate. The concept behind this soundscape was to construct a sonic interpretation of 24 hours in Lincoln.
The piece is made up of 3 sections, Cathedral, Town and Evenings. Each section presents the listener with a different part of Lincoln. The recording process was very enjoyable but incredibly time consuming. For the Evening recordings I would often find myself sat in the cold recording traffic and such for hours on end or I would be in the Cathedral for hours at a time, listening closely for interesting sounds as they echoed around the Cathedral walls. I wanted to make sure I had every recording I needed to piece this 25 minute project together so I could layer these sounds up and paint an audio picture of Lincoln.
Having my piece showcased in a gallery was fantastic. Seeing people walk around the space, listening to my piece made me feel as if I had really accomplished something. But also working for a client was a great experience and it has definitely given me confidence in knowing that I can go out and get my work showcased.
Overall, the project process was a great learning curve and I’d love to do another sometime soon.

Listen to Alex’s Soundscape here:

Nightingale & Violin Duet – 90 Years Since 1st Outside Broadcast

Screen Shot 2014-05-18 at 18.33.53

90 years ago today (19th May), the BBC transmitted its very first outside broadcast and the stars of the show were cellist Beatrice Harrison accompanied by a very tuneful nightingale. To celebrate the anniversary, Senior Lecturer in radio Dylan Roys and I accepted the challenge from Reverend Mark Holden (Wragby parishes) to recreate the historic event in Lincolnshire.

Violinist Janet Welsh kindly agreed to join us in this quest and we headed off to Whisby Nature Reserve to recruit a willing nightingale to our band. As dusk fell, Graham Hopwood, our expert guide and one of Whisby’s wardens, attuned our ears to the song of the nightingale amongst the many other voices of blackcap, whitethroat and robin.

With our avian talent spotted and in full song, Janet began to play the tune Londonderry Air, just as Beatrice Harrison had done 90 years earlier. Dylan and I captured the beautiful duet with our modern hand-held recorders as trains whizzed by and jumbo jets cruised overhead.

The recordings reflect the massive changes in our sonic landscape since 1924. Graham chatted to us about the dwindling numbers of nightingales visiting Britain due to changes in their landscape and habitat too. However, the nightingale’s song remains as beautiful now as it did then.

Dylan Roys’s radio piece documenting the recording will be broadcast on May 19th on Siren FM. Listen here:

Listen to a clip of the recording here.

The piece was broadcast on William Wright’s BBC Radio Lincolnshire show on the 19th May 2014. Exactly 90 years after the original broadcast. Hear it here:

Huge thanks to Janet and Graham for helping to make this happen.

Screen Shot 2014-05-18 at 18.34.14

David Attenborough: My Life in Sound

http://www.bbc.co.uk/programmes/b03lnzxh

MONDAY 16th December 2013 at 11.00am on BBC Radio 4

In an exclusive interview for Radio 4 David Attenborough talks to Chris Watson about his life in sound.

One of Sir David’s first jobs in natural history film making was as a wildlife sound recordist. Recorded in Qatar, David Attenborough is with wildlife sound recordist Chris Watson, there to make a film about a group of birds he is passionate about, The Bird of Paradise. It is in Qatar where the worlds largest captive breeding population is and it is in this setting Chris Watson takes Sir David back to the 1950’s and his early recording escapades, right through to today where David Attenborough narrates a series of Tweet of the Day’s on Radio 4 across the Christmas and New Year period

Guest Lecture – Mike Harding – Touch

Screen Shot 2013-10-22 at 12.11.31

Blog post by Senior Lecturer Dr Dean Lockwood.

On Friday 18th October, Mike Harding, founder and supremo of Touch, came to LSM to talk to Audio Production students taking the critical studies module, Auditory Culture. Given that some of the key concerns of the module are debates around the concepts of noise and the soundscape, it was a great opportunity to talk to someone intimately involved with a label which has specialized in promoting artists exploring precisely these areas. As quickly became clear, Touch has a philosophical orientation which propels it way beyond the narrow exigencies of the music industry. Touch has always been conceived as an art project rather than simply a label. Because of its obsessively experimental ethos, it has survived pretty much on its own terms and has never fit well with the complacent mainstream and its genre categories. As Mike explained, Touch was established in the early eighties in the wake of punk. Capitalizing on the energies generated by the so-called ‘New Wave’ independent scene, Touch was a key post-punk project, its first releases heavily involved in early cassette culture and the ‘mixtape’ phenomenon. With cassette magazines such as Feature Mist and Ritual: Magnetic North, Touch presented sophisticated cut-ups and powerful work by bands such as New Order, Einstürzende Neubauten and Cabaret Voltaire, as well as musics from around the world (before such a thing as ‘World Music’ existed). ‘No one ever said no’, which stands as a great testament to the label’s reputation and integrity. Mike took us, in the first part of his talk, through the early history of Touch, spicing things up with personal anecdotes, and in the second part addressed Touch’s present concerns. It is the home of artists such as Christian Fennesz, Bruce Gilbert, Ryoji Ikeda and Chris Watson. Mike played us a good selection of pieces which some of these artists have put out on Touch. These artists have in common, I would suggest, what we might term an ecological sensibility, a particular attention to the relations which comprise acoustic space, sometimes through glitch aesthetics, sometimes through field recordings or other means.

On Saturday 19th, Touch presented two world premieres at Lincoln Cathedral as part of the Frequency festival. The evening, after Mike’s introduction, commenced with Anna Von Hausswolff’s performance of an austere, resonant new score for the organ, titled Källan. Chris Watson and Hildur Guðnadóttir then presented a stunning new collaborative multi-channel sound work, titled Sönghellir (The Cave of Song), which I think captivated everyone present. Touch’s website describes the work as ‘a sound journey from under the waters of Faxafloi, Iceland, alongside some of the largest animals on the planet. Up, onto the lava beach, across the lava fields and reindeer moss to the foot of the snow mountain, Snaefellsnes. The journey continues up and then into the mountain, ending inside Sönghellir, the song cave…’ It was a perfect example of the art of acoustic space that Touch releases exemplify.

Screen Shot 2013-10-22 at 12.52.38

Guest Lecture – Ken Blair

Screen Shot 2013-01-28 at 15.07.22

Today’s level 3 project guest lecture was given by recording engineer Ken Blair. Ken is a freelance sound recordist who’s company BMP Recording specialise in classical, jazz and acoustic music.

Ken came to talk about his typical week of recording, editing, mixing and filling in tax returns! He described a typical orchestral recording session and how it can differ from a pop/rock recording session, in that a lot of these sessions are still recorded straight to stereo – especially if its a live event recording. This means a lot of time is spent positioning microphones and balancing levels into the recording device. This also means there’s no room for error, both in terms of the musicians’ performance and in terms of the recording levels and mix balance. Nerve wracking stuff!

Ken also talked about how his background and recording experience led him to the place he’s at now. After leaving school in Scotland, Ken studied the Tonmeister course at the University of Surrey and spent a year on work placement at Air studios in Montserrat. He also gave our students some great advice regarding building their portfolios and creating a skills based CV.

Many of our students will go on to be freelance workers across the very broad range of audio production careers. Ken’s lecture was a great insight into the day to day activities of just one of these fields. Really useful stuff!

Guest Lecture – Jez Riley French

I was really pleased to welcome back Jez Riley French for this month’s guest lecture.

Jez is a field-recordist, sound artist and sonic experimenter and I invited him to talk to our L3 project students about his varied and interesting work. Jez specialises in recording hidden sonic worlds such as building structures, underwater environments and the micro perspective of the insect world.

We had great fun discussing and testing Jez’s (often self-made) kit such as hyrophones, geophones, contact and parabolic microphones.

Jez has two exciting projects coming up; a field-recording trip to Iceland with Chris Watson and a Tate Modern commission: audible silence: a headphone piece exploring the hidden sounds of the Tate modern building (february 2013). We’re also trying to arrange a field-recording trip around Lincoln for AP students – watch this space!