Halsey Burgund: I am speaking today with Michael Epstein of Walking Cinema. Our paths crossed a few months ago back in Cambridge, MA at the Media in Transition 10 Conference at MIT where I was doing a performance and Michael was moderating a conversation called Journalism, News, and Civic Participation. He attended our event called Democracy Performed and had some fascinating questions about audio in space and how we were thinking about moving bits of audio around the room.

So I wanted to talk further with Michael to learn more about what he is doing with Walking Cinema and how he is using various audio AR techniques to accomplish narrative, journalistic and documentary goals. The first question I wanted to ask you is a high level one: Do you have a definition of audio AR?

Audio is the most visual medium. – Ira Glass

Michael Epstein: No, I don’t really have a strict definition of what it is. And I think we had a discussion before about how I might not fit into the interactive audio category of audio AR in that most of the audio that I produce is linear and it can be triggered by your GPS location, but it’s not really reactive to the environment. But I do think, and this is an Ira Glass quote, that “Audio is the most visual medium” in that when you’re listening to a really good audio story, your mind gets triggered and starts visualizing what’s going on. That to me has really been a strong thread for the kind of productions that I do and in the richness of the sound bed that we try to create. In outdoor media projects, we really try to use audio to make the environment feel complicit with the story that’s being told. And I will go more into that later, but I think that’s really my general definition of how I work with audio and in the real world.

HB: Yeah, that’s great. I’ve been having a hard time specifically defining audio AR and I do think that the interactivity is often times a part of it. I would argue that a lot of your work that I’ve become aware of does have interactivity in the sense that audio is location-triggered and related to that specific location. There is something simply with geo-located audio that I think falls into the category of audio AR. It’s not quite as in depth as directional audio, perhaps, which is another possible layer, but there is a direct interaction between the listener and their environment.

When we emailed back and forth you used the phrase “AR audio” and I always call it “audio AR” and I think those are actually different things. A lot of your work seems to be audio which I would define more as audio that enhances a visual AR experience versus audio that is itself the primary content of an augmented experience. Am I getting that right in terms of your work?

ME: Yeah, I mean, I’d say it’s more of audio plus AR, is the way define it. And the AR being a visual overlay to your camera’s view finder that either has its own audio associated with it or works with the audio soundtrack that’s playing during the experience. So I put a little plus sign in there. But AR audio also fits what we’re doing. I think what’s interesting about this moment in what you’re doing with AudioAR.org and in these discussions is that nobody’s really great from what I’ve seen with fusing an environment, especially like a real live moving environment, with augmented reality, especially with putting something visual on top of your real world surroundings.

I’m sure you’re familiar with some projects where it’s like, wow, you can do this really cool augmented reality projection onto your living room table and you’ll see these little characters running around and you’ll experience a piece of history or an animation or a cartoon. You know what I’m talking about; the kind of standard, Magic Leap demos. And it’s just your actual living room, or possibly where you might actually be standing in front of a historical site, but either way it tends to be a really clunky integration of the real and the overlay. And so I think whether you call it “audio AR” or “AR audio”, I think the main challenge is how is that audio working in a specific environment?

And with your work, with discussions around democracy and things like that, it was really cool how you hijacked that classroom. You did quite a bit of prep to make that space work for you in terms of  the way that we’d hear the sound. But also I think in some ways, to almost plot out where certain characters or questions or voices could be heard and how they could be spatially set in that room to really work. And it’s the same thing in my work when I’m saying, okay, well now we’re going to have a story being told in these specific places. I’m thinking about, what does it feel like for somebody to be in that place? Is that an intimate place? Is it wide open space, and also what kind of serendipitous things might happen in that space that we could integrate into the story?

Audio AR for Narrative

What you start doing is making people aware – when they move through the world, when they look at things and start thinking more critically – even if it isn’t specifically part of the production.

So really what is the maximum for our storytelling is when the whole world seems like it’s in cahoots with what’s being told rather than as a kind of stage that’s been taken hostage for your little show to go on. And that to me is really important with nonfiction; is that you’re trying to get, at least with Walking Cinema productions, the idea that there are very important stories to be told in history and in current events, and if people can feel like those things are not remote – they’re not something that’s in a show or on TV, they’re things that are in the world around you – what you start doing is making people more aware when they move through the world, when they look at things and they start thinking more critically. Even if it isn’t specifically part of the production, the production attunes you to that fact that we’re not so distant from the things that we care about or the pieces of history that we want to learn more about.

So really what is the maximum for our storytelling is when the whole world seems like it’s in cahoots with what’s being told rather than as a kind of stage that’s been taken hostage for your little show to go on.

HB: That’s really interesting to hear you say about that about stuff that happens in the real world around the people who are experiencing your pieces. You can’t control that. Obviously there’s this integration with the real world, with location-based controls and perhaps some directional controls, but you don’t have control over what cars drive by and what other stuff happens. But therein lies the excitement of building something that integrates with a kind of diversity of possibilities that might happen. I know I try to build in those “error bars” and put them into the composition process.

I love what you said about this notion of the world being in cahoots with your personal experience at the time. And I think that’s a really beautiful way of thinking about it. It’s interesting because a lot of times people use audio – and I think this is a very wonderful use of audio as well – but, they use audio to sort of pull, to sort of create a place to make you transport from wherever you actually are to somewhere else. Like, oh, there’s the beach, I’m hearing the wind and the waves or whatever. That can be very effective, of course. I think that gets back to the Ira Glass quote a little bit, but it’s interesting because I think what you and I both do is figure out how not to locationally transport people, but rather how to enhance the physical nature of where they are at a particular moment.

ME: And I think one of the surprising techniques in making audio more immersive is thinning it out in certain places. Rather than having that urge to create an entire environment with the audio. If you’re in a very active and activated environment and do something really minimal with just a synth, you can bring that space more alive by doing less with the audio, by letting it just be sort of a mood bed rather than information or a highly dense composition. And I think there are moments for both in what we produce. We definitely have a story to tell. We definitely need to do narrative. But what we work on a lot after we get the story down is kind of ripping open of this story and creating these gaps where people can just be walking through space and letting some of what they’ve heard sink in; or letting the environment speak to the story rather than the actual audio.

HB: Oh, I love that. That’s the interstitial moments; like you said, the walking between or transitional ones. Can you give an example maybe in Museum of the Hidden City or another project where you’ve had to really think about where the moments are where less is the right approach and where more is the right approach?

And so the discovery of this jewel right in the middle of the city and being able to walk through that is really important. It’s also important to give people space as we complicate that environment with the story of what used to be there, how it used to be a thriving, multi-generational African American community.

ME: Yeah, well one moment comes when audiences walk through a housing development that was designed after Frank Lloyd Wright, that has this beautiful cedar lined lane with a series of fountains sort of bubbling in the middle of the sidewalk. Partly what we’re trying to show is that the designers wanted to create the tranquility of the suburbs right in the middle of the city. That it was like this big coup that they’d bring back white collar workers to the city by creating a semi-suburban environment. So letting people enjoy the peace and the surprise of this place that a lot of native San Franciscans didn’t even know existed. And so the discovery of this jewel right in the middle of the city and being able to walk through, that is really important. It’s also important to give people space as we complicate that environment with the story of what used to be there, how it used to be a thriving, multi-generational African American community. So those two contrasts are, I think, both accentuated by the sound of the fountains. We do have some fountain in our actual sound bed, but we also leave a lot of space for just the real fountains to talk and respond to the stories that are being told.

HB: Right. And the real fountains are “talking” perhaps in a somewhat unpredictable way? I don’t know if there’s a schedule for the fountains or if there are times when the fountains are off?

ME: They are pretty much on all the time. The other thing too is that it’s a story of a failed utopia. And part of that Utopian thought was that people didn’t want to be on noisy, busy, congested streets. They wanted to be in these idyllic green spaces. You’ve probably heard the reference of Towers in the Park, which was what these projects initially were designed as. Oh, wow, we can go vertical and build a lot of housing units in these high towers and then around them we’ll have this park-like space where people will play. And it just didn’t work that way because for people to go to spaces, they actually need to buy stuff or have commerce going on and traffic actually helps add a little more mood and life. So these spaces that they created, these beautiful green spaces right in the middle of the city, are empty. I’ve walked this route a hundred times; they’re always empty. And so allowing that emptiness to also penetrate the soundtrack is important too. You could say it’s really empty and nobody uses it, but if you just use some silence and ask people to look around them and see what’s happening and 99% of the time it’s nothing? That is so much more effective.

HB: So you actually use silence. Do you bring it down to some very mundane drone or very slight sound? Or do you actually just pull it out completely to silence sometimes? I imagine that’s quite an effect. When you’ve been immersed by audio from this experience for a while, then to pull it all out.

ME: We never pull it all out. Though talking to you, I think that would be a good idea.

HB: Yeah. Who knows? It might be. It’s all about experimenting. That’s really cool.

Bringing Audio AR Off-Site

HB: One of the things that I was interested in with your Museum of the Hidden City app was that I experienced it when I was not in San Francisco. I didn’t have the pleasure of walking around with these fountains and all the other locations. But your app still does communicate certain bits of content and it still does some of the visual AR. I’m wondering if that must’ve taken a lot of deep thought about how to, and whether to, allow for this because accessibility is so important, right? You don’t want to build all this wonderful content that  people have to fly to San Francisco to experience, but you do want to have people who are there be able to get this really extra amazing sort of confluence of physicality and audio. Can you walk me through a little bit of your thought process there? 

That’s really cool. One of the things that I was interested in with your Museum of the Hidden City app was that I experienced it when I was not in San Francisco. I didn’t have the pleasure of walking around with these fountains and all the other locations. But your app still does communicate certain bits of content and it still does some of the visual AR. I’m wondering if that must’ve taken a lot of deep thought about how to, and whether to, allow for this because accessibility is so important, right? You don’t want to build all this wonderful content that  people have to fly to San Francisco to experience, but you do want to have people who are there be able to get this really extra amazing sort of confluence of physicality and audio. Can you walk me through a little bit of your thought process there? 

ME: Yeah, it’s really a big challenge and it’s a really important one because if there is a fairly simple way to create a remote experience of location-based audio or location-based AR, that’s huge. That’s what broadcast media is all about, broadcasting all over the world, right? And it’s easy to do that now, but it’s not easy necessarily to retrofit something that is specifically built to a site to be experienced off site. So there are three ideas that we have around this. We got funding just to do the location based version, but I do think in the next couple of months you’re going to see a non location-based version of it. And the simplest one is using any kind of 360 video platform, be it YouTube or some other platform, where we basically film the entire walk in 360 so that somebody who is listening remotely, either through the app or online, can hear the story, see the augmented reality layered on top of this video, and basically do their own armchair walkthrough of the experience. Not great, not ideal, but probably better than what you got just listening to the audio and maybe looking at some pictures of the site as you’re sitting at home. 

HB: So would that be a fixed video, essentially a 360 video, but a fixed one? Or would that have the ability for the user to pause in certain areas as they would if they were walking through physical space?

ME: I think it would be scenes where actually people have 360 degree stills, that maybe have some movement to them, and they’d basically be looking around at the different stops and be able to hear what’s going on and maybe be able to move through like a church or down a street, things like that. Some people really enjoy it. I’ve done it with other projects and the augmented reality is really cool.

HB: Totally. And with 360 stills, the resolution is so much better than the video that in some ways a 360 still with audio that “moves” is way more interesting than the distraction that can happen with lots of moving parts in video.

Complementary Formats

ME: I think the other one is a transmedia approach and I don’t know, do you still use that term transmedia?

HB: Oh gosh, I don’t know. Maybe that makes me sound old to use that! I don’t know exactly.

ME: Well you sound old and nobody’s quite sure what you’re talking about anymore, basically. Good transmedia doesn’t repeat a story from one platform to another. So it’s not like you’re just re-versioning, you’re actually extending the story world. And so the other possibility, and this is something we’ve discussed with Audible, is doing a podcast or audio book version of the story that you heard, that’s site-based but really digs into a couple of the characters that are mentioned there in the site-based experience and go deeper with them. Specifically the character of Justin Herman is really interesting, and in the on-site version, he pops up as this well-intentioned, but tone deaf, developer who really just wants to turn the Fillmore into middle and upper middle class housing and screw over the African American population.

But his story is much more complicated than that. We weren’t able to get into it in the walkable version. So something like a podcast is a chance to really go into this complex character; a character that’s still being grappled with in San Francisco 50 years later. In fact, they just changed the name last year of Justin Herman Plaza to Embarcadero Plaza as part of a protest against what he did 50 years ago. So his story is still being debated and I think there’s something very topical that could be done on him.

So it’s not like you’re just reversioning, you’re actually extending the story world.

There’s also the character of Wilbur Hamilton, who sadly just recently passed away. A retrospective of his perspective and his relationship with the man who turned into his brother-in-law is a fascinating story too. These are two African American, civil rights leaders in their communities, but they stand completely on opposite sides of the fence with regard to a redevelopment of the Fillmore neighborhood.

HB: Wow. So you basically you have to do a marathon if you wanted to locationally experience the depth of these two stories. So the notion of bringing those into a more narrative form, i.e a podcast made it all make sense. 

ME: Yeah. And I think people should get mad about this statement, but I think it’s very hard right now to tell complex, nuanced stories that need quite a bit of framing in a medium where many of your users are just doing it for the first time and trying to figure it out. So those are some of the reasons we shied away from the gray areas of the characters.

HB: I think about that so much. I build apps for my work as well and I always go under the assumption that there is no such thing as a “power user”. Everybody will be using this for the first time. Just assume that and present things as simply as possible, both functionally and from a content perspective. My work is probably generally a little more obtuse than some of  what I’ve experienced of yours, but I think that’s pretty smart. Hopefully, maybe in a few more years, people will get more comfortable with audio AR generally.

People are certainly getting more comfortable with podcasting but that format is, as you know, not all that different from broadcast radio of course, other than the asynchronicity of it. But there’s something new about the podcast players that people seem to just take a little while to get used. I was actually listening to a podcast yesterday, which was a discussion between, Michael Lewis, the author and Malcolm Gladwell. They went into this big discussion, which I found fascinating, about how there are some stories that really make sense to tell in a podcast and could not be told in a written piece on the written page, and others that are vice versa, that can absolutely be told on the written page very effectively, but would not work at all on a podcast.

Against the Rules: Bonus Live Episode: Michael Lewis and Malcolm Gladwell

Being authors who are used to the written page and then getting into podcasts, they had this interesting discussion about what those differences were. And it had a lot to do with character and how in some cases, during a podcast, you can just let an interesting character, someone with an interesting voice and style, just be recorded. Whereas trying to bring those characteristics out in words that aren’t spoken by the character can be very difficult even for them, both clearly very accomplished writers.

I think that speaks to the transmedia approach; some formats are much more effective for certain things and not for others. For me, I like to think about it from the start of a project to the extent possible, you know, given funding and all the other restrictions that exist in the real world, so then you can design it with that in mind rather than – I think you used the word “retrofit” a little while back – retrofitting which is always problematic, at least in my world. I understand the desire and I understand the tendency to want to; if you do something successful in one medium, you want to make it more accessible to more people. It’s like, oh well we could just take this and do that with it or whatever. But I think what you’re saying is yes, you can, but then it has to be different too. There has to be some kind of additional pay off taking advantage of the new media that you’re transferring into because they all have their own benefits.

ME: Yeah. Well, one medium that also seems to be a good fit for this particular story is a live documentary like Sam Green‘s work. What Sam Green does is he makes a documentary film that audiences watch in a movie theater, but the film is designed to be narrated live, to have live music accompaniment and sometimes other additional live experiences on the stage in front of the movie screen. And I actually think that creating a film, first out of just the footage from the neighborhood, the place itself, but then also some of the historic video that we found, could work really well in the movie theater experience. People would want to watch that film. But what’s great is the talent that we have doing the voicing for the project, our two young dynamic spoken word poets. So them actually performing original spoken word poetry that they composed for this piece that plays out in the site-based version could work very well as a live theatrical version.

Also the original score was done by a producer who does quite a few live performances himself. So there is the possibility that this could be fairly easily converted into a live documentary experience.

HB: That is really interesting. What about immersive theater? I mean, there’s all these projects where you’re walking through the street and then real actors come up to you and interact to convey a narrative. This of course starts narrowing down the accessibility in an extreme way, which is always unfortunate, but can be a remarkable experience.

ME: It’s weird. We’re on the same wavelength here. I think immersive theater takes a lot of work to do it well. Though I’ve experienced things that look like they weren’t workshopped for too long and came out pretty well, but if you really want to do it right, I think you need to spend quite a bit of time getting the site for the immersive theater ready. And one site that appeared to be a great piece of the story that we couldn’t really get into in the walkable version is the story of Jimbo’s Bop City and Waffle House, which was one of about 30 jazz clubs that ran up and down Fillmore Street. They were just dying out at the point where the story that we’re telling starts in the early sixties.

And these clubs were interesting in a few ways. One was that they usually opened around midnight because they featured Black musicians who would go and play in the white clubs in downtown San Francisco and then do a second show late at night at a jazz club for their Black audiences on Fillmore Street. They played from say midnight to 6:00 AM. One of the best venues was a place called Jimbo’s Bop City. They literally served fried chicken and waffles at a small lunch counter. And then in another room (always decorated for Christmas for some reason) was a small stage with seating for about 20 or 30 people. There was a house band that played with the visiting musicians but it was really competitive. If you weren’t with the beat, Jimbo would come over and take you off stage.

HB: Well, you gotta be off the beat just the right amount, right?

And so the place became important because it was supposed to be torn down as part of redevelopment. But it was one of the Victorian buildings that was literally lifted up, put onto a giant platform and moved down the street out of the way of the wrecking ball. And so it ties in to the story that we’re talking about. The destruction and the attempted salvation of the culture and the neighborhood.

ME: Yeah, right, you gotta be in between the notes not off the notes. And so the place became important because it was supposed to be torn down as part of redevelopment, but it was one of the Victorian buildings that was literally lifted up, put onto a giant platform, and moved down the street out of the way of the wrecking ball. So it ties into the story that we’re talking about, the destruction and the attempted salvation of the culture and the neighborhood. And you could imagine that recreating Jimbo’s Bop city in some sort of space in San Francisco, where you’re not just experiencing a show of live jazz and maybe food and drink, but you’re also getting hints, a la Sleep No More, where you’ve got a bunch of documents kind of strewn around the place of who’s who and who’s trying to do what.

The club is facing trouble by both redevelopment and by the decline of the African American population in the neighborhood. Many of them were leaving San Francisco because the jobs were drying up after the war-time industries began to ramp down. So there’s a big opportunity to do something like that too. I’d love to do that. But you know, just getting one platform done at a time is hard enough!

HB: That sounds so amazing. What I really like about those sorts of productions is when you come out of something like that, you’re done with it, so to speak, but you’re totally not done with it. The whole world-building that happened within the immersive performance, as soon as you go back to the world that hasn’t been built for you, it still feels like it has been. You’re left with this ability to notice more and to think about things more closely and with a new perspective. And I think that’s what makes art so great; those kinds of experiences have the ability to stick and can be super, super powerful if done right. But like you said, this is hard stuff.

So how might journalism make use of audio AR? 

ME: What interests me in journalism is that there are some big stories that I think audiences can feel detached from just because the solution seems so far away. You could talk about environmental crises this way, or the housing crisis. A lot of readers and people who follow the news are aware of these issues, but the solution seems so distant that they disengage. I think there’s an opportunity with location-based AR and location-based audio to give people a physical and first-person experience of an issue right in front of them. And in some ways it’s exciting and it’s deep and it’s cool to get immersed in that issue without feeling a weight of “Oh my God, I have to slog through a long form article on all the problems that are happening with the environment to be aware of it”. And then step away from it more depressed than feeling like there are solutions. I think that there’s opportunity to do something like that in journalism with big stories. Again, you run into the problem of to what degree is it site-based or isn’t it site-based? Are there ways you could do it in your own living room? That would be interesting. Those are all questions still to be experimented with, but I think this idea of journalism taking on big issues and using new platforms that can draw audiences is promising.

HB: Definitely. And I think the absolute versus relative location is one of these big sorts of questions that you just hit on. Is there a way to translate a location-based experience to a location wherever the listener wants to be and still be meaningful? I’ve been doing a lot of experimenting and thinking about that. I think there are fruitful ways of progressing down this line that may yield really interesting stuff.

ME: One example – this came from our proposal we started for a large utility company in Massachusetts a few years back – the idea was they send out a reminder every winter of ways that you can insulate your house. That could easily be a mini AR documentary that runs in your home. Where can you find a window? Can you find a door? And then one could layer this story on top of those household objects which moves you into this process of properly insulating your home. Something like the environment and recycling and what you can do and where greenhouse gases come from; all that could also probably fit into anyone’s home. Similarly, if you’re going to do an exposé on a company like Walmart, audiences could go to any Walmart and probably find similar objects and spaces that could be designed for AR.

HB: I was thinking about this idea as a piece on sports; go to a baseball field or something like that and press play when you’re at home plate. I think there are certain physical objects or physical spaces that are stamped out in our current society that could be used to tell a more general story in a more ubiquitous and accessible, yet still place-aware way.

One final question, can you let us know where best to find more information about your work and experience some of your projects?

ME: The website is walkingcinema.org. There you can see past projects and current projects there. The project we discussed in San Francisco, Museum of the Hidden City, is www.seehidden.city. I also have some non-interactive work up on Audible and Amazon Prime.  

HB: And are there other apps that are downloadable that people can access?

ME: At this point, that’s the only one we have in the App Store. There are two other projects in the Boston area we are releasing later this year, but again, if you just look for Walking Cinema on the App Store, you’ll find it. And then in the Android Play Store, Museum of the Hidden City should be coming out in September of 2019.