Q: For Computers Watching Movies, you developed the concept, built the software system, and produced the visual sketches. What was the most satisfying part of the process for you? What was the most difficult?
The most satisfying part of the project is getting to see these movies from a different perspective. The clips I chose are all from movies I admire and know well; yet when I watch how the computer sees them, my own experience of the movies is changed. I see things I didn’t see, pay attention to transitions and framings I wasn’t focused on, and make new connections between the aural and visual material each film presents.
I suppose the most difficult part of the process may have been coming up with the idea in the first place. Not that this seemed difficult when I did conceive the work, but it took my entire life up until that point to lead me there. However, thinking more about the creative process of making the work, I’d say one of the more difficult parts was in selecting the clips. I tried many more clips than are seen in the final video. Interestingly, some clips I thought would produce something dramatic didn’t produce much at all. In other words, some of what I thought was interesting the computer did not.
Q: How does the computer “see” the movie scenes? Can the computer make viewing choices while it is “watching” a clip? If so, what type of choices?
Where the computer “looks” is determined by simple artificial intelligence algorithms that give the system some degree of agency. It uses this ability to discern and look for areas of prominence. In some cases, this means it will use face detection techniques to find and track faces across the frame (the clip from Taxi Driver is a good example). In others clips, faces aren’t prominent, and so the computer looks for other patterns/forms that suggest importance (the bag in American Beauty or the exploding building bits in Inception are good examples). Prioritisation derives from the search for prominence: items deemed important are looked at, the rest is ignored.
I would suggest that these methods the software employs—namely, the search for prominence based on pattern—is not that dissimilar to our core vision-based methods for scanning a film (or any kind of scene). Yet we don’t see things the way the computer does, so what’s the difference? I contend that one thing this work reveals is how our vision of film is culturally developed. We draw on our understandings of narrative and our famililarity with conventions of filmmaking/cinematography just as much as we do our core vision capabilities. Without our cultural understandings, perhaps we’d see things more like the computer does.
Q: What level of control did you have over the sketching process? Was the final outcome expected or a surprise?
My primary methods of control over the sketching process is through the design of the algorithms for drawing (e.g. thickness/opacity of lines, how colors are chosen) and “looking” (what does it look at and how does it decide), and in the selection of the movies the computer gets to watch. However, while those methods produce much of the system’s potential, the output it creates can be quite varied depending on what the computer chooses to do.
I had general ideas of what this work might produce, but I was surprised by a number of aspects.
For example, I wasn’t expecting the output to so clearly reveal differences in style between genres, eras, and directors. The more recent sci-fi clips show lots of rapid visual changes—for example, actors and objects are often moving quickly (or get moved quickly through frequent camera angle changes) in The Matrix and Inception. Yet the older clips, such as 2001, Annie Hall, and Taxi Driver, represent a different approach to film making, one with much fewer camera changes and much less active movement on screen. These don’t operate in isolation, however. I think the drawings lead us to infer differences in director style from, say, American Beauty and 2001 vs. Inception and The Matrix. In other words, it’s not just era that is revealed here, but compositional intent.
From an aesthetics point of view, I’m most struck by the drawing process of American Beauty, how it begins to mimic—through density of color, thickness of line, and placement of mark—the output of artists who draw by hand.
From a human vision vs. computer vision point of view, I think the results from Inception most clearly illustrate how humans can and do easily consolidate rapid changes in their visual field into singular events. The computer sees the explosions halfway through that clip as a series of small rapid changes, so many that its representation of those changes obliterates the drawings it created before that point. But when watching that clip myself, I watch mostly the origins of the explosions and as much as anything else, focus on those aspects of the frame that aren’t moving. I’m left wondering why I and the computer see things so differently?
Q: Computers Watching Movies includes selected scenes from: 2001: A Space Odyssey, American Beauty, Inception, Taxi Driver, The Matrix, and Annie Hall. Why did you choose these particular movies and scenes? Is there a connection between these six selections?
A practical consideration was that I wanted a chance for all viewers of the work to have at least one movie they had seen amongst the sources. So this is one reason the clips are varied in genre and age yet drawn from popular films.
Another reason for the variance in age of the clips is that I was interested to see how changes in film composition and direction over time might be revealed through the computer’s ways of seeing. If you compare the system’s output from watching Taxi Driver and Annie Hall—the two oldest films—to what it produces in response to Inception and The Matrix, you start to see radical shifts in in density, perspective, camera movement, and more. In this way, the range of clips, as seen through the system, start to illustrate changes in directorial style.
Finally, all of the clips I chose are in some way related to how we see and are seen, often in relation to machines and systems (broadly defined).
Q: In this work, you remove a movie scene and replace it with the computer’s visual illustration of the same scene. The original audio remains completely intact and anchors the viewer’s experience. How does this crucial audio element affect the viewer’s interaction with the work?
An important part of the experience of viewing Computers Watching Movies is that moment when the viewer draws a connection between what they see the system focusing on and their own visual memory of the same clip. It is at this moment that the viewer can most clearly draw personal comparisons between how they see vs. how the computer sees. For this experience to happen at all, the viewer needs some sensory clue that ties the abstract visuals they’re watching to their temporal memory of the original film.
Q: What fundamental questions do you hope to answer through your artistic investigations?
We now live in a world run by software. For example, software filters our social media news feeds, determines if we get a loan, tells us what is relevant and available on the web, keeps our refrigerator up to date, guides the planes we fly on, and helps us find our life partners. Despite such broad impacts, many still think of software as neutral. Yet all software is designed by humans, and as such, reflect and project the biases and ideologies of those who write, fund, and direct its production. I create interactive machines, systems, and interventions that investigate how software functions in culture, challenging this assumption of neutrality. How does an interface that foregrounds our friend count change our conceptions of friendship? What does it mean for human creativity when a computer can make its own artworks? In what ways are computational surveillance systems altering how and with whom we communicate? These types of questions help elicit the role of software in culture and software as culture, all of which ultimately helps us better understand ourselves.