Mantas Lilis

Interaction Design, Machine Learning, RunwayML - Playing With The Machine

Year: 2018
Project Type: Educational
Institution: CIID


︎ CIID | Interaction Design Programme’18

Role: Interaction Designer
Team: Anna Smeragliuolo

The project is created at the Copenhagen Institute of Interaction Design (CIID) during the Introduction to Machine Learning workshop with Andreas Refsgaard and Gene Kogan.

The objective of the workshop was to explore different ways of using machine learning tools in the design process. Within the team, we decided to work with Runway, the software that enables to run pre-trained models like im2txt, OpenPose, YOLO, GazeCapture directly on your computer. 

Playing charades with a machine

Learning Machine Learning invites people to play charades-like game with the machine.

The machine observes the current environment through a built-in camera and displays a generated caption for humans to act out. The players are told to act like “a group of people sitting on a couch playing video games,” and they then try to replicate the scenario until the machine can regenerate that caption.

As you play more and more, you start to find tricks to fool the machine. As you scramble to find the required props you learn to use substitutes: a ruler as a tie, a striped shirt as a zebra. These tricks reveal subjectivity and information about the dataset and the types of images within its classifications.

“a man in a tie and glasses is smiling” 

Experience In Detail

The machine generates the caption for a player to act it out. 

The player tries to act out the caption in front of the camera.

After the success, machine will generate a new caption.

Behind The Scene

A current prototype of Learning Machine Learning is built in Processing. To enable the game with machine learning capabilities we used pre-trained im2txt model developed by Google. The model runs in Runway and parses the generated caption to Processing as an OSC message.

Processing & Runway


We had a very short time to understand the capabilities and limitations of the model we are using for our project. To gain that understanding and get more inspiration on what we can do with it we tested the limits of its knowledge by taking it outside for a walk around the streets of Copenhagen. We found it was quite skilled at identifying larger-scale scenes, like “a bus parked on the side of a street.”

In more specific environments, like a drug store or art gallery, it tends to misinterpret items, describing pantyhose packaging as scissors or paintings as birthday cakes. Most interestingly, the algorithm can recognize content within an identified object, e.g. “A black and white photo of a woman holding a cell phone.” But it is not flawless in this determination and sometimes will label a scene like “a woman reading a book” as “a photo of a woman reading a book.”

We wanted to know what properties of an image caused the software to make this distinction between reality and representation. We tried to show the computer magazines, postcards, still images, videos of ourselves holding printed images, and even printed images of images, but we still couldn’t figure it out. However, this exploration also revealed even more surprisingly specific distinctions the algorithm could make, like “a fashion magazine” from “a magazine.”

As a result of these attempts, we found the computer also made unexpected, sometimes hilarious errors. We kept trying to trick the camera into repeating its own misidentifications and realized it was like playing a game of charades with a machine.

But instead of the machine performing for us, suddenly we were performing for it.