a game:

Learning Machine

A game of charades based on machine error.
played by Anna Smeragliuolo and Mantas Lilis

This is a project about computer perception.
About understanding that machines do not operate without error, and in order for them to bend to our wills, we must also bend a little bit to theirs. 

The game is based on an algorithm called im2txt. It is able to analyze pieces of video, and generate live captions describing what is going on in the scene.

here’s the game:
The machine generates the caption for a player to act it out.
The player tries to act out the caption in front of the camera.
After a success, the machine will generate a new caption.

here’s the process:

To test the limits of im2txt, we took a computer for a walk outside to see how it would perform. We noticed some curious errors and misidentifications. What was it about input that caused it to mislabel the scene?

Sometimes the errors made sense according to human perception - legs as scissors, thick paint as cake, stripes as zebras - other times not so much.  

here’s a bit about representational representations:

Most interestingly, the algorithm can recognize content within an identified object, e.g. “A black and white photo of a woman holding a cell phone,” or “A man taking a selfie in the mirror.”

We wanted to know what properties of an image caused the software to make this distinction between reality and representation.

here’s the kicker:

We tried to trick the algorithm into repeating its own misidentifications; it felt like playing a game of charades.

Performing to make it understand, we no longer had taught the machine. The machine had begun to teach us.