︎



hmu

asmeragliuolo@gmail.com

a game of AI charades:

Learning Machine Learning


A game of charades based on machine error.
played by Anna Smeragliuolo and Mantas Lilis


This is a project about computer perception.
About understanding that machines do not operate without error, and in order for them to bend to our wills, we must also bend a little bit to theirs. 


The game is based on an algorithm called im2txt. It is able to analyze pieces of video, and generate live captions describing what is going on in the scene.


here’s the game:

The machine generates the caption for a player to act it out.
The player tries to act out the caption in front of the camera.
After a success, the machine will generate a new caption.


here’s the process:

To test the limits of im2txt, we took a computer for a walk outside to see how it would perform. We noticed some curious errors and misidentifications. What was it about input that caused it to mislabel the scene?


We tried to trick the algorithm into repeating its own misidentifications, and we realized it felt like playing a game of charades.
Sometimes the errors made sense according to human perception - legs as scissors, thick paint as cake, stripes as zebras - other times not so much.  




here’s the kicker:

As we perform to make the computer understand, we bend to its will -- trying to adjust our behavior to translate its perception into something we can understand. In trying to replicate its point of view, we are no longer teaching the machine; the machine has begun to teach us.