The Treachery of Avatars

About The Artwork

Inspired by the 1929 painting, The Treachery of Images, by René Magritte, this coding final project aims to elicit the same message that René conveyed regarding the deceptiveness of representation.

With the advent of the avatar, we discover a rare opportunity for the reinvention of 'self'. No longer bound to our physical vessel, we are invited to project imaginative boundaries, or reflect and re-assimilate our perceived and actualized 'me'. This double-edged sword offers a unique choice to the participant; a fork in the road based on intention and personal engagement.

Is the avatar an act of deception, or a more honest representative reality? Like most technology, the avatar is amoral in nature until wielded from a participant's personal context and worldview. We discover each side of the coin. Is this fragmentation, deception, or liberation?

Step 1: Integrating p5.js with ML5 Posenet

I started by creating a p5.js canvas and integrating it with ml5.js. ml5.js is an open source project that builds off of Tensorflow.js to make browser based machine learning accessible.

It's maintained by "NYU's Interactive Telecommunications/Interactive Media Arts program and by artists, designers, students, technologists, and developers from all over the world." -ml5js website .

Creating my own ml5.js posenet tracking

With Posenet working effectively inside p5.js, as depicted in this documentation, I was able to log the x,y coordinates of all my movements at 24 fps.

I also was able to place the Magritte inspired apple logo over the user's face. This apple is both ironic as it is a technological icon, and pays homage to magrittes hidden figures, again pushing the analogy of representative identity.

The graphic tracks over our webcam screen via the js nose class in Posenet. It grows and shrinks based on user distance to the camera. I achieved this by following the ratio of distance from the nose class to ear class. I also tracked the relative body gestures via a red outline created within p5, joined with the ml5.js data. More information about this process is available via web tutorial here

Bridging the Technologies

In order to achieve control of a model through webcam movement, I needed to utilize a THREE.js canvas that changes based on a browser event. I am indebted to this tutorial from codepen on animating a 3d model based on the location of the mouse-pointer within the viewport.

I 'piggybacked' off of this technique but instead of using the pointer as a measure of direction for model chest and face movement, I took x/y location data of the 'head', 'arms', and 'neck' attributes from our Posenet. I measured the ratio of the camera viewport and corresponded it to the measure of movement across the entire window. I then inputted these classes to replace the mousepointer values for the head and torso rotation.

For the arm movement I correlated the lifting of the arms with the elbow and forearm keypoints in our ml5.js canvas.

Custom Models

To create my own dancing avatar group, I downloaded 4 rigged models from Adobe Mixamo that all had a dancing animation rig attached. I prepped these models and created a single file with saved fbx animations as well as textures. I also decimated the number of polygons for these models to save space, via Blender.

THREE.JS Final Scene Setup

I set the THREE.js stage to pay homage to Magritte's Pipe painting. I added dance music and confetti to the animation, in order to give it comedic impact. I selected one of my models as the main dancer and made sure that their animation skeleton had classes that corresponded to the keypoints within ml5.js.

In JS, I made sure that the data recieved for placement of these limbs reacted on a frame by frame basis with our posenet measurements.

Final Output

After making sure that the data from ml5.js Posenet was correctly bridging with THREE.js via the console, my model interacted directly with the user! I was particularly impressed by how close to real time the animation reacted webcam data.

This proved to me that using different canvases and JS frameworks within the same browser window is possible, and that the data from one can impact the other. This will become ever more important with the advent of machine learning inputs.

Moving forward I will clean up the jitter on the arm movement of my reactive character. I will also better compress my models and 3d environment so they are browser friendly. Finally, I will increase the movement outputs to other areas of the body (legs)

Try Full Project

Compatible browser with webcam required

The Treachery of Avatars