Body Remixer connects real people through virtual bodies

In 2020, the virtual Recto VRso Art&VR Gallery exhibited virtual and mixed reality artworks exploring the theme: “Real body / Virtual body”. John Desnoyers-Stewart, an engineer, designer, artist, and PhD student at Simon Fraser University in Vancouver, Canada, took part in the project with an artwork. He created Body Remixer along with a team from the iSpace Lab (Katerina Stepanova, Bernhard Riecke, and Patrick Pennefather). Body Remixer is an immersive installation that extends bodies and stimulates social connection. We interviewed him, and asked him about his artwork, its link to Recto VRso, and his upcoming projects and how he adapted to 2020.

Can you describe your artwork, Body Remixer?

Body Remixer is an interactive immersive installation that transforms bodies into particle systems. We use a Kinect to track bodies in space – tracking up to 6 people. During the experience, people see their reflection in a virtual mirror projected on the walls, so they can interact with the system directly. They can also put on a VR headset, and all the people around them become these virtual particle bodies. They don’t see the identity of the people around them anymore. Everybody kind of looks the same, they’re just in different colours.

From there, whether they use the headset or the projection, they can high-five each other. By high-fiving, they connect their bodies together and the particles start to flow between them, creating a sense of connection. They can also do a two-hand high-five. When they do the two-hand high-five, they swap their bodies and they get merged together. There is a sort of overlay, so you can see your own body somewhat, but also the other person’s body too. This draws your attention towards the difference between your movements.

The idea is to encourage them to move together, to respond to one another. Further, to reinforce that synchrony of movements, we added a sound response to that synchrony – a drum soundtrack. With whoever you are paired up with, the closer your movement is towards one another, the louder the drum soundtrack gets. The sound is louder when you are in synchrony, and it’s quieter when you move differently. That’s how the users can play with the system. The idea is that it also lets you learn and explore the system.

You applied to Recto VRso 2020. How does your work relate to the theme “Real body / Virtual body”?

One of the main things we’re trying to imagine with Body Remixer, instead of replacing or escaping reality, is that we want to expand it and build upon your real body to create this virtual presence that coincides with it. So there is a link between the real and the virtual body rather than a separation. It’s a mixed-reality experience more than a virtual reality experience that includes touch, something which is often missing from virtual reality.

What’s interesting is this merge between the virtual and the real body. But also this divergence between the two. It was our intention to create a neutral body that didn’t have any clear gender, that anyone can adopt as their own body. We also did that so that people would hopefully see a unifying presence. One of the messages we wanted to communicate is that underneath, we are all the same.

You participated in Recto VRso @Vancouver last September. Did you exhibit the same artwork? Have you modified your installation?

Since the original system is centered around touch and physical contact, it’s not well suited to the current social distancing situation. So, I tried to recreate it for online telepresent use. Effectively, it’s the same system but instead of going to a gallery and seeing a physical installation, you just download the program at home. When you launch it, it connects automatically to a server that pairs people up. Instead of being in the same physical space, you get paired up with someone in a virtual space.

There, you see completely anonymous people. In this telepresent version, you have no idea who you are interacting with. We tried the experience with colleagues from the iSpace Lab, and we had no idea who each other was. It’s a very anonymous interaction. Apart from that, it still works similarly; you have the high-fiving to connect. I also added gestures: based on the input on the controller, you can make a thumbs up or wave for example. This is the only way to communicate – you don’t have any audio, you can’t talk to anybody. It still creates an interesting space and environment.

Photo credits: (left) Photo by Stephen Crocker / (right) Composite by John Desnoyers-Stewart. Original photos © Andreas Psaltis, 2019.

You also took part in the creation of the Recto VRso virtual art gallery this year. How did you adapt Body Remixer to a virtual and non-interactive exhibition?

In the virtual gallery, there was no real way to interact with the artworks. I tried my best to capture what it looked like in the original version. I used some animations and created a representation of these two dancers who respond to each other. So you can see virtual dancers who randomly chose different movements to do. They move between trying to synchronize or not. It’s supposed to give you some idea of how the system works. But it was not possible to recreate the interactivity of the original version. But as the two virtual bodies synchronize and desynchronize, the music responds to it. So you still get that drum beat coming in and out. It still represents the real thing, even if there is a lot missing.

If we look back at the year 2020, how did the health crisis impact you and your way to work?

It definitely changed things a lot. I’m working from home constantly now. I haven’t been to the university campus since March. My work is definitely going in this telepresence direction that I am investigating during my PhD. It was centered around the co-presence and physical interaction at first. With COVID, it changed a bit and I look into how people interact with each other when they are physically present and when they’re not.

The adaptation of Body Remixer is the first step. We are also working on a prototype of an embodied telepresence connection platform where we’re going to use biofeedback as well as some forms of verbal communication to try to connect people. It’s going to work a lot like Body Remixer, with an online server that connects people randomly, but there will be a lot more channels to communicate through. But we’re aiming to restrict it to focus on certain things you wouldn’t normally pay attention to like heartbeat and breathing. I’m also working on a project with Margherita Bergamo.

Can you tell me a bit more about this project with Margherita Bergamo?

I first found out about Recto VRso in 2019 at the Recto VRso @Vancouver where I saw Margherita’s work, Eve, dance in an unplaceable place. When I saw her work, I wanted to get in touch with her because we have similar goals to get people to connect with each other and express themselves. Then, when the Laval Virtual physical event was canceled, I was invited to speak at Recto VRso [during the Laval Virtual World, ed.] and I gave a talk about Body Remixer. I met Margherita there and we started talking about how we could collaborate.

Now, we are working on this project that is kind of an iteration of Eve – it’s a performance dance piece with a lot more interactive components. The original was made of 360-degree videos that show the participant the choreography they have to follow. There will be a portion where we will have generative real time graphics. It will be a very abstract representation for about half of the performance which will respond to the participants’ movements. It will be a mixture between our previous work and visions.

What do you and Margherita want to explore with this upcoming experience?

With this project, the hope is to get the participants to go from not moving, to following the image, and finally moving on their own. Some responses will be there to encourage them during the third part. In that piece, there are 6 different stories being told. Each of them are related to different emotional mental challenges that people deal with like anxiety or depression. There are the stories of 6 different characters dealing with those challenges. The idea is to encourage empathy through this performance.

The original concept is to have 6 people going through the 6 different stories simultaneously and the dancer is in the center, facilitating each one of them. The dancer performs what we call “embodiments” and touches the participants. Like Body Remixer, touch is central to this experience. And at some point, there will be a connection between the participants themselves. This is the ideal implementation.

But I’m guessing you have to adapt the project and create a covid-friendly version…

Yes, like you are rightly saying, we have to adapt. But I think it’s particularly important to still come up with this ideal version, because when things get back to normal, people will need things that bring them back together. It’s going to be strange to go back to normal… There are a lot of social interactions that are missing right now. We hope we can create these artworks that encourage people to connect and overcome the barriers that we are putting up right now.

In the meanwhile, we need to come up with ways of exhibiting and benefiting people in the circumstances that they’re in. So we’re developing an at-home version at the same time. The viewer is going to see the same thing and going through the same experience, but you will do it with someone you are close to in your everyday life: your partner, your roommate, your sister, etc. One person puts the VR headset on, learns the dance performance, and gives the headset to the other person. Then the first person performs, with her television screen or laptop to guide her through the performance, and the other participant is going through the full experience in the VR headset.

What are your research project themes? What topics are you particularly exploring in art creation?

What my research group is mostly looking into is transformative and self-transcendent experiences that go beyond your normal body boundaries. This is what Body Remixer is about. Self-transcendent experiences also include social experiences that connect you to other people and make you feel less like a lone individual. I focus on developing this type of artwork to make people feel they are part of a larger whole and not isolated. It’s even more important right now.