Markerless MoCap Installation

2024.7

Unreal Engine-Based Interactive Installation for Brand-Community Engagement

Unreal Engine, Motion Capture, Captury, Python, LED Wall

During my internship, we collaborated with the liqueur brand Fernet-Branca to create a pop-up shop experience in New Orleans. Using a markerless motion capture system, and an LED wall covering almost floor to ceiling, edge to edge, we built an immersive experience where fans of the liqueur can meet and “become” the brand’s mascot, Fernando. To let the fans help spread the word, we also recorded their interaction with each other and with Fernando, which were sent to them via Email, ready for sharing on social media.

My Work

I was in charge of scouting the installation space and deciding on the placement for motion capture cameras, setting up and calibrating the motion capture system. I also developed the Python script that handles user input of name and email on a webpage, starts recording using OBS, and sends the recorded video via the Gmail API. I also helped with assembling the LED wall, and the visual design of the Unreal Engine environment.

Setting Up Markerless Motion Capture for Interactive Installation

One of the strengths of a markerless motion capture system, like the Captury system that we use, is not needing to wear any special equipment to be tracked. This makes it especially suitable for creating interactive installations.

Several settings were configured to ensure the motion capture system works as stable and seamless as possible. The first is target management. By setting up the automatic detection volume and specifying the maximum amount of targets that can be simultaneously tracked, it can be made sure that the experience will always operate with the designed capacity, and will not be interrupted by the unexpected person entering/exiting the area. In addition, the proximity of the tracking area to the LED wall means the lights emitted from the wall can affect the tracking quality. Therefore, camera exposure settings were adjusted accordingly to compensate for different lighting conditions.

Working with Constraints

The space we installed our experience is a vintage pizza oven with brick walls that cannot handle being drilled into. The LED wall covering an entire side of a wall further limited the mounting options for motion capture cameras that need to have good coverage of the entire volume.

Using my experience with the Captury system, I analyzed the space and figured out a camera placement plan that would avoid unavailable places and require the least amount of cameras. By working closely with fabricators, we created innovative ways to mount the cameras, including hanging plywood or clamping on the side of the LED wall.

Automating the Process

With the client’s goal being to increase the recognition and online presence of their mascot character, while also connecting with the community, we designed a system that automatically records the experience and sends the video to fans for social media.

An action camera is placed in the space. The camera feed, along with screen capture, was recorded by OBS, which is controlled via WebSocket by Python. Using the name and email entered by users on the webpage for the experience, I programmed the Python script to use the Gmail API to send out the videos.

Enlin Li

Kanagawa, Japan