TeleWindow

How might we bridge the gap between technical possibility and human connection in real-time telepresence?

Project Overview

TeleWindow is an ongoing research initiative led by Michael Naimark that reimagines video calls through the lens of spatial computing. The system creates a "window" between spaces, enabling natural eye contact and spatial presence through real-time 3D reconstruction.

Research Focus

My six-month research focused on two critical challenges: improving visual fidelity through surface reconstruction and developing more naturalistic lighting systems. This work sits at the intersection of technical innovation and human-centered design, asking how we can make telepresence feel more like presence.

Current System

Multi-camera coverage reducing occlusion issues

The system employs four Intel RealSense cameras for volumetric capture, creating real-time point cloud representations that enable natural spatial presence and eye contact.

📷

Capture

Four-camera array for complete volumetric coverage

🔄

Processing

Real-time point cloud fusion and perspective correction

👁️

Display

Eye-tracked stereoscopic output via SeeFront

Real environment capture (monoscopic view)

Real-time movement in virtual space

Research Challenges

Visual Coherence

Despite using static transformation matrices calculated through ICP registration, perfect point cloud alignment remains elusive. The challenge intensifies with closer objects, where perspective distortion and sensor noise become more pronounced.

Point cloud alignment artifacts in close-range capture

When the coverage of each camera has too much overlap, the point clouds generated from each camera will have significant misalignments and cause z-fighting artifacts.

Volumetric Representation

Point clouds, while computationally efficient, lack true volumetric properties. Without associated geometry, they cannot fully participate in virtual environments through lighting interactions or physical behaviors.

Research Approaches

1. Surface Reconstruction

While state-of-the-art systems like Microsoft's Fusion4D achieve high-quality reconstruction, their approaches rely on proprietary algorithms and different camera configurations. Our unique setup, with closely-positioned cameras, required alternative solutions.

Fusion4D Pipeline (Microsoft Research)

We explored Marching Cubes as a GPU-accelerated alternative, which previous research suggested could achieve real-time performance with our data scale.

Our GPU-accelerated Marching Cubes pipeline

2. Normal-Based Relighting

💡

Challenge

Adding realistic lighting to point clouds without full reconstruction

🔍

Approach

Generating normal vectors from depth maps for lighting calculations

⚡

Implementation

Custom shaders for real-time normal calculation and lighting

Normal vectors enabling lighting calculations Normal map generation from depth data

Normal map generation from depth data

Implementation Results

Surface Reconstruction

Our implementation, while limited by memory constraints to 256³ voxel resolution, revealed important insights. The reconstructed meshes offered improved rendering performance and better integration with virtual environments, despite lower visual fidelity.

Comparison of reconstruction results

Real-time Relighting

Normal map generation from depth data

The normal-based lighting system successfully added volumetric characteristics to point clouds, though camera discrepancies introduced visible noise in normal calculations.

Directional lighting on captured objects

Human subject under dynamic lighting

Research Insights

🔍

Technical Boundaries

Simple solutions for visual artifacts remain elusive

💡

Alternative Approaches

Point cloud post-processing offers promising feature possibilities

🎯

Future Direction

Focus on novel user experiences over perfect visual fidelity

Multi-camera capture demonstrating system capabilities

This research revealed that while perfect visual reconstruction remains challenging, the combination of real-time 3D capture and stereoscopic display creates compelling telepresence experiences. The future of this work lies in exploring novel interactions rather than pursuing perfect visual fidelity.

Future Directions

I'm currently exploring these insights through a new project using Looking Glass Factory's holographic display technology, continuing to investigate the intersection of technical capability and human experience in telepresence systems.

Acknowledgments

Special thanks to Michael Naimark and Dave Santiano for their guidance and mentorship throughout this research journey.