Some Thoughts on Spatial Representation: From AI Chips to Embodied Intelligence

Di Yang

Ph.D. student, Data Science, William & Mary  ·  dyang06@wm.edu

October 11, 2026  ·  research reflections  ·  prose polished with ChatGPT  ·  - views

Recently, I've been working on 3D/4D representations，world models and visual & spatial reasoning. Along the way, I've found myself thinking more about a simple question: Are we spending too much compute processing information that doesn't really matter?

My background is actually in microelectronics and integrated circuits. After finishing my master's degree, I spent about a year and a half at Cambricon working as a logic engineer for AI chips.

That experience made me particularly sensitive to computational efficiency.

When ChatGPT 3.5 came out in late 2022, I was convinced that compute would become one of the most valuable resources in the AI era. Behind compute are two fundamental things: chips and energy.

For chip designers, performance per watt has always been a central concern. As traditional semiconductor scaling becomes increasingly difficult, the industry continues exploring new computing architectures, advanced packaging, and alternative approaches to improve efficiency.

The goal is largely the same: getting more useful computation out of limited resources.

But when I moved into AI research, I started thinking about this problem from another perspective.

Scaling laws have shown us how powerful additional compute and data can be. Still, given a fixed compute budget, shouldn't we also think about how much of that compute is actually spent processing useful information?

It's somewhat similar to hardware-software co-design. Instead of simply adding more hardware, we should also rethink how information is represented and processed.

Why Representation Matters

One thing that has always fascinated me is how much the choice of representation can influence what a model learns.

Language itself is already a highly abstract representation of the world. A simple sentence can communicate objects, relationships, and concepts without describing every visual detail.

Images and videos are quite different. They contain enormous amounts of information, much of which may be redundant or irrelevant to the task.

We've already seen some beautiful work exploring this problem.

Latent Diffusion Models [1] showed how moving generation into a compressed latent space could dramatically reduce computational costs. Yann LeCun and his colleagues took another fascinating direction with I-JEPA [2] and later V-JEPA [3], exploring how models could learn meaningful representations by predicting abstract features rather than reconstructing every pixel.

I find these works particularly inspiring. Although compression and predictive representation learning are different problems, they make me think about the same question:

How much of what we observe is actually necessary for understanding, reasoning, and acting?

This becomes especially interesting when we move toward embodied intelligence.

What Does a Robot Actually Need?

Imagine asking a household robot to pick up a glass and place it on a shelf.

Understanding the instruction is relatively straightforward. Executing it is another story.

The robot needs to understand the glass's position, orientation, surrounding geometry, and potential collisions. These are fundamentally spatial questions.

Current VLMs have demonstrated impressive semantic understanding, but their spatial reasoning capabilities remain imperfect, as highlighted by works such as SpatialVLM [4].

This is why I find explicit spatial representations interesting.

Rather than relying entirely on visual tokens, could we provide models with a compact, structured representation of the physical environment?

Of course, this doesn't mean RGB cameras are insufficient, or that additional sensors are unnecessary. I actually find Tesla's vision-centric philosophy quite interesting, although I wouldn't claim that camera-only perception is inherently superior.

My interest lies more in how efficiently we can extract and organize useful spatial information from observations.

After all, household robots won't necessarily have access to powerful GPUs or unlimited energy budgets.

From 3D to 4D Representation

Another problem is that explicit spatial representations can themselves be expensive.

Think about a robot observing a living room. The walls, floor, and furniture might remain unchanged for hours. Only a few objects are moving.

Do we really need to encode the entire scene again at every timestamp?

Probably not.

I recently read an interview with Yufei Wang and Zhihao Li about compression, representation learning, and world models. Many of their thoughts resonated with my own.

Their work on Sparc3D [5], along with efforts like TRELLIS [6], shows how structured sparse representations can make 3D modeling more efficient.

And this connects closely to our recent work, Sparc4D: A Compact Explicit 4D Representation for Dynamic Scenes [7].

In Sparc4D, we explore how to encode dynamic scenes into compact, explicit 4D states. Instead of independently representing every frame, we share static information across time and compress dynamic features using spatially anchored temporal slots.

Our experiments show that temporal slots substantially reduce the storage required for time-varying features while maintaining reconstruction quality. We also explored using the learned representation for robotic manipulation, with encouraging initial results.

Of course, Sparc4D is still far from a complete solution. It focuses on compact dynamic-scene reconstruction rather than general physical prediction.

But it has made me increasingly interested in what comes next.

Beyond Reconstruction

One thing I've been thinking about is that reconstruction quality doesn't necessarily translate into better reasoning or control.

A representation might preserve beautiful textures while missing a tiny obstacle that could cause a robot to fail.

Conversely, a relatively coarse representation might be perfectly adequate for navigation if it captures the relevant geometry.

This makes me wonder whether spatial representations should be designed around what an agent actually needs, rather than how faithfully they reproduce every observation.

There's already fascinating work in this direction.

DreamerV3 [8] demonstrated how agents can learn through imagined trajectories in compact latent states. More recently, DINO-WM [9] and V-JEPA 2 [10] have explored predicting future visual representations for planning without generating complete future images.

I see these as different approaches to a question that interests me:

Can we develop a compact, geometrically grounded representation that preserves the information needed for spatial reasoning and action-conditioned prediction?

And does a world model really need to predict the entire future scene, or just the changes relevant to the agent's next decision?

These are some of the questions we're beginning to explore.

Some Final Thoughts

Looking back, my research interests haven't changed as much as I initially thought.

When I was designing AI chips, I cared about how to perform more useful computation with limited hardware resources.

Now, working on spatial representations and world models, I'm thinking about how to preserve more useful information with limited computation.

I'm certainly not the first person to think about representation efficiency. Many researchers have explored these ideas much more deeply, from latent diffusion and JEPA to structured 3D representations and latent world models.

I don't believe compression alone leads to intelligence, nor do I think explicit 3D representations will solve every problem in embodied AI.

But I do think there's something worth exploring at the intersection of representation efficiency, spatial understanding, and embodied reasoning.

After all, processing more information doesn't necessarily mean understanding more.

Sparc4D is just one of our early attempts in this direction. There's still a lot to figure out, and some of our current assumptions may turn out to be wrong.

For now, I just wanted to write these thoughts down.

More on world models, spatial reasoning, and agents in future posts.

Note on Writing

The ideas, research perspectives, and structure of this post are my own; the English wording and prose were polished with the assistance of ChatGPT.

References

1. R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. In CVPR, 2022. arXiv:2112.10752
2. M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, N. Ballas. Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture (I-JEPA). In CVPR, 2023. arXiv:2301.08243
3. A. Bardes et al. Revisiting Feature Prediction for Learning Visual Representations from Video (V-JEPA). TMLR, 2024. arXiv:2404.08471
4. B. Chen et al. SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities. In CVPR, 2024. arXiv:2401.12168
5. Z. Li, Y. Wang, H. Zheng, Y. Luo, B. Wen. Sparc3D: Sparse Representation and Construction for High-Resolution 3D Shapes Modeling. In NeurIPS, 2025. arXiv:2505.14521
6. J. Xiang et al. Structured 3D Latents for Scalable and Versatile 3D Generation (TRELLIS). In CVPR, 2025. arXiv:2412.01506
7. D. Yang, Z. Li, Y. Xiong, Y. Wang. Sparc4D: A Compact Explicit 4D Representation for Dynamic Scenes. arXiv preprint, 2026. arXiv:2610.01229 · [PDF]
8. D. Hafner, J. Pasukonis, J. Ba, T. Lillicrap. Mastering Diverse Domains through World Models (DreamerV3). arXiv preprint, 2023. arXiv:2301.04104
9. G. Zhou, H. Pan, Y. LeCun, L. Pinto. DINO-WM: World Models on Pre-trained Visual Features Enable Zero-Shot Planning. arXiv preprint, 2024. arXiv:2411.04983
10. M. Assran et al. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning. arXiv preprint, 2025. arXiv:2506.09985

← All posts  ·  Home