The blog post doesn't seem to mention what strikes me as the most interesting application of a model like this, namely extracting semantic information from its latent space. It mentions robotics applications, but only in the context of generating realistic world models for simulation.
If you have a robot deployed in an environment, generating synthetic views of the environment you're in doesn't have any obvious value. What does have obvious value is the latent knowledge that the model could have used to generate those synthetic views.
For instance, the fact that Atlas is capable of identifying regions of the input images that look like "floors", and smoothly interpolating them and filling in gaps with more floor, suggests that it has a concept of "floor-like walkability" which it's learned from the examples in its training data. And being able to identify the regions of 3D space that correspond to that semantic label would obviously be useful for robot path planning.
There's plenty of literature about e.g. using neural networks to estimate walkable areas from a point cloud. And you could imagine just bolting one of those methods to the front of Atlas, using the synthesized point cloud (instead of traditional photogrammetry or LIDAR) as input. But that seems like it's throwing away a lot of potentially useful semantic information, on top of being needlessly inefficient.
As a World Labs customer, I suspect they are using existing 3D assets when creating their environments. For one thing, a vendor of 3d models for AI mentioned Word Labs as a customer on a webinar.
I don't know if that undercuts what you're saying entirely. But if they're just protecting the splat in real-time based on the camera path through a 3d world, would the model truly understand that it's a floor?
I am not sure why creating synthetic view of the environment should not have any value.
When you plan your path to go somewhere you can imagine how it would look like based on what you know and when new information comes in that contradicts it it gets updated to what it's really like.
The robot could do the same thing. It could create images/views of what path it wants to take and start acting on it without knowing the real situation yet.
Granted, the 3D representation that atlas also provides is probably more useful than just an image but the idea remains the same, we are giving robots a kind of idea how the current world around them most likely is based on previously learned data of other locations
What is the scenario you're thinking about? This is kinda what happens with VLAs, the latent knowledge of an LLM can help in action generation pipelines, but it's also kinda slow.
This is incredible. One potential application that I'm thinking about already is the rapid iteration of video-game map blocking. Being able to drop in some 'initial state' configuration and then have it procedurally generate a handful of alternative configurations could make rapid prototyping a significantly quicker experience, especially if you wanted to see what a potential end result could look like.
Furthermore, being able to extract and process world geometry and 3D objects from Atlas could reduce friction in the early stages of indy development, where developer time is stretched thinner.
I'm very excited about AI tooling moving forward if this is a glimpse into the future.
Yeah where I've been previously skeptical for using AI in creative ways, basically when these tools get to the point where you have a tight iteration loop, it enables a great creative flow.
You still need the technical proficiency to be able to make edits beyond that, but I think studios would be foolish to let go of their experts in the first place.
Atlas will complement Marble, our current product that can be used for applications like this nicely :) These are indeed great applications for the model. Atlas gives us unprecedented levels of control as well as generative capabilities and is 3d consistent.
This is completely useless for video game map blockout. You need collisions for level blockout, which this does not provide. It makes gaussian splats, not geometry.
Furthermore, AI can be used to generate 3D primitives from Gaussian splats and point clouds. Sure, it would require an intermediary stage should the Atlas team not build it into a first-class offering, but the process is not exactly untenable.
Maybe a dumb question, but when navigating the worlds on the site, I notice that the space is 3D, but the objects (the drum set, or the vending machine) are flat 2D layers that one cannot walk around or examine from a different angle. Is this the inherit limitation of the approach or future features? Thanks.
One of the biggest differences is the conditioning signal. Genie 3 and similar input raw keyboard commands (WASD + arrow keys), while Atlas inputs camera poses. This small difference means that Genie 3 has no 3D whatsoever; the model needs to learn an internal mapping between keyboard commands, world states, and pixels; and with Genie 3 there is no clear way to control the generated world aside from the input image and text prompt. Since Atlas makes camera pose explicit it can use posed input frames to shape the generated world, giving you a lot more creative control.
Another big differentiator is multimodality. Genie 3 only outputs pixels. Atlas also outputs pixels, but it can also output explicit 3D for the cases where you need it (such as plugging into game engines, simulators, or VFX workflows)
Unfortunately that's a complex question... this depends on the number of diffusion steps, the size of the context, the image resolution, and the type and number of inference devices we use. There are lots of knobs to trade off speed, quality, latency, throughput, and cost.
An ideal workflow would be something semi-interactive that you can use to quickly iterate on an idea, followed by a longer offline bake-out to generate final production-quality assets.
Could this be used to replace photogrammtry when accuracy is needed? Photogrammetry requires lots of images and can be brittle, and is slow to compute.
spacial context feature is cool - what are the limitations, if any? What would it take to geo and rotation tag every photo ever taken , combine it into a mass spatial context, run it through atlas and build an entire 3D model of the world?
Atlas is an auto-regressive diffusion model, so context length limitations apply similar to LLMs and video models.
Where Atlas has an edge is that its context comprised of an arbitrary sequence of images with camera poses, which lends itself to managing the context in creative ways (we called this "context juggling" in our RTFM blog, https://www.worldlabs.ai/blog/rtfm). So yes through clever context management you could potentially build an entire 3D model of the world.
Yes, as long as the input images are "poseable" -- if they were taken in the same space they need to have some overlap, where the same object or part of the scene is visible in multiple views so the pose can be predicted.
You can also manually position the input images in 3D space to create scenes generatively; we show examples of this in the "generating with spatial context" section
It's an overloaded term that different people use for different things. At World Labs we think about three major categories of world models:
- Renderers output pixels or views of a world, either the real world or a generated world
- Simulators predict how the world responds to actions
- Planners predict actions for an agent to make to affect some change in the world
Under this taxonomy, Atlas is somewhere between renderer and simulator. It predicts high-quality views of a consistent 3D world (renderer) but it can also predict explicit 3D to aid in simulation. Atlas doesn't currently predict actions to directly drive a robot, so it isn't natively a planner; but its simulation abilities can help train another model to be planner.
It's an overloaded term for AI models that have spatial reasoning LLMs currently lack.
Best definition I've heard is: AI systems that can build an internal map of their surroundings to anticipate what happens next and make decisions based on their predictions about the consequences the different actions they can take would have.
There's a bunch of different approaches people are trying:
- World labs (linked in this post) is going down the route of neural 3D representation work (NeRFs, 3D Gaussian Splatting)
- Yann LeCun is pretty famously betting on JEPA architectures (check out the excellent Welch Labs videos for more)
- Google is betting on generative video
- Karl Friston was pursuing 'active interference,' which is just traditional RL techniques with different reward functions
Small correction, World Labs isn't betting exclusively on explicit 3D -- many of the Atlas results in this post are generated frames without any explicit underlying 3D representation. But Atlas can (optionally) output explicit 3D for the cases where you know you want it.
However essentially a world model is something that has the understanding of 3d world and can generate novel view point given either text or image input.
The use I have seen is for robotics. You feed in the current view and describe the action you want it to do, and then it plans the arm movements. (really useful for softbody manipulation.
There are other meanings. but essentially a world model is able to reason in 3d, rather than text.
It's a promising approach - and the demo goes to show just how advanced and robust "3D from 2D" reconstruction is now.
Dedicated depth sensors used to be a must on advanced robotics platforms - the only way to get anything close to reliable 3D point clouds was to spin a LiDAR. But by now, I wouldn't be surprised to see more and more robots ship with smartphone-like camera blocks - varying FoVs and focal depths, but not a lot of explicit depth sensing, if any at all.
Also, I wonder if this very model can be retrofit into a true robotics VLA? If it already takes text and image guidance, performs autoregressive diffusion of novel views, and handles temporal dynamics - why not diffusion of actions too?
These models will open so many new possibilities for 3d workflows. Reminds me of a few years ago when I wad beta testing Stable Diffusion and realising it was going to change the way people design, visualize and present things forever. These models feel similar (if they work as advertised). I tested the first version of the World Labs model and it was OK but quite limited (quality of reconstruction, type of inputs and outputs) this one feels like a big breakthrough happened, and the model is probably much bigger. Looking forward to test it.
This seems like by far the best model yet for reconstructing 3D spaces from sparse images. It looks like you could reconstruct your whole house with pretty good fidelity from a dozen or so images taken on your phone.
They show it working with videos that have motion, but it seems like time is always frozen while the camera is moving, and they always return to a ground truth camera view before advancing time again. Maybe the temporal consistency isn't very good? This surprises me given how well it understands space. I guess modeling physics and time is the next step in the development of this kind of model.
In addition to frozen time, Atlas can handle some scene motion - see for example the candy New York City in the Camera-Controlled Generation section where cars are moving, or some subtle motion of the waves toward the end of the 1-minute video example.
But this is certainly one area we plan to improve going forward.
I build an AR CAD app for iOS (http://graphite3d.com), and the gap I keep hitting is that world models are great at reconstructing space but bad at editable geometry — users want parametric surfaces they can dimension, not meshes. Curious whether Atlas exposes structured scene primitives.
Looks very very interesting. For my immediate use-case to simulate a real-world 3D space, would it work with exact dimensions? Like can I give it exact dimensions and then can it render it truthfully? For example, I want to say I have walls of 8ftx5ft, and I have N paintings of so-and-so specific sizes (I have the exact sizes), is it able to use the dimensions exactly?
Assuming you had a 1:1 scale, what would be the difference between you taking a camera of an environment and measuring everything the same way vs. it being generated artificially in a 3d space?
Wouldn't the inputs be more or less the same wrt training a model with the synthetic data vs live video and the data captured there?
It can operate in different modes for different applications.
If you only want to reconstruct the content in the input images without any imagination, Atlas can do sparse 3D reconstruction where it predicts a depth map for each input pixel, without predicting anything occluded or not visible in the input views. This could be used to implement your "fog of war" view.
But sometimes you want the model to stitch together views and imagine a coherent completion; this is a powerful tool for building and authoring generative worlds. For example the 1 minute video we show in the post was built by positioning seven unrelated input images in space and prompting the model to interpolate between them along a hand-specified camera path.
Yes, one of the main goals of this model was to keep 3D consistency as the camera moves around; it can do this even without relying on an explicit 3D point cloud or Gaussian splat representation of the scene.
I believe so. This is not a model that generates pixels frame by frame from user input like genie 3. Instead, there’s an actual 3D scene / structure generated (point cloud, 3dgs) from the input images.
“It generates both image frames from novel views and explicit 3D outputs”
Model can indeed generate novel views but I don’t think in real time (I could be wrong). If you want to navigate a space in real time as user above was asking gotta rely on the 3D output and the explicit representation will provide continuity. User likely was referring to the limitations we’ve seen on video-style world models like genie and precursors.
> For robotics, reconstruction is only half the job: as a simulated robot moves through space, Atlas also generates the RGB and depth data its sensors would observe along the way. The world and the robot's view of it come from the same model.
Potentially very significant for accelerating the data flywheel challenge for robotics
It is so good and probably only become better from here but for now I can see things appearing and disappearing (aka inconsistent hallucinations). See the dominos falling scene, it is easy to see dominos appearing and disappearing into thin air.
How fast can it generate a frame? If its fast enough for real time then you wouldn't need to pipe the outputs to a different novel view synthesis method like Gaussian splats.
Unfortunately I can see law enforcement and legal systems using this to make "crime scene recreation" videos that are believable enough to convince jurors despite being pure fabrication.
well, just to illustrate to the jury that the "recreation" is just an illusion built by prosecution, the defense should be able to present the same video with the lead prosecutor in the role of the perpetrator of the crime.
The blog post doesn't seem to mention what strikes me as the most interesting application of a model like this, namely extracting semantic information from its latent space. It mentions robotics applications, but only in the context of generating realistic world models for simulation.
If you have a robot deployed in an environment, generating synthetic views of the environment you're in doesn't have any obvious value. What does have obvious value is the latent knowledge that the model could have used to generate those synthetic views.
For instance, the fact that Atlas is capable of identifying regions of the input images that look like "floors", and smoothly interpolating them and filling in gaps with more floor, suggests that it has a concept of "floor-like walkability" which it's learned from the examples in its training data. And being able to identify the regions of 3D space that correspond to that semantic label would obviously be useful for robot path planning.
There's plenty of literature about e.g. using neural networks to estimate walkable areas from a point cloud. And you could imagine just bolting one of those methods to the front of Atlas, using the synthesized point cloud (instead of traditional photogrammetry or LIDAR) as input. But that seems like it's throwing away a lot of potentially useful semantic information, on top of being needlessly inefficient.
As a World Labs customer, I suspect they are using existing 3D assets when creating their environments. For one thing, a vendor of 3d models for AI mentioned Word Labs as a customer on a webinar.
I don't know if that undercuts what you're saying entirely. But if they're just protecting the splat in real-time based on the camera path through a 3d world, would the model truly understand that it's a floor?
I am not sure why creating synthetic view of the environment should not have any value.
When you plan your path to go somewhere you can imagine how it would look like based on what you know and when new information comes in that contradicts it it gets updated to what it's really like.
The robot could do the same thing. It could create images/views of what path it wants to take and start acting on it without knowing the real situation yet.
Granted, the 3D representation that atlas also provides is probably more useful than just an image but the idea remains the same, we are giving robots a kind of idea how the current world around them most likely is based on previously learned data of other locations
What is the scenario you're thinking about? This is kinda what happens with VLAs, the latent knowledge of an LLM can help in action generation pipelines, but it's also kinda slow.
Sounds like something Atlas probably is a great base model for.
This is incredible. One potential application that I'm thinking about already is the rapid iteration of video-game map blocking. Being able to drop in some 'initial state' configuration and then have it procedurally generate a handful of alternative configurations could make rapid prototyping a significantly quicker experience, especially if you wanted to see what a potential end result could look like.
Furthermore, being able to extract and process world geometry and 3D objects from Atlas could reduce friction in the early stages of indy development, where developer time is stretched thinner.
I'm very excited about AI tooling moving forward if this is a glimpse into the future.
Yeah where I've been previously skeptical for using AI in creative ways, basically when these tools get to the point where you have a tight iteration loop, it enables a great creative flow.
You still need the technical proficiency to be able to make edits beyond that, but I think studios would be foolish to let go of their experts in the first place.
Atlas will complement Marble, our current product that can be used for applications like this nicely :) These are indeed great applications for the model. Atlas gives us unprecedented levels of control as well as generative capabilities and is 3d consistent.
This is completely useless for video game map blockout. You need collisions for level blockout, which this does not provide. It makes gaussian splats, not geometry.
3DGS to Mesh conversion does exist: https://www.kiriengine.app/blog/what-is-3dgs-to-mesh
Furthermore, AI can be used to generate 3D primitives from Gaussian splats and point clouds. Sure, it would require an intermediary stage should the Atlas team not build it into a first-class offering, but the process is not exactly untenable.
I'm a cofounder at World Labs - happy to answer questions about Atlas!
Maybe a dumb question, but when navigating the worlds on the site, I notice that the space is 3D, but the objects (the drum set, or the vending machine) are flat 2D layers that one cannot walk around or examine from a different angle. Is this the inherit limitation of the approach or future features? Thanks.
How does it compare to Google's Genie 3 or other similar models? At first glance it does look much more impressive visually
One of the biggest differences is the conditioning signal. Genie 3 and similar input raw keyboard commands (WASD + arrow keys), while Atlas inputs camera poses. This small difference means that Genie 3 has no 3D whatsoever; the model needs to learn an internal mapping between keyboard commands, world states, and pixels; and with Genie 3 there is no clear way to control the generated world aside from the input image and text prompt. Since Atlas makes camera pose explicit it can use posed input frames to shape the generated world, giving you a lot more creative control.
Another big differentiator is multimodality. Genie 3 only outputs pixels. Atlas also outputs pixels, but it can also output explicit 3D for the cases where you need it (such as plugging into game engines, simulators, or VFX workflows)
Thanks, those are great points, now I'm definitely excited for the full release to try it out
Are y'all using any sort of self-distillation similar to https://self-evo.github.io/ to sharpen representations?
How long does it take to process from input images to model output?
Unfortunately that's a complex question... this depends on the number of diffusion steps, the size of the context, the image resolution, and the type and number of inference devices we use. There are lots of knobs to trade off speed, quality, latency, throughput, and cost.
An ideal workflow would be something semi-interactive that you can use to quickly iterate on an idea, followed by a longer offline bake-out to generate final production-quality assets.
Could this be used to replace photogrammtry when accuracy is needed? Photogrammetry requires lots of images and can be brittle, and is slow to compute.
spacial context feature is cool - what are the limitations, if any? What would it take to geo and rotation tag every photo ever taken , combine it into a mass spatial context, run it through atlas and build an entire 3D model of the world?
Atlas project lead here.
Atlas is an auto-regressive diffusion model, so context length limitations apply similar to LLMs and video models.
Where Atlas has an edge is that its context comprised of an arbitrary sequence of images with camera poses, which lends itself to managing the context in creative ways (we called this "context juggling" in our RTFM blog, https://www.worldlabs.ai/blog/rtfm). So yes through clever context management you could potentially build an entire 3D model of the world.
Would it be more reasonable to take images from movies and create worlds of various IPs?
My first thought is a detailed Hogwarts that is fully explorable using scenes from the movies (or even descriptions from the books?)
can atlas also generate 3D without pose information attached to the input images?
Yes, as long as the input images are "poseable" -- if they were taken in the same space they need to have some overlap, where the same object or part of the scene is visible in multiple views so the pose can be predicted.
You can also manually position the input images in 3D space to create scenes generatively; we show examples of this in the "generating with spatial context" section
Please give us access :) We are doing super useful things for museums and travelers and this would be a huge unlock!
Also, big fan of Dr.Li!
[dead]
What exactly does world model mean? Ive seen it used so many times in so many ways to just describe SOTA anything its lost its meaning.
It's an overloaded term that different people use for different things. At World Labs we think about three major categories of world models:
- Renderers output pixels or views of a world, either the real world or a generated world
- Simulators predict how the world responds to actions
- Planners predict actions for an agent to make to affect some change in the world
Under this taxonomy, Atlas is somewhere between renderer and simulator. It predicts high-quality views of a consistent 3D world (renderer) but it can also predict explicit 3D to aid in simulation. Atlas doesn't currently predict actions to directly drive a robot, so it isn't natively a planner; but its simulation abilities can help train another model to be planner.
Source: https://www.worldlabs.ai/blog/taxonomy-of-world-models
It's an overloaded term for AI models that have spatial reasoning LLMs currently lack.
Best definition I've heard is: AI systems that can build an internal map of their surroundings to anticipate what happens next and make decisions based on their predictions about the consequences the different actions they can take would have.
There's a bunch of different approaches people are trying:
- World labs (linked in this post) is going down the route of neural 3D representation work (NeRFs, 3D Gaussian Splatting)
- Yann LeCun is pretty famously betting on JEPA architectures (check out the excellent Welch Labs videos for more)
- Google is betting on generative video
- Karl Friston was pursuing 'active interference,' which is just traditional RL techniques with different reward functions
Small correction, World Labs isn't betting exclusively on explicit 3D -- many of the Atlas results in this post are generated frames without any explicit underlying 3D representation. But Atlas can (optionally) output explicit 3D for the cases where you know you want it.
It means everything to everyone.
However essentially a world model is something that has the understanding of 3d world and can generate novel view point given either text or image input.
The use I have seen is for robotics. You feed in the current view and describe the action you want it to do, and then it plans the arm movements. (really useful for softbody manipulation.
There are other meanings. but essentially a world model is able to reason in 3d, rather than text.
It means it builds internal representation of the world it understands (can do physics on/predict/modify) and then renders it.
It's a promising approach - and the demo goes to show just how advanced and robust "3D from 2D" reconstruction is now.
Dedicated depth sensors used to be a must on advanced robotics platforms - the only way to get anything close to reliable 3D point clouds was to spin a LiDAR. But by now, I wouldn't be surprised to see more and more robots ship with smartphone-like camera blocks - varying FoVs and focal depths, but not a lot of explicit depth sensing, if any at all.
Also, I wonder if this very model can be retrofit into a true robotics VLA? If it already takes text and image guidance, performs autoregressive diffusion of novel views, and handles temporal dynamics - why not diffusion of actions too?
These models will open so many new possibilities for 3d workflows. Reminds me of a few years ago when I wad beta testing Stable Diffusion and realising it was going to change the way people design, visualize and present things forever. These models feel similar (if they work as advertised). I tested the first version of the World Labs model and it was OK but quite limited (quality of reconstruction, type of inputs and outputs) this one feels like a big breakthrough happened, and the model is probably much bigger. Looking forward to test it.
This seems like by far the best model yet for reconstructing 3D spaces from sparse images. It looks like you could reconstruct your whole house with pretty good fidelity from a dozen or so images taken on your phone.
They show it working with videos that have motion, but it seems like time is always frozen while the camera is moving, and they always return to a ground truth camera view before advancing time again. Maybe the temporal consistency isn't very good? This surprises me given how well it understands space. I guess modeling physics and time is the next step in the development of this kind of model.
In addition to frozen time, Atlas can handle some scene motion - see for example the candy New York City in the Camera-Controlled Generation section where cars are moving, or some subtle motion of the waves toward the end of the 1-minute video example.
But this is certainly one area we plan to improve going forward.
I build an AR CAD app for iOS (http://graphite3d.com), and the gap I keep hitting is that world models are great at reconstructing space but bad at editable geometry — users want parametric surfaces they can dimension, not meshes. Curious whether Atlas exposes structured scene primitives.
Looks very very interesting. For my immediate use-case to simulate a real-world 3D space, would it work with exact dimensions? Like can I give it exact dimensions and then can it render it truthfully? For example, I want to say I have walls of 8ftx5ft, and I have N paintings of so-and-so specific sizes (I have the exact sizes), is it able to use the dimensions exactly?
Assuming you had a 1:1 scale, what would be the difference between you taking a camera of an environment and measuring everything the same way vs. it being generated artificially in a 3d space?
Wouldn't the inputs be more or less the same wrt training a model with the synthetic data vs live video and the data captured there?
Can a reconstruct distinct areas of a larger area with a "fog of war" in between, or will it hallucinate the unknown areas?
It can operate in different modes for different applications.
If you only want to reconstruct the content in the input images without any imagination, Atlas can do sparse 3D reconstruction where it predicts a depth map for each input pixel, without predicting anything occluded or not visible in the input views. This could be used to implement your "fog of war" view.
But sometimes you want the model to stitch together views and imagine a coherent completion; this is a powerful tool for building and authoring generative worlds. For example the 1 minute video we show in the post was built by positioning seven unrelated input images in space and prompting the model to interpolate between them along a hand-specified camera path.
Any ideas if it keeps continuity - when you turn around, is the invented chair still there, or does everything shift like in a dream?
Yes, one of the main goals of this model was to keep 3D consistency as the camera moves around; it can do this even without relying on an explicit 3D point cloud or Gaussian splat representation of the scene.
Amazing! Thanks for explaining!
I believe so. This is not a model that generates pixels frame by frame from user input like genie 3. Instead, there’s an actual 3D scene / structure generated (point cloud, 3dgs) from the input images.
Wrong, it does go straight to generating images.
From blog post:
“It generates both image frames from novel views and explicit 3D outputs”
Model can indeed generate novel views but I don’t think in real time (I could be wrong). If you want to navigate a space in real time as user above was asking gotta rely on the 3D output and the explicit representation will provide continuity. User likely was referring to the limitations we’ve seen on video-style world models like genie and precursors.
> For robotics, reconstruction is only half the job: as a simulated robot moves through space, Atlas also generates the RGB and depth data its sensors would observe along the way. The world and the robot's view of it come from the same model.
Potentially very significant for accelerating the data flywheel challenge for robotics
It is so good and probably only become better from here but for now I can see things appearing and disappearing (aka inconsistent hallucinations). See the dominos falling scene, it is easy to see dominos appearing and disappearing into thin air.
How fast can it generate a frame? If its fast enough for real time then you wouldn't need to pipe the outputs to a different novel view synthesis method like Gaussian splats.
Looking forward to the next step, where the reconstructed/hallucinated environment is dynamic and has a physics model etc
Any plans for characters or is this strictly environments with the basic motion you see in the video?
A camera moving through a 3D space the world model understands is getting much closer to real robotics applications
Will it be cheap to generate Splat outputs with this to use in Spark?
China, DeepAtlas please!
"Can reconstruct [scenes from Unreal Engine]"
Synthetic training data is easier to acquire
Indeed.
[flagged]
Unfortunately I can see law enforcement and legal systems using this to make "crime scene recreation" videos that are believable enough to convince jurors despite being pure fabrication.
well, just to illustrate to the jury that the "recreation" is just an illusion built by prosecution, the defense should be able to present the same video with the lead prosecutor in the role of the perpetrator of the crime.