Back at it
-
So now that visionOS 26 lets you spawn immersive scenes from UIKit apps, I had Codex implement me an immersive scene using Metal and CompositorServices that mirrors the in-window viewport and lets you live in your scene
It's real frickin cool.
The raytracer might be off limits for visionOS, but there's a lot of interesting stuff to do in other areas
@stroughtonsmith that is extremely cool
-
So now that visionOS 26 lets you spawn immersive scenes from UIKit apps, I had Codex implement me an immersive scene using Metal and CompositorServices that mirrors the in-window viewport and lets you live in your scene
It's real frickin cool.
The raytracer might be off limits for visionOS, but there's a lot of interesting stuff to do in other areas
@stroughtonsmith to be honest, that concept might be one of the greatest use cases for Vision Pro. Imagine two controllers in your hand for much more precise input and you've got an awesome product
-
@stroughtonsmith I knew you will do it
@gklka I blame you
-
I love how visionOS, uniquely, *explodes* when rendering goes wrong.
Hello [MacBook] Neo.
@stroughtonsmith I imagine this screen still haunts some Apple engineers
-
I figured I needed some kind of glass shader
@stroughtonsmith but is it Liquid Glass
-
So now that visionOS 26 lets you spawn immersive scenes from UIKit apps, I had Codex implement me an immersive scene using Metal and CompositorServices that mirrors the in-window viewport and lets you live in your scene
It's real frickin cool.
The raytracer might be off limits for visionOS, but there's a lot of interesting stuff to do in other areas
I figured why not use RealityKit for the material previews, so now they are actual spheres.
Miraculously, it all still works — the Metal viewport, the Metal immersive scene, and the RealityKit UI elements, but it's very clear the Vision Pro (M2) doesn't have much headroom to build an actual app around this stuff
-
I figured why not use RealityKit for the material previews, so now they are actual spheres.
Miraculously, it all still works — the Metal viewport, the Metal immersive scene, and the RealityKit UI elements, but it's very clear the Vision Pro (M2) doesn't have much headroom to build an actual app around this stuff
The raytracer is, for now, a no-go on visionOS. It's possible I could throttle it and stay within visionOS' systemwide render budget. But it's probably worth improving the RT performance a bunch on its own first before I come back and try it here. I might run out of steam on this prototype before then.
This entire app project is still in my 'Temp' folder, where throwaway projects live
-
The raytracer is, for now, a no-go on visionOS. It's possible I could throttle it and stay within visionOS' systemwide render budget. But it's probably worth improving the RT performance a bunch on its own first before I come back and try it here. I might run out of steam on this prototype before then.
This entire app project is still in my 'Temp' folder, where throwaway projects live
This project, which runs on iPhone, iPad, Mac, and Vision Pro (with Immersive Space), is now 16.5K lines of code
-
The raytracer is, for now, a no-go on visionOS. It's possible I could throttle it and stay within visionOS' systemwide render budget. But it's probably worth improving the RT performance a bunch on its own first before I come back and try it here. I might run out of steam on this prototype before then.
This entire app project is still in my 'Temp' folder, where throwaway projects live
@stroughtonsmith “This entire app project is still in my 'Temp' folder, where throwaway projects live”
-
Pretty much everything I've worked on with Codex up to now has been stuff I could have built myself, within my area of expertise (or learnable), it just would have taken weeks or months.
This 3D scene app is something I never would have been able to build myself. I would have needed a team of rendering experts with domain-specific knowledge and human-years of research
@stroughtonsmith interestingly for me it is the reverse: a basic modeler with a basic built in ray tracer would take me maybe a day or two to build on my own with an imgui based UI.
But all of the native iOS/iPadOS/visionOS stuff would have required me to have dedicated Apple frameworks experts help out on. My experience with writing Apple native apps is super limited. But with Codex, I’ve now made several nice custom, just-for-me native apps to replace things that were just scripts before.
-
@stroughtonsmith interestingly for me it is the reverse: a basic modeler with a basic built in ray tracer would take me maybe a day or two to build on my own with an imgui based UI.
But all of the native iOS/iPadOS/visionOS stuff would have required me to have dedicated Apple frameworks experts help out on. My experience with writing Apple native apps is super limited. But with Codex, I’ve now made several nice custom, just-for-me native apps to replace things that were just scripts before.
@yiningkarlli you might be a special case, all things considered
-
@jlaase I did blog about it, and it has one prompt session transcript: https://highcaffeinecontent.com/blog/20260301-A-Month-With-OpenAIs-Codex
@stroughtonsmith@mastodon.sociadang, I guess I missed it. Thank you for pointing it out for me!
-
@yiningkarlli you might be a special case, all things considered
@stroughtonsmith Haha, that's fair. You might be too though in Apple stuff; not many other developers reverse engineered all of Catalyst out of thin air before they even announced Catalyst.
Anyhow all of this is very exciting/cool. I have a sort of researchy project that I was previously planning on just making into your usual kind of crappy janky graphics programmer demo app, but now I'm more ambitious and want to make the whole thing a really nice native Mac app with some help from Codex.
-
The raytracer is, for now, a no-go on visionOS. It's possible I could throttle it and stay within visionOS' systemwide render budget. But it's probably worth improving the RT performance a bunch on its own first before I come back and try it here. I might run out of steam on this prototype before then.
This entire app project is still in my 'Temp' folder, where throwaway projects live
@stroughtonsmith it’s really impressive and beautiful how this one is coming together.
-
So now that visionOS 26 lets you spawn immersive scenes from UIKit apps, I had Codex implement me an immersive scene using Metal and CompositorServices that mirrors the in-window viewport and lets you live in your scene
It's real frickin cool.
The raytracer might be off limits for visionOS, but there's a lot of interesting stuff to do in other areas
@stroughtonsmith This is getting scary
-
This project, which runs on iPhone, iPad, Mac, and Vision Pro (with Immersive Space), is now 16.5K lines of code
I thought it was finally time to add vertex editing and subdivision. Now it's a 3D modeling tool and not just a raytracer
-
I thought it was finally time to add vertex editing and subdivision. Now it's a 3D modeling tool and not just a raytracer
@stroughtonsmith Boolean solid operations next?
-
@stroughtonsmith Boolean solid operations next?
@juandesant good idea
-
I thought it was finally time to add vertex editing and subdivision. Now it's a 3D modeling tool and not just a raytracer
Some more things to show off here on this iPad mini 6!
• Longpress band gesture
• Multi-select
• Vertex editing
• Subdividing
• My 'generate a Cornell Box' button
• (And the raytracer, of course)There is a lot of really neat stuff in this app. Still using Codex 5.3 Medium, still haven't touched a line of code myself
-
Some more things to show off here on this iPad mini 6!
• Longpress band gesture
• Multi-select
• Vertex editing
• Subdividing
• My 'generate a Cornell Box' button
• (And the raytracer, of course)There is a lot of really neat stuff in this app. Still using Codex 5.3 Medium, still haven't touched a line of code myself
All of this still works great on iPhone too
