Back at it
-
@stroughtonsmith your results are amazing, but got me curious about Codex writing Metal, so I asked it to make a solar system simulation
After two hours of back-and-forth it still can't seem to texture a sphere in a way that doesn't cause the lighting and texture to rotate with the view
I gave it six goes at trying to get the bloom effect correct, but in the end had to paste an entire blog post into its context about how to render bloom in Metal
@simsaens I could suggest things, but if you want to take this offline, DM me your iMessage maybe
-
Pretty much everything I've worked on with Codex up to now has been stuff I could have built myself, within my area of expertise (or learnable), it just would have taken weeks or months.
This 3D scene app is something I never would have been able to build myself. I would have needed a team of rendering experts with domain-specific knowledge and human-years of research
@stroughtonsmith watching you build these things over the past few weeks has been amazing. Do you think there is a chance you will create a blog post or have a thread with more details about how you are promoting the AI agents?
-
@stroughtonsmith watching you build these things over the past few weeks has been amazing. Do you think there is a chance you will create a blog post or have a thread with more details about how you are promoting the AI agents?
@jlaase I did blog about it, and it has one prompt session transcript: https://highcaffeinecontent.com/blog/20260301-A-Month-With-OpenAIs-Codex
-
Pretty much everything I've worked on with Codex up to now has been stuff I could have built myself, within my area of expertise (or learnable), it just would have taken weeks or months.
This 3D scene app is something I never would have been able to build myself. I would have needed a team of rendering experts with domain-specific knowledge and human-years of research
@stroughtonsmith for my use, the "vibe Coding" is most helpful for edge cases where it would be a huge rabbit whole to implement a relatively small feature, so we're aligned there. Thank you for sharing your work online
-
I love how visionOS, uniquely, *explodes* when rendering goes wrong.
Hello [MacBook] Neo.
So now that visionOS 26 lets you spawn immersive scenes from UIKit apps, I had Codex implement me an immersive scene using Metal and CompositorServices that mirrors the in-window viewport and lets you live in your scene
It's real frickin cool.
The raytracer might be off limits for visionOS, but there's a lot of interesting stuff to do in other areas
-
So now that visionOS 26 lets you spawn immersive scenes from UIKit apps, I had Codex implement me an immersive scene using Metal and CompositorServices that mirrors the in-window viewport and lets you live in your scene
It's real frickin cool.
The raytracer might be off limits for visionOS, but there's a lot of interesting stuff to do in other areas
@stroughtonsmith I knew you will do it
-
So now that visionOS 26 lets you spawn immersive scenes from UIKit apps, I had Codex implement me an immersive scene using Metal and CompositorServices that mirrors the in-window viewport and lets you live in your scene
It's real frickin cool.
The raytracer might be off limits for visionOS, but there's a lot of interesting stuff to do in other areas
@stroughtonsmith that is extremely cool
-
So now that visionOS 26 lets you spawn immersive scenes from UIKit apps, I had Codex implement me an immersive scene using Metal and CompositorServices that mirrors the in-window viewport and lets you live in your scene
It's real frickin cool.
The raytracer might be off limits for visionOS, but there's a lot of interesting stuff to do in other areas
@stroughtonsmith to be honest, that concept might be one of the greatest use cases for Vision Pro. Imagine two controllers in your hand for much more precise input and you've got an awesome product
-
@stroughtonsmith I knew you will do it
@gklka I blame you
-
I love how visionOS, uniquely, *explodes* when rendering goes wrong.
Hello [MacBook] Neo.
@stroughtonsmith I imagine this screen still haunts some Apple engineers
-
I figured I needed some kind of glass shader
@stroughtonsmith but is it Liquid Glass
-
So now that visionOS 26 lets you spawn immersive scenes from UIKit apps, I had Codex implement me an immersive scene using Metal and CompositorServices that mirrors the in-window viewport and lets you live in your scene
It's real frickin cool.
The raytracer might be off limits for visionOS, but there's a lot of interesting stuff to do in other areas
I figured why not use RealityKit for the material previews, so now they are actual spheres.
Miraculously, it all still works — the Metal viewport, the Metal immersive scene, and the RealityKit UI elements, but it's very clear the Vision Pro (M2) doesn't have much headroom to build an actual app around this stuff
-
I figured why not use RealityKit for the material previews, so now they are actual spheres.
Miraculously, it all still works — the Metal viewport, the Metal immersive scene, and the RealityKit UI elements, but it's very clear the Vision Pro (M2) doesn't have much headroom to build an actual app around this stuff
The raytracer is, for now, a no-go on visionOS. It's possible I could throttle it and stay within visionOS' systemwide render budget. But it's probably worth improving the RT performance a bunch on its own first before I come back and try it here. I might run out of steam on this prototype before then.
This entire app project is still in my 'Temp' folder, where throwaway projects live
-
The raytracer is, for now, a no-go on visionOS. It's possible I could throttle it and stay within visionOS' systemwide render budget. But it's probably worth improving the RT performance a bunch on its own first before I come back and try it here. I might run out of steam on this prototype before then.
This entire app project is still in my 'Temp' folder, where throwaway projects live
This project, which runs on iPhone, iPad, Mac, and Vision Pro (with Immersive Space), is now 16.5K lines of code
-
The raytracer is, for now, a no-go on visionOS. It's possible I could throttle it and stay within visionOS' systemwide render budget. But it's probably worth improving the RT performance a bunch on its own first before I come back and try it here. I might run out of steam on this prototype before then.
This entire app project is still in my 'Temp' folder, where throwaway projects live
@stroughtonsmith “This entire app project is still in my 'Temp' folder, where throwaway projects live”
-
Pretty much everything I've worked on with Codex up to now has been stuff I could have built myself, within my area of expertise (or learnable), it just would have taken weeks or months.
This 3D scene app is something I never would have been able to build myself. I would have needed a team of rendering experts with domain-specific knowledge and human-years of research
@stroughtonsmith interestingly for me it is the reverse: a basic modeler with a basic built in ray tracer would take me maybe a day or two to build on my own with an imgui based UI.
But all of the native iOS/iPadOS/visionOS stuff would have required me to have dedicated Apple frameworks experts help out on. My experience with writing Apple native apps is super limited. But with Codex, I’ve now made several nice custom, just-for-me native apps to replace things that were just scripts before.
-
@stroughtonsmith interestingly for me it is the reverse: a basic modeler with a basic built in ray tracer would take me maybe a day or two to build on my own with an imgui based UI.
But all of the native iOS/iPadOS/visionOS stuff would have required me to have dedicated Apple frameworks experts help out on. My experience with writing Apple native apps is super limited. But with Codex, I’ve now made several nice custom, just-for-me native apps to replace things that were just scripts before.
@yiningkarlli you might be a special case, all things considered
-
@jlaase I did blog about it, and it has one prompt session transcript: https://highcaffeinecontent.com/blog/20260301-A-Month-With-OpenAIs-Codex
@stroughtonsmith@mastodon.sociadang, I guess I missed it. Thank you for pointing it out for me!
-
@yiningkarlli you might be a special case, all things considered
@stroughtonsmith Haha, that's fair. You might be too though in Apple stuff; not many other developers reverse engineered all of Catalyst out of thin air before they even announced Catalyst.
Anyhow all of this is very exciting/cool. I have a sort of researchy project that I was previously planning on just making into your usual kind of crappy janky graphics programmer demo app, but now I'm more ambitious and want to make the whole thing a really nice native Mac app with some help from Codex.
-
The raytracer is, for now, a no-go on visionOS. It's possible I could throttle it and stay within visionOS' systemwide render budget. But it's probably worth improving the RT performance a bunch on its own first before I come back and try it here. I might run out of steam on this prototype before then.
This entire app project is still in my 'Temp' folder, where throwaway projects live
@stroughtonsmith it’s really impressive and beautiful how this one is coming together.
