Do you ever daydream about the incredibly wild optimizations a garbage collector could do if it had the direct control over virtual memory layout an operating system does
-
Do you ever daydream about the incredibly wild optimizations a garbage collector could do if it had the direct control over virtual memory layout an operating system does
@mcc i recently saw this, which i thought was a clever use of only userspace-level control over virtual memory layout https://github.com/plasma-umass/Mesh
-
@glyph@mastodon.social @mcc@mastodon.social That's just called shell scripting.
-
@mcc i recently saw this, which i thought was a clever use of only userspace-level control over virtual memory layout https://github.com/plasma-umass/Mesh
-
@glyph One thing I wonder is if we'd even need protected memory in that world. Per se. Or if we could throw out process protected memory and use the protected memory capabilities of the CPU for something entirely other
@mcc on the one hand, in principle, of course not! you can validate everything once at the garbage collector level, put the GC in the kerrnel, use formal methods or whatever and bob's your uncle, everything would be so fast!
on the other hand, in practice, I have been in the room (usually at PyCon) where garbage collector bugs are being diagnosed and fixed, and from that experience, "no, this must never happen"
-
@mcc on the one hand, in principle, of course not! you can validate everything once at the garbage collector level, put the GC in the kerrnel, use formal methods or whatever and bob's your uncle, everything would be so fast!
on the other hand, in practice, I have been in the room (usually at PyCon) where garbage collector bugs are being diagnosed and fixed, and from that experience, "no, this must never happen"
@glyph I've been thinking a lot in recent years about the unikernel environment
-
-
@glyph I've been thinking a lot in recent years about the unikernel environment
-
@petrillic @glyph I once got to see Dave Ungar using his laptop. It was a life-changing experience
-
@petrillic @glyph I once got to see Dave Ungar using his laptop. It was a life-changing experience
-
@mcc @petrillic I am aware *that* Genera sort of petered out, but I still don't really understand *why* it did. It was too expensive for the hardware of the day, but probably *less* too-expensive than, say, NeXTSTEP, which famously endured despite a worse start. Has anyone written a history of Symbolics? I wonder if there are good reasons for this.
-
@mcc @petrillic I am aware *that* Genera sort of petered out, but I still don't really understand *why* it did. It was too expensive for the hardware of the day, but probably *less* too-expensive than, say, NeXTSTEP, which famously endured despite a worse start. Has anyone written a history of Symbolics? I wonder if there are good reasons for this.
@glyph @mcc Asianometry did a video on the whole boom and bust of the industry. It was part of the AI winter that happened.
- YouTube
Auf YouTube findest du die angesagtesten Videos und Tracks. Außerdem kannst du eigene Inhalte hochladen und mit Freunden oder gleich der ganzen Welt teilen.
(www.youtube.com)
-
@glyph @mcc Asianometry did a video on the whole boom and bust of the industry. It was part of the AI winter that happened.
- YouTube
Auf YouTube findest du die angesagtesten Videos und Tracks. Außerdem kannst du eigene Inhalte hochladen und mit Freunden oder gleich der ganzen Welt teilen.
(www.youtube.com)
@petrillic @mcc what promising technology will this one destroy, I wonder
-
Do you ever daydream about the incredibly wild optimizations a garbage collector could do if it had the direct control over virtual memory layout an operating system does
@mcc now I am
-
@petrillic @mcc @glyph have I ever mentioned on here my insane plot to get Smalltalk-80 running on a C64 with GeoRAM? (I swear it's somehow possible, every few months I go back to work on it a little more -- last time I got stuck on optimizing a modulo operation in the method dispatcher)
-
You can use MAP_FIXED to get a range on a specific address. The address you specify that way does get rounded down to a multiple of the page size, but other than that it's used as is.
A zero value for address has a special meaning, so if you absolutely want to map at address 0 you need to ask for address 1 rounded down. Though some kernels won't permit that in the default configuration.
You can ask for more memory ahead of time. I am pretty sure the kernel only allocates the physical memory on the first write. However things get a little tricky with respect to over-commitment and such. As I understand it, the kernel will refuse the allocation if there is no way it could ever give you all of that memory. But I think the default is that when there is any doubt the kernel will let the allocation go through and kill the process later if it doesn't have memory after all. (I am not saying that's a good default.)
There is also the possibility of allocating a memory range with no permissions and then use mprotect to make parts of the range read and writable later. In that case it would make sense to me if the kernel only updates the count of committed memory once you make it writable, I don't know if that's actually what happens, but it should be easy to test.
I have used the approach of using mmap to allocate a range with no read or write permissions and then make a small range in the middle read-write with mprotect. My reason for using it has been to have guard pages around certain buffers as a security measure. It provides an extra layer of protection against buffer overflow vulnerabilities.
For some advanced use cases it can make sense to map a range with no privileges and later change the protection of the range from within a SIGSEGV handler when that address is accessed.
-
@kasperd Thank you, this is not something I have read up on and is a great basis for future research. Here's an important question: Do *both* Linux mmap and the Windows equivalents offer these capabilities? Because my use cases invariably need to target both
-
@kasperd Thank you, this is not something I have read up on and is a great basis for future research. Here's an important question: Do *both* Linux mmap and the Windows equivalents offer these capabilities? Because my use cases invariably need to target both
@mcc @kasperd I thiiiiink the Windows equivalent is VirtualAlloc https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-virtualalloc
I'm a bit rusty on this, but I'm pretty sure you should be able to ask for an arbitrarily large virtual address space up front and change the page mapping on the fly.
-
@mcc @kasperd I thiiiiink the Windows equivalent is VirtualAlloc https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-virtualalloc
I'm a bit rusty on this, but I'm pretty sure you should be able to ask for an arbitrarily large virtual address space up front and change the page mapping on the fly.
-
You can use MAP_FIXED to get a range on a specific address. The address you specify that way does get rounded down to a multiple of the page size, but other than that it's used as is.
A zero value for address has a special meaning, so if you absolutely want to map at address 0 you need to ask for address 1 rounded down. Though some kernels won't permit that in the default configuration.
You can ask for more memory ahead of time. I am pretty sure the kernel only allocates the physical memory on the first write. However things get a little tricky with respect to over-commitment and such. As I understand it, the kernel will refuse the allocation if there is no way it could ever give you all of that memory. But I think the default is that when there is any doubt the kernel will let the allocation go through and kill the process later if it doesn't have memory after all. (I am not saying that's a good default.)
There is also the possibility of allocating a memory range with no permissions and then use mprotect to make parts of the range read and writable later. In that case it would make sense to me if the kernel only updates the count of committed memory once you make it writable, I don't know if that's actually what happens, but it should be easy to test.
I have used the approach of using mmap to allocate a range with no read or write permissions and then make a small range in the middle read-write with mprotect. My reason for using it has been to have guard pages around certain buffers as a security measure. It provides an extra layer of protection against buffer overflow vulnerabilities.
For some advanced use cases it can make sense to map a range with no privileges and later change the protection of the range from within a SIGSEGV handler when that address is accessed.
@kasperd @mcc not just is it possible to use
MAP_FIXED, it is often required. ELF files can specify fixed addresses for loading, and executables do so unless they're specifically compiled for PIE. All binaries will want various pieces of memory mapped from both the file and anonymous backings in specific offsets from each other, so the linker will map aPROT_NONEmapping to reserve parts of the virtual address space then map parts of the file and memory on top of the shadowed mapping. -
@kasperd @mcc not just is it possible to use
MAP_FIXED, it is often required. ELF files can specify fixed addresses for loading, and executables do so unless they're specifically compiled for PIE. All binaries will want various pieces of memory mapped from both the file and anonymous backings in specific offsets from each other, so the linker will map aPROT_NONEmapping to reserve parts of the virtual address space then map parts of the file and memory on top of the shadowed mapping.
