Do you ever daydream about the incredibly wild optimizations a garbage collector could do if it had the direct control over virtual memory layout an operating system does
-
@glyph @mcc Asianometry did a video on the whole boom and bust of the industry. It was part of the AI winter that happened.
- YouTube
Auf YouTube findest du die angesagtesten Videos und Tracks. Außerdem kannst du eigene Inhalte hochladen und mit Freunden oder gleich der ganzen Welt teilen.
(www.youtube.com)
@petrillic @mcc what promising technology will this one destroy, I wonder
-
Do you ever daydream about the incredibly wild optimizations a garbage collector could do if it had the direct control over virtual memory layout an operating system does
@mcc now I am
-
@petrillic @mcc @glyph have I ever mentioned on here my insane plot to get Smalltalk-80 running on a C64 with GeoRAM? (I swear it's somehow possible, every few months I go back to work on it a little more -- last time I got stuck on optimizing a modulo operation in the method dispatcher)
-
You can use MAP_FIXED to get a range on a specific address. The address you specify that way does get rounded down to a multiple of the page size, but other than that it's used as is.
A zero value for address has a special meaning, so if you absolutely want to map at address 0 you need to ask for address 1 rounded down. Though some kernels won't permit that in the default configuration.
You can ask for more memory ahead of time. I am pretty sure the kernel only allocates the physical memory on the first write. However things get a little tricky with respect to over-commitment and such. As I understand it, the kernel will refuse the allocation if there is no way it could ever give you all of that memory. But I think the default is that when there is any doubt the kernel will let the allocation go through and kill the process later if it doesn't have memory after all. (I am not saying that's a good default.)
There is also the possibility of allocating a memory range with no permissions and then use mprotect to make parts of the range read and writable later. In that case it would make sense to me if the kernel only updates the count of committed memory once you make it writable, I don't know if that's actually what happens, but it should be easy to test.
I have used the approach of using mmap to allocate a range with no read or write permissions and then make a small range in the middle read-write with mprotect. My reason for using it has been to have guard pages around certain buffers as a security measure. It provides an extra layer of protection against buffer overflow vulnerabilities.
For some advanced use cases it can make sense to map a range with no privileges and later change the protection of the range from within a SIGSEGV handler when that address is accessed.
-
@kasperd Thank you, this is not something I have read up on and is a great basis for future research. Here's an important question: Do *both* Linux mmap and the Windows equivalents offer these capabilities? Because my use cases invariably need to target both
-
@kasperd Thank you, this is not something I have read up on and is a great basis for future research. Here's an important question: Do *both* Linux mmap and the Windows equivalents offer these capabilities? Because my use cases invariably need to target both
@mcc @kasperd I thiiiiink the Windows equivalent is VirtualAlloc https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-virtualalloc
I'm a bit rusty on this, but I'm pretty sure you should be able to ask for an arbitrarily large virtual address space up front and change the page mapping on the fly.
-
@mcc @kasperd I thiiiiink the Windows equivalent is VirtualAlloc https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-virtualalloc
I'm a bit rusty on this, but I'm pretty sure you should be able to ask for an arbitrarily large virtual address space up front and change the page mapping on the fly.
-
You can use MAP_FIXED to get a range on a specific address. The address you specify that way does get rounded down to a multiple of the page size, but other than that it's used as is.
A zero value for address has a special meaning, so if you absolutely want to map at address 0 you need to ask for address 1 rounded down. Though some kernels won't permit that in the default configuration.
You can ask for more memory ahead of time. I am pretty sure the kernel only allocates the physical memory on the first write. However things get a little tricky with respect to over-commitment and such. As I understand it, the kernel will refuse the allocation if there is no way it could ever give you all of that memory. But I think the default is that when there is any doubt the kernel will let the allocation go through and kill the process later if it doesn't have memory after all. (I am not saying that's a good default.)
There is also the possibility of allocating a memory range with no permissions and then use mprotect to make parts of the range read and writable later. In that case it would make sense to me if the kernel only updates the count of committed memory once you make it writable, I don't know if that's actually what happens, but it should be easy to test.
I have used the approach of using mmap to allocate a range with no read or write permissions and then make a small range in the middle read-write with mprotect. My reason for using it has been to have guard pages around certain buffers as a security measure. It provides an extra layer of protection against buffer overflow vulnerabilities.
For some advanced use cases it can make sense to map a range with no privileges and later change the protection of the range from within a SIGSEGV handler when that address is accessed.
@kasperd @mcc not just is it possible to use
MAP_FIXED, it is often required. ELF files can specify fixed addresses for loading, and executables do so unless they're specifically compiled for PIE. All binaries will want various pieces of memory mapped from both the file and anonymous backings in specific offsets from each other, so the linker will map aPROT_NONEmapping to reserve parts of the virtual address space then map parts of the file and memory on top of the shadowed mapping. -
@kasperd @mcc not just is it possible to use
MAP_FIXED, it is often required. ELF files can specify fixed addresses for loading, and executables do so unless they're specifically compiled for PIE. All binaries will want various pieces of memory mapped from both the file and anonymous backings in specific offsets from each other, so the linker will map aPROT_NONEmapping to reserve parts of the virtual address space then map parts of the file and memory on top of the shadowed mapping. -
@glyph @mcc Asianometry did a video on the whole boom and bust of the industry. It was part of the AI winter that happened.
- YouTube
Auf YouTube findest du die angesagtesten Videos und Tracks. Außerdem kannst du eigene Inhalte hochladen und mit Freunden oder gleich der ganzen Welt teilen.
(www.youtube.com)
@petrillic
Ok that was cool.I never heard of the TI Apple II/Lisp chip Explorer before. Wow. I wonder if any survived...
I knew someone who was a Lisp die hard, worked on Symbolics workstations. Quite the war stories. Lisp afficionados are the hardest of the hard core.
@glyph @mcc -
Do you ever daydream about the incredibly wild optimizations a garbage collector could do if it had the direct control over virtual memory layout an operating system does
@mcc one of my very favorite papers! https://web.cs.umass.edu/publication/docs/2004/UM-CS-2004-016.pdf
-
@kasperd @mcc not just is it possible to use
MAP_FIXED, it is often required. ELF files can specify fixed addresses for loading, and executables do so unless they're specifically compiled for PIE. All binaries will want various pieces of memory mapped from both the file and anonymous backings in specific offsets from each other, so the linker will map aPROT_NONEmapping to reserve parts of the virtual address space then map parts of the file and memory on top of the shadowed mapping.@artemist @kasperd @mcc yes but using MAP_FIXED in practice when you’re not the dynamic linker is fraught with perils like “there’s malloc block there and so i can’t use that address” or worse “there was a malloc block there and i just yeeted it out of existence” and things like “the OS has ASLR’d libc over where I wanted to map my ting”
what is practical is asking for a big block and suballocating within it.
-
@artemist @kasperd @mcc yes but using MAP_FIXED in practice when you’re not the dynamic linker is fraught with perils like “there’s malloc block there and so i can’t use that address” or worse “there was a malloc block there and i just yeeted it out of existence” and things like “the OS has ASLR’d libc over where I wanted to map my ting”
what is practical is asking for a big block and suballocating within it.
@erincandescent @artemist @kasperd but what if i need to grow the block later
-
@erincandescent @artemist @kasperd but what if i need to grow the block later
@mcc @erincandescent @kasperd don't. allocate 1TiB of memory as
PROT_NONE(or on windowsMEM_RESERVE). if you're on a 64-bit system it works fine, chrome does that (though i expect most of that is guard pages) -
@artemist @kasperd @mcc yes but using MAP_FIXED in practice when you’re not the dynamic linker is fraught with perils like “there’s malloc block there and so i can’t use that address” or worse “there was a malloc block there and i just yeeted it out of existence” and things like “the OS has ASLR’d libc over where I wanted to map my ting”
what is practical is asking for a big block and suballocating within it.
@erincandescent @mcc @artemist @kasperd there are definitely ways to make this work on linux specifically (e.g. non-PIE statically-linked stub which blocks out the address space you want to reserve before loading into a more "normal" environment)
WINE has a.... poorly-documented and apparently not required "preloader" which does something vaguely of this nature
-
@kasperd Thank you, this is not something I have read up on and is a great basis for future research. Here's an important question: Do *both* Linux mmap and the Windows equivalents offer these capabilities? Because my use cases invariably need to target both
I know nothing about Windows APIs. Most of the code I have done using this was on Linux. And even on Linux I am not entirely sure about the details surrounding accounting of committed memory.
-
R relay@relay.infosec.exchange shared this topic
