Overcoming Address Space Limitations

The question overcoming the mathematical hardware RAM limit of 64bit operating systems? was closed while I was composing my answer.  I still think my answer is worth sharing, so I'm posting it here.  If anyone is able to share this answer with the user who asked, please do. 

---

(The question):

I read that the limit RAM memory for a 32-bit OS is 4GB, whereas for a 64-bit machine it seems to be 16TB (or is it 16EB?). Are OSes like Linux planning ways to overcome this 64-bit limitation in similar ways to Physical Address Extension (PAE) for 32-bit processors? My googling was unsuccessful.

---

(My answer:)

The short answer is no, because it won't be useful for the foreseeable future, and by the time it is everything will have changed anyway.  The long answer is, well, long...



The 16TiB limit on 64-bit addresses is not analogous to the 4GiB limit on 32-bit addresses.  2³²bytes = 4GiB, so 4GiB is the limit on what can be addressed with 32-bit addresses.  The corresponding limit on 64-bit addresses is 2⁶⁴bytes = 16EiB, much greater than the 2⁴⁴bytes = 16TiB limit you cite.  The 16TiB limit is imposed by some implementation, not by the address size.  Wikipedia indicates the 16TiB limit is imposed by Windows, with AMD CPUs imposing a 48-usable-bits limit of 2⁴⁸bytes = 256TiB.  These are limits on process address space, not on physical memory, which is much more limited by what you can physically attach to the motherboard - the physical memory limit right now might be “only” 768GiB.

It might be possible for an OS to overcome the CPU implementation limit on process address space.  Using more than 48 bits of an address will trigger a general protection fault, and the exception handler will be able to identify the offending instruction.  (See AMD64 System Programming Manual, pages 4, 212, and 216.) The handler could, in principle, update the process state as if the instruction had been executed using some indirect expanded address space, then return control to the following instruction.  This mechanism would have to emulate every instruction that touches memory, and would impose the exception handling overhead in addition to using what amounts to an alternative virtual memory storage.  I am also unable to find any indication of any effort to implement such a thing – I expect that for now it's easier to refactor to just use the alternative storage explicitly, but it might be a useful backward-compatibility mechanism after newer CPUs have more usable bits in their address selectors.

PAE is a hardware feature where the CPU memory bus has more bits than the process address selector.  On 32-bit CPUs with PAE, each process has a 32bit address space (and corresponding 32-bit address selectors), so each process can only use at most 4GB RAM, but the physical addresses used by the CPU are larger so multiple processes can exhaust their address space without having the resort to virtual memory swapping.  PAE cannot be implemented in software, but must be supported by the OS to be useful.  It's described pretty well, if not concisely, on the Wikipedia page.  PAE will not be useful on 64-bit CPUs until physically connecting 16EiB of RAM to your motherboard is no longer a limiting factor.

PAE increases the system-addressable physical memory, without changing the address space available to any process.  To increase the address space available to a process beyond the size of the address selector (without imposing some intermediate API, such as for disk storage), you would have to change the size of a memory address selector, for which you'd have to make a complete software virtual machine and not run natively on the CPU.  (Similar to what's done by the JVM, .NET's CLI, CPython, YARV, etc.)  It might be possible for an emulator to run multiple processes and take advantage of a hardware PAE to provide a larger emulated address space without resorting to swapping (e.g. by rearranging shared memory).

Since the OS virtual memory management must be able to handle allocated memory of many times the physical address space, it might be possible for an OS to integrate support for larger virtual address spaces into its virtual memory system.  This would simplify overcoming the CPU implementation limit as described above, and might make it easier for emulators or native applications to use the native memory address space as a cache for their larger data set, relying on the OS memory management rather than requiring a mostly-redundant alternative.  I'm not sure how to search for such a thing, but I would not be surprised if there is some similar work being done in a research lab, especially one working on memory virtualization across clusters.

Today, just one of the detectors at the LHC generates about 1PiB/second, of which only a bit more than 100MiB/s (or 1PiB/year) is actually stored.  Experiments like this are pushing the limits of data storage & analysis, but are not limited by the 16TiB address space.  Overcoming the limit on process address space might be nice in some contexts, but is currently too expensive to be viable - you can think of the expense as going toward overcoming other storage limits which are more problematic.  As those limits expand, address space can easily expand up to 16EiB (an increase of … approximately 16EiB).  I don't think overcoming the 16EiB limit will be useful until common long-term storage devices have capacities in at least the ZiB (maybe the YiB), which is too far off to be usefully predicted.