Memory management with larger memory sizes of around a couple of
terabyte leads to a number of issues that make life hard. This talk is
a look at the issues and at the work in progress on these issues.
First, huge pages are not supported for the page cache which leads to
numerous performance issues. It begins with the long time it takes to
just copy a file of a few gigs in size (minor compared to the size of
the memory in the system of a few terabytes after all...) to the
performance problems when accessing large amounts of 4k pages from a
memory mapped file. Since huge pages are not supported TLB trashing
often becomes an issue. Devices have issues writing large blocks of
data to disk due to large scatter gather lists having to be processed
in memory. Reclaim becomes slow since the system is literally churning
through billions of 4k page to figure out which ones to evict.
Second, huge pages cause a memory overhead that surprises users. The
kernel manages 4k chunks of the huge page in many cases and needs to
have metadata structures for each 4k page. Other metadata also exists.
This results in gigabytes of memory just not being present. The user
bought a system with a lot of memory but the Linux just cannot use it.
Third, fragmentation becomes an issue in particular if huge pages and
4k pages are frequently allocated and freed (like we have to because
the page cache does not support 4k after all). So over time systems
have to be rebooted to allow the allocation of huge pages. Huge pages
are actually only a small problem because devices on large systems
often want to allocate large contiguous buffers for I/O (like
specialized networking devices using RDMA).
Christoph Lameter is working as a lead in research and development for Jump Trading LLC (an algorithmic trading company) in Chicago and maintains the slab allocators and the per cpu subsystems in the Linux Kernel. He contributed to a number of Linux projects since the initial kernel releases in the early 90s. As a kernel developer at SGI he helped pioneer the use of Linux for Supercomputing and developed the necessary kernel capabilities for HPC applications. Currently he is working on improving Linux through the use of new faster APIs to a variety of high performance devices and is evaluating new technologies that allow faster processing.