Open Radar

 
Summary:
The header sys/resource.h defines both RLIMIT_AS and RLIMIT_RSS (as equivalents) to limit the amount of virtual memory that may be used by a process. Launchd also exposes this functionality using the ResidentSetSize launchd.plist key, in both SoftResourceLimits and HardResourceLimits.

Unfortunately, none of these options have any effect in the kernel. In xnu-2422.90.20/bsd/kern/kern_resource.c:dosetrlimit(), the kernel does not handle RLIMIT_AS or RLIMIT_RSS in the switch(which) section. It merely records the updated limit by setting |*alimp = *limp| on line 922. Effectively, this means the operation is a no-op.

Without proper support for setrlimit(RLIMIT_AS, ...) or RLIMIT_RSS, it is not possible to restrict the size of a process's virtual address space. It also means that the aforementioned launchd.plist keys have zero effect, too.

Steps to Reproduce:
I've attached a sample program (rlimit-as.c) that should fail to allocate in a loop. Instead on OS X, the process terminates without issue. Running this on another POSIX system (e.g. Linux) results in the allocation loop failing.

Output on Linux:
RLIMIT_AS : cur=18446744073709551615, max=18446744073709551615
RLIMIT_AS set to 51200
RLIMIT_AS : cur=51200, max=51200
Failed to allocate at chunk 0
Failed to allocate at chunk 1
Failed to allocate at chunk 2
Failed to allocate at chunk 3
Failed to allocate at chunk 4
Failed to allocate at chunk 5
Failed to allocate at chunk 6
Failed to allocate at chunk 7
Failed to allocate at chunk 8
Failed to allocate at chunk 9
Failed to allocate at chunk 10
Failed to allocate at chunk 11
Failed to allocate at chunk 12
Failed to allocate at chunk 13
Failed to allocate at chunk 14
Failed to allocate at chunk 15
Failed to allocate at chunk 16
Failed to allocate at chunk 17
Failed to allocate at chunk 18
Failed to allocate at chunk 19
Failed to allocate at chunk 20
Failed to allocate at chunk 21
Failed to allocate at chunk 22
Failed to allocate at chunk 23
Failed to allocate at chunk 24
Failed to allocate at chunk 25
Failed to allocate at chunk 26
Failed to allocate at chunk 27
Failed to allocate at chunk 28
Failed to allocate at chunk 29
Finished allocating!

Output on OS X:
RLIMIT_AS : cur=9223372036854775807, max=9223372036854775807
RLIMIT_AS set to 51200
RLIMIT_AS : cur=51200, max=51200
Finished allocating!

The Linux output is desired.

Expected Results:
Since setrlimit(RLIMIT_AS, ...) is defined in sys/resource.h, I'd expect it to work as advertised in the man page.

Actual Results:
setrlimit has no effect on the process's address space size.