The data here was gathered using a CoProcessing module which loaded up ParaView, dumped the information from the status and maps files, then quit. Now that we have some tools at our disposal to try and figure out where our memory is going, let's look at ParaView itself. On 32 bit machines, this is gap is instead 4 KB since the address space is at a much higher premium. On a 64 bit x86 machine, each library uses 2MB for this buffer which seems like a lot until you realize that the memory is never actually realized in silicon (since any use causes a segfault) and 64 bit machines have access to at least 256TB of virtual memory to play with (and can be extended to the full 16EB if the need ever arises). This memory is used to help catch buffer overflows from trampling over the code and to assist in aligning memory to ease sharing across processes and are created by the binutils toolchain to separate the code section ( r-xp) from the writeable data section ( rw-p) in shared libraries. The oddball one is that one " -p" sections which is pretty large (two megabytes). The " r-xp", " rw-p", and " r-p" pages are fairly standard (executable code, writeable memory, and read-only data memory). The other field which interests us is the second one which is the permission bits for that section of memory. The first two should be fairly obvious, but the other two are created by the kernel and those are the pages where the code which actually talks to the kernel lives. Other things to take note of are the special, ,, and sections. After that, it's mostly the C library and the linker which are unavoidable for the most part. We can see here that the vast majority is that /usr/lib/locale/locale-archive file (which we can avoid see below). To find out how much memory ParaView is causing a process to consume, I used a small CoProcessing module and modified it to initialize ParaView and then print out its accounting information from /proc/self/status and /proc/self/maps.įirst, looking at the status file, we see something like (trimmed for brevity): Information on other files may be found in the proc(5) manpage. The maps file contains a list of all of the memory regions mapped to a process, their permissions, and what file they come from. The status file contains things like memory usage, historical memory usage, signal masks, capabilities, and more. Here, the two files which are important to us are the status and maps files. This filesystem is implemented with a little bit of kernel knowledge and magic to allow you to introspect what is going on in a process. On Linux, there is a filesystem called procfs mounted at the /proc directory. But what is actually taking up all of this memory when the build artifacts only take up a total of 150MB? To find out, we need to dig into how a kernel (specifically, Linux here) does its memory accounting.įirst, I'll go over the methods used to find out memory usage and what is using it. In a CoProcessing module, just loading up ParaView shoots your virtual memory space up by 500MB. When loading it up to use for handling data on a super computer, its memory footprint seem to be gargantuan. It includes over 3 million lines of C and C++ code and 250,000 lines of Python with a smattering of other languages such as FORTRAN, Tcl, and Java (accoring to the wonderful sloccount tool) in its build, and the build has more than 18,000 things to do when Python bindings are enabled.
0 Comments
Leave a Reply. |