Paging file conundrum

[Zero Angel]

I'm running XP home on my desktop. My computer has 3GB of RAM, and i've allocated 2.7GB of space to the paging file (which is probably not even needed at all). My antivirus is Avast! and I don't run a lot of tasks in the background. So i'm curious as to why I get warnings every once in awhile that my virtual memory is low.

What kind of programs might I use to find out where all the RAM is allocated? Task Manager doesnt report everything (if I added together all of the memory usage values of everything on task manager, I might get ~500MB or so of total usage, out of 3GB RAM+2GB paging)

[Zero Angel]

I downloaded a program called Meminfo, which does little else than display memory information in the system tray, 'defragment' memory, and launch task manager. It reports:

Physical memory: 533/3,071 MB
Paging file: 2,946/5,645 MB

Which seems much saner. I'm still interested in knowing where that ~3GB of system memory is going. My best guess is into caching various things for quicker retrieval later, but I want to be sure.

[OvermindDL1]

Oh freaking heck, another VM thread.  Ignore my tone but absorb my message, I have said this all way to many times and it is getting way too repetitive.

First of all:  USING VIRTUAL MEMORY IS NOT A BAD THING
Now, continuing with that thought, first of all, you *MUST* have at least as much virtual memory as you have of RAM as Windows will perform a ram dump upon any kernel level crash to the VM, without enough VM to hold all the RAM it crashes during that process saving no information.

Even if all of your running apps fit nicely into the monstrous amount of RAM you have, Windows will still page things out that are not often used for the purpose of being able to rearrange ram into better chunks for passing to the cpu.

All of the running programs also use dynamic libraries (even if it is just the basic must-link-to kernel library), and all library code is loaded only once into RAM, meaning that although a single program may be taking up, say, 100 megs, 20 megs of that may be from dynamic libraries that other programs are also SHARING, hence why you may manually count up more then what windows is otherwise telling you, but believe me, the computer can count a whole lot better then you can, trust the number Windows actually gives you for the overall RAM usage.

Windows will also use unused RAM (or paged out RAM) to cache things like, you know, the Icons on your desktop and all sorts of other things, if it had to redraw every single icon in every single place of every single program on your computer, you would then be able to actually watch it redraw everything (which it can still do if you use up so much memory so fast then it has to dump the cache and rebuild it next time an Icon is viewed).  It also cache's far more important things then just Icons such as pages of files and so forth.

Anyone who uses a program that 'manages' RAM on a windows computer that is NT4 based or higher (NT4, 2k, XP, 2k3, Vista, etc... and of course linux/unix/bsd machines) are (sorry to put this blunt) an idiot for being fooled by the makers of the thing.  First of all RAM is, hence the name RANDOM ACCESS MEMORY, it cannot get fragmented, hence a fragmentation will not work for a variety of reasons, the biggest few being that the RAM paging is controlled by the kernel, no program running inside of Windows could hope to control that, it is quite literally not possible, and the way they 'try' to work is usually something monumentally stupid by allocating a metric ton of memory really fast, forcing it to page near everything out to the VM while dumping all the cache Windows has managed to build up so far, then free it, causing everything to have to be paged back in and to rebuild the cache, it vastly harms efficiency.

It is also monumentally stupid (except in *VERY* rare server-like only circumstances, certainly nothing any of you would run into) to set a specific amount of RAM, *always* let windows (on 2000, XP, etc... and higher) manage it itself, it does a far better job then you ever could since it knows how the memory will be used, most used periods, etc... and so forth.

An example, I only have 2 gigs of ram, but I have used enough memory on my computer at one time once so that it had to dynamically allocate over 10 gigs, if I did not have windows controlling it then most of my programs would have started dieing due to out-of-memory exceptions being thrown all over the place.  Right now (I restarted my computer less then an hour ago, I usually restart once every ~40 days) I am using 1 gig of RAM, and 1.5 gigs of page file, reason being that many of my processes and services are idle so they have been paged out, and the 1 'unused' gig is now being used for a cache for a lot of things (and I do mean a lot of things on my computer right now).

VM is also useful for another reason, many programs that need heavy file optimization access of very large files, well, the programmer would be monumentally stupid to access a file using the usual methods because the usual methods are very slow, kernel switches and other such uglyness.  However, there is a way (which I use quite often) to tell Windows (and *nix/BSD/Unix/etc...) to actually 'map' a file on the drive into the VM space of the program, then you can access the file just like you access memory.  The reason why this is faster is because VM switches occur at interrupt level, a level even lower then the kernel and usually does not incur a costly world switch like normal file access causes.  That alone is not even the extent of the optimization as when a program access a file using normal means it has to do costly lookups to see if any other program is accessing it (and it has to check every time even if a tiny word part is read from a file), as when it is mapped directly to memory it is treated as normal VM at that point, with the full capabilities that implies, such as *very* fast paging switching, you can 'load' a file larger then the amount of RAM you have (you cannot do that using normal file access without a lot of ugly and slow state keeping), and you have exclusive access to it (it locks out normal file access), *BUT* any other app can also mount it into its VM as well, and they share it, including all changes, they can use it like normal shared process memory, at RAM speed, and when they finally close it out the Windows VM manager writes it back out to disk and closes it, just as if it was paging a normal piece of RAM to the normal VM store, and it is a faster operation then normal disk access as it bypasses all the normal crud.

And there are even more things I could say, but I bet you can get the gist of it.  Basically, Windows is a lot better then you with the RAM, trust it, including with counting it up, and do not trust *ANY* program that says they help out in any way shape or form with RAM or VM usage.

And yes, the Task Manager is telling you the correct amount of RAM used, remember, programs share a lot of their code, and in some cases, even share a lot of their memory.

If you are getting virtual memory is low messages, just make sure windows is set to allocate it itself (do not set a predefined amount), and if windows needs more it will allocate more (usually a one-time affair, ever, but can freeze the computer for a good 30 second while it reshuffles stuff around, trust it).
Generated by OvermindDL1's Signature Auto-Add Script that OvermindDL1 did manually since Greasemonkey does not work in Firefox 3.1 yet...

[Zero Angel]

My original question was about finding out where the additional RAM was going. I was hoping for a program or method I could use to find out where it was going (ie: various caches, etc) -- rather than simply 'well this is being used'

It is also monumentally stupid (except in *VERY* rare server-like only circumstances, certainly nothing any of you would run into) to set a specific amount of RAM, *always* let windows (on 2000, XP, etc... and higher) manage it itself, it does a far better job then you ever could since it knows how the memory will be used, most used periods, etc... and so forth.

How is it "monumentally" stupid? Does it somehow measure up to the same level as "rm -rf ." or "format c:"? I've heard from multiple sources (including Lockergnome, Lab with Leo, etc) which advise setting static paging file sizes, with the underlying logic here being to prevent swap file and from an ever growing swapfile from fragmenting other files. It made sense to me at the time.

Oh, and i'm aware that defragmenting RAM is useless since the 'seek' time or whatever other 'proper' definition you want to give to it is virtually nil, compared to a hard drive. Hence the "quotes" around the word "defragment" -- NOTE: quotes around things sometimes imply "sarcasm" in case you didnt guess. 

(BTW: What is the proper policy for walking on eggshells here when it comes to using proper terminology? or can I use a best fitting description, assuming that you have some kind of basic intuitive ability to understand what i'm saying?)

An example, I only have 2 gigs of ram, but I have used enough memory on my computer at one time once so that it had to dynamically allocate over 10 gigs, if I did not have windows controlling it then most of my programs would have started dieing due to out-of-memory exceptions being thrown all over the place.  Right now (I restarted my computer less then an hour ago, I usually restart once every ~40 days) I am using 1 gig of RAM, and 1.5 gigs of page file, reason being that many of my processes and services are idle so they have been paged out, and the 1 'unused' gig is now being used for a cache for a lot of things (and I do mean a lot of things on my computer right now).

I understand that Windows XP and later will attempt to utilize all available swappable space, but shouldnt windows purge the least needed parts of the cache when it reaches a certain amount of RAM/Paging usage rather than expanding the swap file infinitely?

Systems with 512MB of RAM use far smaller smaller paging files than the (I assumed copious) 3GB that I allocated.

Secondly, why is everyone an 'idiot' for not having several years of IT experience? Learned knowledge != intelligence (aka: capacity to learn new things, solve problems, understand complex concepts).  Frankly your pedantry and smug self superiority is irritating.

Next time just point me to the thread where you previously discussed this and keep your condescending tone to yourself.

[OvermindDL1]

My original question was about finding out where the additional RAM was going. I was hoping for a program or method I could use to find out where it was going (ie: various caches, etc) -- rather than simply 'well this is being used'

It is being used by far too many things to be able to list in any concise way is the issue, most of it is just binary blobs that any program would not be able to make sense of anyway.

How is it "monumentally" stupid? Does it somehow measure up to the same level as "rm -rf ." or "format c:"? I've heard from multiple sources (including Lockergnome, Lab with Leo, etc) which advise setting static paging file sizes, with the underlying logic here being to prevent swap file and from an ever growing swapfile from fragmenting other files. It made sense to me at the time.

Monumentally in the way that there is no reasoning to not have it dynamic, and many reasons to have it as such on modern systems.  Back in the 9x system days the dynamic method was so poor and buggy that it was preferred to have it static, but on NT based systems those issues did not exist.  You should always be defragmenting your drive anyway (if not *nix/bsd/etc..), and considering that the paging file rarely grows, as stated, generally only once through the life of a computer, that is a non-issue

Oh, and i'm aware that defragmenting RAM is useless since the 'seek' time or whatever other 'proper' definition you want to give to it is virtually nil, compared to a hard drive. Hence the "quotes" around the word "defragment" -- NOTE: quotes around things sometimes imply "sarcasm" in case you didnt guess. 

There is no seek time, not that it is nill, the very definition of random accessible memory means that every position has the same lookup time from any position, it is as fast as the silicon can switch.

I understand that Windows XP and later will attempt to utilize all available swappable space, but shouldnt windows purge the least needed parts of the cache when it reaches a certain amount of RAM/Paging usage rather than expanding the swap file infinitely?

And indeed it does.  Windows does feel free to dump the cache to make room for actual programs, and it will not grow the VM file just for more cache, it only uses unused space for cache, and will only grow the VM if actual programs require it.

Systems with 512MB of RAM use far smaller smaller paging files than the (I assumed copious) 3GB that I allocated.

It all depends on the programs you are using and how much memory they require to run, but as stated, the allocated VM must always be at minimum the size of the RAM for Windows to function efficiently.  Also, when the computer has less then or equal to 512 megs of ram it runs in a different mode where it does very little caching, hence one reason why you can get such a speed boost in windows going from 512mb- to 1gig+, but no where anywhere near such a boost going to any other amount.

Secondly, why is everyone an 'idiot' for not having several years of IT experience? Learned knowledge != intelligence (aka: capacity to learn new things, solve problems, understand complex concepts).  Frankly your pedantry and smug self superiority is irritating.

Nothing to do with IT experience, it is just that this question has been beaten to death by so many people by now that it surprises me that anyone that knows what RAM is and would know enough to ask these questions do not know the actual answers first with how often they have been stated.  It is just something that is considered 'common knowledge' now.

Next time just point me to the thread where you previously discussed this and keep your condescending tone to yourself.

Those old threads were deleted by CmptrWz not long ago if you do recall, hence it is a little difficult to link to non-existent threads.

And for note, putting quotes around something generally indicates something to focus on (Spelling, design, exact wording, etc...), not sarcasm.  Sarcasm usually involves emoticons and so forth.
Generated by OvermindDL1's Signature Auto-Add Script that OvermindDL1 did manually since Greasemonkey does not work in Firefox 3.1 yet...

[AcneVulgaris]

"It is just something that is considered 'common knowledge' now."

As are 'social skills' 

[Red Devil]

Oh, don't mind OM.  Just think of him as Brainiac on meth.    Besides, he did say right up front to ignore the tone.   

Ever since GSH told me I should have 3 times as much paging as RAM, my rig has run very well.  I even have a separate drive that holds the paging file so that paging is not hampered by drive contention.

BTW, I read once that we don't need a paging since so much RAM is available now, so I tried setting it to zero and it failed miserably after a very short time, which supports what OM is saying.

[TheJamsh]

what id like, is a nice 'OM stylee' tutorial on how to get crappy 32-bit vista just plain HOGGING resources...

i mean, computers must ran faster than mine...

[Avatar]

Lol...

Treat OM like you would (STNG) Data...  no emotional subtext involved on a interpersonal level.

IOW embrace the message, ignore the messenger, it's nothing personal.   

***

Good stuff OM...  danke.

-Av-

[OvermindDL1]

Yea, bad work day along with a few other things (seems our division may be closed down soon and I do *NOT* want to be relocated to the other area for a variety of reasons), sorry about all that.

I am not sure about Vista.  I have had many years to find, experiment, and just all out break XP many times so I know just about all the best ways to make it run better.  My computer, when it starts up, has well over 120 processes running (various servers for various things), it only has two gigs of ram, and my video card is not that new, and yet I still get higher fps and greater responsiveness on maximum quality settings then my friends with better/faster computers.  I am very cheap, so I like to squeeze as much capability out of this thing as I can.  However, with Vista, every time I use Vista it just seems bog slow (and no, not the interface, I tend to do a lot more things to a machine then most anyone else does), and the only things I have noticed that is better is the TCP stack (I keep killing the one on my XP machine, forcing the stack to reset, thus losing network connectivity for a good 30 seconds), and thread switching.

I find it funny though in that Windows XP runs much faster then Vista on newer hardware, and yet Mac OSX 10.5 runs much faster then 10.4 on newer hardware; Windows is just going backwards (you would think that with newer hardware the older OS would not take better advantage of then the newer OS, as with Macs, the newer ones have more advantage with newer hardware, hence why the newer ones do not always run or run well on the older hardware, and yet Vista is just all around slower then XP on newer or older hardware, something is wrong with that...).

So yea, if you want your computer to be faster, get XP back.

I should have chronicled everything I have done to my windows install.  This install is 6 years old now and still running strong (and better then a new install), so if I ever need to re-install I will be pretty screwed in trying to remember everything I have ever done to this install.  That is why I do *not* want to re-install XP.  If I need to re-install an OS on this thing, it will be something else.  My XP install will remain until XP is an invalid (at this rate, will probably install Kubuntu...).
Generated by OvermindDL1's Signature Auto-Add Script that OvermindDL1 did manually since Greasemonkey does not work in Firefox 3.1 yet...

[Zero Angel]

I liked Kubuntu before they decided to change it soley to KDE4 (Intrepid Ibex). KDE4 still has some usability issues that need to be addressed, hopefully those will be worked out in the next LTS release (in April).

[OvermindDL1]

Yep, KDE4 is still in development, hence why I expect by the time I would convert solely to something like that, it will be well and done.
Generated by OvermindDL1's Signature Auto-Add Script that OvermindDL1 did manually since Greasemonkey does not work in Firefox 3.1 yet...

[Zero Angel]

Hmm, found some ways to tweak the system to make the most of my 3GBs of RAM. Posted them here

[OvermindDL1]

Useful.  I question the DLL caching section because there are some DLL's like the microsoft CRT's that are not used by really anything in Windows, but are used by a good portion of user programs out there, where reloading the CRT DLL can actually be quite costly and hit memory hard.  Windows will also only cache DLLs that are globally visible.  DLLs in an application directory will not be cached since they are generally only used by the app in the directory anyway.

Also, [HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Control/Session Manager/Memory Management/Prefetch/Parameters/EnablePrefetcher]  should be [HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Control/Session Manager/Memory Management/PrefetchParameters/EnablePrefetcher].  It is PrefetchParameters, not Prefetch/Parameters. 
Generated by OvermindDL1's Signature Auto-Add Script that OvermindDL1 did manually since Greasemonkey does not work in Firefox 3.1 yet...

[Zero Angel]

It is PrefetchParameters, not Prefetch/Parameters. 

That's what we call 'being a pedant'