Nonpaged Pool Resource Allocation Error (SRV Error 2019)

I currently have a clients server that every so often experiences a condition where it stops responding to network requests.

You can still ping the server, and it still processes some basic commands, like a shutdown command, but it does not allow browsing files or folders, and you cannot RDP to the server. You will also be unable to launch applications if you manage to logon to the server locally.

You may have experienced a similar condition after regaining access to the server you will most likely see that the System log is full of Errors. In my experience this will be Event ID 2019, Source, SRV.

The Server was unable to allocate from the system Nonpaged pool because the pool was empty.

I have seen this occur on Servers and Workstations, there are a lot of fixes suggested online, but so far nothing that has helped me track the usage of the Nonpaged pool.

This is somewhat of a pain because the error seems to happen with no early warning, you simply get a call one day from the client saying ‘can’t access the X drive’ and after a few minutes trying to RDP in, you resolve to logon to the DRAC/ILO and reboot.

I know to reboot the server now, because after trying many solutions i have not found one that will reverse the position, nor have i so far found the cause.

However, the purpose of this post is to show you that there is indeed an early warning of these events, if you look in the right place.

I am not suggesting this to be a long term fix or solution, just to allow you to ‘manage’ a reboot in advance of a server crash, and maybe save face if consistent ‘unexplainable’ crashes have started to make your client a little cranky.

First off you can use Process Explorer, one of the best monitoring tools around, and one I’m sure we have all used.

Process Explorer runs as an EXE and does not require installation, you can run it directly from the Microsoft website, or you can download it and run it.

When you load it up you get a detailed view of what is running on your system, more detailed than the standard task manager.

You can add a column to view the Nonpaged Pool usage, (Under View, Select Columns, Process Memory Tab)

Adding this column can help you track which process has a large amount of memory in the Nonpaged pool.

All of this is quite straight forward, and of course we could also use the Performance Manager tool to show us the current state of Nonpaged pool memory, or even Task Manager.

Process Explorer has the advantage of showing us a process by process breakdown, which i personally like more than Task Manager, as a process like Lsass.exe can have multiple things running within it and you cannot find that with Task Manager.

Just hover over the process you are interested in, and you can see some more information.

I am side tracking slightly by going into this much detail on Process Explorer, but if you are new to it, there are some great posts out there on how to track performance issues, and malware just with Process Explorer.

It can also show us a very crucial piece of information, if you are having problems with the Nonpaged Pool. That is, the limit of your Nonpaged Pool.

Click On the View menu, and then click System Information. Notice you see a lot of process and memory counters here, far more than i understand at least. The most interesting items are highlighted below.

As you can see it currently shows ‘No Symbols’. In order to get this to show useful data you will need to install the Debugging Tools, from the Windows SDK. You can view more info on how to configure this, in the excellent post by Mark Russinovich here. You may also find this KB article useful for the Symbol Server Address.

With Process Explorer correctly configured, you can view your Nonpaged Pool limit. Which is the missing piece of the puzzle if you want to know how long you have before another meltdown.

Knowing the limit of your Nonpaged Pool, can help you keep track of the memory usage, and using tools like GFI’s RMM you can set an alert on the Nonpaged Pool so that when it reaches X % full, you will have time to react. Also you may be able to better trace the fault when the Pool is almost depleted.

Once again, i am not suggesting this information be used as a permanent solution, but i also know that for a busy techie you may not have the time required for the detailed analysis needed to trace a problem like this, and this can possibly help you out.

The reason i set out to write this post was because the subject is something i am not particularly experienced with so i enlisted the help of David Overton to recommend any other tools i may have overlooked. Being the friendly chap he is, of course he obliged. Poolmon, was his recommendation. I realise now if i had bothered to sit down and read through the entire post i recommended earlier by Mark Russinovich i would have come across Poolmon anyway, but a recommendation from David is far better than me having to actually read something… right?

Actually that is another reason i wrote this up, whilst Mark Russinovich’s posts are incredibly detailed, they are a little overwhelming to me, as someone not understanding computers to that depth, so perhaps you can think of this as an executive summary, ok ok, a junior executive summary…

So back to Poolmon.

Seriously, you should go back and read this post the section ‘Tracking Pool Leaks’ is pretty cool and explains again in huge detail how you can track down a pool leak.

To get Poolmon installed you will need to download the Windows Driver Kit. This downloads as an ISO, so either burn it to a CD, (or DVD) or like me, extract it out using MagicISO. Once extracted, run the file KitSetup.exe

Great splash screen…

Choose to install the Tools. You may also want to chose to install the Debugging Tools if you have not installed those already.

You will need to chose an install path, and then the install should complete.

Navigate to the folder you installed to, and find the Tools folder.

Find the Other folder.

Inside here you will find a folder named after the CPU type you have, either amd64 or i386 etc..

Finally inside here you will find the famous Poolmon.exe

You can use the Command Prompt Here trick to launch a CMD window at this path, and then using this resource launch Poolmon with the relevant settings.

For a Nonpaged pool issue, i would suggest launching Poolmon.exe /b

Your CMD window will change to a lovely Blue, and you will have all sorts of info on screen i cant possibly explain to you however using the P key you can sort the output by Paged and Nonpaged type.

Again, pointing you back to this post we can use Poolmon to search for the difference between Allocations and Frees and hopefully one will jump out as taking up a large amount of memory.

I’m going to be employing these techniques myself over the next few weeks to see if i can indeed track down the faulting application or driver on my clients system, and ill post up the results if i do.

Update – Ok so i am updating this prior to even posting it. But wanted to add on that i just followed all of this on my clients server and may already have identified an issue.

There’s an update available for the Broadcom network teaming software that’s in use which i will apply asap.

Thanks to David Overton & Mark Russinovich for their advice and excellent blog posts for help on this one.