Here’s a little something that stumped us for a few days and might be worth posting to save others time.
Following a move to IIS8.5, we started seeing “Out of resource” errors on a server that did not appear to be bottlenecked by disk, CPU or RAM.
It turns out that since a previous version of IIS, the Application Pool service doesn’t grab GDI handles as it runs as a non-interactive service, so anything relying on that, such as a DLL with GDI dependencies, like an image resizing library, only gets the non-interactive desktop heap for graphical services. As soon as you get enough calls into that DLL, the heap fills and the program crashes with the “Out of resources” error.
So you recreate the issue in the debugger, attached to IIS Express, running in user space, with the full interactive desktop heap, and you can’t recreate the issue.
To fix the problem, you need to carefully adjust the heap limit in one of the ugliest registry values in Windows. Have a read here to find out what the Desktop Heap is and the registry key that controls it, then up the 3rd SharedSection value (the non-interactive heap) in small increments (lest you put in a value too high, break the interactive heap and lose the ability to log on).
And then find a way to rewrite the DLL.