.NET StackOverflowException is a special type of exception. Starting with .NET 2.0 you can’t catch this kind of exception with a try/catch block. Usually, it is a result of recursive method calls. Your program is out of free memory, and any clean-up does not make sense. Every time the .NET program has started, an instance of StackOverflowException is created “just in case”. This way framework can throw it (and terminate itself immediately) even if there is no free memory.
Tracking StackOverflowException can be very laborious and challenging. As I will show you below…
IIS (w3wp.exe) crashes
The default IIS App configuration will restart the website after a crash. It is even possible to schedule a periodic automated restart manually. So every time the app crashes, it will respawn with a fresh copy. Additionally, after a crash, an EventLog entry will be added. Unfortunately, from the log, I was not able to determine what has caused StackOverflowException. We need more info to diagnose and fix the problem.
Firsts approach - jit-debugger
Microsoft recommendation is to install Visual Studio Just_In-Time Debugger. After that, when a crash occurs, you will be presented with the following message.
When you decide to debug a Visual Studio instance will be opened, and you can examine threads, memory, load .pdb files and conduct more tests.
Unfortunately (again!) in this case, Visual Studio could not display the StackCall and I ended up with a pretty similar message as in EventLog.
WinDbg - the ultimate tool
WinDbg is a swiss army knife in Windows engineering ecosystem. Imho it is not a tool used on a daily basis by .net engineers, but it’s good to know about it. It has helped me a lot, also in the current case.
First, we need to have a process memory dump. We can capture it using Visual Studio when jit-debugger kicks in. In Debug menu there is a Save MemoryDump option.
We load the dump in WinDbg and run the following commands:
- .loadby sos clr - load .net symbols
- !sos.threads - show threads present in the dump
- ~<threadId>s - switch to the interesting thread
- !EEStack -EE
At this point, I was able to finally see which part of my code was causing the crashes.
Summary
In my case, the StackOverflowException was caused by a buggy LINQ Expression called every 5 mins by an automated process… Regardless of user actions on the website. To confirm that this is the actual issue, I’ve commented it out completely - the crashes were gone.
Extras - .NET diagnostics tools
Here is a list of .NET tools I usually use for various investigations.
- dotTrace
- dotMemory
- PerfMon
- Log Parser + GUI
- Process Explorer
- WinDbg