Most of you probably know that when compiling release, and starting the debugger (F5) you get different default memory allocation than when just running.
This is the debugger heap. It is not the Debug CRT heap.
It's actually quite nice, but can be slow if you're allocating/freeing a lot of memory. This is something OpenSG's 3ds-importer does, and the difference is a 3-5 times slowdown when debugging.
Usually I haven't had performance problems, but rather Heisenbugs that disappear when debugging (since allocated memory is 0xbaadfood instead of pseudo-random), which I've solved by attaching the debugger afterwards. Those are not many, so it's been ok to do that at times.
However, our current app was very slow to load when starting with the debugger. We finally managed to prod Google in the right direction, and ended up here.
The solution is to set _NO_DEBUG_HEAP=1 in your environment. (A user-specific setting on the startup project is probably the best.)