Wild World of Visual Studio -- Proxies confront Hooks

How can Internet Explorer settings interfere with productivity add-in to Visual Studio, such as ReSharper? And even lead to Visual Studio crash when doing some seemingly unrelated operations, like checking out source code from Team Foundation Server?

Story 3 -- Proxies confront Hooks

We've got a number of reports about Visual Studio crash when doing repository operations using TFS. Of course, when ReSharper is disabled, no crashes occurs, which make users think ReSharper is guilty. We couldn't reproduce the problem in the labs, and we again tried to assert/log/verify everything we can think about. We even got some dumps with the crash, but could not find anything really useful there, besides evidence that there are numerous SecurityExceptions instances in the managed heap waiting to be collected. We needed reproducible case with WinDbg attached. Thanks to Vasily, we got one.

It is funny how we were debugging this crash. Since we were not able to reproduce the problem in the labs, we needed access to the faulty computer. However, due to corporate rules on the user's side, we were not able to arrange real remote session. But we had WinDbg with command line interface, and we had instant messaging. So I was sending WinDbg commands, he copied them into WinDbg, executed, and then copied back the result. We laughed about it and were kidding about extension to WinDbg which would be Jabber client :)

After several days being in this copy-paste process, we finally found the problem. Three components playing together made it explosive:

• Connecting to web site with HTTP when automatic proxy configuration script is enabled (It is basically JScript function which gives proxy address for given host)
  
• Managed hooks installed (here is ReSharper's role in the problem, but could be any other plugin)
  
• Pumping messages during JScript code execution
  

Remove one, and all is fine again. So, what is happening in our case? Now, watch my hands.

1. TeamFoundation client is going to communicate to server.
   
2. It initiates connection to web service.
   
3. HttpWebRequest is asking for proxy information.
   
4. WebProxy detects that automatic proxy configuration script is enabled.
   
5. AutoWebProxyScriptEngine downloads script and uses JScript engine to execute it
   
6. JScript engine restricts code permissions to execute only.
   
7. Exception happens while executing JScript and dialog is going to be shown with error information. This is who pumps messages.
   
8. Our managed hook is executed and attempts to use CallNextHookEx to pass data to other hooks.
   
9. Code Access Security system demands UnmanagedCode permission.
   
10. Due to restricted mode in which JScript executes, demand fails and SecurityException is rised.
    
11. Exception causes corruption of some internal Visual Studio COM objects, which instantly leads to Access Violation.
    

Well, actually, this one was our fault. I added SecurityPermissionAttribute asserting UnmanagedCode permission to the hook procedure and ReSharper is shining like a diamond again.

Case closed, lessons learned.

Wild World of Visual Studio -- Mysterious Component

This one was bugging us for a long time, until we got message from Jay (thanks!) who had similar problem and was willing to help.

Story 2 -- Mysterious Component

The problem itself is mysterious. From time to time it happens that Visual Studio stops loading any add-ins. ReSharper is installed, but its menus are suddenly all gray and it is not listed in the "Tools \ Add-in manager" dialog. It happens after installing various products, it could be service pack, SQL server tools, Visual Studio components - anything! Some users install them and all works fine. Some users loose their favorite productivity tool.

Logging, debugging, trying to reproduce for may be several years(!) led to no result. ReSharper was simply not loaded into the process, though all registration information appears to be in the right place. We checked encodings, verified files are not corrupted, verified registry access rights - all we can think of. As you can guess now, the reason was not anywhere near.

After lots of email exchanges with Jay, including mini-dumps, Process Monitor traces, registry excerpts, configuration files and such we suddenly found ourselves staring at the right thing. It was msxml6.dll. Actually, it was the fact that there were no msxml6.dll in the call stack. Instead, there was msxml3.dll.

We use .addin files to register our add-in in Visual Studio. It is XML file describing add-in, containing information about primary assembly, descriptions, load options and Visual Studio version compatibility information. The latter was that msxml3 was not able to process and thus Visual Studio refused add-in as non-compatible. In fact, msxml6 was there in System32, but it somehow happened to be not registered as COM object. Using regsvr32 on msxml6.dll repaired the system, resurrected ReSharper and enabled customers to enjoy our productivity add-in again.

Case closed.

Wild World of Visual Studio -- Unfriendly Package

While developing "The Most Intelligent Add-In To Visual Studio" we often face issues that originates from the outside of our code. Being that Visual Studio itself, other packages and add-ins installed, OS components or other things added to the mix - it is always extremely hard to reproduce, debug and understand. Usually we have to perform post-mortem debugging, when all we have is mini-dump from the user experiencing the crash. And we are very happy when we have mini-dump! Often we have to resort to psychic debugging, speculating about what could have happened, trying this and that until we find the cause and issue the fix.

I though I could share some stories of pure insanity we happened to participate in recently. May be another add-in developer is scratching his head in an attempt to understand the source of strange sounds from beneath the ground, and these stories will help. Or it may be just interesting reading for you. Or not :)

The first one is the most recent one and is pretty simple.

Story 1 -- Unfriendly Package

Our customers were complaining about Visual Studio crashing on start with ReSharper installed when running non-admin. Disabling ReSharper allows Visual Studio to load normally. When you see such behavior, you are absolutely sure ReSharper is guilty, aren't you?

We've learned to be cautious in such cases. So I asked user (thanks Jon!) to capture mini-dump and send it to me for investigation. While downloading it, I was preparing myself to long hours of WinDbg magic, but !ClrStack revealed it in a second. SnippetDesigner was not able to access its files, since it placed it in privileged folder. It did so in the background thread and didn't catch exceptions. The result is CLR termination, which get Visual Studio to nowhere with itself.

But wait! Why disabling ReSharper helped to avoid crash? Well, actually crash happens during SnippetDesigner package loading. Normally, it occurs when its window is opened (and it indeed crashes VS at this point without ReSharper). However, ReSharper uses Visual Studio API to get some configuration information, which can be extended with packages. So, by calling this API ReSharper caused those packages to load, essentially triggering the bug in SnippetDesigner on startup.

Case closed, opened another one.