How to Set Up Kernel Debugging For Windows in VMWare ESX

You’ll know from my previous blog posts that we have an extensive virtual machine infrastructure here at Electric Cloud (the Continuous Delivery company) to support our build and test process (as many of our customers do). Thought I’d share a neat trick I picked up while rolling out our system.

Most of us won’t need to debug at the kernel level with any real frequency. However, when you do need to debug at the kernel level in Windows virtual machines, this may come in handy. The instructions given below will allow you to set up a Windows kernel debug session in a virtual environment.

The Virtual Machines

Getting started: You will need two virtual machines, both deployed on the same physical ESX server. That may mean that you manually deploy them to a specific ESX server VMware Infrastructure Client, or use a LabManager Configuration that contains both machines, or specifically target an ESX server in LabManager as you deploy the configuration.

The Debugee

This is the machine that we will connect to and that is being debugged. The general setup is fairly straight forward:

Log into VMware Infrastructure Client:

  1. Deploy/locate the machine you want to debug in the tree view at the left.
  2. Click on that name in the tree view.
  3. Make sure the virtual machine is powered off. If there is a link in the Getting Started panel for this virtual machine, you need to either connect to the console of this machine and power it off, or press the Power off virtual machine link in the Getting Started panel.
  4. Click on Edit virtual machine settings in the Getting Started panel for this virtual machine.
  5. On the Hardware tab, click the Add… button to add a new device.
  6. Choose Serial Port and push Next.
  7. Choose Connect to named pipe and push Next
  8. Enter a name for the pipe (for example: mypipe).
  9. Pick Server for the near end.
  10. Pick A virtual machine for the far end.
  11. Make sure you choose Connect at power on.
  12. Deselect the Yield on CPU Poll checkbox.
  13. Push the Next button.
  14. Check the summary and push the Finish button.
  15. Push the Ok button in the main Settings dialog (for me the serial port label remains in “…adding” forever, I assume that is just a refresh problem).
  16. Deploy the virtual machine (same ESX server as the other machine).

Log into the machine as Administrator:

  1. Right click on My Computer.
  2. Choose the Properties menu item.
  3. Click on the Hardware tab.
  4. Click on the Device Manager button.
  5. Expand the Ports tree item.
  6. Right click on the newly created COM port and choose Properties. Sometimes these are numbered out of order, depending on what is already there, but usually this will be COM1. If there are multiple ports, and you are not sure which one, I recommend just changing all of them.
  7. Click on the Port Settings tab.
  8. Choose Baud Rate: 115200.
  9. Choose 8, None, 1, None for the other parameters (defaults).
  10. Push Ok and close all the dialogs.
  11. Edit c:boot.ini (and add the /debugport and /baudrate options):
    • multi(0)disk(0)rdisk(0)partition(1)WINDOWS=”Microsoft Windows XP Professional” /noexecute=optin /fastdetect /debugport=com1 /baudrate=115200
  12. Reboot the machine

The Debugger

The machine that runs the kernel level debugger controls the machine being debugged. Again… once you know what you need, the setup is fairly straight forward:

Log into VMware Infrastructure Client

  1. Deploy/locate the machine you want to debug in the tree view at the left.
  2. Click on that name in the tree view.
  3. Make sure the virtual machine is powered off. If there is a link in the Getting Started panel for this virtual machine, you need to either connect to the console of this machine and power it off, or press the Power off virtual machine link in the Getting Started panel.
  4. Click on Edit virtual machine settings in the Getting Started panel for this virtual machine.
  5. On the Hardware tab, click the Add… button to add a new device.
  6. Choose Serial Port and push Next.
  7. Choose Connect to named pipe and push Next
  8. Enter a name for the pipe (for example: mypipe. IMPORTANT! This is the same as the other machine).
  9. Pick Client for the near end (IMPORTANT! This is different than the other machine).
  10. Pick A virtual machine for the far end.
  11. Make sure you choose Connect at power on.
  12. Deselect the Yield on CPU Poll checkbox.
  13. Push the Next button.
  14. Check the summary and push the Finish button.
  15. Push the Ok button in the main Settings dialog (for me the serial port label remains in “…adding” forever, I assume that is just a refresh problem).
  16. Deploy the virtual machine (same ESX server as the other machine).

Log into the machine as Administrator:

  1. Right click on My Computer.
  2. Choose the Properties menu item.
  3. Click on the Hardware tab.
  4. Click on the Device Manager button.
  5. Expand the Ports tree item.
  6. Right click on the newly created COM port (sometimes these are numbered out of order, depending on what is already there, but usually this will be COM1), and choose Properties.
  7. Click on the Port Settings tab.
  8. Choose Baud Rate: 115200.
  9. Choose 8, None, 1, None for the other parameters (defaults).
  10. Push Ok and close all the dialogs.
  11. Download and install windbg.

Debugging

So now we have two machines, eagerly awaiting debugging.

  1. Go to the debugger machine and start windbg.
  2. Click on the File menu button.
  3. Choose the menu item Kernel Debug.
  4. Enter the proper baud rate (from before), and COM port.
  5. You can now press Control-Alt-K to initiate a connection
  6. Have fun looking at assembler code, threads etc.

Interesting Stuff

One interesting thing to note is that on the ESX server (which is Linux based), named pipes are currently always created under /var/log/vmware, and should be just regular file names. During my research I created a named pipe there directly from the OS, which resulted in VMWare throwing up a dialog box stating that the named pipe was already there. It then offered to overwrite the named pipe, which I did.

Based upon my research I expect that you could do the whole setup falling back to the physical serial port on different ESX servers and then use a serial cable, but I haven’t tried that. Happy debugging!

Electric Cloud
Follow us

Electric Cloud

Electric Cloud is the leader in DevOps Release Automation and Continuous Delivery. We help organizations like E*TRADE, Gap, HPE, Intel and Lockheed Martin deliver better software faster by orchestrating, automating, and accelerating application releases.
Electric Cloud
Follow us

Share this:

4 responses to “How to Set Up Kernel Debugging For Windows in VMWare ESX”

  1. steve heller says:

    I’m trying to follow these steps to connect a kernel debugger between two ESX virtual machines, but I don’t know exactly what to put in the windbg startup parameters to connect to the target virtual machine. The problem seems to be the com part. Right now I have that as:
    -k com:port=\109091584855338,pipe,baud=115200,resets=0,reconnect

    But that pipe name seems to be invalid. I’ve already tried what I would use for the pipe name if using two Hyper-V VMs, namely “\.pipe109091584855338”, with no success. What should I have for the -k setting?

    Thanks!

  2. sdelmas says:

    The named pipe you create with the above steps won’t be exposed to windbg. At the application (windbg) level you will just use the created com ports. So I would expect more something like -k com:port=com2 (where com2 is the newly created com port).

    I tried doing a shared pipe setup myself for some time, but that just never worked properly. I hope this helps…

  3. steve heller says:

    Thanks for the reply. By the way, I found some additional instructions here:

    http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007086

    and am trying it at present. I’ll let you know what I find out.

  4. There may be few questions rising in your thoughts concerning machinery. Where I can buy this equipment? Methods to get one of the best inexpensive costs? The right way to test the standard of the machinery? Yes, these questions are obvious and you’re going to get the solutions as properly on this article.

Leave a Reply

Your email address will not be published. Required fields are marked *

Subscribe

Subscribe via RSS
Click here to subscribe to the Electric Cloud Blog via RSS

Subscribe to Blog via Email
Enter your email address to subscribe to this blog and receive notifications of new posts by email.