Windows Drivers Debugging

In this tutorial we are going to see how to setup a Debugging Environment for our Drivers. This is not a complete guide, it’s just a quick tour intended to give a fast overview of Windbg and problems involved into Driver Debugging.

The Problem

Setting up a full working Kernel Debugging Environment is often seen, by novices, as a pretty challenging task. Difficulties comes essentially by the fact that working with a Debugger at Kernel Level implies that entire machine is completely locked for a certain amount of time.

Here pop up the necessity to work with Two Machines ( a physical and a virtual target machine ) in order to enstablish the following relationship:

  • Host System used to debug and Remote Machine.
  • Guest System that acts as ther Remote Machine to be debugged.

Became clear that we need a debugger that supports:

  • Kernel Level Debugging
  • Remote Debugging

WinDbg debugger fits all our needs, so we must setup correct communication layer between Host and Guest and successively use Remote Debugging functionalities of our debugger.

Setting Up Communication Between Host and Guest

The approach that we will follow is identical to the one described by Lords of Ring0. Where the idea is to setup a Virtual Serial Port Connection that will be used by the Host that have WinDbg to access the remote machine.

You can follow the proposed step by step guide also in case you want to use VirtualBox VM.

When you have completed these steps, let’s build a Batch File that contains the necessary command line arguments to start a kernel mode debugging session:

windbg -b -k com:pipe,port=\\.\pipe\com_1,resets=0

The batch file need to be placed in the same directory where is located windbg.exe

Our First Session

If your installation is correct after executing windbg you should see something like this:

Wdd_1

Last line tells you that WinDbg is now holding the control of your Guest System, indeed by switching to the Virtual Machine, if things went well OS should be completely frozen.

But as you can see, the debugger alert us that there are some errors about Symbols, don’t worry it’s normal, we need to setup correct Debugging Symbols now.

Setting Up Symbols

Debugging Symbols are extremely important components that Correlates the Assembly with Source Code Level Informations, a great amount of informations necessary for a Correct and Painless debugging session. It’s important to distinguish between:

  • Public Symbols
  • Private Symbols

First thing is to verify that Windbg can find symbols, this is done by setting the Symbol Path, from command line:

kd> .sympath SRV*d:\DebugSymbols*http://msdl.microsoft.com/download/symbols

The above command assumes that the Symbol Path is D:\DebugSymbols\

Successively would be good practice to increase verbosity level of Symbol Loading Operations, we can use the following command:

kd> !sym noisy

And finally reload symbols:

kd> .reload /f

At this point Symbols if symbol reload and resolution goes well, our Windbg is working with the correct symbols.

Debugging Our Driver

System is now ready to debug our driver, we just need to deal with our Driver Project which is composed by:

  • Source Codes
  • Binary Driver File

The first step is to place the entire driver project (Src+Bin) into a shared folder between host and guest.
Wdd_2
Otherwise, the .sys need to be loaded into Guest Machine.

Just a side note, our debugger still holds the control of the target machine, to release control just press F5 until you see ‘Debuggae Running’.

Time now to instruct Windbg to hold control of our driver, let’s assume you want to Debug from early stages of Driver Execution, we must put a BreakPoint on DriverEntry() function.

So before starting starting the driver we must set a breakpoint in this way:

kd> bu module!name

Assuming that driver is called Test.sys

kd> bu Test!DriverEntry

Windbg still holds the control of the target machine, to release control just press F5 until you see ‘Debuggae Running’. We can now Load the driver with OSROnline DriverLoader:

Wdd_dl

After loading it, guest machine is again frozen, because Windbg stopped execution at DriverEntry point.

Organizing Work Space

Windbg is rich of views that can greatly improve debugging experience, here a screenshot of most handy ones:

200px-Wdd_view_scaled

Windbg Commands

In the following subparagraphs we will talk about Commands and Functionalities of Windbg that could come handy during the analysis.

The essential aim of these paragraphs is not to give a complete syntax description, but more easly to inform the reader about the existence of certain features.

It’s strongly suggested to people that wants further informations about Windbg Commands to read debugger.chm help file, delivered with Debbugging Tools.

Some of the commands samples reported are taken from kerned_debugging_tutorial.doc a file that can be found in the working directory of Windbg.

Managing BreakPoints

The simples for of breakpoint is the bp one, that can be used in the following forms:

kd> bp Module!Name

or

kd> bp address

As you have previously seen, to break into our driver we used bu, this is a Deferred Breakpoint.

If a breakpoint is set for a routine name that has not been loaded, the breakpoint is called a deferred, virtual, or unresolved breakpoint. Unresolved breakpoints are not associated with any specific load of a module. Every time that a new application is loaded, it is checked for this routine name. If this routine appears, the debugger computes the actual coded address of the virtual breakpoint and enables the breakpoint.

If you set a breakpoint by using the bu command, the breakpoint is automatically considered unresolved. If this breakpoint is in a loaded module, the breakpoint is still enabled and functions normally. However, if the module is later unloaded and reloaded, this breakpoint does not vanish. On the other hand, a breakpoint that you set with bp is immediately resolved to an address.

Should be now clear why we used in our sample an unresolved breakpoint instead of bp.

We can also see all breakpoints we defined, via bl (Breakpoint List Command)

kd> bl

e stands for enabled and d for disabled.

Breakpoints can be disabled via bd (Breakpoint Disable)

kd> bd bp_index

Otherwise can be permanently removed with bc (Breakpoint Clear)

kd> bc bp_index

It’s also truly important to mention Conditional Breakpoints. They cause a break to occur only if a specific condition is satisfied.

A conditional breakpoint is created by combining a breakpoint command with either the j (Execute If – Else) command or the .if token, followed by the gc (Go from Conditional Breakpoint) command.

Generically:

kd> bp Address “j (Condition) ‘OptionalCommands’; ‘gc’ “

A pratical sample:

or

kd> bp mydriver!myFunction “.if @eax = 0xa3 {} .else {gc}”

Break on myFunction only if eax register is 0xa3.

It’s also pretty interesting to put in evidence the fact that Windbg supports per Process and per Thread break conditions.

bp /p 0x81234000 SIoctl!SioctlDeviceControl+0x103

or

bp /t 0xff234000 SIoctl!SioctlDeviceControl+0x103

Respectively, Stop at the indicated point only if the EPROCESS is at 0x81234000, and Stop at the indicated point only if the ETHREAD is at 0xFF234000.

Data Displaying

We can Display the content of a range of memory in various ways, commands designed for that task have the following form:

d* [Options] [Range]

  • da -> ASCII characters.
  • du -> Unicode characters.
  • db -> Byte values and ASCII characters.
  • dw -> Word values (2 bytes).
  • dd -> Double-word values (4 bytes).

Thanks to Symbols support we can directly display also the values of local variables:

dv [Flags] [Pattern]

From a pratical sample:

kd> dv Irp
Irp = 0xff70fbc0

Pretty useful is also dt that displays information about a local variable, global variable or data type. This can display information about simple data types, as well as structures and unions.

kd> dt Irp
Local var @ 0xf795bc48 Type _IRP*
0xff70fbc0
+0x000 Type : 6
+0x002 Size : 0x94
+0x004 MdlAddress : (null)

another very common way to use dt is:

kd> dt module!struct

Example:

kd> dt nt!_ETHREAD

Data Editing

In the previous paragraph we have seen how to read data, but undoubtedly we have to know also how to modify data.

For this scope, windbg offer us:

kd> e* Address [Values]

pattern is similar to display command:

  • eb -> Byte values.
  • ew -> Word values (2 bytes).
  • ed -> Double-word values (4 bytes).

If our editings needs require to fill entire blocks of memory with some static pattern, we can use f (fill) command:

f Range Pattern

Examining a BugCheck

Investigating causes of a crash is a core aspect of driver debugging. We have at disposition a pretty handy debugging extension called !analyze. This extension displays information about the current exception or bug check, most common way to use it, is:

kd> !analyze -v

!analyze is a great starting point to investigate

  • Live Debugging Issues
  • PostMortem (DumpAnalysis) Issues

Here a quick real case of !analyze usage:

Windows XP Version 2600 (Service Pack 2) UP Free x86 compatible
Product: WinNt, suite: SingleUserTS
Machine Name:
Debug session time: Sun Feb 20 04:44:08.000 2011 (UTC – 4:00)
System Uptime: not available
Process Uptime: 245 days 0:21:09.000
Kernel time: 0 days 0:00:02.000
User time: 0 days 0:00:16.000
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************

As you can see we have and handy summary of the Faulting Module plus Details about the Fault. We have enough points to start an in depth analysis, here the starting points that we should keep always in mind:

  • Bucket ID
  • Stack Trace
  • Exception Parameters

Let’s see now the commands that can help us to analyze these points.
Stack Backtrace can be displayed with:

kd> k
ChildEBP RetAddr
f7428ba8 f889b54a SIoctl!PrintIrpInfo+0x6 [d:\winddk\3790.1824\src\general\ioctl\sys\sioctl.c @ 708]
f7428c3c 804e0e0d SIoctl!SioctlDeviceControl+0xfa [d:\winddk\3790.1824\src\general\ioctl\sys\sioctl.c @ 337]
WARNING: Stack unwind information not available. Following frames may be wrong.
f7428c60 80580e2a nt!IofCallDriver+0x33
f7428d00 805876c2 nt!CcFastCopyRead+0x3c3
f7428d34 804e7a8c nt!NtDeviceIoControlFile+0x28
f7428d64 00000000 nt!ZwYieldExecution+0xaa9

Now suppose that we are working on a MultiThreaded target, we can introduce at this point ~ which is a Per-Thread-Identifier:

  • ~. -> The current thread.
  • ~# -> The thread that caused the current exception or debug event.
  • ~* -> All threads in the process.

In our case:

kd> ~*k

Will show the stack backtrace of each thread, awesome isn’t it? ๐Ÿ™‚

Exception analysis is another important aspect of fault analysis. The core command is .ecxr that displays the context record that is associated with the current exception. Please note that .ecxr is available on Crash Dumps only (minidumps)

A dump can be created with .dump command:

kd> .dump Options FileName

Finally here a list of commands that could come really handy during debugging
Useful Commands

Debugger Extensions

Together with commands we have also a large set of Extensions, they are of great help in case of Well Defined Specific Debugging Scenarios. Where we can detect a well defined pattern, we can use or write extensions.

Extension commands are exposed by DLLs distinct from the debugger.

An extension is Loaded with:

kd> .load DLLName

and Unloaded with:

kd> .unload DLLName

To List Loaded Extensions:

kd> .chain

It’s also pretty easy to distinguish between a classical command and an extension:

  • Commands starts with .
  • Extensions starts with !

Debugger extensions are invoked by the following syntax:

![module.]extension [arguments]

Handy Extensions

In this subparagraph we will see some available extension specific for Driver Debugging.

The !process extension displays information about the specified process, or about all processes, including the EPROCESS block.

kd> !process [/s Session] [/m Module] 0 Flags ImageName

and here a classical usage + output:

kd> !process 0 0
**** NT ACTIVE PROCESS DUMP ****
PROCESS 80a02a60 Cid: 0002 Peb: 00000000 ParentCid: 0000
DirBase: 00006e05 ObjectTable: 80a03788 TableSize: 150.
Image: System
PROCESS 80986f40 Cid: 0012 Peb: 7ffde000 ParentCid: 0002
DirBase: 000bd605 ObjectTable: 8098fce8 TableSize: 38.
Image: smss.exe
PROCESS 80958020 Cid: 001a Peb: 7ffde000 ParentCid: 0012
DirBase: 0008b205 ObjectTable: 809782a8 TableSize: 150.
Image: csrss.exe

Pretty similar is the !thread extension, that displays summary information about a thread on the target system, including the ETHREAD block.

kd> !thread [-p] [-t] [Address [Flags]]

The !drvobj extension displays detailed information about a DRIVER_OBJECT.

kd> !drvobj DriverObject [Flags]

The !devobj extension displays detailed information about a DEVICE_OBJECT structure.

kd> !devobj DeviceObject

The !irp extension displays information about an I/O request packet (IRP).

kd> !irp Address [Detail]

The !handle extension displays information about a handle or handles that one or all processes in the target system own.

kd> !handle [Handle [KMFlags [Process [TypeName]]]]

The !pool extension displays information about a specific pool allocation or about the entire system-wide pool.

kd> !pool [Address [Flags]]

Driver Specific Extensions

Windows is composed by a wide range of drivers typologies, we have a number of extensions designed to help in that sense, here a quick list:

  • !ndiskd -> NDIS Debugging
  • !wdfkd -> Kernel-Mode Driver Framework Debugging
  • !wudfext -> User-Mode Driver Framework

and so on, for a complete list you can check this link Specialized Extensions

Link References

OSROnline
Analyze-v
NtDebugging
WinDbg-Info
Collection of Links

The End

Here ends this little tutorial, again would like to put in evidence that this is only a reference tutorial, intended to offer a path to guide an hypothetical novice and give him, searching starting points.