Category Archives: Uncategorized

Using Sysinternals ProcMon to capture registry changes

Posted on by

​I came across an issue to which  I needed to change a setting via Group Policy in Internet Explorer but could not find the matching GPO setting. It simply may have been under my nose in the GPO but I simply could not find it. So I went back to an old trick using Sysinternals ProcMon.

I figured if I can sleuth out what the registry key was then I could create a GPO preference which would obtain the results I needed to correct the issue. To find out what the registry entry was I simply opened up IE and navigated to the setting I needed to change. But before I made the change in IE I also opened up ProcMon in parallel and created a few filters to reduce some of the noise such as the registry icon….

!cid_image003_png@01CFCBBA

As well I wanted to see only the RegSetValue Operation as I was only interested in registry changes so I excluded all other registry queries etc….

!cid_image004_png@01CFCBBA

As mentioned I had IE open at the same time. As you see below I wanted to change the “Preserve Favorites website data” setting. I selected/deselected the parameter in IE and I was able to capture what the registry modification was…..

!cid_image005_png@01CFCBBA

Now that I know what the registry entry is I created a GPO preference to "deselect" the setting. Once again GPO’s are awesome.

Debugging XenApp Crash Errors

Posted on by

 

We have been getting reports of more and more IE and Outlook crashes. So to try and get a little more insight into this I am invoking the application crash dumps to try to see the issue. This is not limited to Citrix servers but can be done via Windows 7 or 2008 so if you are having issues with application crashes this may help you find the problem. Here is how I did this:
 
Enable Crash dumps on XenApp
 
Based on this article http://support.citrix.com/article/CTX118614 I created a GPO preference to enable this feature. Here is the configurations:

image

As you can see once I did  GPUPDATE I started to see some .dmp files. 
 

image

Install Debugging Tools

Then one needs to install the debugging tools. This can be found at http://msdn.microsoft.com/en-us/windows/hardware/hh852365 and the link within that page is http://www.microsoft.com/en-us/download/confirmation.aspx?id=8279 (Debugging Tools for Windows (WinDbg)
 

Executing the WindDebug command

Once you have the tools installed you navigate to the tools folder "c:\Program Files\Debugging Tools for Windows (x64)>cd "c:\Program Files\Debugging Tools for Windows (x64)".

The command to execute the tools is 

c:\Program Files\Debugging Tools for Windows (x64)>windbg -y srv*c:\symbols*http://msdl.microsoft.com/download/symbols -i c:\windows\i386 -z <path to dump files>

The path in red indicates that I simply used the Microsoft provided symbols online. The green I simply mapped a drive to the location of the files. In this case it is the D drive on the XenApp server. So my command looks like:
 
c:\Program Files\Debugging Tools for Windows (x64)>windbg -y srv*c:\symbols*http://msdl.microsoft.com/download/symbols -i c:\windows\i386 –z D:\Citrix\Crashdumps\iexplore.exe.22840.dmp
 
Once the command is executed the debug screen will appear like below:

image

 Analyzing the Crash Dump

Once the Windebugging tool executes you want to peform a more verbose output. This is done by simply entering in !analyze -v on the bottom line

image

The reason I created the preferences is that my XenApp servers are provisioned and read-only. As well that I will disable the preferences when I am done troubleshooting. I have a D: drive that I use which is local physical hardware to capture the dumps which also is where my vDiskCache, page file, etc live.
 
As you can see once I did  GPUPDATE I started to see some .dmp files.

 

Microsoft (R) Windows Debugger Version 6.12.0002.633 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [y:\Citrix\Crashdumps\OUTLOOK.EXE.25360.dmp]
User Mini Dump File with Full Memory: Only application data is available

WARNING: Inaccessible path: ‘c:\windows\i386’
Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: c:\windows\i386
Windows 7 Version 7601 (Service Pack 1) MP (24 procs) Free x86 compatible
Product: Server, suite: Enterprise TerminalServer
Machine Name:
Debug session time: Fri Mar  7 09:50:09.000 2014 (UTC – 8:00)
System Uptime: 0 days 2:15:23.903
Process Uptime: 0 days 0:00:08.000
……………………………………………………….
……………………………………………………….
……………….
Loading unloaded module list
………..
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(6310.5adc): Unknown exception – code c0000374 (first/second chance not available)
eax=00000000 ebx=00000000 ecx=7fffffff edx=00000000 esi=08940000 edi=00006310
eip=77bef8d1 esp=0804e6e0 ebp=0804e764 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!NtWaitForSingleObject+0x15:
77bef8d1 83c404          add     esp,4
0:023> !analyze -v
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

*** ERROR: Symbol file could not be found.  Defaulted to export symbols for rsintcor32.dll –
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for MSO.DLL –
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for OUTLOOK.EXE –
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for csma_ldr32.dll –
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for OLMAPI32.DLL –
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for SOCIALCONNECTOR.DLL –
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for EMSMDB32.DLL –
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for BCSAddin.dll –
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for Valkyrie.dll –
GetPageUrlData failed, server returned HTTP status 404

FAULTING_IP:
ntdll!RtlReportCriticalFailure+57
77c9e753 eb12            jmp     ntdll!RtlReportCriticalFailure+0x6b (77c9e767)

EXCEPTION_RECORD:  ffffffff — (.exr 0xffffffffffffffff)
ExceptionAddress: 77c9e753 (ntdll!RtlReportCriticalFailure+0x00000057)
   ExceptionCode: c0000374
  ExceptionFlags: 00000001
NumberParameters: 1
   Parameter[0]: 77cd4270

PROCESS_NAME:  OUTLOOK.EXE

ERROR_CODE: (NTSTATUS) 0xc0000374 – A heap has been corrupted.

EXCEPTION_CODE: (NTSTATUS) 0xc0000374 – A heap has been corrupted.

EXCEPTION_PARAMETER1:  77cd4270

MOD_LIST: <ANALYSIS/>

NTGLOBALFLAG:  0

APPLICATION_VERIFIER_FLAGS:  0

FAULTING_THREAD:  00005adc

BUGCHECK_STR:  APPLICATION_FAULT_ACTIONABLE_HEAP_CORRUPTION_heap_failure_block_not_busy_DOUBLE_FREE

PRIMARY_PROBLEM_CLASS:  ACTIONABLE_HEAP_CORRUPTION_heap_failure_block_not_busy

DEFAULT_BUCKET_ID:  ACTIONABLE_HEAP_CORRUPTION_heap_failure_block_not_busy

LAST_CONTROL_TRANSFER:  from 77c9f659 to 77c9e753

STACK_TEXT: 
0804ecb4 77c9f659 c0000374 77cd4270 0804ecf8 ntdll!RtlReportCriticalFailure+0x57
0804ecc4 77c9f739 00000002 7fb27b0e 00000000 ntdll!RtlpReportHeapFailure+0x21
0804ecf8 77c4e045 00000008 00710000 05579ee0 ntdll!RtlpLogHeapFailure+0xa1
0804ed28 77c12714 00710000 00000002 05579ee8 ntdll!RtlFreeHeap+0x64
0804ee40 77c125df 05579ee8 0000000c 05579ee8 ntdll!RtlpReAllocateHeap+0x190
0804eeb4 751e675b 00710000 00000000 05579ee8 ntdll!RtlReAllocateHeap+0x2c5
0804ef08 751e62e1 05579ee8 0000000c 751e6289 msvcr90!realloc+0x35f
0804ef24 6c2fbb5e 05579ee8 00000003 00000004 msvcr90!_recalloc+0x58
WARNING: Stack unwind information not available. Following frames may be wrong.
0804ef58 6c300b7c 02f0d378 6c31288c 00000007 rsintcor32!RslLoadedTerm+0x3ce2
0804efa0 6c2fcd1f 00000001 48a83599 02f0d378 rsintcor32!RslLoadedTerm+0x8d00
0804efd8 6c2fdaab 05578fe8 00000000 06b1c540 rsintcor32!RslLoadedTerm+0x4ea3
0804f020 6c2fd9dc 02f0d378 00000001 05578fe8 rsintcor32!RslLoadedTerm+0x5c2f
0804f058 6c30092a 02f0d378 00000001 00001b84 rsintcor32!RslLoadedTerm+0x5b60
0804f148 75c34c05 00001b84 06b1c540 00000087 rsintcor32!RslLoadedTerm+0x8aae
0804f194 75c349c8 080914b8 0804f1e0 0043b410 wininet!ICSocket::Send_Start+0x21d
0804f1ac 75c306a8 0043b410 00000000 0043b410 wininet!CFsm_SocketSend::RunSM+0x2e
0804f1f8 75c3496f 080835c8 00000000 00000000 wininet!CFsm::Run+0x159
0804f220 75c31d33 06b1c540 00000087 00000020 wininet!ICSocket::Send+0x106
0804f278 75c319ed 0804f2c0 00456968 080835c8 wininet!HTTP_REQUEST_HANDLE_OBJECT::SendRequest_Fsm+0x329
0804f28c 75c306a8 00456968 06b1e5d8 00000000 wininet!CFsm_SendRequest::RunSM+0x28
0804f2d8 75c32970 080835c8 00000000 00000000 wininet!CFsm::Run+0x159
0804f498 75c313a6 0809e630 06b1e5d8 0804f500 wininet!HTTP_REQUEST_HANDLE_OBJECT::HttpSendRequest_Start+0x66a
0804f4cc 75c306a8 00000000 00000000 06b1e5d8 wininet!CFsm_HttpSendRequest::RunSM+0x38
0804f518 75c30a96 080835c8 00000000 00000000 wininet!CFsm::Run+0x159
0804f538 75c7bbba 06b66970 06b66a28 00000000 wininet!DoFsm+0x55
0804f594 75c7da91 08098b58 0000000d 00000000 wininet!HttpWrapSendRequest+0x429
0804f5b8 75cf3a6f 00cc002c 08098b58 0000000d wininet!InternalHttpSendRequestA+0x2f
0804f70c 75c306a8 06b66970 00000000 06b66970 wininet!ParseHttpUrl_Fsm+0x257
0804f758 75c30a96 080835c8 00000000 00000000 wininet!CFsm::Run+0x159
0804f778 75ccea58 0804f800 06b6a8b8 75cc6218 wininet!DoFsm+0x55
0804f7b4 75ccf279 00000000 0804f800 06b6a8b8 wininet!ParseUrlForHttp_Fsm+0x27b
0804f7cc 75c306a8 06b6a8b8 00000000 06b6a8b8 wininet!CFsm_ParseUrlForHttp::RunSM+0x54
0804f818 75c30a96 080835c8 00000000 00000000 wininet!CFsm::Run+0x159
0804f838 75ccd13f 00000000 0804f8b8 0804f95c wininet!DoFsm+0x55
0804f890 75ccd255 00cc0024 06affde0 75cc6488 wininet!InternalInternetOpenUrlA+0x218
0804f8f0 75cf226e 00cc0024 06affde0 75cc6488 wininet!InternetOpenUrlA+0x3e
0804f930 75cf1ff1 00cc0024 06affde0 00000000 wininet!WininetProxySupport::OpenUrl+0xa4
0804f984 75d150a6 06b31428 08097748 00000001 wininet!WininetProxySupport::DownloadFile+0xc3
0804f9ac 75d14e57 08097748 00000000 00000000 wininet!DownloadScript+0x51
0804f9d8 75d158b6 08097748 00000000 00000000 wininet!DownloadProxyInfo+0x58
0804fa58 75c939f5 004aee60 06b65378 0804fadc wininet!SwpadWpad+0x299
0804fa68 77c39512 06b6ea70 7fb26d2a 0040f6f8 wininet!RefCountWorkItemThread+0xe
0804fadc 77c24429 06b6ea70 06b65378 7fb26bca ntdll!RtlpTpWorkCallback+0x11d
0804fc3c 7576336a 0040f6f0 0804fc88 77c09f72 ntdll!TppWorkerThread+0x572
0804fc48 77c09f72 0040f6f0 7fb26b7e 00000000 kernel32!BaseThreadInitThunk+0xe
0804fc88 77c09f45 77c23e85 0040f6f0 ffffffff ntdll!__RtlUserThreadStart+0x70
0804fca0 00000000 77c23e85 0040f6f0 00000000 ntdll!_RtlUserThreadStart+0x1b

FOLLOWUP_IP:
msvcr90!realloc+35f
751e675b 8bf8            mov     edi,eax

SYMBOL_STACK_INDEX:  6

SYMBOL_NAME:  msvcr90!realloc+35f

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: msvcr90

IMAGE_NAME:  msvcr90.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  4ca2ef57

STACK_COMMAND:  ~23s; .ecxr ; kb

FAILURE_BUCKET_ID:  ACTIONABLE_HEAP_CORRUPTION_heap_failure_block_not_busy_c0000374_msvcr90.dll!realloc

BUCKET_ID:  APPLICATION_FAULT_ACTIONABLE_HEAP_CORRUPTION_heap_failure_block_not_busy_DOUBLE_FREE_msvcr90!realloc+35f

WATSON_STAGEONE_URL: 

Followup: MachineOwner
———