Beechglen Development Inc.

Beechglen Development Inc.
3862 Race Road - Cincinnati, Ohio 45211 - (513) 922-0509 - Fax (513) 347-2834

Using the Standalone Analysis Tool (SAT)

All too often we get support call, or email about a system interruption with very scant details. A string of Bxxx DEAD hex codes or a system abort # with subsystems and status codes. The worst case is that of a system hang with no status codes at all. The severity of the hang may range from a running system that no one can access, to total eclipse, where even a ctrl-b is ignored.

Generally the consensus is that ‘I won’t take the time for a DUMP until it happens again’. Taking this stance is acceptable if a reasonable amount of time passes until the next event, but can come back to bite you if ‘the problem’ starts to reoccur with a vengeance. Then at least some amount of data from the first event would have been very valuable.

So the question is, what are the alternatives when:

Time is of the essence, and
There is too much data to manually collect given issue #1.

The HP3000 platform has a powerful combination to address these issues. Namely, the remote console facility and the Standalone Analysis Tool (SAT). This article will focus on using remote console and SAT as the first line of defense diagnostic tools. For a more comprehensive article on how to handle unexpected system interruptions you are encouraged to review the article titled "Handling System Aborts and System Failures" at http://www.beechglen.com/mpe/technotes/sysaborts.html.

The remote console facility has two major features: One is, as its name implies, it allows console control of the system remotely. And secondly, using a terminal emulator, it allows large amounts of data to be collected in a short amount of time. Please contact us if you need help on configuring or the remote console access on your system.

The SAT offline utility allows us to look at the operating system and current process status to determine the nature of the interruption. It does not involve a tape mount like DUMP so it is very fast, and if we are prepared in advance with the working knowledge of the remote console facility, we don’t have to manually write anything down. So lets look at a system abort using SAT: for each CPU we want to get a trace. In the following example we have a three CPU system so we need to get traces for all three processors, then exit. The commands to be executed are in red:

cpu 0 ; tr,i,d
cpu 1 ; tr,i,d
cpu 2 ; tr,i,d
exit

After a system abort, the console shows in inverse video:

SYSTEM ABORT 0 FROM SUBSYSTEM 0
SYSTEM HALT 7, $0000

With Status Codes of:

FLT DEAD FLT B907 FLT 0100

At this point, press CTRL-B (9x9 systems ensure that the key is in SERVICE, on 9x8 system verify the toggle switch on the back of the system is in the SERVICE postion.)

CM> tc

Wait for the system to reset. Interrupt the AUTOBOOT process if necessary by pressing any key within 10 seconds when prompted.

Main Menu: Enter command or menu > boot pri
Interact with IPL (Y or N)?> y
ISL> sat
...
Processor: 00 HPA: fffa0000 IVA: 00148000 Config: TRUE
Processor: 01 HPA: fffa2000 IVA: 00970000 Config: TRUE
Processor: 02 HPA: fffa4000 IVA: 00976000 Config: TRUE

Current CPU: 0 Original CPU: 0 Monarch CPU: 0 MP array at: c8000

Main memory: 80000000
HPDIR table: 1000000, len 1000000 HPDIROF table: 2000000, len 800000
IPDIR table: 2822000
RGLOB: 0 ICS: 0
Last PIN: 5 ON ICS DISP running
Processing dumpworthy file NMLOGMON.PUB.SYS, SID $128, #946 symbols...Done
Processing dumpworthy file NMCONSOL.PUB.SYS, SID $129, #456 symbols...Done

$2 ($0) nmsat > cpu 0 ; tr,i,d
PC=a.00178648 idle_disable_int+$8
NM* 0) SP=81e41390 RP=a.002c5c44 dispatcher+$790
NM 1) SP=81e41390 RP=a.00177800 iexit
--- Interrupt Marker
(end of NM stack)

$4 ($0) nmsat > cpu 1 ; tr,i,d
PC=a.00178648 idle_disable_int+$8
NM* 0) SP=81fd0390 RP=a.002c5c44 dispatcher+$790
NM 1) SP=81fd0390 RP=a.00177800 iexit
--- Interrupt Marker
(end of NM stack)

$6 ($0) nmsat > cpu 2 ; tr,i,d
PC=a.0018304c system_abort
NM* 0) SP=41855428 RP=a.00182dd8 ?system_abort+$8
export stub: a.008b61a0 make_pcall_from_debug+$330
NM 1) SP=41855428 RP=a.007a5d2c func_code_eval+$998
NM 2) SP=41855128 RP=a.007aa868 func_evaluate+$594
NM 3) SP=41854ee8 RP=a.007c7dd0 operand_search+$6a4
NM 4) SP=41852428 RP=a.007cada8 getvalue+$558
NM 5) SP=4184c7e8 RP=a.007c9e20 getaddrvalue+$d0
NM 6) SP=4184b6a8 RP=a.007c9900 getfactor+$c4
NM 7) SP=4184a568 RP=a.007c93cc getterm+$c0
NM 8) SP=41848c28 RP=a.007c8eb0 getsimpexpr+$1b0
NM 9) SP=41847ae8 RP=a.007c8934 getanyexpression+$f0
NM a) SP=418469a8 RP=a.007a28b0 scn_calc+$44
NM b) SP=41845068 RP=a.00726ca0 do_the_command+$104
NM c) SP=41844768 RP=a.00727b94 secondary_cmd_loop+$204
NM d) SP=41844668 RP=a.007280f8 main_cmd_loop+$98
NM e) SP=418443e8 RP=a.010465c4 nm_debug+$ca0
NM f) SP=41843ae8 RP=a.00e2d098 dbg_tell_the_owner+$36c
NM 10) SP=418438a8 RP=a.001ae65c dbg_break_handler+$688
NM 11) SP=418436e8 RP=a.0036c858 hpe_debug+$504
NM 12) SP=418435a8 RP=a.00383cd4 recovery_counter+$5c
NM 13) SP=41843468 RP=a.0013b038 hpe_interrupt_marker_stub
--- Interrupt Marker
NM 1) SP=418433e8 RP=a.00a5a064 hxdebug+$e4
--- End Interrupt Marker Frame ---
$a ($2f) nmsat > exit

ISL> start norecovery

Gathering information from SAT in this manner will only take a few minutes and is surely worth the effort. Consider that taking a memory dump can take an hour or more and is largely dependent upon the amount of memory in the system, the number and type of disks in the system volume set that contain virtual memory, and the speed of the tape drive in which memory will be dumped. As you can see from the sample output above there is a large amount of information contained here to write down. Clearly connecting to the remote console (or telnet to the GSP on an A-class or N-class system) via a terminal emulator is the way to go. You can capture the entire screen contents and email them to us to record the event and for analysis. Or we can log on to the remote console ourselves and collect the data directly