Significant Bugs Discovered on Release 6.5 and above

Significant Bugs Discovered on Release 6.5 and above

We have identified and developed a workaround for a significant bug in a network patch on systems running MPE/iX Release 6.5 and above. The problem surfaces if you attempt to reboot your system while it is out of disk space in the system volume set. The reboot process will hang while attempting to build the next NMLG (network log) file. In other words, rebooting with no available disk space will render your system unbootable. Our workaround involves running stand-alone offline diagnostic to patch the volume information. This is a delicate and time consuming process, but certainly beats the alternative of a re-install.

You know you have encountered this condition if during the reboot sequence these messages appear after the “mount all volumes” step.

-- Time spent in MOUNT_ALL_VOLUMES -> 40
Leaving - Mount all volumes
The current boot command has been saved on the system master.
NMLOGMON can't create a new NMLG file, the system is out of disc
space. Will retry to create NMLG file. (NMERR 79)
NMLOGMON can't create a new NMLG file, the system is out of disc
space. Will retry to create NMLG file. (NMERR 79)

The NMERR 79 message will repeat indefinitely and the bootup process will not continue. There are no special boot options available to solve this problem. The only supported method is to re-install MPE and restore the system volume set.. We have developed an alternative solution that requires editing the disk with offline utilities that we are willing to use as a last resort.

Please check your HPSWINFO.PUB.SYS file to determine whether you are at risk for this situation. Systems that have one of the following patches installed are susceptible.

Release 6.5: NMSGDT1A
Release 7.0: NMSGDT2A
Release 7.5: NMSGDV1A

HP has been notified about this bug and has assigned SR 8606351808 to produce a patch to correct it. Patches have been created to address this situation.
Of course, the patch has to be installed, before the issue occurs.

If you have a system with one of the listed patches installed and run out of disk space in the system volume set, it is highly recommend every effort is made to correct the situation prior to rebooting. Completely filling the available permanent disk space in the system volume set can happen more easily than one might expect. Inadvertently restoring a copy of a production database, batch processes that loop, to name a few, each consuming all remaining permanent disc space.

This ‘bug’ is just one example of the many types of problems that disk mirroring and disk arrays don’t provide protection from. This issue provides a good excuse to review backup strategies and procedures for completeness. Confirm you are storing the entire system and not excluding important file sets. If you make use of the “;ONVS=” parameter of STORE, verify you are specifying all volume sets, including MPEXL_SYSTEM_VOLUME_SET. As disk prices become more economical, it is easier to take advantage of user volumes by splitting the system volume set up into multiple user volume sets. Revisit the topic of IMAGE logging to keep transaction logs of database modifications on separate disks to create an alternative recovery method. The overhead of Image logging is minimal, and disk space is so inexpensive there should be no barriers to prevent instituting Image logging. We can assist in implementing user volumes and setting up IMAGE logging. Feel free to call on us to review your backup strategy for completeness and offer suggestions based on your specific methods for validating backups.

You may be familiar with our freeware utility script PRESHUT which is recommended to be incorporated into your system shutdown procedures. PRESHUT has many functions, all of which are geared towards giving the system the best chance to reboot without incident. In light of this problem, soon a new feature will be added to PRESHUT to check the available disk space in the system volume set. Systems where ESP/3000 has been installed should already have a PRESHUT.PUB.ESP file.The latest version of PRESHUT can be downloaded from here.

Bug found in STORE program.

Another recent bug that you should be aware of is in the STORE program. A specific version of STORE can write tapes where some of the files cannot be restored. This particular problem has already been fixed. Here are some excerpts from the patch description:

Some files may fail to RESTORE or VERIFY(VSTORE) with backups taken on systems that have MPEMXB6 or MPEMXH6 installed. The file showing the first error message cannot be recovered, but files following it on the backup – although also showing errors -can be recovered.

The error messages are:

/var/stm/logs/os/                 NOT VERIFIED: TERMINATING BEFORE FILE WAS VERIFIED

or

/DEVDB/PUB/DEVLG001               NOT RESTORED: TERMINATING BEFORE FILE WAS RESTORED

This has been duplicated only with Labeled Tape backups, but the analysis indicates that it is theoretically also possible with unlabeled tapes.

MPEMXB6 introduced a problem with STORE’s internal IO buffer handling, intermittently causing stale data to be written to the tape instead of valid file headers or file data.

Note that this is a fix to a problem in STORE. Tapes which have been created with the above versions of store and have experienced this problem will continue to see the problem when restored or vstored once this patch is installed.

As I read this description several thoughts come to mind. First, if you find that you are running one of the affected versions of STORE and have tapes experiencing the problem, they will continue to experience the problem even after patches are applied. The problem is with the STORE process, not the RESTORE process. Second, it yet again underscores how necessary it is to validate backups. Please take advantage of our free service for backup validation testing where errors such as this can be found proactively.

To determine if you are vulnerable to this STORE problem check the STORE banner where it contains the version number and compare to the version numbers in the table below. One easy way is to enter STORE command like this:

:file t=$null
:store loadmap.pub.sys;*t
>> TURBO-STORE/RESTORE VERSION C.70.18 B5151AA <<
(C) 1986 HEWLETT-PACKARD CO.

Refer to the following table to see which STORE versions contain the bug, and the patch level where it was fixed. Store versions that are older or newer do not include this bug.

Contact a Beechglen support representative if you would like further guidance to determine if your system is vulnerable to these bugs, or if you require assistance in obtaining and installing the appropriate bug fix for the STORE problem.