COPYRIGHT (C) 1984-2005 MERRILL CONSULTANTS DALLAS TEXAS USA
Newsletter THIRTY
/* COPYRIGHT (C) 1984-2005 MERRILL CONSULTANTS, DALLAS, TEXAS */
****************NEWSLETTER=THIRTY***************************************
MXG NEWSLETTER NUMBER THIRTY September 10, 1996
Technical Newsletter for Users of MXG : Merrill's Expanded Guide to CPE
TABLE OF CONTENTS Page
I. MXG Software Version 14.07 is now available upon request. 3
II. MXG Technical Notes 6
1. Data Mining the Original Data Warehouse - 25 years + 1 million 6
2. MXGTMNT, the Tape Mount and Tape Allocation Monitor Program 20
3. "RMFINTRV" data for both detail and hourly intervals 20
4. Some allegations that the KEEPALL=NO option in %VMXGSUM "costs" 20
5. BUILDPDB processes SMF data at 41MB per 3090-300S CPU-minute. 21
III. MVS Technical Notes. 21
1. APAR OW16564 corrects zero value in PCHANBY variable in TYPE73 21
2. APAR OW16847 and OW19029 corrects TYPE30 variables EXCPTOTL 22
3. APAR OW17491 corrects TYPE21 variable TAPCUSER Tape Unit Serial 22
4. APAR OW17121 reports incorrect values for TYPE6 variables JOB 22
5. APAR OW17469 reports excessive count of type 80 SMF records 22
6. APAR OW18822 corrects the invalid JESNR in type 26 SMF records 22
7. ISPF Find and Change commands use the equal-sign as a wild card 22
8. Hitachi 9021 Skyline Processors trash TYPE73 (Channel Path Busy) 22
9. APAR OW19482 corrects variable TTRN in dataset TYPE1415 22
10. APAR OW03645 reports extremely high disconnect time in type 74 22
11. CA's CA-SPOOL Release 1.8 creates massive numbers of records 22
12. APAR OW01699 corrects TYPE72MN (type 72 subtype 2 RMF Mon II) 23
13. APAR OW20318 corrects TYPE42TO (type 42 subtype 1) DURATM 23
15. CPUTM in interval type 30s does not match step type 30s 23
16. APAR OW18543 corrects blank values for Service Class & Report 24
17. APAR OW20904/OW22093 (negative or zero disconnect time) 24
18. SMF dump job will run faster with less resources with BUFND=46 24
19. Boole & Babbage's Cache RMF Reporter (user SMF record) wrong. 25
20. APAR OW20844 increases the maximum number of job numbers 25
21. Experiments with BUFNO for Sequential Access (BSAM,QSAM) 25
22. APAR PN87013 (open) reports TCP/IP TELNET LOGN SMF error. 26
23. APAR OW20823 for RACF type 80 record corrects missing data 26
24. CA's CA-1 (TMS) product APAR G081058 for CA1 5.1. 26
25. APAR OW20956 reports that TYPE1415 variable TRKSALOC wrong. 26
26. APAR OW21083 fixes large values in CPUUNITS & SERVICE variables. 26
27. Sites with MVS/ESA 5.2.2 have noticed negative PCTPNCHA values. 26
28. To create SMFINTRV (TYPE30_V) globally synchronized. 26
29. APAR OW19074 corrects three distinct RMF ABENDS 26
30. APAR OW22119 for DCOLLECT record type 'VL' corrects system 26
IV. DB2 Technical Notes. 27
1. DB2 records its media manager VSAM I/O counts in EXCP buckets. 27
V. IMS Technical Notes. 27
1. One site using Candle's ITRF reports their experiences: 27
2. IMS FASTPATH DEBD I/O counts via the VSAM Media Manager counted. 27
COPYRIGHT (C) 1996 BY MERRILL CONSULTANTS DALLAS TEXAS
VI. SAS Technical Notes. 27
1. SAS USER ABEND 0315 (physical I/O error to a SAS data library) 27
2. Further comments about resources in PROC SQL versus DATA step. 27
3. USER ABEND 0318 may occur if your site upgrades STOPX37 27
4. OUT OF MEMORY conditions and REGION=0M. 28
5. SAS can enter either a CPU or wait loop with broken VBS. 28
6. Permanent ARRAYs with large numbers of elements (over 4096?) 28
7. SAS 6.09E Maintenance TS450 has no reported problems with MXG. 29
VII. CICS Technical Notes. 29
1. CICS data volume reduction with EXCLUDE, CICS Blocksize, 29
2. Support for Landmark's TMON for CICS/ESA 1.3. 30
3. APAR PN79698 closes a problem with TASCPUTM. 31
4. CICS/ESA 4.1, IBM creates a CICSTRAN obs for undefined trans 31
VIII. Incompatibilities and Installation of MXG 14.07. 31
IX. Online Documentation of MXG Software. 33
X. Changes Log 34
Alphabetical list of important changes 35
Changes 14.209 thru 14.001 37-77
========================================================================
So what's the big picture? Who needs to request and install MXG 14.07?
1. Anyone installing these products, which are INCOMPATIBLE (i.e., the
new records cause an error), needs at least the MXG version listed:
CICS/ESA 5.1.0 14.07 ASTEX 2.1 14.04
TMON/DB2 Version 3 14.07 IMS APAR PN76410 14.04
Omegmaon/VTAM V200 14.06 OS/400 Release 3.6 14.04
MODEL204 Release 3.2.1 14.06 Thruput Manager #V041238 14.03
SoftAudit Version 5.1 14.06 CRR in Type 74 Subtype 4 14.06
2. Leading edge sites installing OS/390 Release 2 (or MVS/ESA 5.2.2):
MXG 13.13 and later tolerate OS/390 Release 2, but to capture
the several new variables and new subtypes of type 74 and 89,
you must install MXG 14.05 or later.
3. Anyone with DB2 4.1: Although MXG 13.13 handles the important DB2
accounting and most statistics records correctly, some of the new
statistics information on global buffers and remotes, and protection
for IBM skipping sequence numbers were not correct until MXG 14.07.
4. Anyone else. There are lots of enhancements (like TYPE80A for RACF,
which decodes many more RACF commands for simplified reporting) in
the 209+ changes since January's 13.13. I plan to ship the next
Annual MXG Version (to be numbered 14.14) in early 1997 to all sites.
So what's in near-future development?
Support for TRENDSMP data (September, 1996)
Windows NT PERFMON collector and MXG support (4th Quarter, 1996)
I. MXG Software Version Status.
1. MXG Software Version 14.07, dated Sep 10, 1996, is now available,
free, upon request. No tape was shipped with NEWSLETTER THIRTY.
Major enhancements added in MXG 14.06 dated Aug 20, 1996:
Support for CICS/ESA 5.1.0 (INCOMPATIBLE).
Support for TMON/DB2 Version 3 (INCOMPATIBLE).
Support for Boole and Babbage's PRO/SMS SMF record.
Major enhancements added in MXG 14.06 dated Aug 20, 1996:
Support for Omegamon/VTAM V200 (INCOMPATIBLE).
Support for MODEL204 Release 3.2.1 (INCOMPATIBLE).
Support for SoftAudit Version 5.1 (INCOMPATIBLE).
Support for APAR OW15406 for RMF adds support for Year 2000.
Support for Tandem Controller and Line records added.
Sample code to read Network General's Sniffer Network Monitor data.
VM Print sent to JES2 is now merged in PDB.JOBS.
BUILDPD3 now sums JES3 type 25 MDS Tape Mounts/Fetches.
More RACF Reports for Command Events decoded by TYPE80A.
DB2 4.1 DB2STATS interval lost due to QWHSISEQ skipped values.
CICINTRV restored to pre-14.04 version, fixed for CICS 4.1.
Redesigned TRNDTALO to "SPIN" active allocations.
SMF Simulator (ANALSMF) now tests a CISIZE of 18432 for 3390s.
Major enhancements added in MXG 14.05 dated Jul 15, 1996:
Support for OS/390 Version 1 Release 2 (COMPATIBLE).
MXG 13.13 and later tolerate OS/390 Release 2, but to capture
the several new variables and new subtypes of type 74 and 89,
you must install MXG 14.05 or later.
Support for SMF type 89 subtype 2 (Measured Usage Product Summary).
Support for DB2 trace data written to GTF instead of SMF.
Support for HP MeasureWare for HP-UX platform
Support for RDS's EOS Enterprise Output Solution
Support for Landmark TMON/MVS spanned records.
Support for RMF type 74 subtype 5 Cache RMF Reporter.
Support for Anacomp, Inc's XSTAR product's SMF record
Support for DFSORT Release 13 APAR PN71337.
New JCLADHOC example of MXG ad hoc job to select specific data.
Revised MXGSAS JCL procedure adds TAILORNG= symbolic parameter.
New DB2 trace datasets to hold all SQL text are created.
MXG JCL examples now specify REGION=0M
VMXGTAPE utility macro to determine if lib/dsn is on tape.
UDEBLOCK utility to create valid RECFM=U on MVS from PC data.
ASMIMSLG/ASMIMSL5 SLOTS table was moved above the 16MB line.
Major enhancements added in MXG 14.04 dated Jun 15, 1996:
Support for ASTEX 2.1 (INCOMPATIBLE)
Support for NDM 1.4 (compatible) new variables
Support for IMS APAR PN76410 (INCOMPATIBLE) for ASMIMSLG processing.
Support for APAR PN78083 to SMF type 42 (ADSM) required no change.
Enhanced CICINTRV was installed as default (but removed in 14.06).
Major enhancements added in MXG 14.03 dated May 27, 1996:
Support for RACF 1.10 (compatible) - toleration of new records.
Support for NETSPY Release 4.7.
Support (partial) for AS/400,OS/400 Release 3.6 (INCOMPATIBLE).
Support for Thruput Manager #V041238 (INCOMPATIBLE).
All datetime constants '01JAN00:...' were changed to '01JAN1900:....'
Corrections to errors that were only in MXG 14.02:
DIFFDB2 14.108 BY VARIABLES ARE NOT PROPERLY SORTED DB2STATR
TYPE37 14.107 INPUT STATEMENT EXCEEDED ID=37
TYPE72 14.102 INPUT STATEMENT EXCEEDED ID=72
TYPENSPY 14.097 Zero obs in NSPYLU.
Major enhancements added in MXG 14.02 dated April 25, 1996:
ASMTAPES MAINTLEV 9, monitor no longer quits writing, TMNT013I msg.
Support for IBM's Cache RMF Reporter CRR Version 1.7.
Support for Netview FTP (File Transfer) SMF subtype 51x record.
Support for second length STK HSC Subtype 08 record.
Support for Shared Page Groups statistics in TYPE71.
Support for STK's NearOAM user SMF record.
Support for IBM's RMDS Version 2.2 (no change).
Support for NPM APARs OW08565/OW10584 for 3746/900.
Major enhancements added in MXG 14.01 dated March 7, 1996:
Support for OS/390 Release 1.1.0 (already in MXG 13.13).
Support for FACOM MSPE/EX PTF 93061 ID=127 SMF record.
Support for SMF type 6's ESS segment added and externalized.
MAINTLEV 8 of the MXG Tape Mount and Tape Allocation Monitor
INPUT EXCEEDED for NETSPY 4.6, type A record.
INPUT EXCEEDED for STK HSC subtype 8 record corrected.
INPUT EXCEEDED for DB2 4.1 type 101 subtype 2 (packages).
INPUT EXCEEDED for DFSMS/rmm type "O" record.
INPUT EXCEEDED for EREP type '40'X record.
INPUT EXCEEDED for PSF 6 SMF, PSF wrote truncated record.
INPUT EXCEEDED for VSAM 64 SMF, CF Cache Structure segment.
NOTSORTED error for PDB.CICS in WEEKBLD, WEEKBLDT, and MONTHBLD.
ASMVTOC failed to assemble.
INVALID DATA FOR HH,MM,SS with SAMS SMF record.
VARIABLE SYSTEM uninitialized in ASMIMSLG processing.
Hipercache SMF record values for VSAM segment wrong.
NDM/Connect Direct timestamps missing, data wrong.
TLMS dates were not decoded correctly.
NPM dataset NPMVSVVR variables were trashed.
All of these enhancements are described in the Change Log, below.
Table of availability dates for the IBM products and MXG version:
Availability MXG Version
Product Name Date Required
MVS/ESA 4.1 Oct 26, 1990. 8.8
MVS/ESA 4.2 Mar 29, 1991. 9.9
MVS/ESA 4.2.2 Aug 1991. 9.9
MVS/ESA 4.3 Mar 23 1993. 10.10
MVS/ESA 5.1.0 - compatibility Jun 24, 1994 12.02
MVS/ESA 5.1.0 - Goal Mode May 3, 1995 13.01
MVS/ESA 5.2.0 Jun 15, 1995 13.05
MVS/ESA 5.2.2 Oct 19, 1995 13.09
OS/390 1.1.0 Feb 22, 1996 14.01
OS/390 1.2.0 Sep ?? 1996 14.05
CICS/ESA 3.2 Jun 28, 1991. 9.9
CICS/ESA 3.3 Mar 28, 1992. 10.01
CICS/ESA 4.1 Oct 27, 1994. 13.09
CICS/ESA 5.1 Sep 10, 1996 14.07
CRR 1.6 Jun 24, 1994. 12.02
DB2 2.3.0 Oct 28, 1991. 10.01
DB2 3.1.0 Dec 17, 1993. 13.02A
DB2 4.1.0 Tolerate Nov 7, 1995 13.07
DB2 4.1.0 Full support Nov 7, 1995 14.07
DFSMS/MVS 1.1 Mar 13, 1993. 11.11
DFSMS/MVS 1.2 Jun 24, 1994. 12.02
DFSMS/MVS 1.3 Dec 29, 1995. 13.09
NPM 2.0 Dec 17, 1993. 12.03
NPM 2.2 Aug 29, 1994. 12.05
NPM 2.3, 2.4 ??? ??, 1996. 14.03
RMDS 2.1, 2.2 Dec 12, 1995. 12.12
TCP/IP 3.1 Jun 12, 1995. 12.12
VM/ESA 2.0 Dec 23, 1992. 10.04
VM/ESA 2.1 Jun 27, 1993. 12.02
VM/ESA 2.2 Nov 22, 1994. 12.06
Table MXG support for non-IBM products:
Availability MXG Version
Product Name Date or Change Required
Landmark
The Monitor for DB2 Version 3 14.07
The Monitor for DB2 Version 2 13.06
The Monitor for CICS/ESA 1.2 - 12.12
The Monitor for CICS/ESA 1.3 - 12.12A
The Monitor for MVS/ESA 1.3 - 12.05
Candle
Omegamon for CICS V300 User SMF 12.05
Omegamon for CICS V400 User SMF 13.06
Omegamon for IMS V110 (ITRF) 12.12
Omegamon for IMS V300 (ITRF) 14.04
Omegamon for MVS V300 13.170 13.05
Omegamon for MVS V400 13.201 13.06
Omegamon for DB2 Version 2.1/2.2 13.05
Omegamon for VTAM V160 12.04A
Omegamon for SMS V100/V110 12.03
CA
ASTEX 2.1 14.04
Boole & Babbage
IMF 3.1 (for IMS 5.1) 12.12
Memorex/Telex
LMS 3.1 12.12A
II. MXG Technical Notes.
1. The following Invited Paper will be presented at CMG 96 and SUGI 97.
Data Mining the Original Data Warehouse:
Twenty-Five Years and a Million Lines of SAS Later
H. W. "Barry" Merrill, PhD
ABSTRACT
The author of MXG Software provides a historical perspective of how and
why the SAS System became the pervasive tool for managing and mining of
the original data warehouse, the Performance Data Base, built from SMF
data. The architecture of the MXG implementation is described to show
how MXG, currently 911,749 lines of SAS code in 3,011 files (members),
executes under MVS, VM, UNIX, OS/2, Windows 95 or Windows NT to create
1,908 SAS tables (datasets) with 78,278 columns (variables) from the raw
data records produced by 268 products, where the input volume ranges
from only hundreds of megabytes to fifteen gigabytes per day; some CICS
tables contain in excess of 130 million rows (observations).
Contents
a. Notre Dame - 1959 - WOW! from an IBM 610 digital computer.
b. Purdue, 1964-1967 - IBM 7090/7094 and IBM 360/44.
c. State Farm Mutual Automobile Insurance Company, 1972-1976.
d. Sun Oil Company, 1976-1984.
e. Architecture of MXG Software SMF Processing - Single Record
f. Architecture of Building the Performance Data Base - BUILDPDB
g. Mining costs and tons of warehouse data dug up and delivered:
h. Growth of the MXG Source Library
i. SAS does not stand for Single Authored Software. Acknowledgements.
========================================================================
a. Notre Dame - 1959 - WOW! from an IBM 610 digital computer.
As a Notre Dame sophomore in EE in September, 1959, my first EE lab
experiment was to calculate the determinant of a 4x4 matrix. As the
ancient Lab Instructor finished his instructions, he said "I have to
read this. The IBM corporation has donated a Model 610 dig-it-tal
computer, located in room 240, and students can sign up for hour-long
blocks." Putting down the sheet of paper, he said "those digital things
will never last, but next year, as Juniors, you can learn to use the
Bendix G15 Analog Computer - that's how engineer's solve real problems!"
I went to room 240, looked thru the peephole and saw a large gray box,
a table with typewriter, and what I assumed to be a senior, and opened
the door to enter. As the door unhinged, so did the student, shouting
"Shut that door!" as he strode across the room to the door, flailing
his arms. As he stepped out into the hall shouting "Didn't you read the
damn sign?", he discovered his sign had fallen face down on the floor.
Calming, he informed me that you must get the operator's attention, so
he could put the machine in "QUIESCE/STOP" (which took 5-10 seconds),
and only then was it safe shuffle in, slowly. The vacuum tube machine
was so heat sensitive, that the air currents would cause computation to
fail, requiring a program restart.
He pointed me to the IBM manuals and I began at page one. Several hours
later, I had learned to punch paper tape and print them on the Selectric
and decided to calculate the determinate on my new toy. By Saturday, I
had punched my program, printed it, and was now ready to run my first
computer program. As I watched the paper tape whir thru the reader, the
addresses flickered on the nixie tubes; I crossed my arms and thought
"Wow, it is 1959, I am a sophomore in college and am running a real
program on a digital computer". The paper tape came to the end, the
printer came alive, and I received my first computer output, four
characters: WOW! It took until Sunday to find the senior, who found
that I had sort of missed the difference between "program" and "data",
that the first punch in the tape was a control character that put the
610 in a scan mode, that in the fifth-from-end position there was a
control character to print the tape as machine instructions, and what
had been printed were the code letters for the last four program
instructions: W=Carriage Return, O=Line Feed, W= Carriage Return,
!=Print Accumulator! (Two Carriage Returns were always used to ensure
that the very slow print head was all the way left before print.)
I did finally get the determinant computed, and submitted the first EE
lab problem that used a digital computer at Notre Dame, but I did
nothing further with computers while there. I dropped out of Notre Dame
in 1962, joined the Navy, was in the Cuban blockade on a surface ship,
then on a Diesel submarine, and then won a Navy scholarship that sent
me back to college in EE at Purdue University in 1964.
b. Purdue, 1964-1967 - IBM 7090/7094 and IBM 360/44.
At Purdue, I took a one-hour Fortran II course, using a 7090/7094 and
was hooked. I worked on Linear Programs to model power grids, got a job
in the Tab department wiring plug boards for sorters, collators and
printers, implemented the Fast Fourier Transform from the original
Cooley-Tukey paper, worked for the Laboratory for Agricultural Remote
Sensing (pattern recognition of crops from spectral data which led to
the Earth Resource Technology Satellite), built the ground-truth data
for LARS agronomists, and set fire to our 360/44 Serial #2 (twice!) with
a tight loop in the floating point divide unit that lacked a heat sink.
I showed one PhD candidate in Psychology how pattern recognition and
vector distance could be used to cluster petroleum engineers that found
oil from those that did not, and coded Fortran programs to manipulate
data to invoke the BIMD statistical subroutines for another. I finished
my BSEE and MSEE in August, 1967, but the Navy needed nuclear submarine
drivers, not programmers, so again I set computing aside for a second
masters in Nuclear Propulsion and sightseeing in the Barents Sea, until
shore duty running the airline to Guantanamo Bay Cuba, where I taught
calculus and ran the overseas extension for Old Dominion University.
c. State Farm Mutual Automobile Insurance Company, 1972-1976.
Leaving the Navy in 1972, my Psychologist friend, now working at State
Farm Automobile Insurance in Bloomington, IL, suggested that I might
find a home there. Dave Vitek had gone to the Boole and Babbage User
Group (BBUG, the predecessor of CMG) and decided that maybe, instead of
trusting the IBM salesman as your capacity planner, State Farm could
measure its own computers, and had funded a ten-person Measurement Unit
for a feasibility study. Steve Cullen had drafted an excellent attack
plan to evaluate tools, and in short order we had Kommand/PACES for
accounting, Software Monitors (SYSTEM LEAP and PROGRAM LEAP), Hardware
Monitors (TESDATA XRAY), and Simulation (SAM). Because Kommand was only
for billing, Denny Maguire had started to write PL/1 programs to extract
fields from SMF records, and I had revived an old Plot subroutine from
LARS days, when I found this brief announcement in Datamation:
The Institute of Statistics at North Carolina State University
announces the availability of the Statistical Analysis System, a
package of 100,000 lines, one third each in Fortran, PL/1 and
Assembler, that does printing, analysis and plotting of data.
I wrote for information, and got a typical university document, with
some pages dittoed, some pages typed, some printed, each on paper of a
different color, but immediately saw the power and simplicity of the
INPUT statement for SMF data. However, in the list of supported data
formats, there was no reference to Packed Decimal. You only need to get
seven bytes into an SMF record to encounter a Packed Decimal field, so I
called the Institute and asked Tony Barr, the author of the SAS compiler
about support. "Well, we haven't got around to documenting it yet, but
if you type in PD4. it will work jest fine" he said, so I convinced
State Farm to risk the 1972 purchase price of $100 for the SAS package.
Starting in 1964, Tony Barr and Dr. Jim Goodnight had collaborated to
develop an ANOVA routine for the Department of Agriculture. Tony had
been an IBM developer of the data base for the cold war's Distant Early
Warning (DEW line) radar system, and Jim was a well-known statistician.
Both recognized the weakness of the existing stat packages: they were
only subroutines that had to be invoked by other programs that had to
prepare and manage the data to be analyzed. By creating a language, a
database, and the statistics, the Statistical Analysis System expanded
well beyond the original ANOVA routine and had been tested at several
Agricultural Experimental Stations and other universities, but the 1972
announcement was the first public release of the Statistical Analysis
System, and in October, 1972, State Farm was the first real customer to
install the SAS package from NCSU's Statistics Department.
Within days of receipt of SAS, we were extracting CPU time and PROGRAM
name and K-Core-Hours to produce reports on resource consumption direct
from SMF records, and, because SAS stores in floating point, we found
that Kommand lost hours of CPU time thru truncation. Presentations on
the use of SAS software and the PDB were given to the Bloomington and
Chicago chapters of the ACM and DPMA; the SAS data base was mentioned in
my paper (on the use of the SAS data base to create simulation input for
the System Analysis Machine directly from actual SMF data) presented at
the 1973 SSCS (Symposium on the Simulation of Computer Systems) at NBS,
and at a BOF session at the Seventh Annual Interface Symposium at Iowa
State. Many XRAY hardware monitor users became aware of State Farm's
PDB through the Midwest TESDATA Users Group, which held its inaugural
meeting in 1973 at State Farm. These presentations were only half
technical; we also had to convince attendees that staffing of this new
measurement concept was cost justified by the real dollar savings. John
Chapman had used an XRAY at Standard Oil and invited me to join SHARE's
Computer Measurement and Evaluation (CME) project, and the PDB was
described in a closed session of the CME project at SHARE 42 in Houston
in March of 1974. The first open session presentation on the use of the
SAS System to process SMF data was before to an audience of over 750
(half of the attendees!) at the Chicago SHARE 43 meeting that August.
That session was split with an IBM presentation on their new Statistics
Gathering Package, an FDP that selected a few fields from a few SMF
records. IBM spoke first, then I showed what we had done with SAS at
State Farm. One attendee stood and asked the IBM author of SGP, Bill
Tetzlaff, "Now that you have seen SAS, is there any reason why you would
still recommend your SGP product?" Several hundred SHARE sites acquired
SAS that fall as a result of this SHARE session!
d. Sun Oil Company, 1976-1984.
In 1974, SAS added File 13, SAS.MERRILL, to their distribution tape with
code examples for reading SMF data. In 1976, I completed my course work
at the University of Illinois (65 miles each way on a CB500 Honda), and
when State Farm decided not to rapidly migrate to the new MVS operating
system, I left for Dallas and Sun Oil company, where I demonstrated that
the analysis of SMF with SAS was valid for VS2 as well. In 1979 I wrote
my dissertation, "A Comprehensive Approach to the Measurement of Large
Scale Computer Systems" and received my PhD in EE from U of I. In 1979,
Jane Helwig, director of publications at SAS Institute (which had become
an independent company in 1975, marketing "the SAS System" instead of
the "Statistical Analysis System") said users wanted a book and SAS code
that showed how to measure computers, so we worked together on what was
to be titled "The Analysis of SMF and RMF Data Using the SAS System".
Just before printing, Jane called to say that no one liked the name, and
asked if my ego could handle the title "Merrill's Guide to Computer
Performance Evaluation using the SAS System", which became a 395 page
blue book, sold by SAS with a tape of sample programs for $395. By
1983, MVS/XA loomed with radical changes to SMF data, and many of the
book's users were asking for a real software product, so in 1984, SAS
published "Merrill's Expanded Guide to Computer Performance Evaluation
Using the SAS System", an 835 page red book ($50), and SAS distributed
the new MXG Software tape ($700) that was shipped with the then optional
Merrill Consultant's "Support Subscription" agreement ($500 annually),
and I left Sun Oil. Judy, who had taught Business College and had been
an executive with an apparel firm, said that she would run the business
and I would write and support the software; she does and I do. In 1987,
SAS Institute published the 630 page red book "Merrill's Expanded Guide
Supplement" and in 1991, Merrill Consultants replaced the old Support
Subscription with a License Agreement and took over all distribution of
MXG Software and MXG Books.
MXG Software has been installed at over 5,200 data centers in all states
and 49 countries (although there are only about 3,000 licenses now, due
to data center consolidations), and over 15,000 people bought the books.
e. Architecture of MXG Software SMF Processing - Single Record
So much for history. The design of MXG Software exploits many features
of the SAS System, especially in the DATA steps that are used to convert
raw SMF data into SAS tables (aka "datasets") that are stored in SAS
data libraries (aka SAS "databases"). A simple SAS program to read the
SMF file and decode type 0 (IPL) records is shown in Figure 1:
Figure 1
DATA TYPE0;
INFILE SMF;
LENGTH DEFAULT=&MXGLEN IPLTIME 8;
FORMAT DOWNTM SMCAJWTM TIME12.2
IPLTIME DATETIME21.2
;
INPUT @1 MVSXAFLG PIB1.
@2 ID PIB1.
@3 SMFTIME SMFSTAMP8.
@11 SYSTEM $EBCDIC4.
@;
IF ID=0 THEN DO;
IPLTIME=SMFTIME;
INPUT @15 SMCAJWTM PIB4. /*SMF0JWT*/
@19 SMFBUFF PIB4. /*SMF0BUF*/
@23 VIRTSIZE PIB4. /*SMF0VST*/
@27 SMCAOPT PIB1. /*SMF0OPT*/
@28 REALSIZE PIB4. /*SMF0RST*/
@;
SMCAJWTM=60*SMCAJWTM;
OUTPUT TYPE0;
RETURN;
END;
But to process more than one SMF record, a separate program for each
record type would be needed, and the SMF file would have to be read once
for each SMF record. Instead, for each record type, MXG creates one
source member, VMAC0, that defines two "old-style" substitution MACROs,
_VAR0 and _CDE0 with the code segments that are unique to each record:
MACRO _VAR0 TYPE0 %
MACRO _CDE0 Figure 1 code from LENGTH thru END; %
and one source member, VMACSMF, for the INFILE SMF code segment:
MACRO _SMF INFILE SMF ...; %
and you can construct a SAS program that will read multiple SMF records
in one pass of the SMF data using simple macro references:
%INCLUDE SOURCLIB(VMAC0,VMAC6,VMAC30,VMACSMF);
DATA _VAR0 _VAR6 _VAR26 _VAR30 ; _SMF; _CDE0 _CDE6 _CDE26 _CDE30 ;
to create SAS datasets from type 0, 6, 26 and type 30 SMF records.
These old-style MACRO statements are used simply as shorthand; SAS will
replace the macro name with its contents as SAS reads the source code.
They were not replaced by the newer %MACRO facility, because MACRO can
handle any text string, whereas %MACRO has real problems if the text has
parenthesis, and because %MACROS must be compiled, while MACROs are
simply read and stored. All MXG MACRO names start with an underscore.
The actual MACRO definitions for _VAR0 and _CDE0 in member VMAC0 can
now be examined in detail in Figure 2 to see the SAS features used:
Figure 2
%INCLUDE SOURCLIB(IMAC0); /* DEFINES _LTY0 AND _KTY0 */
MACRO _VAR0
_LTY0 /* TYPE0 */
(LABEL='TYPE 0 IPL SMF'
KEEP=DOWNTM IPLTIME OPTDSETS OPTVOL REALSIZE REC
SMCAJWTM SMFBUFF SYSTEM VIRTSIZE ZDATE
/* ADDED BY MVS/ESA 5.1 */
PRODUCT SYSNAME SYSPLEX
_KTY0
)
%
MACRO _CDE0
IF ID=0 THEN DO;
LABEL
DOWNTM ='ESTIMATED*SYSTEM*DOWNTIME'
IPLTIME ='SMF*RECORD*TIME STAMP'
OPTDSETS='CREATE*DATA SET/DASD*RECORDS?'
OPTVOL ='CREATE*VOLUME*RECORDS(19/69)?'
PRODUCT ='MVS*PRODUCT*NAME'
REALSIZE='REAL*MEMORY*SIZE(KBYTES)'
REC ='TEMPORARY*SCRATCHES*(PERM/ALL)?'
SMCAJWTM='JOB WAIT*(ABEND 522)*LIMIT'
SMFBUFF ='SMF*BUFFER*SIZE(BYTES)'
SYSNAME ='SYSNAME*PARAMETER*FROM IEASYSXX'
SYSPLEX ='SYSPLEX*NAME FROM*COUPLEXX'
SYSTEM ='SYSTEM*ID'
VIRTSIZE='VIRTUAL*MEMORY*SIZE'
;
LENGTH DEFAULT=&MXGLEN IPLTIME 8;
FORMAT DOWNTM SMCAJWTM TIME12.2
IPLTIME DATETIME21.2
ZDATE DATE9.
;
IF LENGTH-OFFSMF NE 31 AND LENGTH-OFFSMF NE 56 THEN DO;
NBAD0+1;
IF NBAD0 LE 3 THEN
PUT '***VMAC0.ERROR. INVALID TYPE 0 RECORD DETECTED. ' LENGTH= /
' BUT TRUE IPL RECORD SHOULD BE EITHER 31 OR 56 BYTES '
' RECORD IS DELETED ' _N_= SYSTEM= /
+16 SMFTIME=;
DELETE;
END;
IPLTIME=SMFTIME;
INPUT @15+OFFSMF SMCAJWTM &PIB.4. /*SMF0JWT*/
@19+OFFSMF SMFBUFF &PIB.4. /*SMF0BUF*/
@23+OFFSMF VIRTSIZE &PIB.4. /*SMF0VST*/
@27+OFFSMF SMCAOPT &PIB.1. /*SMF0OPT*/
@28+OFFSMF REALSIZE &PIB.4. /*SMF0RST*/
@;
IF LENGTH-OFFSMF GE 56 THEN
INPUT @32+OFFSMF +1 /*RESERVED*/
@33+OFFSMF PRODUCT $EBCDIC8. /*MVS*PRODUCT*NAME*/
@41+OFFSMF SYSNAME $EBCDIC8. /*SYSNAME*PARAMETER*FROM IEASYSXX*/
@49+OFFSMF SYSPLEX $EBCDIC8. /*SYSPLEX*NAME FROM*COUPLEXX*/
@;
IF SYSTEM=PREVSYS AND IPLTIME GE PREVTIME THEN DOWNTM=IPLTIME-PREVTIME;
OPTDSETS='NO ';
IF SMCAOPT='...1....'B THEN OPTDSETS='YES';
OPTVOL='NO ';
IF SMCAOPT='...01...'B THEN OPTVOL='YES';
REC='PERM';
IF SMCAOPT='......1.'B THEN REC='ALL';
SMCAJWTM=60*SMCAJWTM;
%%INCLUDE SOURCLIB(EXTY0); /* _LTY0 OUTPUTS TYPE0 */
RETURN;
END; /* END CDE0 */
%
Instead of the TYPE0 dataset name, MACRO _VAR0 contains the MACRO name
_LTY0, the "Library" macro, that is defined in IMAC0, the "Installation
Tailoring member" for the VMAC0 member. The default definition in IMAC0
is MACRO _LTY0 TYPE0 %, to create the WORK.TYPE0 dataset; by using an
externalized macro name in place of a hardcoded name, you can tailor
IMAC0 to send the TYPE0 dataset to tape for a large dataset to
save DASD space, or could rename TYPE0, without ever modifying the MXG
Source Library. By concatenating a tailoring library that contains all
of your changes ahead of the MXG Source Library:
//SOURCLIB DD DSN=MXG.TAILOR.SOURCE.LIBRARY,DISP=SHR
// DD DSN=MXG.MASTER.SOURCE.LIBRARY,DISP=SHR
installing a new version of MXG is little more than replacing the old
MXG Version's Source Library with the new MXG Version's Source Library.
The KEEP= list inside the parenthesis names the variables that are to be
kept in dataset TYPE0. Without the dataset KEEP= operand, all variables
defined in the DATA step would be kept. At the end of the KEEP= list is
the _KTY0 token, defined in IMAC0, which defaults to null. If you wish
to create new variables NEWVAR1 and NEWVAR2 in dataset TYPE0, you would
define MACRO _KTY0 NEWVAR1 NEWVAR2 % to add those variables to KEEP=
list. Moreover, if you want to drop variables that you don't need (to
reduce the stored size of TYPE0), for example OPTDSETS and OPTVOL, you
would define MACRO _KTY0 DROP= OPTDSETS OPTVOL % and those variables
would not be kept in dataset TYPE0, because the DROP= list overrides the
KEEP= list! You can also use the KTY0 macro name to add any dataset
option (for example, COMPRESS=YES) for the TYPE0 dataset.
When variables are listed in a KEEP= list but are not created, those
variables are not kept. This is exploited in CICS type 110 processing,
where optional segments (DL/I, DBCNTL) may or may not exist. MXG names
those variables in the KEEP= list, but the optional processing code (in
IMACICDL,IMACICDB) is commented out, so SAS never sees the creation of
those variables, and they are not kept. In this way, only one member,
the optional IMACs, needs to be tailored to create them. The SAS option
DKROCOND=WARN/NOWARN (Drop/Keep/Rename Output) can be enabled to see if
variable names are listed but not referenced.
Inside the _VAR0 definition, the dataset LABEL= operand describes the
dataset (that label is visible thru PROC CONTENTS), and the KEEP= list
is commented when new variables are added by a new release. Variables
in the KEEP= list are listed alphabetically and collimated for ease in
reading, but that ordering has no actual effect on the built dataset.
Instead, by making the first SAS statement inside the _CDE0 macro to be
the LABEL statement, and by listing variable names alphabetically, the
dataset is created with variables in alphabetical order, so PROC PRINTs
and PROC MEANS will show the variables in order, which is very useful
when the dataset has hundreds of variables.
The LENGTH statement follows the LABEL statement and always contains
DEFAULT=&MXGLEN, causing SAS to store numerics in only 5 stored bytes,
SAS default is 8 bytes, halving the DASD storage required. Four bytes
floating point will store exact integers up to 16,777,216 and will keep
seven significant digits, quite sufficient for most numerics. However,
variables that contain DATETIME stamps require 8 bytes (using only four
bytes for a DATETIME variable will truncate up to 255 seconds), and some
accounting variables (like Service Units) are also kept in 8 bytes (that
will store 16 significant digits, integers up to 72,057,594,037,927,936)
The FORMAT statement assigns the display format for variables, but does
not affect the internal value of the SAS variable. Time variables are
stored internally as seconds and fractions; most SMF durations have
resolution of .01 second so they are FORMATed TIME12.2 (999:59:59.99).
Datetime variables are stored internally as the number of seconds before
or after Jan 1, 1960, the SAS epoch, and are FORMATed as DATETIME21.2
(15AUG2019:12:00:00, the startime of the third Woodstock) and display
the four-digit year. Date variables are stored internally as the number
of days since the SAS epoch and are FORMATed DATE9 (15AUG2019).
Although FORMATs are assigned during creation of the variable, you can
always override in your reports with your own FORMAT statement. This is
especially true of the DATETIME format. If you want to count IPLs by
Date and Hour, you do not have to use DATEPART() and HOUR() functions to
create new variables in a new dataset; instead, you can directly process
the MXG dataset in your report program and shorten the format length:
PROC FREQ DATA=TYPE0; TABLES IPLTIME; FORMAT IPLTIME DATETIME10.;
To count by date, use FORMAT IPLTIME DATETIME7.; Because of this SAS
feature, MXG datasets always have one DATETIME variable instead of two
separate Date and Time variables.
Following the FORMAT statement, variables LENGTH and OFFSMF are tested
to detect invalid type 0 records (SYSPROGs who create SMF record often
have unexpectedly created a record with ID=0). LENGTH and OFFSMF are
created in MACRO _SMF, a portion of which is shown in Figure 3:
Figure 3
MACRO _SMF
;INFILE SMF STOPOVER LENGTH=LENGTH COL=COL START=BEGINCPY
END=ENDOFSMF RECFM=VBS LRECL=32760 JFCB=SMFJFCB;
RETAIN OFFSMF PREVID PREVSYS;
IF OFFSMF=. THEN DO;
IF SUBSTR(SMFJFCB,100,1)='....1...'B THEN OFFSMF=4; /*VSAM*/
ELSE OFFSMF=0;
%%INCLUDE SOURCLIB(IMACZDAT); /* SET ZDATE=TODAY() */
END;
INPUT @1+OFFSMF MVSXAFLG &PIB.1.
@2+OFFSMF ID &PIB.1.
@3+OFFSMF SMFTIME SMFSTAMP8.
@11+OFFSMF SYSTEM $EBCDIC4.
@;
%%INCLUDE SOURCLIB(IMACFILE);
%
The leading semicolon before INFILE is needed to terminate the DATA
statement that will precede it (since the _VARxxxx tokens cannot end
with a semicolon).
The OFFSMF test is initialization logic; once OFFSMF has been set, the
RETAIN statement will keep that value. The JFCB=SMFJFCB operand on the
INFILE puts the SMF Job File Control Block in variable SMFJFCB; if the
fifth bit of the 100th byte of the JFCB is on, your //SMF DD points to
an undumped VSAM SMF file. SMF records in the VSAM SMF file have four
bytes that are not present in the dumped SMF BSAM records; by setting
OFFSMF=4 for VSAM and using +OFFSMF in the INPUT statement, those extra
bytes are skipped, so you can transparently read either dumped BSAM or
un-dumped VSAM SMF data.
IMACZDAT externalizes the code to set variable ZDATE (Zee Date Zee obs
was created) so it can be changed in case of a rerun. The SMF header's
four always-present variables are INPUT, and exit member IMACFILE is
called. IMACFILE can be used to delete or select SMF records by time,
ID, or system, to write selected raw SMF records to a flat file, etc.
Returning to the _CDE0 section, variable SMFTIME that was INPUT in the
_SMF macro is stored into IPLTIME, and then the variables unique to the
type 0 record are INPUT. SAS provides a wide range of SMF INFORMATS
(SMFSTAMP, TODSTAMP and RMFSTAMP) that convert data into SAS datetime
values; these unique INFORMATs work the same on all SAS platforms.
However, INFORMATs IB, PIB, PD, RB, and NUM under MVS must be changed to
S370FIB, S370FPIB, S370FPD, S370FRB, and S370FF when SAS is executed on
ASCII platforms. For these INFORMATs, MXG uses a macro variable name
(&PIB) in its source code, and member VMXGINIT (invoked by the CONFIG
member at startup) executes either %LET PIB=S370FPIB or %LET PIB=PIB,
depending on the execution platform, for transparent execution anywhere!
While numeric variables can be used for fields containing hex values,
using HEX format for display, MXG now INPUTs hex fields into character
variables with INFORMAT $CHAR and assign a $HEX FORMAT, because a one
byte character variable stores in one byte, while a one byte numeric
needs two storage bytes on MVS and three bytes on ASCII platforms, and
a four-byte numeric must be stored in five-bytes to see all of the bits.
MXG detects when IBM has added new data to a record, to make MXG fully
backwards compatible with all levels of MVS. The test for LENGTH-OFFSMF
detects the new data added by MVS/ESA 5.1. While bits in MVSXAFLG might
have been used to identify the MVS version that wrote the record, it is
usually safer to test the record length rather than a version indicator,
because developers must set the length correctly, but do not always
update version numbers!
Although not shown in the _SMF macro in Figure 3, variables PREVSYS and
PREVTIME contain the SYSTEM and SMFTIME from the immediately preceding
SMF record, so that variable DOWNTM, an estimate of the outage, can be
calculated for the TYPE0 (IPL) dataset. Then _CDE0 creates variables,
converts SMCAJWTM to seconds, and the TYPE0 Data Set Exit member EXTY0
is INCLUDEd. That member contains comments and the SAS statement:
OUTPUT _LTY0;
to externalize the OUTPUT of the TYPE0 dataset. In the Data Set Exit
you can create new variables to be output, or you can add logic to
conditionally execute the OUTPUT statement and thereby output only
certain observations. The exit is taken after all variables have been
created, so any criteria can be used for selection. The Data Set Exit
member plus the _L and _K macro definitions in the IMAC member allow
complete user tailoring of the contents of all MXG datasets.
f. Architecture of Building the Performance Data Base - BUILDPDB
I coined the term PDB, for the Performance Data Base, while at State
Farm in the fall of 1972, when I began to regularly use my SAS program
to process weekly SMF data records (from OS/360 MVT Release 20.6!). I
recall a comment that we would need to build a warehouse to store the
SMF data tapes, and a reply that we would let SAS manage the warehouse!
The original PDB used only type 0, 1, 4, 5, 6, and 12 SMF records, and
the weekly PDB, A.PERF155(0), was built on a mountable 3330-1 device;
six weekly PDBs would fit on one 3330 back then! I implemented the daily
PDBs in 1973, at the request of Operations, who had come to like the
weekly reports and wanted daily detail. All trending, capacity planning
and serious analysis must use weekly data, rather than daily or monthly,
to see the true growth, because holiday-containing weeks fall out of a
weekly plot, while monthly data varies according to the number of work
days (and when they fall in the week) and cannot be normalized safely.
The term PDB was in wide-spread use inside State Farm by 1973, when
Mario Morino and Doug Denault came to Bloomington, to install the first
copy of their new TSO/MON product. They arrived Monday morning, and
with the help of Kathy Colbert and Steve Cullen, they had assembled and
started the monitor by noon. They then began to compile their COBOL
report programs, which were still compiling at 6pm when Kathy handed me
the SMF format for the TSO/MON SMF record as they went out to supper. I
wrote the SAS code to decode the new record, added it to the PDB job,
and came in the next morning to find the new code was successful; when
Mario and Doug showed up at 8am, I gave them a SAS plot of the number of
TSO users versus time of day, and thus did Mario learn of the power of
SAS! (Only late that second day did their programs produce any reports!)
The MXG BUILDPDB logic consists of five phases. In the first phase,
the input SMF file is read and multiple SAS datasets are created in the
//WORK data library. The simple macro references for this phase are:
DATA
_VAR0 _VAR0203 _VAR6 _VAR21 _VAR26J2 _VAR30 _VAR7072 _VAR71
_VAR73 _VAR74 _VAR75 _VAR77 _VAR78 _VAR89 _VAR110 _VARDB2
_VARTMNT _VARUSER
_SMF
_CDE0 _CDE0203 _CDE6 _CDE21 _CDE26J2 _CDE30 _CDE7072 _CDE71
_CDE73 _CDE74 _CDE75 _CDE77 _CDE78 _CDE89 _CDE110 _CDEDB2
_CDETMNT _CDEUSER;RUN;
This DATA step creates 116 SAS datasets in the //WORK library, creates
the CICSTRAN dataset (one per CICS transaction, from type 110) and the
DB2ACCT dataset (one per DB2 plan, from type 101) direct to tape (to
minimize DASD space and yet capture each CICS transaction and DB2 plan).
Programs ASUMCICS and ASUMDB2A, executed after BUILDPDB, read the tape
detail transaction datasets to create the summary datasets PDB.CICS and
PDB.ASUMDB2A that are compact for CICS and DB2 response and resources.
Four PDB exits, EXPDBINC, EXPDBVAR, EXPDBCDE, and EXPDBOUT allow other
SMF records to be added to the PDB in the one reading of the SMF file.
In the second phase of BUILDPDB, datasets TYPE0, TYPE0203, TYPE21,
TYPE30_D, TYPE30_6, TYPE30OM, TYPE30MU, TYPETMNT, TYPETSWP and TYPETALO
are SORTed from the //WORK data library into the //PDB data library. By
creating PDB datasets in SORT order, report programs do not have to do
a SORT; instead, they can use BY statements with the report procedures
to save CPU time and I/O activity for reporting. The sort order is
preserved into the WEEKLY and MONTHLY PDB libraries as well.
The third phase SORTs these RMF datasets into the //PDB data library:
TYPE70 TYPE70PR TYPE71 TYPE72 TYPE72DL TYPE72GO TYPE72MN TYPE72SC
TYPE73 TYPE73L TYPE73P TYPE74 TYPE74CF TYPE74ST TYPE74TD TYPE75
TYPE77 TYPE78 TYPE78CF TYPE78CU TYPE78IO TYPE78PA TYPE78SP TYPE78VS
and then invokes member RMFINTRV to read and MERGE these datasets:
TYPE70 TYPE71 TYPE72 TYPE72GO TYPE73P TYPE74 TYPE75 TYPE78
to create one observation per RMF interval in the PDB.RMFINTRV dataset.
PDB.RMFINTRV contains the PCTCPUBY from type 70, and (by using member
IMACWORK to map performance groups or service classes to workloads) type
72 resources are summed in Batch, TSO, CICS, etc workload variables, so
uncaptured CPU time and capture ratio can be measured, and the paging
swapping and I/O activity is contained in a single observation for each
RMF interval, providing hour-by-hour capacity measurement data.
The fourth phase reads the 47 CICS statistics datasets that were written
to the //WORK library and invokes member CICINTRV to summarize them into
the PDB.CICINTRV (interval) and PDB.CICEODRV (shutdown) CICS datasets.
The fifth phase of BUILDPDB operates on the type 30 subtype 1, 4, and 5
records, the type 6, and type 26 records to create accounting, resource
and activity data for Jobs, TSO, STC, APPC, and Open MVS address spaces.
Records for incomplete jobs (i.e., executed but not printed, or still
running when SMF was dumped, etc.) that were written yesterday to the
PDB's //SPIN data library are merged in with today's new SMF data. The
completed jobs are written to the PDB.JOBS (one observation per job, 223
variables), to the PDB.STEPS (one per step, 200 variables), and to the
PDB.PRINT (one per print file, 51 variables) and job accounting fields
are propagated into PDB.STEPS and PDB.PRINT so that resource billing by
account at the job, step, or program level can be done directly from the
PDB. Today's still-incomplete job's records are written to the //SPIN
library for tomorrow's processing. The IMACSPIN member defines SPINCNT,
the number of days BUILDPDB will SPIN records. While the PDB.JOBS,
PDB.STEPS and PDB.PRINT data sets contain observations for completed
jobs, the dataset PDB.SPUNJOBS describes all jobs in the //SPIN library,
so all of yesterday's work can be analyzed. Holding the records in the
//SPIN library until the job has purged produces a single picture of the
job. If records are not SPUN, one day's PDB can have an obs with only
the CPU time from the type 30s, another day's PDB can have an obs with
only print lines from the type6, and yet another day's PDB will have an
obs with just Purge record data, and all three obs from this one job
will have many missing values and an incomplete picture, although no
resources are lost. The individual SPINxxxx data sets are copied from
the SPIN library into the PDB library for backup purposes. You can
always go back to the last successful run, copy the SPINxxxx datasets
from that PDB into the SPIN library, and then restart after an error.
While dataset TYPE70PR (one obs for each PR/SM, MDF, or MLPF LCPUADDR in
each LPAR) contains LPAR utilization, INCLUDing ASUM70PR after BUILDPDB
summarizes TYPE70PR to creates the more usable PDB.ASUM70PR dataset with
resources consumed by each LPAR and with total CPU busy for all LPARS.
g. Mining costs and tons of warehouse data dug up and delivered:
The first phase of BUILDPDB took 1 hour and 1 minute of elapsed time and
16 minutes of CPU time on an IBM 9021-952 to process two 3490 tapes with
596,814 SMF records totaling 2,314 Megabytes (2.3 Gigabytes) of raw SMF
data from three MVS systems. The total BUILDPDB step plus the ASUM70PR,
ASUMDB2A, ASUMCICS and ASUMJOBS summaries took 1 hour 55 minutes elapsed
and 24 CPU minutes. The WORK file required only 256 Megabytes (324
cylinders of 3390).
This Performance Data Base PDB output data library contains 143 datasets
totaling 3,040 Megabytes, but 2,290 of those Megabytes are in CICSTRAN
for its 2,333,338 CICS transactions. The high-volume CICSTRAN dataset
is written directly to tape, to archive each transaction, but using no
DASD space. Then CICSTRAN's 2,290 MB are summarized into only 21 MB in
the 214,088 observations of the PDB.CICS summary dataset by ASUMCICS
(which invokes the VMXGSUM generic summarization %MACRO and took only 22
minutes elapsed and 3 and a half minutes of CPU for that summarization).
So the online DASD PDB required only 910MB (1150 cylinders on a 3390).
Its contents are shown in Figure 4. Most of the volume is taken by only
a few datasets: DB2ACCT (427MB for 266,513 DB2 plan executions, which
could be sent to tape like CICSTRAN), STEPS (83MB for 88,777 steps),
ASUMDB2A (69MB, the summarized output from DB2ACCT), TYPE74 (48MB), JOBS
(34MB for 31,472 job executions), TSOMSYST (30MB), TYPE30MU (22MB) and
CICS (21MB), plus all other RMF data (24 MB) account for 760 MB. But
many important datasets are less than one megabyte; the RMFINTRV summary
dataset has 24 hours of each of the 3 system's 15-minute RMF interval
data (288 obs) in only 452 Kilobytes (9 tracks) of DASD space!
Figure 4
Dataset Obs Vars Len MBYTEs Description
ASUM70PR 288 218 836 RMF PR/SM LPAR INTERVAL SUMMARY
ASUMDB2A 52110 323 1383 69 DB2 DB2ACCT INTERVAL SUMMARY
CICAUSS 4648 31 172 1 CICS AUTOINSTALL TERMINAL USS
CICAUTO 96 43 203 CICS AUTOINSTALL GLOBAL
CICCONMR 2110 37 183 CICS ISC/IRC MODE ENTRY
CICCONSR 2044 48 223 CICS ISC/IRC SYSTEM ENTRY
CICCONSS 90 27 139 CICS ISC CONNECTION - SECURITY
CICDQG 96 38 183 CICS TDQUEUE TRANSIENT DATA GLOB
CICDQR 1992 28 140 CICS TDQUEUE TRANSIENT DATA SPEC
CICDS 96 87 367 CICS DISPATCHER, CPU BY TCB
CICDTB 96 21 115 CICS DYNAMIC TRANSACTION BACKOUT
CICEODRV 96 290 1180 CICS END OF DAY
CICFCR 2352 70 435 1 CICS FILE CONTROL SPECIFIC
CICINTRV 0 290 1180 CICS INTERVALS
CICJCR 196 32 162 CICS JOURNAL CONTROL SPECIFIC
CICLDG 576 43 208 CICS LOADER DOMAIN FOR PROGRAM
CICLDR 165362 28 144 23 CICS LOADER DOMAIN FOR PROGRAM
CICLSRFR 688 25 135 CICS LSRPOOL FILE STATS EACH FIL
CICLSRR 76 294 1220 CICS LSRPOOL POLL STATS EACH POO
CICM 96 27 139 CICS MONITOR DOMAIN GLOBAL
CICPAUTO 96 22 119 CICS AUTOINSTALL PROGRAM
CICS 214088 23 105 21 CICS INTERVAL TRANS SUMMARY
CICSDG 96 21 115 CICS SYSTEM DUMP GLOBAL
CICSDR 496 24 131 CICS SYSTEM DUMP SPECIFIC
CICSEXCE 205 39 194 CICS EXCEPTIONS
CICSMD 15092 35 164 2 CICS STORAGE MANAGER DOMAIN SUBP
CICSMDSA 768 64 335 CICS STORAGE MANAGER DSA AND EDS
CICSMT 384 30 146 CICS STORAGE MANAGER TASK SUBP
CICST 96 22 119 CICS STATISTICS DOMAIN GLOBAL
CICSTRAN 2133338 260 1126 2290 CICS TRANSACTIONS
CICTCR 12476 35 166 2 CICS TERMINAL CONTROL SPECIFIC
CICTDG 96 21 115 CICS TRANSACTION DUMP GLOBAL
CICTDR 1368 24 127 CICS TRANSACTION DUMP SPECIFIC
CICTM 96 53 243 CICS TABLE MANAGER GLOBAL
CICTSQ 96 57 259 CICS TSQUEUE TERMPORARY STORAG
CICUSG 96 24 127 CICS USER DOMAIN STATISTICS
CICUSSRV 147 290 1180 CICS USS INTERVAL SUMMARY
CICVT 96 30 151 CICS VTAM GLOBAL
CICXMC 1050 37 183 CICS TRANSACTION MANAGER TCLASS
CICXMG 96 31 155 CICS TRANSACTION MANAGER GLOBAL
CICXMR 32216 32 168 5 CICS TRANSACTION MANAGER TRANS
DB2ACCT 266513 354 1680 427 DB2 PLAN ACCOUNTING
DB2ACCTB 0 46 294 DB2 PLAN BUFFER POOLS
DB2ACCTG 11 39 39 DB2 PLAN GROUP BPOOLS
DB2ACCTP 0 79 458 DB2 PLAN PACKAGES
DB2GBPAT 0 23 105 DB2 GLOBAL BUFFERS
DB2GBPST 0 44 148 DB2 STAT GB POOLS
DB2STAT0 571 540 2130 1 DB2 STATS SUBTYPE 0
DB2STAT1 571 510 1987 1 DB2 STATS SUBTYPE 1
DB2STAT2 2296 24 111 DB2 STATS SUBTYPE 2
DB2STATB 1150 80 348 DB2 STATS BUFFER POOLS
DB2STATR 357 54 256 DB2 REMOTES
DB2STATS 571 1037 4047 2 DB2 STATISTICS INTERVAL
DDSTATS 0 26 148 TYPE 30 DD SEGMENTS
IPLS 0 14 70 TYPE 0 IPLS
JOBS 31472 223 1121 34 JOBS/TSO/STC/APPC/OMVS
JOBSKED 194 19 89 JOB SCHEDULING CLASS SUMMARY
NJEPURGE 0 62 357 NJE PURGE EVENTS
PRINT 15322 51 363 5 TYPE 6 PRINT EVENTS
RMFINTRV 288 402 1608 RMF INTERVAL SUMMARY
RRTMPSE 718 67 301 ROSCOE ACCOUNTING
SMFINTRV 0 218 1078 SMF INTERVAL ACCOUNTING
STEPS 88777 200 989 83 STEPS FOR JOBS/TSO/STC/APPC/OMVS
TAPEMNTS 0 18 18 MXGTMNT TAPE MOUNT SUMMARY
TAPES 1862 28 124 TYPE 21 TAPE DISMOUNTS
TSOMCALL 9075 110 573 5 TSO/MON USER RESPONSE
TSOMSYST 39372 183 785 30 TSO/MON SYSTEM INTERVAL
TYPE0203 140 5 27 SMF DUMP HEADER/TRAILER
TYPE23 72 22 110 SMF INTERVAL STATISTICS (STATUS)
TYPE30MU 142337 24 165 22 MEASURED USAGE DATA
TYPE30_6 1392 148 636 1 SYSTEM ASID INTERVALS
TYPE37 7941 102 1091 8 NETVIEW HARDWARE MONITOR EXTERNA
TYPE70 288 382 1435 RMF CPU ACTIVITY
TYPE70PR 4800 34 121 RMF PR/SM LPAR ACTIVITY
TYPE71 288 307 1248 RMF PAGING ACTIVITY
TYPE72 15941 89 396 6 RMF PERFORMANCE GROUP
TYPE7204 0 73 321 RMF MONITOR III GOAL MODE
TYPE72DL 0 35 166 RMF GOAL MODE DELAY SAMPLES
TYPE72GO 0 151 797 RMF GOAL MODE SERVICE CLASS
TYPE72MN 14699 81 345 5 RMF MONITOR III COMPAT
TYPE72SC 0 16 97 RMF SERVICE CLASS SERVED
TYPE73 47520 45 197 9 RMF CHANNEL PATH
TYPE74 118712 101 423 48 RMF DEVICE ACTIVITY
TYPE74CA 0 149 483 RMF CRR CACHE CONTROLLER
TYPE74CF 384 188 1050 RMF COUPLING FACILITY
TYPE74OM 0 82 336 OPENMVS KERNEL ACTIVITY
TYPE74ST 1728 56 259 RMF CF STRUCTURE REQUESTS
TYPE74TD 288 11 61 RMF TAPE DRIVES ALLOCATED
TYPE75 1344 40 225 RMF PAGE DATASET ACTIVITY
TYPE77 9917 43 266 3 RMF ENQ ACTIVITY
TYPE78 0 29 128 RMF I/O QUEUEING
TYPE78CF 15080 32 131 2 RMF I/O CONFIGURATION
TYPE78CU 5155 23 110 RMF LCU CU-HDR QUEUEING
TYPE78IO 1152 23 107 RMF IOP QUEUE
TYPE78PA 0 102 569 RMF JOB-LEVEL VIRTUAL STORAGE
TYPE78SP 0 18 109 RMF JOB-LEVEL SUB POOLS
TYPE78VS 288 445 2430 RMF VIRTUAL STORAGE
TYPE89 0 29 188 MEASURED USAGE INTERVAL
TYPETALO 0 42 286 MXG TAPE ALLOCATION MON
TYPETMNT 0 16 82 MXG TAPE MOUNT MONITOR
TYPETSWP 0 7 33 MXG TAPE SWAP EVENT
Total Size in Megabytes: 3200MB
And what about that 130 million observation CICSTRAN dataset? One
massive site read that many CICS transactions from 68 SMF tapes (3490E
compressed) to select 1500 CICS transactions. The job took 30 elapsed
hours and 5 CPU hours!
h. Growth of the MXG Source Library
An Annual MXG Version is created in first quarter of each year, and
throughout the year, interim Versions are created as needed to keep
up with new products and changes to old records. The fourteenth MXG
Annual Version will ship in early 1997, but MXG 14.06, the August, 1996
Current Version, was the 99th MXG Version created! Figure 5 lists the
measurements of each of the Annual MXG Versions and MXG 14.06. Sometime
in 1997, the MXG source library should exceed 1,000,000 source lines.
Figure 5
HISTORY OF MXG VERSIONS AND RELEASES
VERSION DATE MEMBERS LINES BYTES MB PAGES DATASETS VARS
14.06 20AUG96 3011 911,749 72,939,920 69.6 15,196 1908 78,278
13.13 20JAN96 2940 885,344 70,827,520 67.3 14,756 1868 76,219
12.12 20MAR95 2533 776,687 62,134,960 59.3 12,945 1573 66,221
11.11 26MAR94 2286 654,341 52,347,280 49.9 10,975 1360 55,168
10.10 15MAR93 1965 534,902 42,792,160 40.8 9,999 1195 47,296
9.9 1MAR92 1605 388,651 31,092,080 29.6 6,477 1093 41,332
8.8 8APR91 1367 300,000 24,000,000 22.8 5,000 872 30,420
7.7 15FEB90 1100 230,000 18,400,000 17.5 3,834 679 22,277
6.6 15JAN89 910 165,614 13,249,120 12.6 2,760 552 18,048
5.5 01FEB88 785 136,880 10,951,296 10.7 2,281 456 14,909
4.4 01MAR87 661 108,166 8,653,472 8.8 1,803 360 11,770
3.2 01FEB86 537 79,444 6,635,648 6.8 1,346 264 8,632
2.0 01FEB85 413 50,722 4,057,024 4.9 845 169 5,526
1.0 AUG84 289 22,000 1,760,000 3.0 367 73 2,387
The original MXG 3420 Tape Reel contained 99 feet of source code!
i. SAS does not stand for Single Authored Software. Acknowledgements.
While I have designed all and written most of the MXG Software, many
users have contributed code examples, and my three consultants have ably
tested each new release, have covered my technical calls while I am out
teaching, and who have personally contributed significantly to MXG, are
hereby acknowledged specifically:
Chuck Hopf Bruce Widlund Freddie Arie
Overseas, where we are represented by the local SAS Office, there are
also scores of dedicated SAS technicians who have provided local
language and local time of day help with MXG queries.
But our real thanks for our success are to the dedicated MXG users, who
have taken the time to learn those prerequisite skills of SAS, JCL, TSO,
and PC technology, and who have waded thru 1,497 pages in two Books, 730
pages in 30 Newsletters, and 2,994 Changes in 99 Releases to learn how
to use MXG Software to measure and thereby improve the performance of
their company's computer systems - they have made all of us look good!
Chapter 99
This chapter cites and thanks all contributors to MXG Software, who
are hereby officially awarded the title of "Chapter 99 CodeSharks"!
Named by Judith S. "99" Merrill, Vice President and Partner, in honor
of the diligent sleuthing of the MXG code performed by each CodeShark,
she also established our policy that credit should be given to every
MXG user who made any contribution to MXG Software, whether they
provided an entire program, or found a programming error, or offered a
suggestion, or even just found a comma out of place in a comment!
Contributors with Eleven or more Changes - Descending Ranking
Name of Contributor Total Number of Changes
Chuck Hopf 130
Diane Eppestine 50
Norbert Korsche 45
Freddie Arie 40
Tom Parker 34
Joseph Faska 28
Pete Shepard 27
Siegfried Trantes 23
Don Deese 19
Bruce Widlund 18
Rodney L. Reisch 18
Susan Marshall 17
Tom Elbert 16
Jim Gilbert 15
Dan Kaberon 14
Neil Ervin 14
Waldemar Schneider 14
Rod Fernandes 12
Shaheen Pervaiz 12
Bob Rutledge 11
Dan Squillace 11
Dick Whiting 11
Don Friesen 11
George Scott 11
Ray Dickensheets 11
2. MXGTMNT, the Tape Mount and Tape Allocation Monitor Program (in the
member ASMTAPES, in MXG 14.01 and earlier (MAINTLEV 8 and earlier),
can stop writing Tape Allocation subtype SMF records, after first
sending the messages TMNT008E or TMNT013E to SYSLOG, causing zero
observations in dataset TYPETALO.
(The Tape Mount Monitor part of MXGTMNT continued to write Tape
Mount subtypes, and the Allocation monitor can be restarted
using the Modify Command, F MXGTMNT,ALLOC=YES.)
The TMNT013E stoppage resulted when the Allocation Monitor had been
waiting more than 2 seconds for its own lock that controls access
to its own SRB, because we thought a long wait should not occur,
and so to protect against any possible error in our code, we
decided to stop the allocation monitor component. Two sites (both
with very large tape-intensive multi-image environments) reported
stoppages. On investigation with Mike ONeill at Dun and Bradstreet
(we commented out the stoppage and then examined SYSLOG) he found
that across four days there were two instances on two of three MVS
images when five or six TMNT013E messages were written, all in one
minute! So what we had were occasional system 'stalls', but no
real reason to shut down the allocation monitor.
So beginning with MXG 14.02 (See Change 14.086, MAINTLEV 9), the
TMNT013E Error and Stoppage of the Allocation Monitor was replaced
by Informational Message TMNT013I - Possible Stall, and the
Allocation Monitor now continues to write allocation subtypes.
Now that I realize we are detecting system "hangs", or "stalls",
the ASMTAPES will be enhanced to actually report these events by
creating a new subtype=5 "Possible Stall" event record, so that you
can analyze if, how often, and how long these "stalls" occur, and
can investigate (perhaps by examining SYSLOG to see what system
messages immediately preceded the image-wide delay) their cause.
Even in advance of that enhancement, you can use MAINTLEV 9 and
scan SYSLOG for TMNT013I messages to detect these "stalls".
The TMNT008E stoppage occurs if we are unable, after 25 tries, to
schedule our SRB in the address space of a tape-using job. This was
a problem with MAINTLEV 7 and earlier while trying to schedule an
SRB for a swapped-out address space, but this has not occurred with
MXG 13.13, and we no longer expect to see this condition.
3. If you want "RMFINTRV" data for both detail and hourly intervals,
you can use this (rather clumsy!) technique until I the new %MACRO
%RMFINTRV exists. You would need to create "YOUR.SPECIAL.PDS" and
EDIT member IMACRMFI into "YOUR.SPECIAL.PDS" (and would set the
MACRO _DURSET therein for hourly summary - see its comments).
//RMFINTHR EXEC MXGSAS
//PDB DD DSN=&&PDB,UNIT=SYSDA,SPACE=(CYL,(5,5))
//REALPDB DD DSN=YOUR.REAL.PDB,DISP=OLD
//* NOTE: PUT YOUR MODIFIED IMACRMFI MEMBER YOUR.SPECIAL.PDS
//SOURCLIB DD DSN=YOUR.SPECIAL.PDS,DISP=SHR
// DD DSN=YOUR.USERID.SOURCLIB,DISP=SHR
// DD DSN=YOUR.MXG.SOURCLIB,DISP=SHR
PROC COPY IN=REALPDB OUT=PDB;
SELECT TYPE7072 TYPE71 TYPE73 TYPE74 TYPE75 TYPE78;
%INCLUDE SOURCLIB(RMFINTRV);
DATA REALPDB.RMFINTHR; SET PDB.RMFINTRV;
4. Some allegations that the KEEPALL=NO option in %VMXGSUM "costs"
have been counteracted by extensive measurements. The KEEPALL=NO
option (the default in VMXGSUM) will significantly reduce the DASD
work space, CPU time, elapsed time, and I/O consumed by VMXGSUM,
because only the variables that are needed are kept.
ASUM42DS example: 1,133,117 raw observations reduced to 100,628
observations with 25 variables kept:
KEEPALL WALL CPU EXCP DASD WORK SPACE
NO 12:40 2:53 14976 231MB
YES 23:05 4:03 23411 460MB
The only real "cost" of KEEPALL=NO is the cost of the additional
DATA and SORT steps that are used to parse the VMXGSUM arguments to
determine the list of variables to be kept. That cost is a fixed
function of the number of variables that are kept, and is not
affected by the number of observations processed. With a small
number of variables (16) the CPU increase is only 2 seconds, while
for a lot of variables (354) the CPU increase is still only about
15 seconds. The above example shows that small CPU increase is
insignificant when compared with the savings when you actually
perform VMXGSUM summarization of typical MXG datasets.
The KEEPALL=NO option did increase the size of the SAS log because
of the NOTES printed on the log for the additional DATA/SORT steps
(800 more lines in the preceding example), but Change 14.145 has
enhanced VMXGSUM to insert OPTIONS NONOTES; before those parsing
steps (which is reset after parsing) so these extra lines will no
longer appear on the log (although enabling the DEBUG option in
VMXGSUM will print them if ever needed for diagnostics).
5. BUILDPDB processes SMF data at 41MB per CPU-minute on a 3090-300S.
You can separate the Compile cost of BUILDPDB from the Read+Write
cost to estimate the CPU time per megabyte of data. By specifying
//SMF DD DUMMY in your JCL, the CPU cost of Compile with no records
read is measured (use the CPU Time on the SAS log following the
"MXG HAS SUCCESSFULLY READ xxx MB of SMF DATA" message and use that
message for SMF data volume). Then execute BUILDPDB with real
//SMF DD DSN=... to measure Compile+Read+Write CPU time. The delta
between the two runs measures the CPU time to read that amount of
SMF data and create all of the //WORK datasets. For example, on a
3090-300S, the Compile+Read+Write with 30MB took 143 seconds while
the Compile phase took 99 seconds so BUILDPDB took 143-99=44 CPU
seconds for the 30MB, or processes 41MB of SMF data per CPU minute.
III. MVS Technical Notes.
1. APAR OW16564 corrects zero value in PCHANBY variable in TYPE73 that
began with MVS/ESA 5.2.0. One site had added test IF PCHANBY GT 0
THEN before the OUTPUT statement in member EXTY73 (so as to output
TYPE73 observation only if there was channel activity during the
interval) and found that test reduced 16,896 observations from 5.1
to only 8,000 (i.e., half of the time there was no channel
activity), but with the IBM error in MVS/ESA 5.2 data, only 55
observations had non-zero PCHANBY value! (Note that the MXG test
inside VMAC73 to invoke the EXTY73 exit does not test for usage of
the channel, but rather will output only real channel segments.
IBM writes 255 channel segments in each type 73 record. Without
the MXG heuristic to delete the false channel segments, there would
have been 26,456 observations output instead of 16, 896!). You
could add your test to EXTY73 to only output if PCHANBY is non-zero
to reduce observations, but even with 16,896 observations, the size
of a daily TYPE73 dataset is only about 500,000 bytes!
2. APAR OW16847 and OW19029 corrects TYPE30 variables EXCPTOTL and all
EXCPxxxx variables (i.e., both SMF30TEP and SMF30BLK values) for
DB2 I/O. The variables had very large values. The APAR text
reads: "ICYSTOR will now clear the count field, and ICYPFCP will
be changed to put the blocks per track into the SMF count field.
The VSAM Media Manager does all I/O for Linear Datasets. Most DB2
Tablespaces reside on Media Manager controlled linear data sets.
The DBM1 Address Space calls the VSAM Media Manager to perform
these asynchronous database I/O operations." and comments that the
problem was reported with both DB2 3.1 and DB2 4.1. This is the
first printed reference I have seen that confirms that DB2 I/O is
captured in type 30 records; with the APAR, it appears the count is
not only captured but also is now accurate!
3. APAR OW17491 corrects TYPE21 variable TAPCUSER (Tape Unit Serial,
which identifies which unit created the tape, and can be quite
helpful in tracking the source of tape errors). The problem was
caused by DF/SMS 1.3 which inserted a macro call that used Register
zero (which had already been used by IFG0194F to store TAPCUSER).
4. APAR OW17121 reports incorrect values for TYPE6 variables JOB,
READTIME, and LOCLINFO for type 6 records created by PSF for APPC
transaction initiators running under JES2 and printed on the same
JES2 node; the record contains values for the APPC transaction
initiator instead of the transaction itself.
5. APAR OW17469 reports excessive count of type 80 SMF records after
MVS/ESA 5.1 if commands are issued by STCs (e.g., for each START
INIT command issued by JES2, an SMF80 event type 1 qualifier 25
record is created). This fix is in addition to APAR OW14521.
6. APAR OW18822 corrects the invalid JESNR in type 26 SMF records that
was fixed by MXG in Change 13.263. The APAR will make the 4-digit
field zeroes is the JES number is over 9999, but the MXG change has
already corrected the value of JESNR in MXG, so MXG does not need
this APAR to be installed.
7. ISPF Find and Change commands use the equal sign as a wild card in
a picture clause. To find strings starting with ABC that have X
in the 6th position, FIND P'ABC==X' is the valid syntax.
And Picture clauses can be used in Change commands. To blank
columns 73 80 for all non-excluded lines you can use
CHANGE P'=' ' ' 73 80 ALL NX
8. Hitachi 9021 Skyline Processors trash TYPE73 (Channel Path Busy)
until you install HDS Microcode Fix EC SCTC943.
9. APAR OW19482 corrects variable TTRN in dataset TYPE1415 so that it
counts total tracks for striped datasets. Without this APAR, the
TTRN contains only the tracks used in the first stripe.
10. APAR OW03645 reports extremely high disconnect time in type 74
RMF records for 9345 and 9394 devices behind 3990-6 with EC C88981H
or higher.
11. CA's CA-SPOOL Release 1.8 creates massive numbers of type 6 SMF
records that have nothing to do with printing; instead, they are
written when data is moved from the JES2 spool to the CA-SPOOL
spool, and CA support indicates they should not have been written.
CA APAR GS98190 dated 31May96 will suppress creation of these
"spurious" type 6 records. These records all have the JOB name of
the CA-SPOOL address space, but there is no generic flag by which
MXG can identify these spurious records. I have suggested to CA
that if they would set a unique identifier in these type 6 records,
(as they do for EXD and SAR type 6 records), and make the records
optional, they might be useful to measure CA-SPOOL's internal
performance, but their volume is so large that not writing makes
more sense, at least until they can be separated from "real" 6's.
Note that actual printing done by CA-SPOOL is separately recorded
in their user SMF record, supported by MXG member TYPECMA.
12. APAR OW01699 corrects TYPE72MN (type 72 subtype 2 - RMF Monitor II)
swap counts which were out of order, fixes R722LSEF(which was
always zero), and corrects ESCON bits in SMF72PRF.
13. APAR OW20318 corrects TYPE42TO (type 42 subtype 1) variable DURATM;
for PDSE and VSAM/Record Level Sharing IBM put out Timer Units
rather than seconds until you install this APAR.
14. Note 14 was a duplicate of note 6.
15. CPUTM in interval type 30s does not match step type 30s, because
the CPITCTBM/CPISRBTM times (the "initiator TCB and SRB CPU time")
are step-level measurements, and are always zero in the interval
records. These "initiator" buckets record the CPU time used by the
step before the program starts or after the program ends (the other
five CPU fields in all SMF type 30s and in RMF type 72s record the
CPU time used between program start and program end)
It would really have been useful if IBM had provided separate
buckets for the "before" values and the "after" values, since
then we could assign the "initiator" times to correct shift and
time-of-day and include them in interval summaries, but all we
have now is the lumped "initiator" CPU time in the step record.
These "initiator" CPI CPU time values can be significant. They are
the CPU time spent by the step between its initiation and its
program load (this includes data set enqueue in the first step, and
it especially includes the cost of allocation of each DD in the
step - here is where the CPU time to execute your SMS allocation
rules is recorded), and the CPU time spent by the step between its
program end and when its SMF type 30 subtype 4 is written (which
includes DDCONS CPU time, see above, and the time to create the
type 30 subtype 3 and 4 records).
Thus if you match the interval data to the step data you will find
more CPUTM in the steps than in the intervals, because CPITCBTM and
CPISRBTM are not in CPUTM in the interval data.
However, this is true only if there is a step termination record
for each step that had interval records. Even if you IPL-to-IPL
there will be some started tasks that were never "taken down"
and thus had no step termination records, but had many interval
records written, so there the sum of the interval records, while
still missing the "initiator" times, will be greater than the
step records. It is even worse if you just read a day's SMF
data and try to compare the steps data to the interval data:
you'll have step records today whose interval records were in
yesterday's SMF file, and you'll have interval records today
for tasks won't end until tomorrow, and match is impossible!
If you are using only the interval records for billing, you are
missing the "initiator" CPU time that was used, and you should
consider billing off of the sum of the interval data plus the
initiator CPU times from the step records.
This note added to ADOC30. Written Jun 15, 1996.
16. APAR OW18543 corrects blank values for Service Class and Report
Class in type 30 records with MVS/ESA 5.2.
17. APAR OW20904 (negative disconnect time or zero disconnect time
calculated from CMB values in type 74, the "core-clobber" problem)
has been solved in child APAR OW22093. RMF will now move a copy
of the CMB (Channel Measurement Block) into local storage and then
build the type 74 records. Previously, RMF would move one entry
at a time, and CMB updates between individual loads caused the
error. The fix also check sample count before and after copying
of the CMB to validate that it had not changed during copy. This
note was revised October 3, 1996.
18. Your SMF dump job will run lots faster and use less resources if
you disregard the old recommendations in the SMF manual and provide
plenty of buffer space. Although MXG recommends the use of half
track CISIZE for SMF (see "Impact of VSAM CI Size..." in Newsletter
25), benchmarks were run at a site which had an SMF CISIZE of 16K
(16384) on a 3390; using BUFND=46 on the INDD DD statement of the
IFASMFDP program, the run time was HALF of the time with default
buffering:
BUFND (Bufferspace) Wall Time CPU Time EXCP Count
2 23768 5 minutes 6.98 seconds 41892
5 81720 4 minutes 5.67 seconds 32857
46 753665 2.5 minutes 4.68 seconds 24609
92 1507330 2.5 minutes 4.70 seconds 24609
The SMF default is 2 buffers (horrible); the SMF manual's very old
suggestion of using BUFSPACE=81720 (wow, 10*8172!) is better, but
clearly using the VSAM rule of thumb for sequential I/O, "set BUFND
equal to the number of Control Intervals that fit in one Control
Area, plus one" is clearly superior. Note also that the "rule of
thumb" appears to be optimal; doubling BUFND to 92 provided no
improvement from the "CIs in a CA + 1".
With SMF CA size of one cyl and SMF CISIZE of 16384 (16K),three
CI's fit on one 3390 track so you would need 3*15+1=46 buffers for
BUFFERSPACE=753664. With half-track CISIZE of 26624, you would
need 2*15+1=30 buffers for BUFFERSPACE=825344.
You can specify AMP=('BUFND=46') (16K) or AMP=('BUFND=31') (26K) on
your INDD statement in your JCL for IFASMFDP dump program to
significantly speed up the process. Alternatively, you can specify
BUFFERSPACE=753664 or BUFFERSPACE=825344 when you define your SMF
VSAM data set. Note that BUFND can be specified thru JCL (i.e., if
you use a DD statement to statically allocate the SMF VSAM dataset
in your dump job), but if instead you dynamically allocate your
IFASMFDP input file, then only BUFFERSPACE is available for you to
increase, and BUFFERSPACE can only be set when the VSAM dataset is
created - it cannot be altered after the fact.
What about the memory requirements? True, the virtual storage size
is increased, but by less than 1MB, but the real storage occupied
(as measured in pageseconds in the type 30 record) shows that less
real storage is used with 46 buffers than with the SMF default; the
larger bufferspace takes less real memory because the I/O is done
quicker and those memory pages are made available sooner to some
other task:
BUFND AVGWKSET PAGESECS
2 1173 1002
5 1220 823
10 1185 663
15 1204 640
30 1415 728
46 1639 802
92 1674 845
Once again, moving lots of data per I/O saves time, CPU, and real
memory! Large bufferspace for either VSAM or BSAM/QSAM is goodness
for sequential access.
19. Boole & Babbage's Cache RMF Reporter (user SMF record) is wrong.
In their record, they store the REPORLVL=1.6, but the format of
the record is from IBM's Release 1.5. Since MXG believed the
value and expected the 1.5 format, many variables have strange
values. Boole is working on PTF BPM5835 to correct their error.
20. APAR OW20844 increases the maximum allowable value for job numbers
from 32787 to 65534. There is no impact on MXG, as it already will
handle these larger values in variable JESNR. However, the SMF
records created by Software Engineering of America's $AVRS and by
Xerox's XPSM do not yet even support JESNR's greater than 9999, as
they have only a 4-byte numeric field for JESNR.
21. Experiments with BUFNO for Sequential Access (BSAM,QSAM), including
that rare experiment of RTFM (Reading The Fine Manual), discovered
that unstriped SAM is limited to transfer of 240K bytes per I/O,
and is also limited to a maximum of 30 buffers. Thus the maximum
number of SAM buffers (BUFNO=) should be
MIN ( 30 , 240K/BLKSIZE) ;
The above is true for most sequential datasets but NOT for extended
sequential datasets. Even with a stripe value of 1, more data can
be read with a single IO than with an ordinary sequential dataset.
This does not necessarily mean faster, it just means more data is
moved. Whether faster or not depends on the buffers.
The algorithm for building the minimum size of the channel program
for extended datasets with half-track blocks (1 slice = 1 track) is
IF BUFNO LE 1 slice then amount = BUFNO*BLKSIZE
IF BUFNO LT 2 slices then amount = 1 slice
IF BUFNO LT 5 slices then amount = min(bufno/2)
IF BUFNO GE 5 slices then amount = 5 slices
This yields slightly more than the 240K MAX for normal QSAM (272K
for an 80-byte records blocked at 27920.) The default buffers for
an extended sequential dataset can be calculated as:
2*blocks per track*number of stripes
Thus with 8 stripes of half-track blocks 32 buffers is the default.
Just like regular QSAM, these buffers live on low memory unless you
add a DCBE with RMODE31=BUFF and run as RMODE 31.
Data striping can yield significant savings in elapsed time.
In one experiment reading a 2.3GB SMF dataset the results were:
Device Elapsed Connect CPU MB/Sec
6390 - parallel 0:19:15 09:22 18.1 1.85
6390 - ESCON 0:07:58 03:58 18.2 4.47
6390 - 8 stripes 0:01:27 04:49 18.4 26.47
Data striping is enabled by using both a data and a storage class.
The storage class defines the Sustained Data Rate (SDR) desired for
the storage class (in this case a rate of 32MB/sec was used) and
the data class defines the maximum number of stripes that can be
allocated for that data class. Using the SDR and the maximum
number of stripes, SMS calculates the number of stripes to use on
3390 devices as SDR/4.
22. APAR PN87013 (open) reports TCP/IP TELNET LOGN SMF records are not
always written if DEFAULTAPPL parameter is specified.
23. APAR OW20823 for RACF type 80 record corrects missing data in the
relocate section data type 44 (x'2C') that occurred in certain
circumstances described in the APAR text.
24. CA's CA-1 (TMS) product APAR G081058 for CA1 5.1 (on 9604 tape)
will eliminate the IEC507D messages (and the required operator
reply!) that were introduced with 5.1 whenever multiple datasets
are created on one tape. Previously, CA-1 hooked MVS/DFP code to
suppress the issuance of that WTOR (Write to Operator with Reply)
but in release 5.1, CA1 no longer puts hooks via PDZAP; instead,
the hooks are dynamically front-ended via other CA services. This
APAR corrects their oversight in their new release. MXG's MONTHBLD
and WEEKBLD are examples of SAS jobs that create multiple datasets
per tape and exposed this CA-1 error.
25. APAR OW20956 acknowledges that TYPE1415 variable TRKSALOC is not
valid for PDSE datasets (it always contains one) and that in a
future PDSE release, the correct number of tracks will be returned.
26. APAR OW21083 fixes large values in CPUUNITS and SERVICE variables
(SMF30CSU,SMF30SRV) in TYPE30_V subtype 3 interval records for DB2
4.1 DDF address space. IBM's APAR reports "negative" values, but
MXG inputs these variables as PIB (Positive Integer Binary) and the
IBM "negative" values have first bits on, causing large positive
values in MXG datasets (and that is intentional, so that a large
value slaps you in the face, whereas the small negative value might
be overlooked if MXG had INPUT with IB instead of PIB).
27. Several sites with MVS/ESA 5.2.2 have noticed negative values for
the calculated Channel Pending time in TYPE74 data. One specific
device's interval of 556.13 seconds duration with 155 SSCHs shows
Active Time (SMF74ATV/DEVACTTM) of 1.12 seconds and its components
Pending Time (SMF74PEN/DEVPNDTM) 0.07 seconds, Disconnect Time
(SMF74DIS/DEVDISTM) 0.76 seconds, and Connect Time
(SMF74CNN/DEVCONTM) 0.29 which do add to 1.12 seconds. However,
the Pend for Device (SMF74DVB/DLYDEVTM) is 1.94 seconds, while
Pend for CU (SMF74CUB/DLYCUBTM) and Pend for Dir Port (SMF74DPB,
DLYDIRTM) are both zero. The Pend for Channel is calculated by
subtracting the Pend Components from Pend: PEN-(DVB+CUB+DPB), but
with DVB value greater than PEN, the calculation is negative. How
can Pend for Device be greater than total Pend time? How can Pend
for Device be greater than Device Active Time? Awaiting IBM reply.
28. To create SMFINTRV (TYPE30_V) globally synchronized, you must
specify INTVAL(15), SYNCVAL(15), and SYS(EXITS(INTERVAL(SMF,SYNC))
in SMFPRMxx member of SYS1.PARMLIB (for 15 min. in this example)
29. APAR OW19074 corrects three distinct RMF errors in which RMF itself
goes down with ABENDU1204 in ERBSMF1QB, ABENDC0D RC2A000201 in
LOGREC or ABEND0C4 RC10 in ERBFME0Q plus ABEND0FE in ERBMFEVT.
30. APAR OW22119 for DCOLLECT record type 'VL' corrects SMS system
status, MVS System status, and Confirmed SMS status.
IV. DB2 Technical Notes.
1. DB2 will now record its media manager VSAM I/O counts in the EXCP
buckets in the type 30 record. This enhancement was contained in
DB2 Version 4.1, but has now been retrofitted back into DB2 V3 by
APAR PN76648. With V3+APAR or V4, you must specify SMFIO=YES on
the MMSRV CONNECT request and be at DFSMS 1.1 or above. IBM notes
"do NOT be alarmed if you see the DBM1 ASID with EXCP counts that
range into the 'millions'."!
V. IMS Technical Notes.
1. One site using Candle's ITRF reports their experiences:
They were unable to process ITRF records written to SMF because the
Candle extractor program generated OTR037 INPUT LOG SEQUENCE ERROR
and after several iterations with Candle tech support (who thought
the data records needed to be sorted) the error remained until
further information from Candle indicated they strongly recommend
writing to the IMS log instead of SMF.
Switching to writing to the IMS log, they found they have to run
a separate Candle extractor job for each IMS region, but they can
then combine the three output files into a single input for the
MXG processing.
They have observed that for transactions running at the time of
OLDS switch, the response time is garbage (119,302+ hours!).
Candle is still working on that open problem.
2. IMS FASTPATH DEBD I/O counts via the VSAM Media Manager are now
recorded in the EXCP buckets in the type 30 for IMS with APAR
PN74925.
VI. SAS Technical Notes.
1. SAS USER ABEND 0315 (physical I/O error to a SAS data library)
occurs if your SMS Management Class specifies the RELEASE option.
SAS re-opens its data library for each new SAS dataset in that
library, and RELEASE does not work correctly with these multiple
opens. However, if you add the RELEASE parameter to the DD
statement for the SAS data library, the ABEND won't occur, because
SAS can recognize the explicit RELEASE request and is then smart
enough to suppress its multiple opens!
2. Further to my comments in Newsletter TWENTY-NINE about resources
used by PROC SQL, Paul Shipley, Telstra, AUSTRALIA, notes:
PROC SQL does seem to consistently use more CPU resources than
DATA steps and PROC SORT steps. While PROC SQL merges quicker and
can be easier to write (especially with multiple datasets and
keys), normally use PROC SQL only:
- where the computer resources are so small it doesn't matter
- for one-time jobs where the programmer time savings are worth
more than computer resources, or
- for many-to-many joins (such as Cartesian products) which can
be coded very easily in PROC SQL and can be very difficult in
DATA/SORT step logic.
3. USER ABEND 0318 will occur if your site upgrades STOPX37 to Boole &
Babbage's PROSMS replacement and your installer did not read their
instructions to disable PROSMS for PGM=SAS. Just like STOPX37,
PROSMS tries to protect for out of space conditions by allocating
additional space, but if the additional space is on a different
volume, SAS will fail when it tries to use that extra space, as it
does not meet the SAS multi-volume requirements. This can be an
erratic ABEND; the job will fail and them may the rerun may succeed
with no change, depending on where DASD space was found. The real
problem is that your primary space allocation is not large enough,
and the job is living on extents. Increasing the primary Space
allocation so that no secondaries are required will circumvent
the problem, but you must also disable PROSMS for PGM=SAS to
avoid the exposure.
Update Jan 22, 2001: Under SAS Versions 7 and 8 it is still true
that you must disable STOPX37/PROSMS etc. You can configure by
PROGRAM name (SAS, SASXA1, SASHOST, etc. etc), or from SAS Note
001876, "configure the third party product to exclude data sets
accessed via EXCP. SAS uses EXCP processing to access disk format
SAS libraries".
4. OUT OF MEMORY conditions were previously discussed in Newsletter
TWENTY-EIGHT (Section VI.5), but I was unaware of using REGION=0M
to bypass site restrictions in IEASYS. Specifying REGION=56M on
a stress test of an enhanced BUILDPDB failed and IEF374 indicated
I could only get 32M of Extended space. Changing to REGION=0M
allowed the job to successfully run (and it needed 53M). There
were site restrictions in effect when a non-zero region was
requested, but the REGION=0M bypassed the restriction and permitted
the job to successfully execute. Change 14.147 makes REGION=0M
the default in MXG JCL examples.
5. Under rare circumstances, SAS can enter either a CPU loop or wait
loop after trying to read an SMF file with an invalid VBS segment.
VBS errors were thought fixed in SAS 6.08 at TS415 by Usage Note
V6-SYS.FILE-9321, but this new instance spawned Usage Note fix
V6-SYS.FILE-C441, to be available in the first maintenance for
SAS 6.09E (tentatively TS455, but no date has been set). The new
problem occurred reading an SMF multi-volume BSAM disk file that
contained a broken segment. Only two sites are believed to have
experienced the error. Since one of the best features of SAS has
been its ability to detect and correct broken VBS segments, the
closure of this exposure is welcome.
6. Permanent ARRAYs with large numbers of elements (over 4096?) can be
unbelievably expensive in CPU time, both for the compile of the SAS
program and for execution, because permanent arrays create real SAS
variables. You must use a permanent array if you intend to OUTPUT
the variables in the array, but if you are only using the array to
hold counters or temporary strings, use _TEMPORARY_ in the ARRAY
statement and you can save lots of CPU time. In fairness, the
manual "SAS Language" does state (see ARRAY statement) "You can
improve the performance time by using temporary data elements",
but I never knew then what I know now! When Dan Squillace at SAS
observed a CPU time jump in BUILDPDB between MXG 13.13 and 14.05
(with his 137MB SMF file, CPU jumped from 293 seconds to 485!),
Freddie and Bruce ran tests which isolated the jump to MXG 14.04,
and I then isolated the CPU delta to Change 14.121, which had added
two 4096-element arrays to VMAC78 processing. Using my small (5MB)
SMF file with TYPE78 and iterated array sizes shows the impact:
Array elements: 0 512 1024 2048 4096 4096-Temp
TYPE78 CPU sec: 7 8 10 14 24 7
for small array sizes, permanent arrays are not expensive, but with
4096 elements, the CPU cost jumped from 7 to 24 seconds, so using
permanent arrays indeed can be expensive. Further tests using my
small 5MB SMF file to compare permanent with temporary showed:
4096-Perm 4096-Temp Increase
BUILDPDB Compile seconds 98 148 +50
my 5MB Read/Write secs 9 23 +14
Total CPU secs 107 171 +64
BUILDPDB, a 60,000+ line SAS program with 14,694 variables suffered
a +50 second increase in the CPU time in the Compile phase alone,
when the two ARRAY statements with 4096 permanent variables were
added, but it was not the ARRAY statement itself that caused the
increase; rather, it was the creation of 8,172 more SAS variables
that increased the CPU time, with or without an ARRAY statement!
And these new 8,172 variables have to be inserted into the middle
of the "Program Data Vector" which has one entry per variable,
causing it to expand, which increases CPU time because variables
that previously were adjacent now span addressable units (MVS
registers limit SAS to about 40K at a time) so more loads,
branches, etc., are necessary. That's why the compile time jumped!
But I observed also that the simple existence of these 8,172
variables also increased (from 9 to 23 seconds) the CPU time just
to read the 5MB SMF file and create //WORK datasets. Some of this
increase is also loading time to manage the larger "Program Data
Vector", but also, as SAS documents, the ARRAY statement is
culpable because all variables in permanent arrays are set to be
missing with each new record processed (whether or not the block of
code referencing the array was executed for this SMF record),
whereas temporary arrays are automatically retained.
So there is both a per-compile CPU cost and a per-record CPU cost
when 8,172 variables are added to BUILDPDB and permanent variables
are used in an ARRAY statement. Using _TEMPORARY_ instead, in the
ARRAY statement, where possible, can save a lot of CPU time!
Fortunately, MXG ARRAYs in data step code have only a few elements
(mostly 16 or 64). While a few ARRAYs in reporting programs have
999 elements, those programs have only hundreds of variables.
Addressability of the data vector only becomes a performance issue
for large programs with tens of thousands of variables. So I do
not expect any other big paybacks in using temporary rather than
permanent arrays, but over time I intend to replace all permanent
ARRAYs that I can with temporary ARRAYs, mostly because it is the
righteous thing to do. Only when variables in the array must be
kept in SAS datasets, will the array be permanent. Change 14.166
shows the syntax changes to make permanent temporary.
7. SAS 6.09E Maintenance TS450 has no reported problems with MXG, nor
should it, since SAS Institute now executes MXG Software as a part
of their Quality Assurance test stream! Only one impacting error
was not repaired in TS450; (usage note V6-SYS-FILE-C441, broken VBS
record, see above) that is to be fixed in TS455. Tests with
BUILDPDB under TS450 show no noticeable change in CPU resources
when compared with SAS 6.09 TS425.
VII. CICS Technical Notes.
1. CICS data volume reduction with EXCLUDE, CICS Blocksize, etc.
The physical amount of data created by CICS can be reduced by
telling CICS to "exclude fields" from the transaction segments in
DFHMCT. The default transaction segment in CICS 4.1.0 is 572 bytes
long and contains 107 fields, so a maximum of about 56 transactions
fit in one physical 32K byte SMF record.
Your CICS type 110 records will still be about the same length, but
there will be fewer total records. Reducing the segment size by
excluding fields does reduce the volume of data that is written to
SMF and that will definitely reduce the MXG run time (which is
mostly driven by data bytes, not record count).
A maximum type 110 record size of 6000 bytes is not good, because
CICS has to stop every 6000 bytes to hand over a record to the SMF
writer's address space, instead of sending 32K bytes per record.
It takes 5 SVCs at 6K to transfer the same data that takes only one
SVC at 32K, so CICS has to interrupt itself five times as often
with that smaller buffer size in CICS. A larger buffer size in
CICS will also reduce the number of I/Os necessary for the SMF
writer to write, the SMF dump program to read and write, and for
any SMF-processing program to read, and will reduce run time of
those programs as well! (For CICS 2.1 transactions, the default
segment was only 320 bytes and only 61 fields.)
To process "Excluded" SMF records in MXG, exits already exist for
your tailoring. In member IMACEXCL, you comment out the fields
that you have excluded in your DFHMCT and thus cause MXG to not
INPUT those excluded fields (instructions are in comments in
member IMACEXCL).
Then in member IMACCICS, in the _KCICTRN macro definition, you
use the DROP= A B C ... syntax to drop those variables from the
CICSTRAN dataset, thereby reducing the size of the CICSTRAN data
set (see text of Change 10.175 in member CHANGESS for "_K" macro
usage instructions).
In CICS chargeback, the CPU time recorded in the CICSTRAN
transaction observation is often less than half of the CPU time
actually used (which is captured in the region's type 30 SMF record
and in the type 72 for the CICS performance group (or service class
in WLM mode).
That uncaptured CPU time comes from two sources:
- CPU time spend in DB2 on behalf of transactions in this CICS
region, because that CPU time is not captured in the
transaction segment, but is included in the type 30 and the
type 72 data.
- A fixed CPU time per transaction to start and stop the
transaction, no matter what the transaction eventually does.
The DB2-caused CPU time is captured in the DB2 type 101 SMF
record which becomes the DB2ACCT dataset in MXG, so you could get
a good estimate of that fixed-per-transaction cost by calculating
fixed (CPU TIME from 30) - (CPU time from 101s)
cost
per = _____________________________________________
CICS
transaction (Number of Transactions from type 110)
You might want to schedule a startup and shutdown of CICS with no
work and look at its type 30 CPU time and subtract that cost from
the numerator if it is significant. My paper in the CMG 1993
Transactions "MVS/ESA Resource Accounting" which is also in member
NEWSLTRS of the MXG Source Library, has further discussions.
2. Support for Landmark's TMON for CICS/ESA 1.3.
Raw data still must be converted when Landmark's Dump program is
executed, as noted in comments in TYPETMON. Landmark has never
released the internal format of their records, and I have corrected
old comments that implied MXG could read their native records - it
can't.
MXG 13.06+ support for TMON 1.3 is in member TYPETMON, but that new
MXG support requires the MONICICS (input) file to have RECFM=VB in
the DCB. See text of Change 13.204.
MXG 12.12A and MXG 13.01 thru MXG 13.05 supported TMON 1.3 with
their member TYPETMON, but required RECFM=U for MONICICS.
See Change 12.320.
MXG 12.12 did not correctly support TMON 1.3, but the post-shipment
"Flash" letter sent in March, 1995, announced that MXG 12.12A did.
3. APAR PN79698 closes a problem with TASCPUTM (IBM's USRCPUT) field
being zero because an ISV DASD manager (unnamed) was issuing STIMER
REAL requests which were canceling the CICS STIMER (and the CICS
STIMER is needed to record the task CPU times!
4. Beginning with CICS/ESA 4.1, IBM creates a CICSTRAN observation when
an undefined transaction ID (TRANNAME) was typed in by the CICS
user. Variable TRANNAME contains whatever was typed in, and the
PROGRAM variable contains '########'. (With CICS/ESA 4.1 and DTR,
Dynamic Transaction Routing, you no longer have to define all of
your tran names in the TOR). IBM now acknowledges in APAR PN70976
"CICS issues an attach for the transaction, and then attempts to
route the transaction, passing the transid specified on the
DTRTRAN SIT parameter (or CRTX, the default). The monitoring
record will contain the name of the program associated with the
routing transaction in the field TMRPGMN. If DTRTRAN has not
been specified, TMRPGMN will contain "########", which is the
program name of the definition of CRXT, the default DTRTRAN
transaction."
PTF UN79168 creates a new CICS value 'NO' for the CICS SIT parameter
DTRTRAN which causes:
"Dynamic Transaction Routing to not be initiated if an undefined
transaction ID is entered; instead, the CSAC transaction will be
attached, and a monitoring record will be written for this
transid."
so the PTF does not eliminate the undefined transaction record, but
rather changes it to be a TRANNAME=CSAC transaction. Without the
PTF, it would be wise for you to either delete these transactions
with PROGRAM='########' from your CICS reports (your CICS users will
disbelieve reports with trashed TRANNAME values), or separately
count them as Undefined. With the PTF, you will only see more CSAC
transactions and will never know what four letter combinations were
typed in by sloppy (lots) or disgruntled (a few) CICS terminal
operators. About 350 undefined transactions were seen out of
several million CICS transactions in one day at one site.
VIII. Incompatibilities and Installation of MXG 14.07.
1. Incompatibilities introduced in MXG 14.07 (since MXG 13.13):
a- IMACs that were changed (if they exist in your USERID.SOURCLIB, you
must refit your tailoring, starting with the new IMAC member):
NONE
b- Other incompatibility changes:
Dataset TYPE116 (MQM) variable QWHCATYP replaced by QWHCXTYP.
Dataset CACHE90 from VMACACHE now TYPE74CA from VMAC74. CH14.152.
c- These products were incompatibly changed by their vendor, and they
require MXG 14.xx as indicated:
See products listed as INCOMPATIBLE in Section I, above.
2. Installation and re-installation procedures are described in detail;
in member INSTALL (which also lists common Error/Warning messages a
new user might encounter), and sample JCL is in member JCLINSTL:
Summary:
a. Install member MXGSAS as JCL Procedure MXGSAS in your PROCLIB.
b. Allocate a 105-cyl PDS: MXG.V1407.MXG.SOURCLIB, and use IEBUPDTE
to read the MXG tape to create the 2955+ member Source Library.
c. Allocate a 1-cyl PDS: MXG.V1407.USERID.SOURCLIB for your site
"Installation Tailoring" Source Library. Installation specific
tailoring (like telling MXG your shift hours, which performance
groups are TSO, CICS, etc.) is done by copying and modifying MXG
source members into V1407.USERID.SOURCLIB.
d. Allocate a 1-cyl SAS Data Library: MXG.V1407.MXG.FORMATS and
execute SAS to create the library of Formats required by MXG.
e. If this is the initial install of MXG, tailor these members into
your MXG.V1407.USERID.SOURCLIB tailoring library:
IMACACCT (Account Length),
IMACSHFT (Shift Definitions),
IMACWORK (Performance Group to Workload mapping), and
IMACSPIN (for BUILDPDB).
Each IMAC member is self-documenting, and IMACAAAA is the index
of all of the IMACs. You should at least scan IMACAAAA to see
the acronyms MXG uses for the many products MXG supports.
e. If re-installing MXG, copy your existing USERID.SOURCLIB library
members into the MXG.V1407.USERID.SOURCLIB. Then, compare the
members in your USERID.SOURCLIB with the list of members that
were incompatibly changed (above, in this section) in this MXG.
If any of the incompatibly changed members exist in your dataset
MXG.V1407.USERID.SOURCLIB, then you must reinstall your site's
tailoring for that IMAC, starting with the IMAC member from the
MXG 14.07 Source Library.
f. EDIT and submit member JCLTEST6 to verify that your tailoring
did not create any errors.
g. EDIT and submit JCLPDB6 to create a Daily PDB for testing. Or
use the TYPE.... members to process specific data sources, use
the ANAL.... members for report examples, the GRAF.... members
for SAS/GRAPH reports.
You have now installed MXG 14.07 in its own set of libraries. When
parallel testing is complete and are ready to implement MXG 14.06
in production, rename your three current MXG Production Libraries
(MXG.MXG.SOURCLIB, MXG.USERID.SOURCLIB, and MXG.MXG.FORMATS) to
(MXG.BACK.MXG.SOURCLIB, MXG.BACK.USERID.SOURCLIB, MXG.BACK.MXG....)
and rename the MXG.V1407.x.y libraries to their Production names!
Again, detailed installation instructions are in member INSTALL
Always read comments in the CHANGES member for compatibility issues, as
well as for any last minute changes.
Whenever you install changes or test a new version of MXG (or even your
own reports), be extra careful to look on the SAS log for any real error
conditions. Search for all occurrences of "ERROR:", "ERROR :", " NOT "
"UNINITIALIZED", "TRUNCATED", "NEVER BEEN", "NOT FOUND", "CONVERT",
"APPARENT", and "NOT CATLGD", as they usually indicate a serious error.
A PROC PRINT and a PROC MEANS of each new MXG-built SAS dataset can help
you to understand their contents, and should be used to examine any
unusually large, negative, or suspicious values. Print all variables in
the dataset, and read the variable's descriptions in its ADOC member.
IX. Online Documentation of MXG Software.
Since 1994, the contents of the two MXG Books, (the 1984 MXG Guide, and
the 1987 MXG Supplement) are contained in the MXG Source Library, as are
all MXG Technical Newsletters and all MXG Changes, so all MXG
documentation is actually online in the software itself; even the
Installation Instructions are online, in members INSTALL/JCLINSTL!
ACHAPxxx members are the text of the 42 chapters from the two MXG books,
to which the text from newsletters and changes has been added. Some of
these chapters are still rough; while some of the chapters have actually
been completely revised, many of these ACHAPxxx are little more than a
concatenation of the two original chapters, often without the figures
or tables. The revision is work still in progress!
Members ADOCxxxx are what were in Chapter FORTY, and should be the first
place you look for information about MXG variables and/or datasets. The
ADOCxxxx members alphabetically describe each dataset and all variables
that are created by product xxxx, the instructions on how to enable that
product, bibliography of the vendor documentation, sample PROC PRINT and
PROC MEANS of real data, references to MXG reports that use these data,
and the MXG member names that you use to process that product. While
this too is work in progress, the most heavily used data sources,
especially the common SMF records, have been revised and are up to date.
There is an IMACxxxx member for every product supported by MXG. Once
you know the xxxx suffix for a product, you then know the names of all
of the MXG members for that product, because of MXG naming conventions:
IMACxxxx - Defines record IDs, and the _Lyyyzzz and _Kyyyzzz macros
that name the dataset(s) created from product xxxx.
ADOCxxxx - "Chapter FORTY" style dataset and variable documentation of
all datasets created from product xxxx, with sample output.
VMACxxxx - The "real" source code member, often extensively commented.
TYPExxxx - Standalone member to test or process product xxxx records.
ASUMxxxx - Summarization example (only for some products)
TRNDxxxx - Trending example (only for some products)
ANALxxxx - Reporting/analysis example (only for some products)
GRAFxxxx - SAS/GRAPH report example (only for some products)
EXyyyzzz - OUTPUT exit for tailoring of each MXG dataset, not used by
most MXG sites, but powerful if needed. There can be more
than one dataset created from one product. The yyyzzz
suffix of the EXyyyzzz member name is the same as the
suffix of "_L" and "_K" macros defined in the IMACxxxx for
its product. See Using the MXG Exit Facilities in ACHAP33.
Member IMACAAAA is an index of all IMACs, and is the best place to begin
to find what xxxx suffix Merrill chose for which product! You can often
find additional documentation by searching members NEWSLTRS or CHANGESS
for the xxxx suffix.
Member CHANGES identifies this Version and Release of MXG Software, and
describes all changes made in this Release, plus new technical notes.
Member CHANGESS contains each of the CHANGES members from each version
of MXG, so this member contains ALL changes ever made to MXG Software.
Since each MXG change lists the names of the members that were added or
altered, names the new product/version supported by a change, or lists
error messages corrected by a change, this member is designed to be read
online (with SPF BROWSE); you can search for specific product acronyms
(CICS, MVS/ESA, etc.), or the MXG member name or anything else. Many of
the changes are actually mini-tutorials, especially for new products.
Member NEWSLTRS contains the text of all newsletters. You can search
NEWSLTRS for product name or acronym to find all of Dr. Merrill's
published and unpublished technical papers, technical notes announcing
enhancements in new operating systems or subsystems, new datasets and
products, important APARs and PTFs, and other technical information of
importance to MXG users. (Since the Change Log that is printed in each
newsletter is in member CHANGESS, it is not repeated in NEWSLTRS.) MXG
Technical Newsletters are typically published twice a year, with one
printed copy sent to each licensed site's technical addressee.
Member DOCVER lists alphabetically ALL datasets and variables that are
built by this MXG Software Version, abbreviated to a line per variable.
Members DOCVERnn are the "delta-documentation" between MXG versions, and
list only those datasets and variables that were added/deleted/changed
by version "nn", so you can identify when a variable/dataset was added.
Finally, remember that MXG is source code, and you can often find your
answer by BROWSING the source members, especially the VMACxxxx members.
The MXG Variable name is frequently the vendor's field name, or the
vendor's field name is often in a comment adjacent to the variable's
INPUT, so you can cross reference MXG to the vendor's documentation.
The migration from print to online is clearly work in progress, but at
least the two books are now machine-readable! When all 42 chapters
are completely revised and updated in the source library, I will decide
which, if any, will also be made available in printed form, but the
primary media for all future MXG documentation will be these members of
the MXG source library, which can be immediately updated in each new
version of MXG as changes occur.
X. Changes Log
--------------------------Changes Log---------------------------------
You MUST read each Change description to determine if a Change will
impact your site. All changes have been made in this MXG Library.
Member CHANGES of the MXG SOURCLIB will always be more accurate than
the printed changes in a Newsletter, because the software tapes are
created after the newsletter is sent to the printer!
Member CHANGES always identifies the actual version and release of
MXG Software that is contained in that library.
The actual code implementation of some changes in MXG SOURCLIB may be
different than described in the change text (which might have printed
only the critical part of the correction that can be made by paper).
Scan each source member named in any impacting change for any comments
at the beginning of the member for additional documentation, since the
documentation of new datasets, variables, validation status, and notes,
are often found in comments in the source members.
Alphabetical list of important changes after MXG 13.13:
Dataset/
Member Change Description
all 14.019 Support for OS/390 Release 1.0 already in MXG 13.13!
many 14.158 Support for OS/390 Release 2.0 tolerate by MXG 13.13!
ANALCNCR 14.162 FILE WORK.SPLIT DOES NOT EXIST corrected.
ANALCNCR 14.175 Specifying both output dataset and reports failed.
ANALDB2R 14.022 DB2 report PMAUD03, if PDB is on tape, will fail.
ANALDB2R 14.073 VARIABLE QWHSIID NOT FOUND corrected in DB2 reports.
ANALDSET 14.064 Using Tape instead of DASD for ANALDSET fails.
ANALSMF 14.178 SMF Simulator now tests a CISIZE of 18432 for 3390s.
ASMIMSL5 14.129 Support for IMS 5.1 APAR PN76410 (INCOMPATIBLE)
ASMIMSLG 14.148 SLOTS table moved above the 16MB line.
ASMTAPES 14.037 MAINTLEV 8 of MXG Tape Mount and Allocation Monitor.
ASMTAPES 14.086 MAINTLEV 9, monitor does not stop, new TMNT013I.
ASMVTOC 14.003 Archaic assembly member was wrong on MXG 13.13
ASUMAPAF 14.062 SORT ORDER error if you increase number of domains.
BUILDPD3 14.169 JES3 type 25 MDS Tape Mounts/Fetches in BUILDPD3.
BUILDPDB 14.185 VM Print sent to JES2 is now merged in PDB.JOBS.
CICINTRV 14.188 Old CICINTRV replaced CICINTRZ, fixed for CICS 4.1.
DIFFDB2 14.167 DB2 4.1 DB2STATS interval missing due to QWHSISEQ.
DIFFDB2 14.194 Extra obs in DB2STATB/DB2STATR, negative SEQCHECK.
IHDRDCOL 14.027 First of new "IHDRyyyy" - "INFILE header" exits.
IMAC6ESS 14.036 Decoding of SMF type 6 ESS segment is added.
IMACEXCL 14.024 CICS Excluded Field support enhanced for multiples.
IMACICOC 14.123 Omegamon for CICS OMSUPRTM/OMDCOMTM incorrect.
JCLADHOC 14.140 New example for ad hoc job to select specific data.
JCLTMON 14.012 Example JCL for Landmark's The Monitor for CICS.
JCLall 14.147 All MXG JCL examples now specify REGION=0M.
MONTHBLD 14.010 NOTSORTED error with ASUMCICS in monthly logic.
MXGSAS 14.140 Revised MXGSAS JCL procedure adds TAILORNG= parm.
TRNDTALO 14.130 INVALID DO LOOP error if ALOCSTRT=. or ALOCEND=.;
TRNDTALO 14.176 Redesign of TRNDTALO to "SPIN" active allocations.
TYPE102 14.047 DB2 Trace T102S096 vars QW0096SN,SC,SK corrected.
TYPE102 14.138 New datasets with all SQL text added for DB2 trace.
TYPE102 14.206 Dataset T102S231 corrected.
TYPE110 14.089 Support for PN69653 (YYYY digit year in COLLTIME).
TYPE110 14.106 Variables MCTMNTAD/SMFPSRVR added to CICSEXCE.
TYPE110 14.184 CICSTRAN variable TRANTYPE increased to two bytes.
TYPE110 14.209 Support for CICS/ESA 5.1.0 (INCOMPATIBLE).
TYPE116 14.087 Variable QWHCATYP was INCOMPATIBLY renamed QWHCXTYP.
TYPE16 14.150 Support for DFSORT Release 13 APAR PN71337.
TYPE28 14.023 Some NPM VVR (VTAM Virtual Route) variables trashed.
TYPE28 14.065 NPM APARs OW08565 and OW10584 for 3746/900 supported.
TYPE30 14.099 Auto Restart section INPUTs were incorrect.
TYPE30 14.172 Variable EXECTM in TYPE30_V wrong if only subtype 3.
TYPE42 14.063 DASDMPL 1000 times too large in TYPE42DS.
TYPE42 14.131 Support for APAR PN78083 required no change to MXG.
TYPE6 14.009 Truncated PSF type 6 record INPUT STATEMENT EXCEEDED
TYPE64 14.004 INPUT STATEMENT EXCEEDED, CF Cache Structure segment.
TYPE7072 14.051 ELAPSTM added to TYPE72GO, and RMFINTRV for WLM.
TYPE7072 14.059 TYPE72GO variable VALDSAMP and delay PCTs wrong.
TYPE7072 14.180 Variable PERFINDX now created in TYPE72GO.
TYPE71 14.058 Support for Shared Page Groups added.
TYPE72 14.085 MVS/ESA 5.2.2 variables overlooked in TYPE72GO.
TYPE73 14.164 APAR OW15406 for RMF adds support for Year 2000.
TYPE74 14.085 MVS/ESA 5.2.2 variables overlooked in TYPE74OM.
TYPE74 14.152 Support for type 74 subtype 5 Cache RMF Reporter.
TYPE78 14.121 Variable PCTALLBY, LCUIORT added to TYPE78CU dataset.
TYPE78 14.166 ARRAY statement changed to _TEMPORARY_ to save CPU.
TYPE80 14.070 Support for IBM APAR OW19251 (RACF year 2000).
TYPE80 14.114 Support for RACF 1.10 (toleration).
TYPE80A 14.170 More RACF Reports for Command Events decoded.
TYPE88 14.066 INPUT STATEMENT EXCEEDED corrected.
TYPE89 14.158 Support for Subtype 2 (Measured Usage Product Sumry).
TYPE99 14.069 TYPE99_2 now has obs for each period vice just first.
TYPEBETA 14.050 INVALID DATA FOR BETASTRT and BETAEND with 1.6.5.
TYPEBETA 14.084 INPUT STATEMENT EXCEEDED for SUBTYPE=41.
TYPECACH 14.093 Support for IBM's Cache RMF Reporter CRR 1.7
TYPECMF 14.033 MXG now recognizes 3990 model 6 in CMF user SMF.
TYPEDB2 14.011 DB2 4.1 type 101 subtype 2 INPUT STATEMENT EXCEEDED.
TYPEDB2 14.044 Protection for truncated DB2 record.
TYPEDB2 14.071 Dataset DB2STATB now always has observations.
TYPEDB2 14.105 QWSDLR length 8, QWSCIID corruption corrected.
TYPEDB2 14.174 VMACDB2 ERROR ... QWHSIID=230 UNEXPECTED fixed.
TYPEDB2 14.195 DB2STATR, DB2 remote counts, corrected.
TYPEDB2 14.208 Datasets DB2GBPST and DB2GBPAT all BP now output.
TYPEDMON 14.125 Support for ASTEX 2.1 (INCOMPATIBLE).
TYPEEDGS 14.029 DFSMS/rmm type "O" INPUT STATEMENT EXCEEDED RECORD.
TYPEEREP 14.021 INPUT STATEMENT EXCEEDED with EREP CLASRC='40'X.
TYPEF127 14.032 Support for FACOM MSPE/EX PTF 93061 for ID=127 SMF.
TYPEFTP 14.054 Support for FTP subtype 51x SMF record.
TYPEHIPR 14.015 Hipercache VSAM buffer field wrong in MXG 13.13.
TYPEHSM 14.052 Short HSM ABARS FSRTYPE=15 INPUT STATEMENT EXCEEDED.
TYPEIMS 14.030 Early testing IMS log records for IMS 5.1
TYPEIMSA 14.017 VARIABLE SYSTEM IS UNINITIALIZED with ASMIMSLG.
TYPEM204 14.171 Support for MODEL204 Release 3.2.1 (INCOMPATIBLE).
TYPEMOVT 14.168 Support for Omegmaon/VTAM V200 (INCOMPATIBLE).
TYPEMWUX 14.134 Support for HP MeasureWare for HP-UX platform.
TYPENDM 14.034 NDM or Connect Direct timestamps missing, data wrong.
TYPENDM 14.116 Support for NDM 1.4 (compatible) adds variables.
TYPENOAM 14.057 Support for STK's NearOAM user SMF record.
TYPENSPY 14.005 INPUT STATEMENT EXCEEDED, NSPY 4.6, type A, invalid.
TYPENSPY 14.053 LUNETID PCSESSID VILUNAME in dataset NSPYLU trashed.
TYPENSPY 14.111 Support for NETSPY Release 4.7 (compatible).
TYPEOPC 14.077 INVALID MTD SUBTYPE, observations not output.
TYPEORAC 14.103 Accounting data input incorrectly for ORACLE.
TYPEQAPM 14.098 Support for AS/400,OS/400 Release 3.6 (INCOMPATIBLE)
TYPERMDS 14.092 Support for IBM's RMDS Version 2.2 (no change)
TYPESAMS 14.013 INVALID DATA FOR HH,MM,SS with SAMS SMF record.
TYPESFTA 14.179 Support for SoftAudit Version 5.1 (INCOMPATIBLE).
TYPESNIF 14.186 Network General's Sniffer Network Monitor data.
TYPESTC 14.001 INPUT STATEMENT EXCEED for HSC subtype 8 record.
TYPESTC 14.055 STK's HSC Subtype 08 now in two lengths, 38 and 40!
TYPESYNC 14.115 Syncsort variables SORTBEGN/END midnight spanning.
TYPETLMS 14.014 TLMS year 2000 dates were not decoded correctly.
TYPETMDB 14.197 Support for TMON/DB2 Version 3 (INCOMPATIBLE).
TYPETMON 14.042 INVALID DATA for TIGETMCT or TIFREMCT corrected.
TYPETMS5 14.018 TMS datasets TMSRECS,DSNBRECS now deleted from WORK.
TYPETMVS 14.135 Support for Landmark TMON/MVS spanned records.
TYPETPM 14.113 Support for Thruput Manager #V041238 (INCOMPATIBLE).
TYPEVM 14.008 INVALID DATA FOR PWCOUNT in VMID=06 VM Accounting.
TYPEWSF 14.143 Support for RDS's EOS Enterprise Output Solution
TYPEX37 14.091 STOPX37 SMF records changed by Boole, useless now.
TYPEXSTR 14.144 Support for Anacomp, Inc's XSTAR product SMF record.
TYPPROS 14.207 Support for Boole & Babbage's PRO/SMS.
UCICSCNT 14.060 Enhanced CICS diagnostic tool for EXCLUDE/INCLUDE.
UDB2GTF 14.154 Support for DB2 records written to GTF.
UDEBLOCK 14.155 Utility to create valid RECFM=U on MVS from PC data.
UTILGETM 14.018 Type 110 Subtype 2818 recognized and counted.
VMXGSUM 14.177 If DESCENDING was used with KEEPALL=NO, it was lost.
VMXGTAPE 14.153 Utility macro to determine if lib/dsn is on tape.
WEEKBLDT 14.010 NOTSORTED error with ASUMCICS in weekly logic.
YEAR2000 14.100 Use of Date literal '01JAN00' changed to '01JAN1900'
Inverse chronological list of all Changes:
Changes 14.209 thru 14.001 were printed in MXG Newsletter THIRTY,
and are contained in member CHANGESS.