There! I didn’t even have to alter this movie title.

Introduction

Anyway, there’s a reason to return to these pages and write another installment. And the reason is in the title. After some work that we’re going to get to in a minute, I’ve managed to get to the point where I can submit and successfully execute jobs on the COS operating system. As things stand today, this is most likely also the last piece to this series, but more on that later.

The filesystem

Before we get into how batch processing works, lets talk a little bit about how files and the file-system works on COS!

Modern filesystems go to great lengths to organize your files into logical groups. They provide directories, symbolic and hard links, and long file names. At the same time, files themselves are just binary blobs and the interpretation of them is completely delegated to the applications using them.

COS calls files datasets, and (as far as I can tell) gives you very little in terms of organizing them. Permanent dataset names can be up to 15 character long and there are no directories. You could add a rather long note to describe what a dataset was about though. At the same time COS had a rich understanding of the internals of datasets. They are broken into files, and further into records. Records can have variable lengths so can have files.

Datasets are maintained in ‘Catalogs’, which are somewhere in between directories and file-systems in modern terms. The Dataset Catalog (DSC in short) contains an entry for every permanent dataset that is stored on the hard drive(s). There are two optional but highly recommended catalogs, the Master and Backup catalog.

The backup catalog contains entries for datasets stored on backup tapes. In addition to the usual data, like the name of the dataset it also contains an edition number, essentially implementing versioning of backups.

The master catalog serves as a set of links to gather together all editions of a dataset into a single location. It might point to the DSC entry for permanent datasets or to the backup catalog for archived datasets. This feature deeply integrates backup handling into the OS. For example if a job references a dataset that’s not available on permanent storage, an operator is notified to load the right backup tape into one of the tape-drives to make it available for the job to access.

There’s a third interesting feature of the way COS manages datasets. When a job wants to access a dataset, first it must be made ‘local’. When the job is done with the dataset, it has to explicitly release it. This – if I read things correctly – essentially implements atomic file operations, that is either all writes succeed to a permanent dataset or none do.

Batch processing

As you probably already know, the COS was a batch operating system. It doesn’t mean that it couldn’t do multi-processing, but it does mean that – at least in the way you would normally interact with it – you didn’t get a command line of any sorts. You couldn’t work on the machine interactively. What you did instead was to sit in front of a workstation hooked up to another mainframe, like a CDC Cyber or maybe an early VAX/VMS system. You run something, called the Station software. This was your interactive working environment, where you assembled your jobs into a batch, and submitted them for processing to the Cray mainframe. The mainframe then scheduled and eventually executed your job, produced the results and sent them back to your station for you to look at.

These batches of operations were described in a special language, called the Job Control Language, or JCL. To give you an idea how such a description might have looked like, here’s a job file:

Now, this might look foreign and strange, but in essence it’s not that much different from a .BAT file or a shell script. The syntax is funny, but the idea is the same: you invoke commands, some built-in, some external and pass parameters to them. There are a few specialties worth mentioning:

Since we’re operating in batch mode, what you would call today STDIN and STDOUT don’t directly exist. $IN, the equivalent of STDIN gets mapped to the job file itself and $OUT (STDOUT) gets redirected to a temporary file. When the job terminates, $OUT (along with $LOG where the system collects all sorts of statistics about what your job did) gets shipped over to your station, where it most likely gets printed.

Job files identified themselves and the user account they run under. These are the first two lines of the job file above.

Since you’re operating in an environment where your station software is running on a different machine than your actual code, you have a bit of a schizophrenia to deal with: datasets can live in two places and depending on what piece of code we’re talking about they see one set or the other. Special operations are available to transfer files between the two systems (one of these is the DISPOSE command above).

Now, I’ve mentioned earlier that datasets can contain multiple files. This feature becomes handy with job processing: you can concatenate the input to the various commands into a single dataset – one that now contains multiple files. When a new command is executed, COS advances $IN to the next file in the job dataset. Here is one example:

This job invokes the GENCAT utility, which in turn reads its commands from $IN. So the dataset contains a second file with the following content:

These four lines inform GENCAT to generate a backup and master dataset catalogs discussed above.

Jobs can be put into job classes so they can be summarily controlled, and the number of concurrent jobs could be set as well.

The processing of the JCL file that you submit is the job of the ‘Control Statement Processor’ or CSP module. Whenever a new job is created a section of memory is carved out and set up for it by the OS (the STP component to be more specific). Since the Crays had only very rudimentary memory protection features (a per process base address offset and address space length could be set), each process needs to operate in a completely separate address space. Shared libraries are not supported. Because of this, the command processor, needs to be copied into this memory space that is reserved for the job. When additional programs get invoked (like the Fortran compiler) they also get loaded into the jobs address space.

A sample session

We start with a fully booted system (either through install or deadstart). Once, you have the system fully booted, you should see something like this on the console:

you type in the following commands:

CLASS,ALL,ON

LIMIT,5

These enable all job classes and sets the number of parallel jobs to 5.

Now, to see some status, we can enter the command ‘STATUS’:

At this point you can submit a job. There are a few on the expander disk, most of them are rather useless at least at this stage. One of those is JTEST30. Useless, maybe, but demonstrates how things work and doesn’t cause any harm. Se let’s submit it:

SUBMIT,JTEST30

At this point you’ll see the job queue – which is displayed on the screen – list your just submitted job as it paces through the various stages of execution.

Once it’s done, it will disappear from the list. So where are the results? I’ve told you that normally $OUT and $LOG goes to the printer on the station you’re using. As it turns out there was a local printer attached to the IOPs expander on this configuration. This is the location where the information was sent. The simulator dumps the output intended to the printer into PR0.TXT. So, let’s take a look at what it contains!

You can see no output from the job and the log is rather concise as well. It informs you about an error, which is that TEST30 is missing (PD000 and PD009 messages). Why is that? Let’s take a look at the job source:

Line 5, the submit statement says: load dataset TEST30 from mass storage and execute it. In our system, mass storage is empty, there’s no TEST30 to launch. Most of the jobs on the disk would behave similarly. There’s one that’s useful: JGENCAT. It generates the afore-mentioned master and backup catalogs. You can launch that too and see it eventually succeed. I’ve created a short video about job submission and execution as well, in which I do show how JGENCAT works:

Interactive mode

Did I say COS was a batch operating system and you didn’t get any sort of command prompt? Well, I’ve lied. In fact there was a way to work with the machine interactively. This mode is very similar to what you would see on a modern Unix-like machine. CSP, the command processor runs on the mainframe, but $IN, the input dataset is redirected to your console. You enter your job line-by-line, which gets immediately interpreted and executed by CSP. $OUT is also redirected to your console, so you immediately see any output. Most programs were not designed to work in this interactive environment, though there are notable exceptions: TEDI, the interactive editor, and SID, the symbolic debugger. One of these, TEDI happens to be on the expander disk, so we can see how that worked.

Getting into interactive mode is a bit more involved. First, you have to start something called the ‘interactive concentrator’ on the MIOP console. You do that, by typing in IAIOP LOG into the kernel prompt:

The concentrator – or as sometimes called the front-end interface – was a piece of hardware and software that allowed for remote stations to connect to the Crays. A non-interactive version of this was automatically started, that’s why we have been able to use the STATION command to start a station. Now, we have the interactive version running as well.

You will have type in all the usual incantations to get the OS booted and functional, most importantly, enable classes and set the job limit:

CLASS,ALL,ON

LIMIT,5

At this point we’re ready to switch over the station to interactive mode. We do that, by typing IAC into the station prompt:

Once initialization done, you’re screen goes black and all you’re left with is a prompt at the bottom of the screen. Next, you’ll have to logon – again – into the mainframe. You do that by typing /LOGON into the prompt (‘local’ commands are escaped with a leading / character):

Now that you’ve established a connection to the mainframe, you can start issuing commands. These commands are the same as they appear in JCL datasets. The first thing you have to do is to identify yourself (you’ve seen logon containing no account information). You do that with an ‘ACCOUNT’ statement. Something like this (all commands must be terminated by a ‘.’):

ACCOUNT,AC=CRAY,APW=XYZZY,UPW=QUASAR.

After this, we’re finally ready to transfer the image of the editor from the station (expander disk) to the mainframe:

FETCH,DN=TEDI,MF=AP,AC=ST,TEXT=BIN/TEDI.

What we’ve said here is to grab the dataset named as ‘BIN/TEDI’ (yes the IO subsystem did support directories) from the expander disk and make it available for the job (our interactive prompt) as TEDI.

At this point we can start the editor:

TEDI.

Eventually you’ll be greeted by this:

Here you have to enter a dataset name. It doesn’t matter too much what the name is (just don’t start it with a ‘$’). If it doesn’t exist, the editor will create a new one:

I’m going to stop hand-holding at this point, partly because I don’t know how to use TEDI either, and partly because it seems to have extensive help built in.

Before we move on, here’s a short video on interactive jobs:

Under the hood

If you’re still with me, you might be interested in what it took to get to this point. Maybe not. But that’s what I’m going to write about anyway, so you have no choice.

There are three important changes that made all the above possible

Bugfixes

Yes, there were bugs. Not too many, but enough to cause issues. They congregated (surprise surprise) in the so far not well-exercised vector instructions.

Watchpoints

I’ve resisted this idea for the longest time, but to track down the above mentioned bugs I needed a way to surgically enable or disable instruction tracing. Watchpoints trigger when a certain memory address is accessed and to implement them, I need to monitor every memory access the CPU makes. This I thought would slow the simulation considerably down but that was not the case – most likely because my simulation is already dog slow. Now that they are implemented, you can enable them in the configuration file, using the following syntax (on the global level):

The first entry is the (physical) address, the second is a string. When the specified address is accessed (read or written) an event gets fired with the following text:

By setting an event that matches this message you can turn on logging, or do other interesting things:

Since the <hit count> section gets incremented for every hit (starting at 0), you can also implement counting, so for example you can wait until the 3rd access to turn on tracing.

Now that I think about it, this is probably how I should have implemented breakpoints as well. Oh, well, that will go on the todo pile…

Missing sectors

Remember back in the early days, when I’ve worked on recovering the disk image, I couldn’t successfully extract all the sectors? I’ve left that work off with 672 missing sectors. As it turns out, one of them – at least – was essential for job execution: the CPU calls into code that should be there. So I needed to re-visit the recovery effort and try to extract more sectors.

My original plan was to build on the voting idea that worked well in the previous effort and expand it: instead of using just one scan and ‘fold’ it over several times to line up the various reads of the same physical location, I could use several adjacent scans as well. That would greatly increase the number of samples I have for any particular part of the disk, increasing my chances for a good result. The trouble of course lies in finding the corresponding pieces in the different scans. My approach was that if a scan had at least one recovered sector, that creates an ‘anchor’. I could use these anchors to link the scans together and form a single unified scan. I can then fold them together, and easily get 10+ readings for every original bit.

Cool idea, but did it work? Not a bit. In fact, when I’ve printed the bit-length histograms, they looked horrible. Instead of the peaks getting better, they got almost completely washed out. So what went wrong? Maybe I didn’t do the alignment correctly? After all one bit offset in the wrong direction can affect the histograms a great deal. So, how to find the perfect alignment? I’ve implemented a (binary) correlation function – essentially calculated the average number of differing bits. The smallest this number is, the better the match. So, all I have to do is to find the minimum on this function by varying the offset and voila, I have the best alignment possible. So did that work? Not at all. But now I at least had something to look at, namely the value of this correlation function at the minimum. Turns out it was around 0.1 in most cases. That means that on average one out of every ten bits was different even in the best match. Not a big surprise then that these efforts didn’t pan out: the different readings of the same parts of the disk were quite different.

Back to the drawing board, I started thinking about other ways to recover the sectors. It was clear to me that using multiple samples and statistics will not lead to anything decent, so how can I improve recovery of a single reading? One thing I’ve tried was to make sure recovered sector numbers are sequential. If I’ve read sector 3, the next one should be sector 4. Unless sector 4 is unreadable of course. I honestly don’t fully understand why, but this caused a bunch of sectors to read properly, ones that didn’t before. My working hypothesis is that now the assumed sector starts – points in the bit-stream where I try to recover a sector – changed. Previously I couldn’t recover these sectors because I haven’t even tried. I would have to go back and validate this idea, and if turns out to be correct, see if I can get even better by altering the start point search even further. For now however it’s good enough: I have the missing sector for the boot image which allows for job execution. In fact, at this point I have no missing sectors in the boot images!

In closing

This is already a long article, so time to wrap it up. You can download the newest release from the download page. From here however the future is very uncertain. To gain the full functionality of the system at minimum I would need tools: COS, the assembler, CFT, the Fortran compiler, SEGLDR, the linker and the libraries: $CLIB, $FTLIB, $IOLIB, $PSCLIB, $RATLIB, $SCILIB, $SYSLIB, $UTLIB, maybe others too. I have some leads to some of them, but if you have anything, anything, please let me know. Without at least a subset of these tools and libraries, further work on the project is rather pointless. It would not lead to a more functional system, only a more polished version of the uselessness that it is today.

As an interesting side-note, the disk contains one dataset (JINSTALL) that would restore the full system from backup, trouble is, the tape that it would use to restore the system from is missing. If you look at the 3rd file in that dataset, you get a list of files that comprised the base system:

1. JOBCLASS
2. JGENCAT
3. GENCAT
4. JDIAGGO
5. $DIAGLB
6. LOCALS
7. STARTUP
8. STARTARCHIVE
9. CRAY1SYSTEMDUMP
10. JSYSDIR
11. $ADMLIB
12. $APTEXT
13. $ARLIB
14. $CLIB
15. $DBCTL
16. $DBHELP
17. $DBTXT
18. $DIAGLB
19. $FTLIB
20. $IOLIB
21. $LEXLB
22. $PDDERR
23. PROCLIB
24. $PSCLIB
25. $RATDEF
26. $RATLIB
27. $RATXVS
28. $SCILIB
29. $SID
30. $SYSDEF
31. $SYSLIB
32. $SYSTXT
33. $TDGTOC
34. $UTLDEF
35. $UTLIB
36. $UTLTXT
37. $WTOOLS
38. $YYPLB
39. ACCOUNT
40. ACCTDEF
41. ADSTAPE
42. APML
43. AUDIT
44. AUDPL
45. BACKUP
46. BIND
47. BLOCK
48. BUILD
49. BUPIO
50. BVCEDIT
51. CAL
52. CC
53. CFT
54. CFT77
55. CHARGES
56. CLEANUP
57. CLUPIO
58. COMPARE
59. CONNECT
60. COPYD
61. COPYF
62. COPYR
63. COPYU
64. COSDEF
65. COSTXT
66. CPP
67. CSIM
68. DDA
69. DEBUG
70. DRD
71. DSDUMP
72. DUMP
73. DUMPGOS
74. EXTRACT
75. FDUMP
76. FLODUMP
77. FTREF
78. ISP
79. ITEMIZE
80. JCSDEF
81. LDR
82. LD2
83. LOADGOS
84. MANAGE
85. MODSEQ
86. MODSET
87. MTDUMP
88. NEWCAL
89. NOTE
90. PASCAL
91. PDSDUMP
92. PDSLOAD
93. PERFMON
94. PREMULT
95. PRVDEF
96. QSTSUI
97. QUERY
98. RECALL
99. RECIO
100. SLSTACK.RELEASE
101. RELOAD
102. RESTORE
103. RETIRE
104. SLSTACK.REWIND
105. SEGLDR
106. SEGRLS
107. SETOWN
108. SKIPD
109. SKIPF
110. SKIPR
111. SKIPU
112. SKOL
113. SKOLREF
114. SKOLTXT
115. SORT
116. SPAWN
117. SPY
118. STATGOS
119. STATS
120. STOPGOS
121. STEP
122. SYSREF
123. TARGET
124. TDI
125. TDUMP
126. TEDI
127. TG
128. UNB
129. UNBLOCK
130. UPDATE
131. WRITEDS
132. EDITOR

This is going to be my wish-list for Santa this year.

To leave you on a higher note, there was one more thing I’ve uncovered: the line-printer attached to the expander supported a graphical mode. The banner, printed on every page used this mode. In the simulator – since text files don’t have a graphical mode – I’m using very simple bit-map graphics to print these: X for every dark pixel, and a space for every white one. Of course this way the graphics doesn’t line up with the text, but at least you can see it. So what was the graphics that was printed?

Update

With the new 2.0 release, finally AUDIT works as intended: