There are many differences between the entry-level Cray systems (like the J90) and their bigger brothers, like the Y-MP. They differ both in silicon technology (CMOS v.s. ECL), performance, price and mechanical design. Most of these however don’t really matter if one wants to write a simulator for any of these beasts. One important difference that does matter however is their different IO subsystem designs. The big machines used a custom processor design and peripheral interfaces. The super-minis used a VME-based IO subsystem with mostly 3^rd party components.

The old X-MPs used something called IOSD (IOS model D) for the IO subsystem. Later Y-MP class machines used IOSEs, which were significantly different from the ‘D’ variants. J90s had IOSVs (V for VME), based on SPARC-based single-board computers.

For our purposes the IOSD and IOSV designs are the important ones: I have a function IOSD simulator, and I have UNICOS for a machine that used the IOSV. Of course these two systems run different code, used different peripherals and in general were totally different beasts.

Why am I talking about this? Because of the state of documentation. The custom-built Cray IOSD (and ISOE as well) is well documented. The IOSV inside the J90 however is mostly undocumented. This is probably due to the fact that it’s built from 3^rd party components, so Cray probably didn’t even have the right to re-distribute the documentation. As I’ve said, the IOSV was built using the VME bus, and was using (in the J90 anyway) a SPARC processor at its heart. Other third party interfaces provided hard-drive interfaces (mostly in the shape of SCSI cards) and network interfaces (Ethernet, ATM and FDDI). Pretty much the only custom piece was the interface to the mainframe itself: one VME card with some custom chips at the center and some thick cables at the edge. On this card I did find some rather rudimentary documentation, but that’s hardly enough to get the whole IOS simulation up and running.

Of course none of this was news to me, in fact, I’ve been warned about these problems by several people. This is why I put off even looking deeply into simulating the low-end line of Crays: without documentation the simulation of the IOSV is hopeless and without the IOS, simulation of the mainframe is pointless.

So, provided don’t want to just curl up in a corner and cry, what options do I have?

I can’t simulate the IOSV. I can simulate the IOSE, but I don’t have a single line of code to execute on it. I can try to marry the existing X-MP IO subsystem (IOSD) with a J90 mainframe and hope that the inter-processor communication didn’t change all that much over the years. Or, and this is the new idea: Instead of trying to simulate an IOS on the instruction level, I could simulate the IO subsystem on the protocol level: interpret, execute and respond to the commands that the mainframe sends to it. After all, this is what SuperTek must have originally done when they conceived of the precursor of the J90, the SuperTek-1! Plus, that’s the best I have and what do I have to loose other than a few weekends, maybe?

With that plan in place, let’s see how this protocol worked! The operating system, at least the part running on the mainframe didn’t implement I/O natively. Instead, it sent commands over to the IOS to which the IOS responded. This is how COS worked, and I have no reason to believe that this high level concept changed. In fact I have some reason to believe it didn’t: UNICOS on the X-MP class machines used the same IOS code that COS did.

What did change between COS and UNICOS is the complexity of these messages. COS used a very rich set of commands, especially to implement UI and networking. This is all documented – to a certain degree – in two manuals:

One is the COS table descriptions manual
The other is the front-end interface internal reference

I spare you the details; it is complex, mostly because every possible UI interaction (like checking the load of a CPU or displaying the jobs awaiting execution or submitting a new job) was implemented as a separate message exchange.

UNIX (and UNICOS as well) takes a different approach to UI though. Their concept of a user interface is a ‘dumb’ terminal. Their abstraction level is what you see in STDIO: send a string, get a character. That’s about it. The UNICOS concept of disks is a block-device: not much more than a collection of sectors. In this regard, even COS wasn’t all that complex. Tapes were very important in the X-MP days, but were much less so by the J-90 shipped, so even if I can’t get that working, no big loss. Networking? Well, we shall see. That’s a long ways down the road, I’ll worry about it when I get there.

This difference in protocol complexity is why the idea of interpreting the commands coming from the mainframe was not exciting as an option in my COS days but is potentially feasible for UNICOS.

From my X-MP work I’ve seen that the pattern of communication is the following:

There’s a (relatively) low-speed interface which is used to communicate commands and responses between the IOS and the mainframe. The communication is packetized and these packets are relatively short (in the X-MP case fixed at 6 words). They contain things like, ‘please read sector such-and-such from disk something-or-another’.

The actual data-transfer happened through a much higher-speed DMA interface: the IO subsystem directly put the content into the mainframe memory. Of course the buffer pointer was passed as part of the request.

Finally the IOS sent a response packet through the low-speed interface informing the OS that the request was completed with a potential error code if something went wrong.

The type of the packet was denoted by a single byte, but only ASCII characters were used. Upper case letters were used for commands and lower-case ones for replies. So an ‘A’ packet would be a disk access command, and an ‘a’ packet would be the corresponding reply. A ‘D’ packet would be a tape request and it’s response would be a ‘d’ packet.

Simple enough. Of course simple enough only if it’s documented. All what I know about this so far is for X-MP and (mostly) COS, but what about the J90 and UNICOS? So, we’re back to square one: hunting for documentation. I’ve found none online. I’ve found none on the CD. I’ve found none anywhere.

Well, that’s kind of depressing, luckily documentation is just one of the ways to get the information I need. Reverse-engineering is another one, but I would prefer not doing it through execution traces and disassembly. If I had source code, it would be much easier. Unfortunately the CD doesn’t contain much in terms of source code either (not zero, but nothing of real importance).

There are however a few universals about UNIX-style operating systems. One of those is that they contain header files under /usr/include. Not having source is disappointing, but the content of these include files (especially under /usr/include/sys) can be quite revealing about how the system worked.

Now that I can access the raw content of the installation media, I actually do have access to the /usr/include directory. So let’s poke around and see what we can learn!

As I said before, the communication between the IOS and the mainframe is organized into packets, in fact it used to be called the ‘any packet layout’ under COS. So, let’s look for something with that in the name!

OK, so how about ‘packet.h’?

This is somewhat encouraging, especially because the names look familiar. This is an extended list of packet types that are used in the X-MP, under COS. Where are the actual packet layouts though? I’ve found a number of files corresponding to the packet types described in this file. apacket.h for example details the layout for ‘A’ or disk packets:

Another file, bpacket.h details the fields in the ‘B’ or station control packets:

You get the idea. Except when you reach epack.h, it paints a different picture. It talks about model-E IOS structures:

There’s a whole set of epack*.h files, one for each packet type (for example epackz.h for terminals).

We know that we’re dealing with IOSE for the big Y-MPs and IOSV-s for the J90. We also have two possible packet structures, the old one in packet.h and the new ones in epack.h. Which one did IOS-V use? Maybe a third kind? One that’s not documented at all?

To dig deeper, I needed to experiment. And to experiment meant to boot the OS, and booting the OS meant to have a simulator.

So what needs to be done?

I need to modify the simulator to simulate the J-90 instruction set, and whatever else have changed between the X-MP and the J-90 mainframe over the years. I’ll have to yank out all the IO simulation stuff and replace it with stubs to see if/when UNICOS tries to talk to the IOS. This is ‘just’ work, provided I can find the instruction set description for the J-90.

I need to figure out the boot process, at least to the point where I can put the right pieces of code in the right places of the mainframe memory. This is something that I knew from my X-MP work fairly well: the IOS places the kernel and something called the parameter file in the mainframe, patches up the initial exchange packet to point S4 and S7 to the parameter file then releases one of the main CPUs from reset. The mainframe starts executing the kernel, which loads and parses the parameter file, eventually making contact with the IOS using the aforementioned packet interface. For the X-MP, the first few packets establish communication and synchronize the real-time clock between the IOS and the mainframe.

I don’t know if UNICOS uses a similar process, but chances are it does. UNICOS worked on X-MPs with the same IOS firmware that supported COS.

Eventually the OS gets to the point of mounting the root FS and executing the first user-mode process: init. When that happens, we’re 90% there: init will start spawning new processes and enter some run-level. Hopefully single-user, otherwise I’ll have to also figure out how to crack passwords, which wouldn’t be fun!

To summarize where we are and what we need:

1. We need a kernel: check
2. We need a root FS: check
3. We need a parameter file: NOT check
4. We need an actual simulator: NOT check

We also have to figure out how to string all these things together into a functional system. And then, then I can potentially start figuring out how the IO packets flow between the mainframe and the IO subsystem, what they mean and how to respond to/generate them properly.

This is already getting long, so let me stop here. Next, I’m going to look into how to get all this done and start the simulation.