The Cray Files

Even greater news everyone!

Well, maybe not exactly everyone, but to the two people who care about this kind of stuff…

The other day I felt nostalgic and wondered over to the great archive.org site, originally to look for Commodore 64 material. Then, I thought, what the hell! and typed ‘cray’ into the search box. Lo and behold, a lot of hits! There are manuals, a CD image with some marketing or teaching material on it, but among them, two interesting CD images:

Both are UNICOS install images. I’ll have to see if they are complete and what version (and for what machine) they are, but I wanted to share the news: it appears that there is now a publicly available OS image (maybe even two) for my simulator.

Great news everyone!

After a lot of work, I finally can release the next major version of the Cray PVP simulator: You can head over to GitHub for the latest version.

So what’s new? A ton of things, most of which you can’t directly play with. There’s support now for the XMP, YMP-el, J90 and SV1 family of machines. Yes, this means that I have IOS simulation for the super-minis! If you are interested in the back-story of that endeavor, you can read all about it at the ‘return of the Cray files‘. Unfortunately, while I can share the simulation sources for that work, I can’t share the OS images. You can nevertheless play around with it on the on-line simulator.

As far as COS and XMP simulation is concerned, a ton of bugs have been fixed both in the simulator infrastructure and in the simulation itself as well. The vector- and floating-point operations should be much more precise now. There are some rather substantial speed gains as well for the mainframe process simulator.

Great, but what is this all about?

cray_sn_302a

These pages contain my project notes on how to build a Cray supercomputer at home. Computer history is a fascination of me so this project allows me to geek out to the max. I’m writing my story up in hopes that others (well, you, dear reader) will find it interesting. It is somewhat of a living document, a journal of sorts. Though the story starts several years ago – and that part I recount from memory – I intend to add new pages to the files as I progress. When I tell this story I leave out some of the dead-ends and my wanderings in the digital desert, so the story-line feels more linear than it actually was. I don’t think you want to hear about all the dozens of failed attempts at solving something. I’ll only leave some of the most spectacular failures in for entertainment value.

Special thanks to Andy Gelme, Scott G. Taylor, Brian Taylor and Chris Fenton without whom I would not have been able to get this project off the ground!

The picture above shows Cray X-MP serial number 302 being assembled after delivery. I got this picture from Andy who was the source of the OS backup disk that I’ve used for this project. The following picture shows how it arrived to the lab: through a hole in the wall.

cray_sn_302b

The gory details

With that introduction, let’s look at the various stops along this journey:

  1. Prelude
  2. A New Hope
  3. The Hunt for the Red Bootcode
  4. The Return of the Cray
  5. The Matrix
  6. First update
  7. Needle in the Hay-stack
  8. A Brave New World
  9. Multiple Platforms
  10. Jobs
  11. Parallels
  12. Turbo

After this first series, I’ve started a second phase of the project, this time to re-create a Cray J90, something I do have software for. The chronicles of that section of the trip are here:

  1. A Bridge too Far
  2. That Obscure Object of Desire
  3. Exploring the CD
  4. A New Simulator, part 1
  5. A New Simulator, part 2
  6. A New Simulator, part 3
  7. The UNICOS file-system
  8. Oldies but goodies
  9. A debug session
  10. To SSH and beyond

The road from here

Now that the Cray simulator is finally working, what else can be done? The most interesting thing would be to extend the simulator to other machine types. Namely:

  • The Cray 2/3/4: These machines are very different in architecture. I have some OS images for a Cray 2, but I would need firmware for the front-end processors, that I lack. I also have almost no documentation on these front-end processors. I don’t know their instruction set, how they dealt with memory, what was the interaction between them and the back-end processors and I/O? How did they boot? There probably was some internal Cray documentation on all these topics, but I don’t have them.
  • The big brothers of the EL-line: the XMP, YMP and C90 systems. These I believe I have sufficient documentation for, but I lack any and all SW. An OS image for these beasts would be very useful, but so far I have not been able turn up any.
  • The last of the dinosaurs: the T90. This was a rather drastic departure from the previous lines in terms of architecture judging from the little documentation I have. However, for this machine, this is where the trouble starts: not only I have no operating system image, I don’t have detailed enough documentation either. Marketing material does not make a simulator…
  • The end of the line. There was actually one iteration of the low-end line, after the SV1 (which I do support). It was the SV2, a.k.a. X-1. Just as with the T90 though, it’s a significant departure from the previous designs in terms of architecture and just as with those, I lack both documentation and software to make any significant headway in supporting it.

One could also look back and try to revive the COS operating system on the X-MP. For that though, I need more than just the kernel, I have. I need a full OS. I need SW to run on it. I need source code, I need compilers, I need tools. Things that I don’t have. Things without which the OS kernel alone is little more than a test case for the machine and without which the simulator itself is rather pointless.

Finally one could endeavor on re-creating the actual HW in an FPGA of sorts. This is actually an effort that Chris and I started with and even made some progress on. It’s incomplete at this point, stuck in an endless loop of long (indeed days long) simulation/crashing/head-scratching/fixing loop, also known as verification. We might pick this effort up at a later point, but help on this front would be highly appreciated.

If you have or know of anybody who has experience, old backup tapes, disks, anything that can be used for any of these machines, please contact me!

Print Friendly, PDF & Email

76 thoughts on “The Cray Files

  1. Pingback: » Cray-zy progress! We have a booting system!

  2. Pingback: Cray-zy progress! We have boot! » NYC Resistor

  3. You have your timeline a little wrong, I think. The Cray X-MP 2/2 was released before the Cray-2. I ran on a X-MP 4/8 at NASA/Ames for quite a while before they got their first Cray-2.

    • That could be the case, thanks for the input. I’ve seen conflicting information on the web about the release dates of these machines. I’ve tried to reconcile them to the best of my ability, but I could got things wrong. I certainly don’t have any personal memories from those days.

    • Great suggestion, thank you very much! I will contact them – let’s hope they still have some backups.

      • The e-mail you sent to ETH was forwarded to me, which lead me to your site.

        I worked for Cray Research in Zurich, where I was in charge of applications support and optimization. The X-MP of ETH Zurich was already decommissioned when I started working there in ’98, it was on display at the entrance of the computer center for many years. It was replaced with four J90’s, which were later replaced with two SV1’s. These machines were all running UNICOS, not COS. As far as I know the only thing left from this era is the X-MP chassis (which is no longer functional since many components were removed before it was put on display), and a few programming & optimization manuals that I kept as a souvenir.

        The Cray Museum in Chippewa Falls used to have a nice collection of CDC and Cray systems. Maybe someone there can help you — if this museum still exists.

        Good luck!
        Olivier

          • The Chippewa Falls Museum of Industry and Technology’s Coordinator is intrigued by this thread. Please contact us at : info.cfmit@gmail.com and we can try to help. I’m sure you can help me understand some things! I have direct access to some of the people who built, installed or worked with these computers.

            Looking forward to whatever comes of this.

  4. You might try contacting http://www.cray-cyber.org While they don’t list the system your working with. They are restoring some systems to active status and never know they might have some old software in their archives. Worth a shot.

    • Thanks for the suggestion. I’ve been in contact with the Cray-Cyber folks and they don’t have anything X-MP related. The Y-MP EL they have is a substantially different architecture, and last time I’ve checked even that was down.

  5. Pingback: Simulator pentru Cray X-MP - kandrei.ro

  6. You might try contacting the National Museum of Computing. They recently restored a Cray Y-MP EL to working condition: http://www.tnmoc.org/news/current-projects/cray-y-mp-el-restoration-during-2011 The 12/02/2011 update mentions that they created backup copies of all the disks (8×3 GB data, 200 MB system). That might be interesting to get a copy of; it surely has compilers installed (see the second screenshot). The museum has a significant archive of software and disks, as described at the end of this newsletter: http://www.tnmoc.org/sites/default/files/TNMOCnews_200906.pdf The museum’s contact email is info@tnmoc.org

      • Hi Andras,
        I realise this is now a very old part of this thread, but … Did you ever get any response from TNMOC about any Cray COS materials they might have? If not, I have a friend who is heavily involved with restoration projects there and I could ask him about this. I must admit, I can’t recall them having any Cray related items, but, judging from the links in the above (now dead), it looks like they might have some.

        • Hey!

          No, I didn’t get any response and if you have friends there, I would love if you could ask around.

          Thanks again,
          Andras

  7. Hello Andras,

    maybe a stupid question: is there the assembler in the disk pack ? I suppose so.
    So we could also use some source listing, and compile a compiler from the scratch.

    Then, another question, is there a way to create a tape or another medium in order to run a job ? Putting in another way, how can I write and run a job using the simulator ?

    thanks,
    Fausto

    • Unfortunately there isn’t any assembler on it either. I’ve found one dataset, that contains the following utilities:

      • AUDIT
      • TEDI
      • PLD10
      • GENCAT
      • ACCTDEF
      • JCSDEF
      • PRVDEF
      • LOADCAT
      • RECALL
      • RELOAD
      • RECIO
      • DMP10

      That’s all I have.
      There are two ways of transferring files into the simulator. The best probably is to put files on the expander disk, by recreating it. You can use the build_exp_disk.bat file as a template to do that.
      The other way is to rename the file(s) you want to put on the tape to 1.dat, 2.dat, etc. and move them into the ‘boot_tape’ folder. Than they should be accessible from the expander tape.

      The final question is: how to create a job file. It seems to me that these files would need to be part of a dataset, where each line is a separate record. The dataset format is documented in here for example: http://bitsavers.trailing-edge.com/pdf/cray/COS/2240011E_Cray-OS_Ver_1.0_Reference_Jul78.pdf. I have written a tool (dataset_extractor) that can crack a dataset open and extract the individual files and records from it, but I didn’t do the reverse: to assemble a dataset from individual files. That’s on the TODO list.

  8. One thing that struck me when comparing to Chris Fenton’s older site is that you made little mention of xmpsim.

    xmpsim was a Cray simulator / XMP CAL interpreter for x86 MS-DOS PCs, written by Kay A. Robbins and by Steven Robbins of the University of Texas at San Antonio. Version 1.01 was the last known version, apparently dating from March 12, 1988.

    It has some nice features, such as the ability to produce a Gantt chart showing pipeline timing.

    xmpsim is described in the book entitled “The Cray X-MP/Model 24 : A Case Study in Pipelined Architecture and Vector Processing”.
    http://www.amazon.com/exec/obidos/ASIN/0387970894/mixi02-22/

    You can also find some information on how to use the simulator in Japanese here:
    http://www.takeoka.org/~take/supercom/cray-xmp.html

    or an English tutorial here:
    ftp://130.191.3.100/pub/sdscinfo/SUE-notes/xmpsim1.asc

    Even some lecture notes!
    http://download.springer.com/static/pdf/248/bfm%253A978-0-387-34787-5%252F1.pdf?auth66=1390818563_a86420e57d228c5c5c525e90ba821fc5&ext=.pdf

    ——–

    Chris Fenton has xmpsim available for download here:
    http://chrisfenton.com/wp-content/uploads/2010/08/cray1_r2.zip
    http://code.google.com/p/cray-1x/source/browse/trunk/Software/xmpsim/

    I don’t know if you want to contact Steven Robbins or not, don’t know if he can help you and/or Chris at all…

    • Thanks for the comment. I was aware of XMPSIM, I probably should have mentioned it. The difference between what Steven did there and I did is that while his is a fairly accurate simulation of the CPU pipeline, mine is a much less accurate simulation of the whole system – I/O, peripherals, interrupts. I’ve contacted Steven in the past but he was unable to help unfortunately.

      Thanks again,
      Andras

  9. Marvelous effort!
    I am presently restoring a 1972. HP2000 TSB system, real hardware :), and hope to have it running on the internet probably this year.
    Anyway, my major interest in computing was always in a way “parallel processing”, so a Cray is very interesting!
    Why I write to You here (except to congratulate You on what You did) is to mention that I found this Lady on LinkedIn:
    http://www.linkedin.com/pub/janis-johnson/25/156/45b

    On another note, isn’t it absolutely weird how we succeeded to forget all we did in the 80-ies and early 90-ies. All the software lost… Recovering discs from magnetic image scans… We must be a strang civilisation… It is only 20 to 30 years!!!

    Wish you all the best!!!

    BTW: Regarding software development, there is always the “full bootstrap” possibility. Having a specification of the internal workings it is always possible to write a cross-assembler, then to write the assembler in Cray assembly language, transfer it to the simulator, and start up something like described by William Waite in “Implementing Software for Non-numeric Applications”, what he calls STAGE-2, a macro language capable for (cryptic) writing of compilers. I used this approach back in the mid-80ies to boot up an operating system and Pascal compiler for a industrial control multi-micro-computer system I designed using MC6809 processor(s).
    And there is always the GNU … for Cray …

  10. Hello,
    thank you very much for your hard work, its very important to preserve this fantastic machine!

    have you tried to contact http://www.computerhistory.org/ ?

    Or Mr Robert “Bo” Ewald, he worked in Los Alamos when they had 5 Cray….

    please don’t give up!!!!

    Thanks again

    • Thanks! I have talked to the Computer History Museum, they don’t have anything that they can help with. I’ll try to track down Robert. So far Los Alamos could not help with anything either (and they were using CTSS, not COS). Having more leads could never hurt though, so thanks again.

      Andras

  11. Have you considered contacting the people at cray-cyber.org?
    http://www.cray-cyber.org/general/start.php

    They seem dedicated to enthusiasts being able to see how these machines worked. On their terms of usage page it says:
    “These machines are originally intended to preserve the old culture and machines and make them available for everyone who just wants to play around, do software developing or whatever.
    Everyone is welcome, have a great time and do not hestitate to contact us if you have questions. ”

    I bet they would help in any way they can. 🙂

  12. We helped port Unix to the Cray (Bell Labs). I did some command porting. Dennis Ritchie was the lead of this project.

    • This is great, nice to meet you! I assume this was not UNICOS, right? Which machine did you use for the port?

      Thanks,
      Andras

  13. Our system became Unicos on the Cray X-MP. We wrote the C compiler also. I’m checking to see if there’s any trace left of the compiler work.

  14. I met Chris Fenton at a NY Maker Faire a couple of years ago, but I just learned about your work. We loved working on the Cray (I really worked on the Cray – I used to sleep on the circular bench when I had to spend all night fixing disk failures)

  15. The late Dennis Ritchie’s home page has moved. The current URL is
    https://www.bell-labs.com/usr/dmr/www/
    and the article “Experiences with the Cray X/MP” is at:
    https://www.bell-labs.com/usr/dmr/www/cray.html

    As for http://www.cray-cyber.org/, there hasn’t been much activity there in quite a while. As of 2014-03-04, their machines had to be shut down. They were planning to move them, and were hoping to be back some time in 2014 (unfortunately that didn’t happen). There have been no posts on the diary since then.

  16. What happened to cray-1x repo in code.google.com? They are now empty. I searched for cray-1x on GitHub but all clone repos are empty too. There were no repos on sourceforge. Thanks.

    • Unfortunately Google changed their servers around and won’t support (maybe it does now, but didn’t for a while) SubVersion. As a result we’ve moved the repository to a private server. I periodically do source releases on the website though and of course if you wish to contribute I would be more than welcome to give you access.

  17. We’ve got a Cray XMP in our museum collection (see http://nih.pastperfect-online.com/37870cgi/mweb.exe?request=record;id=6B758A78-DF59-4A7C-B6C9-076222561075;type=101 ) and are hoping to put it on permanent display in the National Institutes of Health Clinical Center. It would stay in place, and we would change the interpretation every now and then. I’ll be starting research for the exhibit soon, and will certainly be reading your blog! And if I find that we have any OS stuff you might need, I’ll let you know. I anticipate having many questions, if you wouldn’t mind me asking you.

  18. Hello

    I was wondering i know nothing about the cray’s but is there a version of linux that can be ported to your simulator. If so that would allow a number of programming language available to the simulator as well . This is just a thought i dont know if you have time or even if it could be done but if i could it would be very cool. I would spend a lot of time playing with it if this was able to be done
    Phil

    • Unfortunately no. It would be a *ton* of effort to port Linux over to the Cray architecture, starting with binutils, then GCC. Even then, the rather trivial memory protection scheme that’s available in the Cray architecture (no MMU) would probably prevent anything but uCLinux to be ported. With the current C compiler at least, there’s the additional problem that ‘short’ is a 64-bit integer with 32-bit semantics, that trips a lot of standard programs over, like openssh. That being said, there are several programming languages available on the current UNICOS installation, such as perl, fortran, C, even an early version of C++. Earlier versions of the OS supported Pascal and I know that there existed an ADA compiler as well though I don’t have access to that.

      • Hello

        Thanks for the reply not surprises there LOL i thought that is the way it is. You cant allow the cos operating system come with the simulator can you ?

        Philip

        • Well, there were two OS-es for these old Cray machines. COS, the older one is a non-interactive batch operating system. I have bits and pieces for it, and I actually include that in the distribution. It’s an incomplete setup though so there’s not much you can do with it other than getting a flavor of this class of ancient technology. The newer OS, UNICOS is UNIX-like and while I have several versions of it, I can’t redistribute those. The online simulator has the most recent release (10.0) on it with some extra packages, like the compilers and the cray open source packages (also abbreviated as COS just to make things more confusing). I’ve also compiled several open-source packages, like bash or Midnight Commander to enable a more modern feel. At its heart though, it’s still just an old UNIX port.

          Another note: the installation and configuration of the OS is less than trivial, especially without the operator workstation and it’s SW. So even if I had the rights to distribute the OS, you would need a *lot* of tinkering (or I would need to write a loooong intro) to get it off the ground.

          • Why can you not release UNICOS OS distros? Cray emulators are almost useless without OS system. Is that possible to port Linux or BSD to Cray system?

            I found cray emulator on GitHub and someone is working on UNICOS configuration files recently. It has only copy of COS 1.17 distro but not UNICOS v10 distro.

          • Timothy,

            I can’t give away the Unicos image, because I don’t have the rights to do so. However, someone put the install images on Archive.org. Check them out!

  19. Hello

    I run ubuntu 16.04 on my laptop. I was wondering if i can compile the simulator to run in ubuntu let me know.

    Phil

  20. I am at make 14112 in the SIM_lib and out of memory. How big it this compile job?
    I have centos 7, a hex core Xeon and 8 gigs of ram

    • That’s strange. I’ve never had that problem. The makefile uses link-time code-generation, but that should not impact SIM_lib. In fact, if anything, it should reduce memory usage in that step of the process. 8Gb of RAM should definitely be enough to compile. I suspect you’ve run into a toolset bug, but I of course can’t be sure.

  21. I made headway tonight. (And I am very happy with my pdp-8 project. Fortran IV is very interesting.)
    But I might be stuck:

    [root@power sw]# make CXX=/opt/rh/devtoolset-7/root/usr/bin/gcc -I/opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux/7/32 -lsupc++ -I//opt/rh/devtoolset-7/root/usr/bin/gcc -librt -j8
    make -C sim_lib
    make[1]: Entering directory /home/jonathan/Desktop/cray_build/boost_1_67_0/cray/sw/sim_lib'
    make[1]: warning: jobserver unavailable: using -j1. Add
    +’ to parent make rule.
    make[1]: Nothing to be done for all'.
    make[1]: Leaving directory
    /home/jonathan/Desktop/cray_build/boost_1_67_0/cray/sw/sim_lib’
    make -C bin_patch
    make[1]: Entering directory /home/jonathan/Desktop/cray_build/boost_1_67_0/cray/sw/bin_patch'
    make[1]: warning: jobserver unavailable: using -j1. Add
    +’ to parent make rule.
    g++ -Wl,–relax -Wl,–no-whole-archive -Wl,-O,2 -Wl,–gc-sections -Wl,–whole-archive -lpthread -Wl,–no-whole-archive ../_obj/linux_release/bin_patch.o -L../_lib/linux_release -Wl,–start-group -lsim_lib -l:libboost_system.a -l:libboost_filesystem.a -l:libboost_regex.a -l:libboost_thread.a -l:libboost_timer.a -l:libboost_chrono.a -l:librt.a -lsupc++ -lstdc++ -lpthread -Wl,–end-group -o ../_bin/linux_release/bin_patch
    /usr/local/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../librt.a when searching for -l:librt.a
    /usr/local/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../librt.a when searching for -l:librt.a
    /usr/local/bin/ld: skipping incompatible /lib/librt.a when searching for -l:librt.a
    /usr/local/bin/ld: skipping incompatible /lib/librt.a when searching for -l:librt.a
    /usr/local/bin/ld: skipping incompatible /usr/lib/librt.a when searching for -l:librt.a
    /usr/local/bin/ld: skipping incompatible /usr/lib/librt.a when searching for -l:librt.a
    /usr/local/bin/ld: cannot find -l:librt.a
    /usr/local/bin/ld: cannot find -lsupc++
    collect2: error: ld returned 1 exit status

    I really don’t know anything about Visual Studio Code. I am not sure what is meant by building a 32bit project to get a JSON file. I have used frameworks before, I just don’t see the functionality. Maybe this is because I am using CentOS 7.
    Any suggestions on cross compiling with librt and lsupc++ would be appreciated. I think my compile command is effective, though obviously the -I commands are being ignored.
    Here it is again:
    make CXX=/opt/rh/devtoolset-7/root/usr/bin/gcc -I/opt/rh/devtoolset-7/root/usr/lib/gcc/x86_64-redhat-linux/7/32 -lsupc++ -I//opt/rh/devtoolset-7/root/usr/bin/gcc -librt -j8

  22. Your and Chris Fenton’s work is just amazing! I commend you both and others like you doing similar work.

    I grew up writing assembly language for Control Data Cyber mainframes. The Cybers were also designed by Seymour Cray and had similar instruction sets and architectures to the Cray machines which followed. It seemed all so elegant and clever, making maximal use of every logic gate and instruction word.

    It’s a shame that so much of that technology, especially the software, met the scrap heap so quickly with the PC revolution. I mean in the literal sense, not in just becoming outdated, but that very few people (if any) tried to preserve the hardware and software. Most of what seems to exist today seems to be an accident.

  23. No chance of getting a YMP UNICOS kernel?
    I worked with Cray’s in the late 90’s C90 and T3E, the T3E Alpha processor of course
    I still have some logs, binaries and maybe a kernel as well

    • Peter,

      I unfortunately don’t have kernels (or anything really) for the machines you’ve mentioned. I wouldn’t be surprised if a YMP-el image would work in a full YMP (not the IOS portion of course): I haven’t seen anything that would suggest any functional difference between the mainframe portion of the two machines. But not having one means I can’t be certain.

      I’ve searched high and low for anything C90 related without any luck. Modifying the simulator to support this architecture shouldn’t be that difficult, but again: without something that would actually run on it, it would be.

      What I have are full OS images for the low-end series: YMP-el, J90, SV1.

      If you really have some binaries and kernels for any of the machines, and you’re willing to share, I would *really* appreciate it.

      Thanks,
      Andras

      • Andras,

        I can search my archive and see what I can find, I doubt I saved C90 kernel, it was more strict environment compared to the T3E, would you be able to lend me a a disk image or a kernel to install UNICOS on the cray-sim? I promise not to keep a copy when I hand it back 😉
        The other binaries I have are mainly just calculation projects, what else(?)

        Regards, Peter

      • Andras,

        I also tried the FPGA version on Xilinx, it seems the cray_iop.ucf file is missing to run the Makefile build, “I think” I have the latest version, the one from big-G code with separate IOP directory (your name in the Makefile), or is there a newer?
        I try build from ISE instead.. and add a ucf that match my board.

        Regards, Peter

  24. Andras,

    Do you think the CD images from archive.org can be used to set up a UNICOS installation that can be used with your emulator?

    Regards,
    One of the 2+ people who care about this stuff 😉

  25. I tried cray-cd2 today and it works. I had a little trouble transferring the Install script to the ram-fs. It seems that unless the file exists, it won’t transfer. After doing a touch /install, it transferred properly. After that there was a problem with the UNICOS_exe dir. In the script UNICOS is uppercase, and on that cd image it is lower case. After editing the script, it ran fine. In about two hours, I had a fully installed system.

  26. I discovered a copy of Unicos images on archive.org. Search for ‘unicos’ and you will find two Unicos v10.0.0.2 and v10.0.1.2 images that were uploaded last December.

Leave a Reply to Brian Cancel reply

Your email address will not be published. Required fields are marked *