The Ultimate Install Fest: Linux on the IBM System/390

By: Scott Courtney
Monday, July 31, 2000 07:47:10 AM EST
URL: http://www.linuxplanet.com/linuxplanet/reports/2127/1/

Recycling the Mainframe with Linux

Let's say you've been using a fairly mainstream, commercial operating system for a few years on your company's computer. You keep hearing about this thing called "Linux" and it sounds interesting, but you never used UNIX before and that makes you a little nervous about Linux.

Then one day you hear about a thing called an"Install Fest." Supposedly there will be Linux experts on hand to help get that first, trickiest installation up and running in just a couple of hours. Terrific! Stuff the CPU in the back seat of the car and toss the cables in the trunk, and head on over to your local Linux User Group (LUG). By the end of the day, you're running the latest Linux kernel--coexisting with your old operating system--and you think you just might be starting to understand this Linux phenomenon.

It is a scene that's been repeated endlessly, all around the world. Only this time there is a slight problem: the company mainframe won't fit in the back seat!

By now just about everyone involved with Linux knows that, among its many other ports, Linux is now available for IBM System/390 (S/390) mainframe systems (see accompanying photo). If you've ever had the fun of installing it on an S/390, you know that the first time through the process it can be quite daunting. From the beginning, there is a gap in terminology. If you are a mainframe administrator, you know all about how to allocate DASD and LPARs, how to set up IUCV and SNA communications, and how to IPL a few VMs--but the concept of grepping the stdout of sed sounds like line noise to you! If, instead, you come from the UNIX side of the house, the same logic applies but in reverse. The problem with putting Linux on a mainframe is that you really need to know both UNIX-like commands and mainframe administration.

To help alleviate the knowledge gap, IBM engineers and management in Germany came up with the idea of a Linux for S/390 "Install Fest" which would work almost like the local install fests run by LUGs around the world. The leader of the Install Fest project is Alex Stark, Manager of Linux for S/390 Design and Development. At 35, Stark has the enviable job of leading the team that develops new code for Linux on S/390 hardware. Stark's team of 25 developers is fully dedicated to the Linux on S/390 effort, and they saw this project as a way to get closer to the customer and learn how to improve the installation process. At the same time they hope to spur even more interest in the Linux for S/390 port, which has already seen over four thousand downloads.

The Install Fest itself was done in a virtual meeting space, over a combination of Internet and telephone. Customers signed up in advance, specifying what days were best for their schedule, and IBM set up conference calls with four to six customers and as many IBM engineers as needed. According to Kathy Grabarits, Manager of IBM's System/390 New Technology Center, there were around twenty IBM engineers on call during each Install Fest teleconference. A small core team would moderate and lead the conference, but they could call on any of the twenty to get an immediate resolution for any technical problems. Grabarits says the installation process usually takes between thirty minutes and four hours, depending on the knowledge level of the customer.

Customers, of course, had to do the actual installation on their own mainframes. There was a pre-installation checklist and site survey, and the customer also had to make a virtual tape image (or a real tape) from the SuSE Linux CD media sent out from IBM. According to Grabarits, there have been very few Linux-specific trouble reports during the installations so far. A few customers, she says, needed some offline assistance with the preparatory phase. To keep from holding up everyone on a conference call, these folks are transferred to another engineering team, in effect becoming their own one-customer Install Fest.

In keeping with the "festival" theme of the event, the package IBM sent out to each participant included party hats and other toys--so much for IBM's blue-suited corporate image! No one on the Linux-390 e-mail list has yet admitted to actually wearing the party hats, but maybe they're just not telling. The most important item, though, was a brand-new boxed distribution of SuSE Linux, including both Intel and S/390 ports on a half-dozen CDROMs.

The Install Fest conference calls began on July 15 and will end around the first of August. IBM's Alex Stark was ecstatic about the success of the program, and says there is a very good chance it will be repeated. Over 150 customers signed up for the Install Fest from countries as diverse as Austria, France, United Arab Emirates, Israel, Switzerland, Canada, and the United States. According to Stark, at least thirty of these sites were from IBM's "Top 1500" list of corporate customers.

Is IBM Really Serious About Linux?

It is fairly common in the software industry for a company to make a loud and strong commitment to a technology, only to back down later when the world seems to be going in another direction. We can all think of examples; there is no need to list them here. So, is IBM serious about Linux? Or is it just blowing with the wind, and with as much constancy?

The answer seems to be that yes, IBM is very serious about Linux. Peter McCaffrey, System/390 Program Director, says, "Customer response has taken us by surprise," and IBM did not expect as much interest in Linux on the mainframe as it has actually seen. Many companies have just downloaded the free code to gain some experience with it and to see what it can do. Yet some large customers are looking very hard at Linux for real-world applications.

IBM has been getting the message loud and clear from its customers that Linux on S/390 is a hot product--and that IBM needed to refine the installation process if people were going to make it work as anything other than a lab toy. McCaffrey says customers weren't happy with the cryptic, sparsely-documented installation instructions, or with some of IBM's pricing policies that made Linux a poor business choice for some installations. The message, according to McCaffrey, was, "You've got to make this easier, and you've got to make this less expensive!"

The Install Fest was one of IBM's responses to the issue of installation complexity. Another was IBM's recently announced partnership with SuSE and TurboLinux, both adding System/390 as a supported hardware platform for their mainstream Linux distributions. The SuSE product is currently considered to be at late-beta stage, while the TurboLinux offering is scheduled for release this fall. Customers will be able to get support for their S/390 installations of Linux from three sources: the distribution provider, IBM Global Services (which will support both SuSE and TurboLinux), and the Internet itself. Linux on S/390 is very compatible (at the source code level) with other Linux ports, so many of the HOWTOs, newsgroup posts, and other online resources are just as applicable to mainframe Linux as they are to Intel or MIPS or Alpha ports.

IBM Announces Linux-Specific 390 Hardware

As for the pricing issue, IBM has responded by introducing two new products, one hardware and one software. Both of these will be announced officially on August 2.

The Integrated Facility for Linux, or "IFL," is a dedicated processor node that plugs into a System/390 mainframe and runs only Linux. The instruction set is fully binary-compatible with other S/390 processors, but the microcode won't let it run IBM's OS390 or VM software. This reduced generality doesn't affect Linux users, except to lower the price of the board by about 65%, from US$375,000 to US$125,000, approximately. Linux will still run on the normal S/390 processor, but the new IFL is much more cost-effective if Linux is the desired operating system.

The IFL processor works just like a standard S/390 CPU module, and can be allocated in Logical PARtitions (LPARs) to allow dynamic resource management. Like other S/390 CPUs, the IFL shares the memory and I/O channels of the machine. IBM's "G6" class machines accept up to twelve CPUs total, of which eleven can be IFLs. Up to fifteen LPARs can be allocated on a single machine. If you need more, you can combine G6s into a giant Sysplex whose size is practically unlimited (you will run out of money first!). The system administrator can carve up those LPARs to assign any supported number of virtual CPUs to each Linux instance on the machine.

Virtual Image Facility: A Cheaper VM
The second new product from IBM is called Virtual Image Facility, or VIF. Readers of my first S/390 article (http://linuxplanet.com/linuxplanet/reports/1532/1/) will recall that IBM's Virtual Machine (VM) hypervisor allows many hundreds--or even a few thousand--instances of Linux to run on a single physical CPU or LPAR. This is great for sites that already have a VM license for other applications, but VM is expensive, and Linux customers had trouble cost-justifying the VM purchase.

In response, IBM will announce the Virtual Image Facility as a low-end alternative to VM. VM is more than just a hypervisor to allocate virtualized resources; it is a full-blown operating system that runs large-scale applications. With VIF, IBM has stripped off the general-use parts of VM, leaving only the hypervisor core and some simple management tools, and reducing the price accordingly. VIF isn't nearly as versatile as VM: you can't run OS390 inside VIF, for example, nor can VIF run inside VIF. On the other hand, VIF is a one-time license priced at around US$20,000, a fraction of the ongoing software lease price for VM itself.

VIF is not only cheaper than a full VM license, but also easier to install, according to Pete McCaffrey. He also asserts that the one-time-fee pricing model--uncommon for mainframe software--is a direct response to demand from customers. McCaffrey adds that much of the IBM middleware for Linux will also use the one-time model when it is released over the next several months. Middleware products, in IBM parlance, include DB2 Universal Database, MQ Series messaging, and Tivoli system administration and backup tools, among others.

Performance: "The Answer is Forty-Two!"

After my first article, many of the questions I received via e-mail concerned the cost effectiveness of deploying a single large S/390 mainframe, versus a few racks of Intel or RISC equipment. Although the S/390's I/O and memory bandwidth far exceeds any PC-class hardware, the raw speed of the CPU itself--per processor--is higher but not orders-of-magnitude higher. Some readers questioned whether it makes sense to even consider the S/390 for Internet or intranet applications.

Since Alex Stark is IBM's key technical lead for Linux design on the 390, I asked him to respond to these questions. What, I asked, are the classes of application where the 390 makes sense, and where is it overkill or not cost effective?

Stark answered by pulling a quote from Douglas Adams' famous Hitchhiker's Guide to the Galaxy, responding, "The answer is forty-two!"

He went on to explain that the performance issues are heavily dependent on the application--no surprise to anyone who knows about hardware optimization. Stark also includes other items in the "performance" category besides the throughput of the machine itself.

Stark lists four key areas for assessing performance: speed of porting new operating system versions, speed of porting new and existing applications, execution speed of the applications individually, and overall execution speed of the total software environment.

The first two aspects, he says, are as good for Linux on S/390 as they are for any other Linux port. IBM is committed to changing nothing about the base architecture of Linux beyond the hardware-specific items that are needed for any port to a new platform. Stark says that any application that is free from architecture-specific dependencies--such as a sensitivity to big-endian or little-endian byte ordering--should run on Linux for S/390 with not much more than a recompile from source. IBM is working with the Linux kernel team to incorporate S/390 into the official kernel; Alan Cox is a frequent contributor to the Linux-390 e-mail discussion list. According to Stark, if a developer's application compiles and runs on other non-Intel processors it will most likely compile and run on the S/390 as well.

Who Are the Customers? In terms of application speed, Pete McCaffrey, Program Director for S/390, acknowledges that there are some applications where a 390 may not be a good choice. These include scientific number-crunching or ray tracing--areas where I/O is often less important than CPU speed. For such applications, McCaffrey recommends Intel or RISC platforms, preferring, of course, the IBM Netfinity line.

On the other hand, says McCaffrey, tests run in IBM labs and by customers evaluating Linux for S/390 indicate that other classes of application perform extremely well on the S/390. As an example, McCaffrey cites a very large e-commerce hosting company that is serious about deploying S/390 for a multi-customer "virtual storefront"web farm. He says that the company likes the idea of being able to rapidly redeploy CPU and memory resources from one customer to another, based on a very dynamic loading model, while still retaining complete isolation of each customer's environment from the others. LPARs are allocated a variable percentage of a variable number of physical processors, and these values can be adjusted while the machine is running. This is the equivalent of upgrading--or downgrading--the CPUs in a rack full of Intel machines, allocating them where they are most needed.

McCaffrey says that there are still areas where the performance characteristics are not yet well-defined. IBM is currently engaged in some internal tests to study the horizontal scalability (multiple applications) and vertical scalability (many users on each application) of their middleware on the Linux for S/390 platform.

Pete McCaffrey lists internet content providers, application service providers, and colocation hosting providers among the best candidates for S/390 Linux. One example would be the case of a Web hosting company that has many small customers who generate a lot of traffic in aggregate but little as individual sites. Each customer's server sees a very "bursty" load profile, which tends to work well on the S/390 hardware. The provider can dedicate a Linux instance to providing common services such as DNS or POP3 e-mail accounts to all customers, and could even offer a single instance of OS390-based DB2 as a shared database server (with each customer getting its own database for security reasons). Among the middleware soon to be available for S/390 Linux is the "DB2 Connect" client, which provides APIs for connecting to DB2 Universal Database over TCP/IP. If the OS390 instance hosting DB2 is on the same physical mainframe as the customer's Linux instance, then the database connections can be made over the machine's internal memory bus, on a "virtual LAN". And the various customers can purchase more or less CPU power from the provider, with the changes made in near realtime and with little administrative cost to the provider.

I asked Alex Stark about the famous David Boyes experiment, in which over 41,000 instances of Linux ran on a single System/390 machine. Is that really practical, in the real world?

As one would expect, Stark's answer was an emphatic "No." He was familiar with Boyes' work (and impressed by it) but said that production installations just wouldn't be done that way. More likely, Stark says, there would be one instance of Linux per application function rather than one per user. Stark does feel, though, that a few hundred instances are realistic even if tens of thousands are not.

Of course, as with everything else, the customer application determines what is and is not realistic. Pete McCaffrey cited universities as one case where thousands of Linux instances might realistically be used, because each student needs to have an isolated, private operating system to tweak--and to break--as they learn. At the same time, most of the students' CPU loading would be small, averaged over time.

Making Money from Free Beer

The IBM team that has built Linux on S/390 reminds me a lot of the team that built the original IBM PC back around 1980. Both teams seem, at least from the outside, to have a certain "renegade" feel to them. Clearly the senior management at IBM is behind the Linux team now, but one has to wonder--as one wondered in the days when the PC was new--were the managers on board from the beginning?

Even while Linux for S/390, and Linux on IBM's other hardware, opens up new markets for the company, there has to be some concern over what this will do to its other markets. What if Linux cuts into sales of OS390 and OS390 applications? Will the one-time pricing model that seems normal in the PC industry, or the zero-pricing model from open source, undercut IBM's margins in other areas through price pressure?

I asked Alex Stark if his team was feeling pressure from other parts of IBM because of these issues, but he says it is not seen as a problem. Stark emphasizes that IBM views Linux--on S/390 or elsewhere--as one part of a very large market strategy. He says that Linux may help to drive more S/390 sales, but that this is no more important than the ability to market a single operating system on all IBM hardware, from laptops to mainframes.

Stark expects IBM's overall business to grow, because Linux versions of IBM database, messaging, and system management products will be strong revenue generators. IBM also hopes to profit from value-added support services provided through IBM Global Services, and from consulting contracts.

I asked Pete McCaffrey just where the minimum entry point is, pricing-wise, for the S/390 line. It is lower now, with the introduction of Integration Facility for Linux and of Virtual Integration Facility, than it was a week ago, but it is still high--especially since there must be at least one general-purpose 390 CPU in the machine, even if Linux will be the only operating system. On the other hand, McCaffrey says that Linux for S/390 will also run on IBM's smaller (and cheaper) Multiprise 3000 midrange machines (shown in the accompanying figure).

Because Linux for S/390 is so compatible with other ports, developers can deploy their systems on Multiprise or even Intel PC hardware and still have a growth path to full System/390 hardware as their needs increase.

IBM, Open Standards, and Open Source

IBM has a long history of setting standards and then trying to force the world to adopt them. For years, SNA was the network architecture as far as they were concerned, and EBCDIC was the character encoding standard. Microsoft is famous for such behavior today, but didn't invent it.

Times have changed, though, both outside and inside IBM. Over the past decade or so, IBM has been whacked on the head with the open-standards stick by so many customers that the message finally sank in. And once it did, the philosophy has spread like wildfire within the company just as it has in the world at large. Witness, for example, the ubiquity of TCP/IP network support. How recent it was that well-informed people were sure IBM would never support TCP/IP on a mainframe--IBM pushed SNA, and it pushed Token Ring, not the Ethernet that most other companies used. Now just about every piece of IBM hardware--including the System/390--supports both of these standards, right out of the box.

So it has been with software, as well. Says Alex Stark, "It's tremendous how IBM understands the value of open standards. The old days are gone....It's really fun to work here in [this] company....I can see changes every day sometimes."

Not that it doesn't care if IBM-specific products are the mainstream standard. Stark is quick to add, "We want to make what we're doing here a success. That means customers using and valuing the 390 platform and product." But he doesn't see the desire to sell IBM products as being at odds with the open source philosophy. "We want to have very good relations with the open source world....It is a world of give-and-take. We have taken something from the open source world, and we want to give something back." Among other code, IBM has recently released its Journaled File System (JFS) from the AIX environment into the open source community.

Stark emphasizes the participation of IBM developers in the open source arena at large, not just producing IBM products. IBM programmers are active on the Internet discussion forums and project teams, and "discuss items that don't even affect [IBM] directly." He says that there is a quality difference in the code produced in an open environment:"Open source is a world where you have to be good to succeed. Everyone can read your code!"

He adds that the IBM programmers--himself included--enjoy the collaborative atmosphere surrounding Linux and other open source projects. That's one of the objectives Stark has for his team: "We also want to have fun here." The programmers in Boeblingen know their stuff: I thought I was a pretty good hardware geek until I started e-mailing with some of IBM's mainframe wizards. I am now much more humble. And they seem passionate about what they do. Although nongeeks might disagree with the definition of the word "fun", these IBMers seem to be having some of it.

Looking Ahead for Linux on S/390

IBM wants more mainframe customers to give Linux for S/390 a try, for a variety of reasons. Obviously, it makes a lot of money selling S/390 hardware and services, so the more uses that can be found for that big iron, the more it will, presumably, sell.

Linux on the mainframe is an image booster, too. People who work with mainframes know that they've enjoyed a renaissance of innovation in the past few years--but the image of big iron still carries an outdated, negative connotation in some circles. Linux on S/390 helps shore up the machine's image as a fully capable player in the new eCommerce industry. Furthermore, since IBM is pushing Linux and AIX on its other hardware platforms such as Netfinity servers, having Linux on the S/390 helps IBM to make the case for UNIX-like systems as a common standard for multiple levels of hardware.

So, for a variety of reasons, IBM has gotten very serious indeed about Linux. A couple of years ago, the same thing seemed to be happening with Java, and many people doubted whether IBM was enlisted for the duration. Today, IBM is one of the biggest players in the Java marketplace, and the last count I heard was that there are more Java programmers inside IBM than at Sun Microsystems. If the same thing happens with Linux over the coming year or two, IBM could push Linux into markets that it might otherwise have taken a decade or more to reach.

Say what you will about this giant corporation being a strange bedfellow for wild-and-wooly open source teams, but the undeniable fact is that corporate executives love IBM. Those same corporate executives set standards for technology within their companies, and they make large purchases. IBM's entry into the Linux arena will undoubtedly influence the open source movement, both for good and for ill, but there is no doubt whatsoever that it has given Linux a new level of credibility in the boardroom.

At the same time, IBM is still working to shred vestiges of the old "800 pound gorilla" image. If IBM can work successfully in an open environment, building open source and commercial code around open standards, then Linux may very well bring a new level of credibility for IBM as well.

Author's Bio
Scott Courtney is a web programmer and feature writer for the Linux and Open Source channel of Internet.com. He has two Bachelor's degrees in engineering and over fourteen years experience working on systems, from embedded microcomputers to mainframes, at a large manufacturing company. Scott can be reached by e-mail to scourtney@internet.com.