Many years ago I worked on IBM DB2 and so I occasionally get the question, “how the heck could you folks possibly have four relational database management system code bases?” Some go on to argue that a single code base would have been much more efficient. That’s certainly true. And, had we moved to a single code base, that engineering resource efficiency improvement would have led to a very different outcome in the database wars. I’m skeptical on this extension of the argument but the question is an interesting one and I wrote up a more detailed answer than usually possible off the cuff.
——————–
IBM Relational Database Code Bases
Few server manufacturers have the inclination and the resources needed to develop a relational database management system and yet IBM has internally developed and continues to support four independent, full-featured relational database products. A production-quality RDBMS with a large customer base is typically well over a million lines of code and represents a multi-year effort of hundreds and, in some cases, thousands of engineers. These are massive undertakings requiring special skills, so the question sometimes comes up, how could IBM possibly end up with four different RDBMS systems that don’t share components?
At least while I was at IBM, there was frequent talk of developing a single RDBMS code base for all supported hardware and operating systems. The reasons why this didn’t happen are at least partly social and historical, but there are also many strong technical challenges that make it difficult rewind the clock and use a single code base. The diversity of the IBM hardware and operating platforms would have made it difficult, the deep exploitation of unique underlying platform characteristics like the single level store on the AS/400 or the Sysplex Data Sharing on System z would make it truly challenging, the implementation languages used by many of the RDBMS code bases don’t exist on all platforms raised yet another road block, and differences in features and functionality across the four IBM database code bases make it even less feasible. After so many years of diverse evolution and unique optimizations, releasing a single code base to rule them all would almost certainly fail to be feature and performance compatible with prior releases. Consequently, IBM has four different relational database management system code lines maintained by four different engineering teams.
DB2/MVS, now called Db2 for z/OS, is a great product optimized for the z/OS operating system supporting unique System z features such as the Sysplex Coupling Facility. Many of IBM’s most important customers still depend on this database system and it would be truly challenging to port to another operating system such as Windows, System I, UNIX or Linux. It would be even more challenging to replace Db2 for z/OS with one of the other IBM relational code bases. Db2 for z/OS will live on for the life of the IBM mainframe and won’t likely be ported to any other platform or ever be replaced by another RDBMS code line from within IBM.
DB2/400, now called Db2 for i, was the IBM relational database for the AS/400. This hardware platform, originally called the System/38, was released way back in 1979 but continues to be an excellent example of many modern operating system features. Now called System i, this server hosts a very advanced operating system with a single level store where memory and disk addresses are indistinguishable and objects can transparently move between disk and memory. It’s a capability-based system where pointers, whether to disk or memory, include the security permissions needed to access the object referenced. The database on the System i exploits these system features, making Db2 for i another system-optimized and non-portable database. As with Db2 for z/OS, this code base will live on for the life of the platform and won’t likely be ported to any other platform or ever be replaced by another RDBMS code line.
There actually is a single DB2 code base for the VM/CMS and the DOS/VSE operating systems. Originally called SQL/Data System or, more commonly, SQL/DS (now officially Db2 for VSE & VM), it is the productization of the original System R research system. Some components such as the execution engine have changed fairly substantially from System R, but most parts of the system evolved directly from the original System R project developed at the IBM San Jose Research Center (later to become IBM Almaden Research Center). This code base is not written in a widely supported or portable programming language and recently it hasn’t had the deep engineering investment the other IBM RDBMS code bases have enjoyed. But it does remain in production use and continues to be fully supported. It wouldn’t be a good choice to port to other IBM platforms and it would be very difficult to replace while maintaining compatibility with the previous releases in production on VM/CMS and DOS/VSE.
For the OS/2 system, IBM wrote yet another relational database system but this time it was written in a portable language and with fewer operating system and hardware dependencies. When IBM needed a fifth RDBMS for the RS/6000, many saw porting the OS/2 DBM code base as the quickest option. As part of this plan, the development of OS/2 Database Manager (also called OS/2 DBM) was transferred from the OS/2 development team to the IBM Software Solutions development lab in Toronto. The mission was both to continue supporting and enhancing OS/2 DBM but also to port the code base to AIX on the RS/6000. We also went on to deliver this code base on Linux, Windows, HP/UX, and Sun Solaris.
My involvement with this project started as we began the transfer of the OS/2 DBM code base to Toronto. It was an exciting time because not only were we going to have a portable RDBMS code base and be able to support multiple platforms but, in what was really unusual for IBM at the time, we would also support non-IBM operating systems. This really felt to me like “being in the database business” rather than being in the systems business with a great database.
One of the first things we discovered was OS/2 DBM was really struggling with our largest customers and they were complaining to the most senior levels at IBM. I remember having to fly into Chicago to meet with an important IBM customer who was very upset with OS/2 Database Manager stability. As I pulled up in front of their building, a helicopter landed on the lawn with the IBM executives who had flown in from headquarters for the meeting. I knew that this was going to be a long and difficult meeting and it was.
We knew we had to get this code stable fast, but we also had made commitments to the IBM Software Solutions leadership to quickly be in production on the RS/6000. The more we learned about the code base, the more difficult the challenge looked. The code base wasn’t stable, didn’t perform well, nor did it scale well in any dimension. It became clear we either had to choose a different code base to build upon or make some big changes fast.
There was a lot to be done and very little time. The pressure was mounting and we were looking at other solutions from a variety of different sources when the IBM Almaden database research team jumped in. They offered to put the entire Almaden database research team on the project, with a goal to both replace the OS/2 DBM optimizer and execution engine with Starburst (Database research project) components and to help solve the scaling and stability problems we were currently experiencing in the field. Taking a research code base is a dangerous step for any development team, but this proposal was different in that the authors would accompany the code base. Pat Selinger of IBM Almaden Research essentially convinced us that we would have a world-class optimizer and execution engine and we would have the full-time commitment from Pat, Bruce Lindsay, Guy Lohman, C. Mohan, Hamid Pirahesh, John McPherson and the rest of the IBM Almaden database research team working shoulder to shoulder with us in making this product successful.
The decision was made to take this path. At around the same time we were making that decision, we had just brought the database up on the RS/6000 and discovered that it was capable of only 6 transactions per second measured using TPC-B. The performance leader on that platform at the time, Informix, was able to deliver 69 tps. This was incredibly difficult news in that the new Starburst research optimizer, although vital for more complex relational workloads, would have virtually no impact on the simple transactional performance of TPC-B.
I remember feeling like quitting as I thought through where this miserable performance would put us as we made a late entrance to the UNIX database market. I dragged myself up out of my chair and walked down to the Janet Perna’s office. Janet was the leader of IBM Database at the time and responsible for all code bases on all platforms. I remember walking into Janet’s office and, more or less without noticing she was already meeting with someone, and blurting out “we have a massive problem.” She asked for the details and Janet, typical of her usual “just get it done” style to all problems, said “well, we’ll just have to get it fixed then. Bring together a team of the best from Toronto and Almaden and report weekly.” Janet is an incredible leader and, without her confidence and support, I’m not sure we would have even started the project. Things just looked too bleak.
Instead of being a punishing or unrewarding long march, the performance improvement project was one of the best experiences of my life. Over the course of six months, the joint Toronto/Almaden team transformed the worse performing database management system to the best. When we published our audited TPC-B performance later that year, it was the best performing database management system on the RISC System/6000 platform.
It was during this performance work that I really came to depend upon Bruce Lindsay. I used to joke that convincing Bruce to do anything was nearly impossible but, once he was believed it was the right thing to do, he could achieve as much by himself as any mid-sized engineering team. I’ve never seen a problem too big for Bruce. He’s saved my butt multiple times over the years and, although I’ve bought him a good many beers, I probably still owe him a few more.
The ad hoc Toronto/Almaden DB2 performance team did amazing work and that early effort not only saved the product in the market but also cemented the trust between the two engineering teams. Over subsequent years, many great features were delivered and much was achieved.
Many of the OS/2 DBM quality and scaling problems were due to a process model where all connected users ran in the same database address space. We knew that needed to change. Matt Huras, Tim Vincent and the teams they led completely replaced the database process model to one where each database connection had its own process each of which could access a large shared buffer pool. This gave the isolation needed to run reliably. They also kept the ability to run in operating system threads, and put in support for greater than 4GB-addressing even though all the operating systems we were using at the time were 32-bit systems. This work was a massive improvement in database performance and stability. It was a breath of fresh air to have the system stabilized at key customer sites so we could focus on moving the product forward and functionally improving it with a much lower customer support burden.
Another problem we faced with this young code base, originally written for OS/2, was that each database table was stored in its own file. There are some downsides to this mode but, generally, it can be made to work fairly well. What was absolutely unworkable was that no table could be more than 2GB. Even back then, a database system where a table could not exceed 2GB is pretty close to doomed.
At this point, we were getting close to our committed delivery date and the collective Toronto and Almaden teams had fixed all the major problems with the original OS/2 DBM code base and had it ported and running well on AIX. We also could support other operating systems and platforms fairly easily. But, the one problem we just hadn’t found a way to address was the 2GB table size limit.
At the time I was lead architect for the product and felt super-strongly that we needed to address the table size limitation of 2GB before we shipped. I was making that argument vociferously but the excellent counter argument was we were simply out of time. Any reasonable redesign would have delayed us significantly from our committed product ship dates. Estimates ranged from 9 to 12 months and many felt bigger slips were likely if we made changes of this magnitude to the storage engine.
I still couldn’t live with the prospect of shipping a UNIX database product with this scaling limitation so I ended up taking a long weekend and writing support for a primitive approach to supporting greater-than-2GB tables. It wasn’t a beautiful solution but the beautiful solutions had been investigated extensively and just couldn’t be implemented quickly enough. What I did was implement a virtualization layer below the physical table manager that allowed a table to be implemented over multiple files. It wasn’t the most elegant of solutions but it certainly was the most expedient. It left most of the storage engine unchanged and, after the files were opened, it had close to no negative impact on performance. Having this code in our hands and it being able to pass our full regression test suite swung the argument the other way and we decided to remove the 2GB table size limit before shipping.
When we released the product, we had the world’s fastest database on AIX measured using TPC-B. We also had the basis for a very available system and the customers that were previously threatening legal action became happy reference customers. Soon after, we shipped the new Starburst optimizer and query engine.
This database system became quite successful and, after working on it for many releases, it remains one of the best engineering experiences of my life. The combined Toronto and Almaden teams are amongst the most selfless and talented group of engineers with which I’ve ever worked. Janet Perna, who headed IBM database at the time, was a unique leader who made us all better, had incredibly high expectations, and yet never was that awful boss you sometimes hear about. Matt Huras, Tim Vincent, Al Comeau, Kathy McKnight, Richard Hedges, Dale Hagen, Bernie Schieffer and the rest of the excellent Toronto DB2 team weren’t afraid of a challenge and knew how to deliver systems that worked reliably for customers. Pat Selinger is an amazing leader who helped rally the world-class Almaden database research team and kept all of us on the product team believing. Bruce Lindsay, C. Mohan, Guy Lohman, John McPherson, Don Chamberlin, the co-inventor of the Structured Query Language, Hamid Pirahesh and the rest of IBM Almaden database research team are all phenomenal database researchers but they are also willing to roll up their sleeves and do the sometimes monotonous work that seems to be about 90% of what it takes to ship high quality production systems. For example, Pat Selinger, an IBM Fellow and inventor of the relational database cost based optimizer, spent vast amounts of her time writing the test plan and some of the tests used to get the system stable and gain confidence that it was ready for production.
IBM continues to earn billions annually from its database offerings so it’s hard to refer to these code bases as anything other than phenomenal successes. An argument might be made that getting to a single code base could have allowed the engineering resources to be applied more efficiently. I suppose that is true, but market share is even more important than engineering efficiency and what would have helped grow market share faster would have been to focus the database engineering, marketing, and sales resources on selling DB2 on non-IBM server platforms earlier and with more focus. It’s certainly true that Windows has long been on the DB2 supported platforms list, but IBM has always been most effective selling on its own platforms. That’s still true today. DB2 is available on the leading cloud computing platform but, again, most IBM sales and engineering resources are still invested in their own competitive cloud platform. On this model, IBM database success will always be tied to IBM server platform market share.