I think I've mentioned this somewhere on my blog before, but last year right before I left for the Miracle Oracle Open World conference, I found myself unexpectedly responsible for a new system. Up until that point, I was working in an 'advisory' role for the project - available to answer questions, offer guidance, etc. I had no authority to direct the work, it was just assumed that the other people involved would actually listen.
About two days prior to my flight, I got my first look at the complete package and it was horrifying. Up until that point, I'd been told everything was fine. The contractor, a self-proclaimed DBA and developer, had written code that address the primary function of the system and it was working. Everyone was happy ... until they realized there was a performance problem. No big deal - they'd have me handle it because they'd 'heard that I was good at that kind of thing.' But this problem went all the way to the foundation: the hardware was wrong, the data model was flat and the code was poorly written. Give me any one of those three issues, and there's something that can be done to make a process faster. It might even be possible to improve the performance enough to meet an easy target in the presence of two of them. But all three? We were doomed and now my name was listed next to this mess as the 'architect'.
This meant I was more than a little bit distracted while at the conference. I didn't know what I would do to fix this situation, but I knew I was going to have to do it with the system up and running and supporting customers. Sure, there would be maintenance windows available but that's not much time when you basically need to change the entire application out from under an existing system. To make matters worse, we'd already been looking for new people and I'd seen no one with the qualifications we needed.
I found myself so distracted, I was having trouble even thinking about the presentations I needed to give. The variance presentation would be fine: I had spent a great deal of time over the past several months testing and measuring results so I practically talked about that one in my sleep. But the root cause presentation - suddenly I wasn't happy with it at all and I knew why. This new application was going to present a whole host of reoccurring problems and I felt powerless to stop it. I hadn't built it but I was going to be on the front line for it, and that ticked me off.
In my angry and somewhat retaliatory mood, I envisioned a new version of my presentation:
The Root Cause of ALL Evil. I would detail every last thing that was wrong about the server, the application, the methods, the assumptions and the contractor. I would have just one slide - a picture of the person that had developed it. I'd channel
Lewis Black and get all my pent up irritation out and I'd feel better. Except I wouldn't really, because pointing at the mistakes wasn't going to change anything and I still had to fix it when I got home.
In the end, I did change the presentation. All that churning about it gave me some ideas for how I could do it better - but I stuck to the original message of applying fixes that will prevent problems from reoccurring, and that was good thing.
But ... at the end of the week, sitting at The Oak Table with several very good friends, I told all the dreadful details, all the problems and all about this contract DBA/developer that I now referred to as '
The Root Cause of ALL Evil'. It made for a pretty good story and better than that, telling the story led to a real solution.
But back to the title of this post ... Why the public apology? You see, after many, many months, I have learned that this particular person was NOT
The Root Cause of ALL Evil. Sure, I still have many issues about how things were done, but
TRCoAE is a pretty heavy label. Plus I have since learned that the mistakes that were made are not all that uncommon.
In truth, this person did exactly what someone had told him to do. He was given a spreadsheet style data set and an interface document, and he wrote code to perform the work described. And the database did perform the function. Very, very, very, very slowly, but it did it.
In hindsight, the people responsible for him got
exactly what they asked for: they had a database on a big, shiny, expensive machine that performed one function. Ok ... one function and two minor data retrievals, but basically
ONE FUNCTION. Months and months of work, lots of money and effort, all to be able to perform one function at the speed of molasses in Iceland.
And the real kicker is that the source data is extremely valuable. There is data that is critical to the customer and very useful, as in core business, money making useful. There was so much more that could be done with it, if only someone had looked at the data for what it was and remembered one simple, basic truth:
Databases exist for the protection and presentation of data
If you don't take the time to recognize the potential within the data, functionality will remain limited and future development will be painful. Most DBAs know that our primary responsibility is the protection of the data, and we associate protecting the data with backups and security. But if you don't think about the value of the data in your trust, you can't optimize the usage of information, and that is part of the job too. In our case, some of those neglected data values were the clues that would have resulted in the database performing it's one function in a reasonable amount of time. Instead, those columns did nothing except take up space and slow the data retrieval down. This wasn't apparent until the details of the data were documented and recorded in a real data model. Not the object model, not an interface document, a good old fashioned data centric view of the data. Most importantly, those ignored data points were the keys to what the customer would need from the system next. Understanding what those values represented and storing them in a third normal form allowed us to respond quickly to requests for the next phases of development.
The data model didn't exist until two months after I returned from Denmark. Once done, it required a great deal of work to transition from the old to the new, but the database is a much better product because it happened. Ten months later, it now performs many, complex functions and it performs them much faster, even though it's still stuck on the old hardware. It also performs the original function 40 times faster.
So I hereby apologize most sincerely for having referred to the former DBA/Developer as the '
The Root Cause of ALL Evil' when he was not
solely responsible for the original evil. I'm sorry I imagined the presentation that I never gave and I'm sorry about all those other things I said (even though they were true). To make amends, his name from this point forward shall be '
The Propagator of Small Evils'. I don't know exactly who or what the
TRCoAE label belongs to in this particular case, but that's not important anymore. We're focused on the solution and there's more work coming ...