Possibly random thoughts of a oddly organized dba with a very short attention span

6.15.2009

on the importance of being agile

If you're expecting something in favor Agile development, you're on the wrong blog. You'll notice that my 'agile' uses a little 'a' - I'm referring to the adjective agile, defined on answers.com as:

1. Characterized by quickness, lightness, and ease of movement; nimble.
2. Mentally quick or alert: an agile mind.

or in the Merriam-Webster dictionary:

1 : marked by ready ability to move with quick easy grace <an agile dancer>
2 : having a quick resourceful and adaptable character <an agile mind>

The software industry needs to be agile. We need to adapt to changing tools, technology, requirements, targets and more, plus we need to be able to respond faster than the other guy or we may miss out on an opportunity. The industry knows that speed is important, and the recognition of the need for speed has had both positive and negative effects. There have been some astounding innovations delivered to market much faster than anticipated, and there has also been a boatload of code released well before it's time, leading to failed projects and products. I'm not going to try to mention which products fall on which side of the line or I'll get sidetracked and forget the whole point of this post. (although I suspect that hardware innovations tend to fall in the first category, software in the second) The point is, agility is an advantage and the software development environment should support and encourage the ability to respond with quickness, but quickness should never upstage effectiveness. I like the Merriam-Webster definition - 'having a quick resourceful and adaptable character' or 'ready ability to move with quick easy grace'.

Generally, when people come up with a new process, it's because the old process is broke. Either the process is failing to provide the desired results OR the process has become such a burden that the results are no longer worth the effort required to maintain the process. Because I prefer to be an optimist about people's motivations, I suspect that Agile development was created with the hopes of improving the software development process. However, the current reality looks like something you would intentionally foist onto a competitor, hoping to completely derail their effectiveness.

So where is the balance? We need to be resourceful and adaptable, aka 'agile', but if we lose the foundations of good development, the resulting product will be of poor quality or even if the first product version is successful, our ability to 'move with a quick easy grace' on subsequent development will be lost.

Of course, I'm a data person so I'll admit to being a little biased, but data is king - applications are built to capture, record, share and manage data. Data doesn't exist so we can have applications. The life span of different bits of data can vary widely: there is data that can live forever, there is financial data that is retained according to law and then there is streaming-type data that may only be significant for a few moments. No matter what the life cycle is, the most critical portion of an adaptable development process is understanding the data and the relationships between the data elements. And the best method of understanding and displaying data that I know is an old fashioned, third normal ERD. I'm not trying to start a battle over the importance of using third normal form or not, but if you are going to denormalize, that decision should come after you understand the normalized view of the data.

Once you understand the data elements, then it's time for data flows. Doing it the other way around doesn't make sense - how can you understand the flow of a data element before you understand the element itself? This is the first place many projects go wrong: the focus is on the flow.

Example: my first Oracle database was a tool for manufacturing to Request Engineering Action. (REA) Data elements included engineering document numbers, manufacturing part numbers, dates, descriptions of issues and the impacted teams. The tool needed to process a request through the appropriate work flow, depending on the functional area of the plane. (structural, hydraulic, electrical) Reports had to track the progress of the request, the number of requests for categories and team, and provide the information on the fix back to the requester.

The application was successful: management appreciated the tracking data, engineers found it easy to use and the manufacturing guys were getting the answers they needed faster. I was approached by a team that worked the flight line: they had something called an Integrated Problem Report (IPR) and they were wondering if they could get an app built for it. They insisted that their document was completely different from the REA: it involved integration issues and the process to review and sign off on the document was completely different. I looked at it from the data perspective. There were a few additional data fields, but the core information was the same: doc numbers, part numbers, dates, teams, people and descriptions. I suggested that they could just use the REA system if I added a few additional supporting tables, but they strongly disagreed, their process was better better and more important than the REA. Rather than argue, I added the tables to the REA database and added a code to the primary table that identified the document type. I built new screens, using all the IPR data labels and set both applications to use the document type to determine which records were displayed. The work flow option were also set to check the doc type. Simple - the flight line team had a new application in about a week. I didn't tell them they were in fact using the REA database and the 95% of the data was common but they were happy.

Management was happy too. Now they had tracking reports the showed how many IPR's were affecting production and the IPR workload each team handling. Perfect ... except for one thing. Many teams handled both IPR's and REA's - management wanted reports on both. Once again, a new use of the data was easy. Clone a subset of the existing reports, set the report to read both document types, and it was done. Almost immediate turnaround on something everyone expected to be a major undertaking. At this point, the flight line guys were told they didn't really have a separate database, and this started a discussion that led to the combining of the two systems, so eventually we eliminated a redundant system.

Had we focused on the data flow, it wouldn't have turned out this way. The flows were different, time requirements were different (once the plane is on the flight line, everything moves fast) and some of the teams were different. However, a data centric view revealed that the two processes were more alike than different, and since the REA foundation was solid, adding to it with a resourceful approach meant we could move quickly and gracefully to a new iteration. Hence, an 'agile' project. Had I followed what currently is described as 'Agile' development, there would have been two distinctly different databases and creating the high-level reports management eventually requested would have required joining data from two systems (eventually three, as another app was worked in later) Possible, but it would have taken longer and been a more complex undertaking.

Customers will always change their requirements - it's a given. While business people who participate in development planning understand their data, they seldom see the potential for everything that could be done with that data until the first few releases of a new tool. If we start with a data centric view and a model that represents the reality of how that data is used, doing more with the data later is easier. If our development is application focused, then when the customer requests a different use of their data, we're starting from scratch. Build your data model first, understand all the data, even the pieces the customer isn't asking you to use ... yet. Then, if you must, use Agile Development process to build the application.

ps. I still don't see the advantage of the Agile development process. Perhaps whatever original process the creator was trying to replace wasn't working and they really did intend to streamline the process. Unfortunately, the end result is just a different kind noose around the development team's collective neck. Anytime the process requires more effort to manage than the actual work being managed, it's time to do some serious process re-engineering...

9 comments:

James Barton said...

I'm not totally pro-Agile, but I think some of the practices are very valuable. The most obviously useful one rom a business point of view, I think, is weekly releases - the fact that nothing gets marked as "done" until it's seen in working in a recent build means that there's much more confidence in project statuses.

John Brady said...

Robyn - Another good post, which I agree with wholeheartedly. "Data is king" as you say.

Another result I see of poor or non-existent data modelling and design is bad scalability. Some developer puts together a few tables with their data as their application processes it (flows again), and demonstrates that it works. Everyone else says that it is good enough (Agile?) and so it becomes permanent and makes it through to the final release. Sometime later this gets deployed and the data volumes grow, and then performance hits that dreaded knee and response times go up, and up and up. And its not an "Application" problem, it is a "Database" problem because all the SQL statements are taking so long to execute.

At this point they get a Database type person to review the data model and the tables, and they can see why it does not scale. The joins are horrible (too many columns), data type choices are bad (dates in character strings), the data is spread over too many tables leading to multi-table joins, and so on.

Solution? A redesign of the data model, as it should have been done in the first place (of course). However, the application team and the support team will kick and scream for short term fixes - an index, or a SQL rewrite - anything to make it suddenly scale.

I came to the conclusion a long time ago that Performance and Scalability is not something you 'bolt on' to an existing application. It is something you design into it from the beginning. In the same way that Safety and Security is designed into many products - cars and planes - from the start, and not just added on at the end as an afterthought. (Look at the problems with Windows and Security, because Microsoft assumed that no security was ever needed in a single user operating system).

John
Database Performance Blog

Martin said...

What makes you think that Agile (with a capital A as you wish) 'prescribes' a data flow perspective? I am an agile practitioner (note the lowercase a) and there's nothing non-agile about thinking about your data first.

Robyn said...

Hi James,

I'm not trying to suggest that everything about Agile development is bad. From what I have seen, the data model is neglected and that will lead to problems regardless of the development approach the team uses.

Weekly releases could be useful from a business point of view - it helps the project manager maintain visibility of the coding progress. However, it is not an approach I would use. IMO, arbitrary due dates result in developers being overly preoccupied with getting it done, while forcing them to shortcut on quality of the final product. And I want better than 'working in a recent build' for our products - I want the code to be optimized and scalable. I'd rather wait a little longer and know I'd given the developer time to explore and test. Once a complex piece of code has been written and released, I find that people are uncomfortable making major changes to it, and that can leave you stuck with suboptimal code.

Good to hear from you again. Hope to see you at UKOUG again next year.

Robyn

Robyn said...

John,

Your comments highlight exactly the problem I have seen, and am afraid I might see again soon. One of the older apps has a sub-optimal data model, yet revising would be a huge undertaking. A new app is being built on the same data model. I haven't seen the details and it's possible that the issues in the old system won't crop up in the new one, depending on which data elements are in use. This was an Agile development project, and there was no process (and no time) to reevaluate the data model.

It's a little like sitting on the tracks and knowing a train is coming - just don't know exactly when :)

thank you for the comments ... Robyn

Robyn said...

Hello Martin,

I would love to hear that there are agile/Agile projects out there that have a data first perspective. That was my original question in the last post: does Agile development really discount the data model, or has the intent of Agile been misunderstood by the implementers of Agile? Because from what I've seen, the data model is being neglected.

My premise is that starting with a data centric approach, and then moving on to a development process that remained cognizant of the need for adaptability, scalability and performance would result in an agile development process. If Agile works for a specific team or project, use it, but don't let the desire to follow the process overtake common sense. Otherwise the process itself will hamper your agility.

I'll follow up shortly by sharing about a project that I'm working on that is achieving agility and I'd like to hear more from you about how your team handles data models within Agile development.

cheers ... Robyn

Martin said...

Robyn,

If your process blocks common sense, that would sure be an anti-pattern.

I get the impression that you feel that agile means that speed is more important than quality.

On the contrary - agile maintains speed by delivering quality. One of the key principles in agile development is "sustainable throughput", which more or less means the team should be able to deliver approximately the same amount of functionality each iteration. This makes delivery predictable (which is usually not the case).

However, I'm a big fan of being able to deliver 'slices' of functionality: maybe the software does not do everything, but what works, works perfectly. This enables high quality feedback - users are able to give feedback based on real software.

A big challenge for the team is imho to help slicing functionality, which may also lead to a 'sliced' data model.

Of course this should not stop you from thinking ahead - agile is no excuse to stop thinking more than a few weeks ahead.

Robyn said...

Martin,

I think you've misunderstood my post. I don't feel that agile means speed is more important than quality. On the contrary, I'm saying that quality and agility go hand in hand.

I'm also not suggesting that using Agile (capital A) development methods always results in speed being valued over quality. I am saying that I am frequently tasked with dealing with performance and scalability issues that result from the lack of a good data model and that many of those systems were developed using Agile methods. Since I wasn't part of the original project, I don't know why the data model was neglected so I'm curious. Is this a common occurrence with Agile development?

Delivering slices of functionality makes perfect sense, especially if each slice 'works perfectly' before adding the code to a production release. That's the approach we're using on a project that is resulting in a well designed database capable of adapting to our customer's changing requests.

But I don't expect code packages to be delivered in consistent intervals. Some functionality is more complex and requires more planning, coding and testing, and deadlines should be adjusted appropriately. If developers are already working hard to complete their slice in a week, how likely will they be to test an alternative approach that could take more work up front, but will scale better in the long run?

Robyn

Allen Shatzer said...

Another great post, Robyn. I believe the problem is in the way that teams choose to implement and "Agile" process. I totally agree that a data centric approach and involvment of a good data modeler / DBA / Developer is important for long term success of an application.

Search This Blog

Loading...

My Blog List

disclaimer ...

The views expressed on this page are mine and do not reflect the opinions of my employer, former employers, other associates or the opinions I may hold 3 days from now.