http://en.wikipedia.org/wiki/User_story

Apparently a user story is nothing more than using words to describe the requirements for someone else to use when building software. I also noticed the mention of using user stories with Agile software development. And I had to stop and think to myself, “The Hell? Do these people think this is new?” If you judge everything just from this wikipedia entry, you might think that a user story is some revolutionary new idea that has evolved from Agile within the past ten years. Well, it isn’t.

Go back to Binoogle and look up “Voice of the Customer” or “VOC”, and check on any Six Sigma references. See, getting feedback from customers, or users, has been around for a long, long time. Probably for centuries. Yet it would appear that only recently have some in the software sector suddenly found out that getting such feedback could actually streamline the development process. Talk to the user before I build something? Genius!

Not sure why I am surprised by this fact but apparently everything old is new again. Even in software development.

“Papa, why [whatever]?”

“I don’t know.”

“Why?”

“Because I don’t know everything.”

“Why?”

“Because no one person can know everything.”

“Why?”

“I don’t know.”

And we would continue that loop a few times until they grew tired of getting the same answers about how I don’t know everything and no one person can know everything. Turns out that my five and six year old were only just showing that they have incredible skills when it comes to Root Cause Analysis (RCA). Asking “why?” is a fundamental part of an RCA as it can help to identify potential root causes.

5 Whys

The ’5 Whys’ is a technique used in RCA and Six Sigma and one we reviewed yesterday during training. I believe that all too often we find ourselves affixing band-aid solutions to symptoms, especially in IT, as opposed to finding and fixing the root cause. The process is quite simple, you just keep asking “why?” until you get to the root cause. So, you may find something like this scenario:

My dedicated log disk filled up last night. Why?

Because someone tried to load ten million rows all at once instead of batching the load into smaller chunks. Why?

Because the package they use for their ETL process does not include logic to batch a transaction size. Why?

Because they do not understand where the option is, or why it is important. Why?

Because no one has explained it to them. Why?

Okay, fine, I’ll go over there after I finish my coffee if you promise to leave me alone. Why?

Stop it. Leave me alone. Why?

And so on until your head will explode. Typically after asking “why?” for the fifth time you might be at the root cause but asking five times is only a rule of thumb. Sometimes you need to ask more, sometimes you can ask less. The key here is to drill down into discovering a potential root cause. You should not consider this tool to be able to easily identify the root cause each and every time. In fact…

Why 5 Why May Not be Effective

Look, no tool is perfect. And the ’5 Why’ tool is not all unicorns and rainbows either. As much as I can appreciate the simplicity of the technique the fact remains that it has a couple of glaring issues. First, it is not repeatable. That means that two different people may have two very different results. Consider the example above, but this time from the viewpoint of application support as opposed to server support.

My dedicated log disk filled up last night. Why?

Because someone tried to load ten million rows all at once instead of batching the load into smaller chunks. Why?

Because the bank sent us a huge ass file instead of the normal file. Why?

Because their process created a huge extract by mistake. Why?

I don’t know, but they said it would not happen again. 

So, a different perspective can lead to a different conclusion altogether. To me that demonstrates that the tool itself is subjective to the user, which means it may not be ideal. However, there is something positive from that as well, and that is if you get the different people together in a room you can start to list out all potential root causes and work together on a solution rather than in silos.

Another issue with the tool is that people are only able to find causes they already are familiar with. In the two examples above we reached two different conclusions, but each conclusion was the result of that persons viewpoint and experience. In other words, you know what you know and you don’t know what you don’t know, and the ’5 Why’ tool will not give you insight into things you don’t know. This also helps to explain why the tool is not a repeatable tool.

A third drawback would be that as a result of the tool only allowing people to find causes they are familiar with means that they will often times stop at symptoms instead of continuing on to the actual cause. All three of this drawbacks I have listed are really just variants of each other, but they are all valid considerations to make when using this technique.

I think the most important aspect to all of this would be to remember that you are only looking for potential root causes. At best, that is what you will find with this tool. And if you keep in mind that the tool is only going to help you get closer to your desired goal, and not the tool that definitively gets you there, then you should be fine. Just consider it as one piece of the puzzle, and not the whole picture.

Fortunately for me I was reminded by Louis Davidson that it is not the feature I dislike, it is the abuse (or misuse) of that feature. This would not be the first time that Louis has reminded me that I was (or about to) make an ass of myself, so I took note of his words very carefully. Indeed, it is not necessarily database triggers that I hate, but rather the fact that they are abused…misused…actually, I hate the fact that they are used and it is so difficult to troubleshoot.

But NULLs, well, they are a different animal. Why do I say that I hate NULLs? I mean, there is nothing to them, right? They do not perform any actions like triggers, they just sit there, in your tables, doing nothing. Some of them are waiting to be discovered and some hope to be updated to have real value. It makes me think of that old Schoolhouse Rock video about that little scrap of paper. So, why so much dislike for them? What harm do they cause?

Why NULLs Can Be Harmful

The harm they cause is usually the result of a person not knowing they are there. It’s that simple, and that is why I hate them. OK, not hate, but I would rather design a database with as few NULL values as possible.

Oh sure, I understand that there are times when you do not know the value of something, and it is easy to simply define that particular column to allow for NULL values and you will come back later to update that value. Sure, sounds great. I always hear about people that plan on coming back later to fix the things they didn’t do right the first time around. I only wish I could say that I actually see that happening as often as I hear it is going to happen. So, you put a column into a table, allow it to have NULL values, and are asking yourself right now “WTF? What’s your problem?”

Again, it is because the end user is not always aware that NULL values exist. And sometimes, the end user is a developer (*ahem*) who may be brought into the middle of a project and be building some widget only to have the widget crap out once a month because their code never accounted for NULL values that get inserted (or updated) after a monthly batch load. Not that I know anyone that has ever experienced such things, but I hear about these people (*ahem*). What’s that? Oh, so my hatred for NULLs is really a hatred for the fact that I am not  someone is not good enough at coding or T-SQL or database design to begin with? Maybe, but consider this…

Query Tools

What tool do you use to write a majority of your T-SQL queries? OK, do you know what the default connection properties are? What I am looking for is: Do you know what your ANSI_NULLS is set to? How about this question: Do you even know what ANSI_NULLS setting you want to use? Did you know you even had that option? Do you know the difference in behavior if ANSI_NULLS is set to ON or OFF? I have a handful of tools at my disposal and surprisingly they have different default settings when making connections to an instance. Some of those differences can be with ANSI_NULLS, which makes no sense to me because the whole idea around ANSI is to have a standard so you would think the rocket surgeons in charge would have thought about mandating a default ANSI_NULLS setting whenever someone makes an application that wants to connect to an database server in order to avoid crap like I have had to deal with. Sure, you want to be able to change the option, but the behavior out of the box should be a standard, in my opinion.

Now, how many DBA’s know everything there is to know about ANSI_NULLS, connection strings, the various applications being used to connect to data these days, and each and every database design in their shop? I have no idea how many of you just raised your hand, but I know mine stayed on my keyboard. Now, let’s go one level away from the DBA, and ask: how many customers understand all of this? I supposed that you could only have your users connect to pristine cubes or data marts, right? Good luck with that, because I know most people want to connect to the raw data so that they can go through it themselves. And guess who will get blamed if it turns out that data is “missing” because the end user missed the fact that a NULL value altered their result set?

So, if database designs minimized the use of NULLs, then we can minimize the amount of issues faced downstream. Ever since I took part in some Six Sigma Black Belt training I have come to understand that what I really hate are inefficient processes. The use of and troubleshooting triggers is inefficient. And the same case can be made for NULL values. Their use can lead to a lot of bad result sets being accepted as a “version of the truth”, which is something we will ultimately be on the hook for at some point. Why not save people some time by looking to minimize their use? That would seem to be more efficient for everyone.

So, maybe I don’t hate NULLs. I just hate how so many groups are all operating under the assumption that everyone knows what they are, why they are there, and how best to use them. Seems to me that with all of those assumptions chances are lots of bad business decisions are being made every day.

Wait a minute…maybe we could blame the economic downturn on the use of NULL values? Hey, maybe I found a reason to love the little buggers after all.

Naysayers will always exist, no matter what.

By now your first question is most likely: why Congress? Well, it is simple. Have you ever been invited to go somewhere, say golfing or a ballgame, and found that you need to check to see if you can get the time off? Not only from work, but from your family as well? My group of coworkers started to refer to our better halves as “Congress”. That way, we could say things like “I wanted to go to the ballgame, but I couldn’t push that bill through Congress”. Or, “I was hoping to buy that new [whatever], but Congress squashed the funding. Next time I will need to hire some lobbyists”. There, I hope that clears everything up for you. Back to the topic that matters.

So Congress asks me for some help putting together a brochure of sorts to be used at an upcoming trade show. She starts by explaining that she wants the customer to feel a certain way after reading the brochure. With my recent Six Sigma training, I translated her words directly into a few acronyms and settled on VOC (Voice Of Customer). At that instant I got up, went to my desk, grabbed a stack of post-its, and came back to the kitchen table.

“We need to gather as many one-word descriptions that you want to convey to your customers in this brochure.” Congress immediately wanted wanted to know what the hell was going on, since she had never seen me so focused on slapping post-its to the kitchen table before. So we spent about twenty minutes coming up with roughly fourteen descriptions that we wanted to convey, things like “professional”, “easy”, “beauty”, “service”, etc.

The next step was to see if the descriptions had a natural affinity. In other words, to group the descriptions together. Once that was done, we went about putting together some actual sentences. We got two or three done fairly quickly, and then decided to benchmark against similar companies by reading their brochures. In about two hours we had all of the meat there, and then spent the rest of the night haggling about order, flow, etc.

The whole experience was eerily similar to what I had read about in the Jelly Effect. In that book, Andy Bounds describes a process for putting together better presentations, and the methods he uses are similar in nature to what we did last night. I have been trying to follow the advice in the book in order to help me put together my own talks, and I think it has helped. To have his suggestions reinforced by techniques learned in Six Sigma training is wonderful. To find myself actually putting them into practice is amazing.

A long time ago I was asked if I would rather be a brick or a sponge. Bricks are hard to change, but sponges can be altered by simply putting them into a bucket of water. Sure, it is still a sponge, but it is also something more than it once was.

I have often been told that I am a sponge, that I absorb a lot around me, and that is one of the reasons I made a good Database Administrator. It is also one of the reasons why I can find myself getting thrown into difficult situations in life but manage to make my way out. To me, the experience last night is just further evidence of my being a sponge, and one of the reasons why I hope to be able to apply everything I learn in Six Sigma training to my everyday lives.

Now, before I go any further, I want to make it clear that I try my hardest to avoid using acronyms whenever possible. Six Sigma has acronyms for everything. There is DMAIC, SIPOC, VOC, VOP, CCR, CTQ, and the list goes on. Many of these terms are only found in a Six Sigma context and it can be hard to follow your first time through. That being said, I will do my best to make certain I explain the acronym before just assuming that the reader knows what it means.

DMAIC stands for Define, Measure, Analyze, Improve, and Control. Those five elements comprise most every type of Six Sigma project (I actually do not know of any that would not have them, but anything is possible). At some point during the week I started thinking about how most DBAs I know already apply these five principles to most of their everyday tasks.

For example, let’s assume that someone (call it a customer), contacts you (as a DBA) and says something akin to “The system is slow. It needs to be faster. Please fix.” Now, being the top-level DBA that you are, you get to work. And what do you do?

First step is to define the problem. You start asking questions, trying to get as many details as possible. What system? Which server(s)? Which databases? What do you mean by faster? When did you want/need this done?

Then, you measure the problem by seeing for yourself how “slow” the system is responding. Again, you ask more questions, looking to further define the issue. Most DBAs just lump this into “Performance Tuning” for the example we are discussing, but in general you would look to gather your own sets of measures for whatever the problem may be. Without any measurements, how would you ever know if things got better, right?

Then, you analyze your results. In most cases the analyze phase takes place during the measure phase, but the best DBAs (in my opinion) would gather as much information as possible, take a step back, and then analyze the data in front of them. Your analysis would lead you to an action that you would take in order to do what?

Well, improve the system, which is the very next step. You would apply your analysis in the previous step, formulate a plan of action, and implement the action in an effort to improve the situation. Perhaps an index is missing, or needs to be rebuilt, for example.

Lastly, you would put a control in place to help identify if the system acts slowly again in the future. Perhaps the controls involve configuring a new set of metrics to help you with any additional measures and analysis that needs to be done if things break again. Or your control could be as simple as a report that shows how long the system is running. That way if the customer calls again to say it is slow you can respond back with “our control report says it took five seconds less than the average run time, so any slowness  is either due to a different issue or your (lack of) perception”.

I could go on, but the point here is that many of the Six Sigma processes are built upon common sense (which is not always common), and many of the DBAs I interact with already act in a similar fashion to those processes. Many of us follow a DMAIC process without knowing anything about Six Sigma. And while not every DBA could simply jump into a Black Belt program (although if I can do it then I am 99 and 44/100% certain anyone could), chances are most DBAs would find the transition quite easy. If nothing else, you would be able to follow the logical nature of the processes and probably not be scared by the math involved.

If there is one thing that almost every Black Belt has that most DBAs need, it is a good set of soft skills. If I can take away tips and tricks for a better set of soft skills by the time this is all done, that will be as valuable as anything else I may learn along the way.

This got me thinking about many, many, many meetings I have been involved with over the years. I have always been aware of the difficulties in working within a group dynamic, but seeing this blog made me realize that I do not spend enough time reviewing how to work with people effectively, especially in a group setting. Those soft skills can set you apart from the herd, which is very important when you need to make your ideas known, or more importantly to make your ideas popular and accepted.

How many times have you known the right course of action on a subject, but were not able to get a group to see things your way? Understanding how group dynamics play a role is the first step toward success in that area. Some people try to get their points across forcefully, and will simply raise their voice, or call into question your merits on the subject, putting you on the defensive. My former boss was very good at handling such situations. He would remain calm and simply re-explain his position. I do not always have the same level of patience, but I am trying to get better.

I am usually a good devil’s advocate, I can see both sides of an issue and often take an opposing view simply to engage someone to understand more about their point of view. I remember one time doing an exercise where we were paired off and given a topic to debate, then we had to switch and argue the counterpoint. I was paired with my supervisor at the time and I won both arguments, which either says something about my skills at persuasion or says very little about hers. At any rate, it was at that moment that I understood that any issue has two sides, and you need to see things from both points of view.

But then I come across this blog and I now see that issues often have more than just two sides. In this case, any one issue could have three or more sides, as each group could see an issue from a distinct point of view. I suppose you could expand this and say that one issue could have so many different points of view that bringing everyone to the same table could make discussions very, very difficult.

I guess that explains why the UN is often times quite useless. If you cannot get twenty people working for the same company to decide upon some basic design concepts, why do we expect the UN would ever be able to solve any issue presented before the Security Council? I am amazed that they are able to function at all. Same thing can be said of Congress, or any Government body. I guess at some point you need to have a person in charge and let them make the decision and everyone follow, hoping that the decision was made with the best intentions for everyone.

Another point of reference with regards to group dynamics can be found in the book “The Jelly Effect“. I read this a few months back and found it fascinating with regards to networking, public speaking, working with groups, etc. If you have some time it is a quick read and well worth the trouble.

I think the most important piece of all of this information is that people will react to you for a variety of reasons. If you start to develop skills that allow you to interpet these reactions, or anticipate them, then you are well on your way to being an effective communicator in any type of setting.

I came across a blog and thanks to my Bat-Google Reader I can keep up with the latest posts as they happen, which is nice. Today I found this one to be interesting, despite not offering a possible solution. I find it interesting to see a handful of examples that describes one or more environments we have all worked in or around. How is Six Sigma supposed to solve this? I have no idea. Why do I care about Six Sigma? Well, to be honest, I see very little use of it in the IT industry. That is not to say it does not exist, just that I have not seen (or heard) about it being used. To drill down even further, I know very few DBAs that employ any type of Six Sigma methodology in their everyday lives.

For me, Six Sigma is all about determining what is a signal versus what is noise, and using that data to help make something more efficient. The best example I can think of right now is the simplest, a database dump job inside of SQL Agent.

Say you have a job that runs weekly to do full dumps of all databases on your server. One week it takes thirty minutes, a week later it takes forty-five, and the third week it takes an hour. Now, as a DBA, would you suspect that there is a problem with the server? After all, the time for the dumps has doubled in less than two weeks. There could be many factors for this, of course, but at what point should you decide to investigate?

What I hope to do is to figure out a ‘best practice’ with regards to time management for such issues. Our time is quite valuable these days, so it is important that we research issues only when the need exists, i.e. we are given a signal. So, perhaps we get alerted that the database dump job is taking too long, so we get notified. But it may be possible that despite the increase, the job is running as expected, and therefore there is nothing for us to investigate, so we would not want to be alerted. Less alerts, less distractions, and more time to spend our energy elsewhere.