Tuesday, October 25, 2005

Open Source and IBM

Martin Fowler's review of JAOO2005 includes this blurb:

"On panels 'Bedarra' Dave Thomas and Brian Barry said that they believed the current spate of Open Source development was unsustainable. Much support for open source is funded by large companies, IBM's support for Eclipse is a long one. They felt that this support wouldn't last, if so there's likely to be quite a collapse in open source activity. (I'll stay silent on this question, I prefer to avoid trying to predict the future.)"

All I can say is WHAT!?!? Someone doesn't understand IBM's business model. They are not a products company, they make their money on services. Therefore, they want to sell things based on thier services expertise. By making products open source, they eliminate the competitive advantages of their opposition. That allows them to compete on an even keel, services vs services. If anything, IBM wants MORE open source products so that they can sell more integration services. Sheesh!

The Value of Domain Knowledge

I recently gave a talk to the computer science department at my undergraduate university. I was trying to inform them about the state of the industry and give them a little heads up on what they were getting themselves into. One thing that I think they found interesting was my statement that we are merely problem solvers. We are like astronomers and computers are our telescopes. Obviously, being able to use a telescope is important, but our ability to solve problems is our most valuable asset. Computer Scientists have replaced mathematicians as the problem solvers of choice. While mathematicians have their formulas and proofs, computer scientists have the ability to solve problems with brute force. Being half academic, I pine more than wonder at our barbaric problem solving abilities, but my other half enjoys the "get it done" mentality.

As a problem solver, one of our most valuable assets is domain knowledge. Any newly minted B.S.C.S. can turn specs into code, but to be able to have empathy for the user and understand the domain comes only with years of experience. When you ask the user what he or she wants and then you come up with more items than the user does, you have achieved domain expertise. It is akin to the astronomer who can predict the number and size of planets based on the wobble of a star. To a company, this is the most valuable an employee can get. Not only can they contribute to a company by "tuning the telescope", they can also help analyze the results and even help construct a telescope that better measures what they wish to measure.

Where does this leave consultants? Well, there are two different kinds of consultants, and it effects each differently. Some consultants make thier living off of training others. Technology changes at such a rapid pace that schools cannot provide adequate training for everything that you might ever see. These consultants work hard to stay abreast of best practices and then transfer their knowledge to those who hire them. It is similar to teaching earth-bound astronomers how to operate the hubble. Obviously, they never envisioned the hubble when they were in school, so that new knowledge has to be imparted to them in some manner. This is a valuable service, and we should think of it as an extension of school rather than as a consultant.

The other form of consultant is an expert telescope user. This consultant is paid to be very good at using telescopes. He cannot tell what he is looking at, but the telescope is guaranteed to take the best pictures possible. Obviously, this consultant needs a lot of hand holding. In many cases, it is easier to just tune the telescope yourself; however, there are times when the person reading the pictures no longer knows how to use the telescope. This position has a value, but it is small compared to the other two positions I described.

So, as you evaluate where you fit in on the spectrum, see if you provide domain knowledge, education, or just an extra hand. If it is the latter, don't be surpised to see your hand replaced by a cheaper one. If it is the former, and you are being replaced, then you have to seriously wonder about the health of the company. And if you are in the middle, enjoy the moment, but keep learning for the future.

Dilbert Blog

Here it is, Scott Adam's Dilbert Blog.

Thursday, October 13, 2005

Reuse

Reuse is a good thing right? Give the option to reuse good code or write your own, you would always pick reuse. We've heard tales of horrible programmers and their NIH (Not Invented Here) syndrome and we're not like those guys. We want to leave our pride at the door and hold up the shining shield of reuse and the burning sword of standardization.

Not so fast.

Reuse, like ANYTHING ELSE, has its tradeoffs. On the one hand, you get "free code". On the other hand, its not free because you have to learn how to use it, debug it, and write integration code to get it into your project. On the one hand, you get "free upgrades" as a separate development team makes upgrades and enhancements. On the other hand, you suseptible to changing interfaces, newly introduced bugs, and silent but deadly logic changes. You are dependent on that code and its uncertain future.

Joel defends NIH for good reason. His basic premise is something I've been saying for quite some time: software development is the practice of managing dependencies and the dependencies that are easiest to manage are those that are not there. Just like in life, the more dependencies you have, the less agile you are (those with children: how easy is it to do something spontaneously?) - and in the tech industry agility is everything.

Often the development cost of the product increases when you eliminate dependencies because of some duplication; however, the ability to improve the product quickly also increases and that will gain enough income to overcome the incresed development costs. Now, don't get me wrong, you shouldn't go rewrite Windows in order to not be dependant on it; there is something to be said for interoperability as well. However, if it is core to what you do or it plays a vital role in your product then you should own it and it should have a one-to-one dependency chain between you and it. The cost will be well worth the rewards.

Large Company Dilemma

Once your tech company reaches a certain size, it has a dilemma on its hands. Usually, the problem is that it cannot increase its profitability without taking on more people. However, the people available to take on are not the same calibur as the people already on board. Therefore, the company can either refuse to take on additional hands and stagnate or take on average employees and eventually capsize. There appears to be no other outcome. The difficulty lies in taking average employees and making them great, when I'm not sure that is possible. Like turning lead into gold, it is just not within our abilities, today. Of the two options, capsizing seems to lead to the most profit and the capsizing doesn't necessarily mean the downfall of the business. Instead, it could just mean massive layoffs followed by restructuring as was the case with IBM and Apple. However, it seems to me that there should be a way to fill the traditional product units with the new mediocre guys, because their profit has been extracted and then spin off a new company with the masterminds, letting them focus on creating "The Next Big Thing." Just a thought...

Thursday, October 06, 2005

Double Entry Bookkeeping

I don't remember if I mentioned this on my blog, or someone else's, but I view TDD as double entry bookkeeping. Apparently, Uncle Bob does, too.

Update: it was on my blog.

Wednesday, October 05, 2005

The 4th Quarter

I've been recently wondering if always having to win the game in the 4th quarter makes you stronger or weaker. When you have to come from behind and fight for your survival, does it give you mental toughness or does it just drain you, emotionally? What happens when you finally lose, is it just another loss, or is it the proverbial straw that breaks the camel's back? How much can you take when it is always on the line? Interesting questions, no answers. Keep coming back for updates.

The trip

So, you're going on a trip from San Diego to Buffalo and you've decided to travel by automobile. You now have two popular options: drive your own car or ride in someone else's. For the moment, we're going to ignore driving someone else's car or riding while someone else drives yours.

If you decide to drive, you get to pick what car you take. You can take the gas guzzling SUV or the environmentally friendly hybrid. You also get to choose who goes with you. I hope you choose who goes with you for good reasons. You might pick Bob because he is friendly and good company. You might pick Ann because she is great with a map. You might pick Fred because you know he doesn't have to stop at a bathroom every 30 miles. Finally, you might pick Jill because she offered to pay for the gas if she could tag along. One thought is to pick people who want to go in the same direction as you. Perhaps they don't want to go to Buffalo, but going to Seattle or Tampa Bay would probably be a little out of the way.

If you decide to ride, your options are different. First, you need to find someone who can take you. It could be that you have to get on a greyhound bus: they are cheap, fast, fairly safe, and if it breaks down you don't have to worry about fixing it. Or, you might have a friend or acquaintance driving their own car and they invited you to come along. Often, the invitation process is stressful and intrusive, but once that is over with the ride can be quite nice. There are additional things to consider, things over which you have no control. For instance, who are they taking with them? Are they taking Reggie, the loud obnoxious guy you can't stand, or are they taking Gina, the cute secretary from across the street? These decisions can definitely affect whether or not you choose to ride. Of course it could be that you don't own a car and Reggie is your only hope, but we try not to be in that situation, don't we. We brush up on our map reading skills, learn how to control our bladder, and try to save enough money to help buy gas. Then again, it could be that your best friend is going to Columbia and has asked you to come along. Yes, he is your best friend, no you don't have your own car; however, going to Columbia (the country) does not help in your quest to get to Buffalo, so it is best to avoid that trip. It also helps to examine the car you will be riding in. Do you think it can get to Buffalo? Are there already too many people on board? Has the driver engaged in preventative maintenance? It could be that you just hop in the first car available and blindly wish for good fortune, but that inevitably leads to disaster. The best thing to do is to know yourself and choose the transportation method that is right for you.

Sunday, September 25, 2005

Paid Apprenticeships?

A minor rant. I hear now and again someone talking about the old craftsman ways and how an apprentice used to pay his master for services. Now, we expect our first job to pay us well and it is a terrible injustice. To that I can only say that we do pay for our apprenticeships, but it is now called college. Whether you agree or disagree that it is a valuable apprenticeship, it is one just the same and probably had the same value now as it did back in the olden days. So, the next time you think you ought to get paid for taking on an apprentice, go teach school.

Thursday, September 22, 2005

More on Company Death

I happen to like the beekeeper story, but I think it is incomplete. The author blames a company's demise on the leader leaving or changing, but I don't necessarily believe that is it. I think projects get larger and mature and they change in nature. Take Office. It doesn't have to get a ton of new whiz bang features out immediately. It has market share and it needs to maintain its dominance while not hurting quality. To do that, you add processes in place to control the change in the product and to ensure integration with other products. I'm curious if this is just natural product evolution. It doesn't have to be company wide. For instance, at MS the XBox, MSN Search, and V. Studio teams are still quite agile and seem to enjoy their jobs, where the larger, integrated systems people are miserable. Is it the leader? Is it integration? Is it complacency (wanting to maintain your product instead of revolutionize it)? Is it all 3? Why do some companies impose process in some areas and not in others whereas other companies impose process over the entire company at one time? What is the survival rate of each?

On complacency, I think that is a valid reason for a while. Office, in the past 5 years, has not had to revolutionize. I think that the open source movement and OpenOffice in particular has forced its hand and required Office 12 to be produced. However, therein lies the problem. When a new challenge faces a process laden product, it can't adapt. The top programmers have left to find projects with less process and processes have been inserted to maintain the quality in their absense. Now, the mediocre programmers are faced with a challenge that is too great for them, create a revolutionary project in Internet time. The processes don't allow it, the programmer's don't have the skills, and the result is a buggy product that is late and not close to what users expect or need.

Do I think MS is on its death throws? Most certainly not. It is too big to go down the tubes over night. However, many people thought the same thing about IBM and it lost huge amounts of market share and almost was divided into many separate, smaller companies. It took Lou Gerstner to turn the company around, eliminate the beurocracy, and get the company agile again. Coincidentally, Lou named his book, Who Says Elephants Can't Dance. It is a must read if you are in the technology sector as it directly addresses a company's shift from beurocracy to agility. Making a large company dance is an amazing feat. In Lou's opinion, it took a leadership change to make it happen. In fact, had he not come in time, they would have spun off many IBM divisions and it would have continued to decline. According to my interpretation of Lou, agility is not something that can come bottom up, it must be a leadership decision.

What about Oracle? How does it handle process? What are the other older and bigger tech companies: SAS, Apple, Sun - what do they do? What about larger tech companies that have failed: DEC, Compaq - what did they do wrong. I've studied Apple's problems and they seemed to stem from arrogance and a fundamental lack of marketplace economics, not process. It was more an unwillingness to change rather than an inability to change. Looks like it is time to hit the books...

The Other Side

I really don't want to promote this guy's blog, but in the interest of fairness, I'll show the other side of the coin. This post describes why a business man believes Microsoft needs more process and more beurocracy, not less.

They just don't get it. There is a known connection between process and productivity, one that until recently I forgot. When labor unions want to strike, but cannot for whatever reason, they often strike "by the book." What this means is that they follow the process manual to the letter. Basically, no work gets done because all of the process gets in the way. It is a very effective tactic. The same holds true for companies, as process is added, productivity drops. The blog talks about the problems with bugs in the old MS software and how process should fix that. WRONG. You can't "process" the bugs out of the system. The delayed release of Vista, Monad, and Office 12 because of bugs shows that process doesn't help that. Instead, it drives away your best developers, causing MORE bugs, not fewer. It just goes to show that business people don't understand the relationship between quality craftsman and quality craftsmanship. They like to think in assembly lines and mass production. I hate to tell them, but the software industry isn't there yet, and may never be.

Of course, many companies don't need the best. They can settle for a prefabricated home instead of a custom built mansion. However, if you're life's work is hosting parties, then you better opt for the mansion and you better make sure it is built well. The same goes with software companies. If you're life's work is making great software, then you better hire and retain the best in the business.

Wednesday, September 21, 2005

The Perils of Integration

Thinking more about mini-microsoft and what has been posted by him and his readers, I wonder how much a company's downfall has to do with integration. When a company decides to integrate its systems into one mammoth system, it seems to spell doom. I think the process goes like this:

First, some senior executive decides that everything needs to be "integrated." This could mean that everything needs to work together, run on the same box, or in the same environment, who knows the exact definition?

This decision spawns a massive project and a few reorgs. The massive project introduces hundreds, if not thousands, of dependencies between teams. Managing these dependencies becomes too much for the teams and therefore process has to take over. The system is fragile and buggy, but any change to the system has to be approved by everyone else, and may cause everyone else to recompile or recode their part. Therefore, changes are to be avoided at all cost. It begins to take longer and longer to put out a release.

At this point, the individual teams have lost all power. The power is centralized into a few leadership individuals who make decisions, but don't know the technical details of those decisions and therefore can't understand the ramifications. It's not their fault, there are too many dependencies for ANYONE to understand all the ramifications. The teams can no longer respond to customer needs; they'd like to, but their hands are tied because if they change the Foo widget then it will cause a recompile of 5 other libraries and 2 months of testing - and the testers are already running on empty.

The best programmer's get frustrated. They can see the problem, but they can't put their finger on what exactly is wrong, plus they spend more time filling out forms and "communicating" then they do programming. They get disgruntled and leave for greener pastures. Those who remain are left maintaining systems that they didn't build and changes to the system are slower and more painful than before.

Finally, one of two things can happen, either the company can fold, or some spark occurs and the old, integrated system is left for dead and a project is started to build a new system that relies on customer teams and independence. The best programmers are hired to create the system and it becomes a success. Then the cycle repeats itself again.

Integrated systems are the hardest things to do correctly. As I've mentioned before it is our job to manage dependencies. The best way to manage a dependency is to eliminate it. With an integrated system, you have the potential to create an exponential number of dependencies, thus ensuring the demise of your company. Now, the question remains: how do you create an integrated system correctly? I'm not sure, but I'm going to think on it, so keep checking back here for the/an answer.

Mini-Microsoft and the Death of a Corporation

Mini-Microsoft is an anonymous Microsoft employee that is trying to change his company by airing its dirty laundry, publicly. I don't know that I agree with his approach, but I certainly feel sorry for him. It is sad to see a corporation leave the expansion phase and go into the recession phase, but all the signs are there. They have reduced or stopped raises, their earnings are slumping, they are using reorgs as a way to increase productivity, they are using process over people, and their best people are abandoning ship while those that stay have low productivity due to low morale. Microsoft had one of the longest expansion periods I've seen. Now, the giant has peaked and it must move into a recession phase. This is not to say that sometime, in the future, it can't expand again, but it must fall hard before that is to occur. Some of its products will decline as well, some have already been in decline. Some of its products may even excel: I'm a huge fan of C# and I think it will do well, though lack of true multi-platform support may hinder it (sorry Mono, you're not there yet). The warning signs are clear and its sad for the 50,000 people that work there. So long and thanks for all the phish.

C++ Tip: lower_bound and functors

Here's an interesting thing I learned recently via necessity. Let's assume you have a vector of string/int combinations.


vector<pair<string, int> > myvec;


Now, lets assume we add data to the vector, in sorted order (based on the string).


myvec.push_back( make_pair( string("A"), 0 ) );
myvec.push_back( make_pair( string("B"), 1 ) );
myvec.push_back( make_pair( string("C"), 2 ) );
myvec.push_back( make_pair( string("D"), 3 ) );


Finally, we want to search for a string key in the sorted vector using lower_bound.
We can't do the following:


lower_bound( myvec.begin(), myvec.end(),
string( "C" ) );


It will complain that there is not an operator<(std::pair<std::string,int>, std::string). Well, that is annoying, but we can write one. However, it is usually better to not clutter the global namespace with such functions and instead we choose to create a comparator function.


bool pair_str_less(const pair<
string,int>& lhs,
const string& rhs)
{
return lhs.first < rhs;
}


Now we pass in the comparator function

lower_bound( myvec.begin(), myvec.end(),
string("C"), pair_str_less );


Ouch! Now it complains that there is no pair_str_less( std::string, std::pair<std::string, int> ). It needs both permutations of function arguments! Unfortunately, we can only pass in one comparator! Perhaps it is not meant to be? Maybe we should just write our own version of lower bound? Or perhaps we should create a dummy std::pair from the string? Or, perhaps we can use functors!


struct pair_str_less
{
bool operator()( const pair<
string, int>& lhs,
const string& rhs ) const
{ /*...*/ }
bool operator()( const string& rhs,
const pair<
string, int >& rhs ) const
{ /*...*/ }
};


Now, we can use the pair_str_less function like


lower_bound( myvec.begin(), myvec.end(),
string("C"), pair_str_less() );

and the right version of operator() is chosen each way. Yeah!

Tuesday, September 20, 2005

Process vs Best Practice

I'm tired of process. Yes, I know it is supposed to help. Yes, I know it encourages responsibility and increases quality and decreases entropy and blah blah blah, but it is WRONG.

If you hire the smartest people, give them the best tools, and let them work, they will do the right thing, intrinsically. You only need process when your people have let you down. In fact, when you institute a mandated process, you are slapping your employees in the face. You are saying, "I don't trust you to get this right, so I, your leader, have mandated that you must follow these steps." The best people will follow the best practices. They will use version control, bug tracking, and change management because it is the right thing to do, not because someone forced them to do it. Moreover, they will also be able to choose when not to use it. The problem with process is that it must be blindly followed. It will bog your people and your company down in its forced standards. It will lower morale due to trust issues. It will lower morale due to developers being forced into the "one right way" (developers naturally resist standards and blind obedience).

Get the right people on the bus and get the process off the bus. Let the people find the best practices and let them have a great time producing great code.

Sunday, September 18, 2005

C# 3.0

There was a recent post on slashdot sending people to view a video of Anders talking about C# 3.0. For those who don't know, Anders Hejlsberg was the creator of Delphi and the Object Pascal language. Microsoft hired him away from Borland to oversee their .NET languages. While I think C++.NET is an abomination, C# is an amazing staticly typed language. It has stayed away from all of Java's problems and has adopted a much more agile flavor: very ruby-ish in nature. I've been reading the O'Reilly book on C# 2.0 and I've been impressed, but 3.0 looks like it will blow all other static typed languages out of the water. Kudos to Anders for giving us a language worth using. Now, I wonder about performance...

Another thing I found interesting about this video was Anders continual discussion of the "shape" of data. I've heard this term before (I think it was in Code Complete), but it had been a while. Acutally using C# to show the difference between rectangular data and heirarchical data was very interesting to me. It is something I'll have to start thinking about on a regular basis.

Dependencies

One of the most important jobs a programmer has is to manage dependencies. In fact, many of the Code Smells that we learn to avoid have their roots in the removal of unnecessary dependencies.

Especially evil are implicit dependencies (i.e., dependencies that are not easily noticed/not explicitly written). For example, once and only once is about the removal of implicit dependencies. Why? Because all instances of duplicated code depend on each other. If you have the following lines

int i = 0;
for( ; i < 10; ++i )
{
...
}
i--;

and you decide that the last i-- is a bug, then you have to remove the last i-- in every instance of the duplicated code - they are all dependent on one another. What's worse is that there is no semantic information associated with this code. It could be that one instance of the duplicated code actually precedes something slightly different so that the i-- is actually needed. Because there is no semantic information tieing all the pieces together, there is no way to know which i-- should stay and which should go.

Magic numbers, being another form of code duplication, have exactly the same problems; however, the problems with lack of semantic information is exacerbated with magic numbers. In the code above, there is no knowledge of what the number 10 represents. When you realize that it should be 15 do you change all the instances of 10 to 15? If not, how do you know which ones to change and which ones not to?

Dave Chelimsky discusses how to Manage Dependencies Not Aesthetics in his blog. He uses the following example:

display.print( person.getAddress().getZipCode() );

and then compares it with

display.print( person.getZipCode() );

and

person.printZipCode(display);

Which one has its dependencies managed correctly? Let's examine where the dependencies are. In example 1, there is a dependency from the main code to display, person, address, and zip code. There is also a dependency between display and zip code. Even though zip code is just a string, there is a logical dependency between the concept of a zip code and display.
In example 2, there is a dependency from the main code to display, person, and zip code. The dependency on address has been eliminated. There is still a logical dependency from display to zip code. In example 3, there is a dependency from the main code to both person and display, the dependency to zip code has been removed (even though the function still says zip code, the zip code is not seen or used in the main code, so there is no dependency). There is an additional dependency between a person and a display object.

Now, elimination of dependencies is not the goal, only the effective management of dependencies; however, it is much easier to manage a dependency that doesn't exist, just like it is much easier to maintain code that doesn't exist. I think we can quickly eliminate example 1 as being the best because it has no advantages over example 2 and an additional dependency. Example 2 does have an additional dependency over Example 3, but there are some trade-offs involved. With example 2, if the zip code changes, then the display class must also change. With example 3, if you want to write to something other than a display, then the person class must change. So, the question becomes which is more likely to change. Obviously this will change on a design by design basis.

Interestingly enough, the comments section of David's blog dives off into a discussion of formatters and visitors and how to isolate the change, but each introduces more dependencies in an effort to find the best dependency combination that is least likely to change. In this case, I might stick to the TDD principle of doing the simplest thing possible and then refactoring when we actually see the change, at that time you can make a more informed decision about what needs to be changed and how the change needs to occur.

Friday, September 16, 2005

J - A - C - K - E - TS

My daughter just cheered at her first game. Kindergartners through fifth graders had the chance to be "mini-cheerleaders" at tonight's football game. My daughter was very excited. She had her cheerleading outfit and her yellowjacket face tatoo. She knew the cheers and she was ready to go. I'll post some pictures of it over the next day or two. She did a great job for only being in Kindergarten. I was very proud of her, as always.

I certainly don't encourage cheerleading. I've known very few cheerleaders that I actually respected (two). I tend to think the "sport" is for the flighty, energetic people that I typically don't associate with because they have way too much energy and way too little intelligence for me. However, I made a commitment many years ago to support her in whatever she wanted to do (as long as it was legal and moral). I'll stand by that even if she wants to be a cheerleader, but I'm sure she won't -- she's way to smart for that ;0)

Tuesday, September 13, 2005

DRY needlessly

DRY, don't repeat yourself, has become a battle cry to many in the programming profession. Wiping out all causes of spatial, temporal, and logical repetition is the aspiration of today's elite. However, zealotry has never suited me, and I tend to think that there exists good repitition as well as bad. Let's look at some examples:

First, Test-Driven Development is full of repetition. The tests and the code define the specification separately, but equally. Yes, you can make up all sorts of excuses for why there is duplication, but the fact remains that both the tests and the code are asserting the same thing.
Second, in Code Complete, Steve McConnell talks about the practice of using a function to check another function's work. For instance, if you are making some difficult calculations, you may wish to write a slow and steady function that you know works and then write a swift and dangerous function that you are unsure of. In debug mode you would run both functions and ensure the identity of the results.
Third, web programming often has client and server side validation of input parameters. Yes, I know you could write a control file and have both generated from that, but I seriously doubt that is a big enough win to justify the programming time involved. The duplication is accepted as useful.

In each of these cases, we're using repitition to enable a sort of "double-entry bookkeeping". We perform our work and then check our work to make sure everything balances out. The duplication is there for a reason, to ensure quality and correctness. The duplication that DRY is intended to eliminate is not there to support itself, but is instead needless. Think of it as a teepee. You have two sticks leaning different directions, supporting one another. There is repetition, you have two sticks instead of one, but they are working together and leaning on one another. This is the key, if you have duplication in your code, make sure the duplication is leaning in opposite directions.

Monday, September 12, 2005

Forty Things Wiki

I've created a wiki where I can post my thoughts and preliminary drafts of Forty Things Every Programmer Should Know...But Doesn't! I'm using phpWiki because it was a breeze to set up and should be just fine for what I intend to do. Please feel free to contribute to The Forty Things Wiki, but be sure to leave your name so I can give you credit!