Wednesday, January 17, 2007

Big brother strikes again

Where do we draw the line? Jeff Jonas details the using of a P300 brain response as allowable evidence in court cases.

I would like to have seen more information about what a P300 response is. Is it possible that some people are "malformed" and have this response even when presented with an unfamiliar image? What if it is your living room and, while the blood is unfamiliar, the rest of the information is. There are too many variables for me to enjoy this.

As for going to jail because you thought about committing a crime, that is ludicrous. The difference between humanity and animals is that we can control our actions. I'm not sure we can control our thoughts. Regardless, as humans, we must be judged on what we can control.

Tuesday, January 16, 2007

Boolean Equation Problem

I'm struggling with the following problem.

I have a series of boolean equations. Only one of these equations can be true at any one time. I need to find an efficient way of determining which of these equations is true. I would also like to know more about these equations, like is it true that only one of the equations can be true and what new equations are needed to make that true.

Here's an example. Let's say that half of the equations deal with the condition where variable A issw equal to the value 'M'. 25% of the equations deal with the condition where variable A is 'm', 25% of the equations deal with the condition where variable A is 'n', 15% where the variable A is '?' and 10% where the variable A is 'N'. The variable B has a situation where 75% of the equations deal with the condition where it is 'M' and the other 25% of the equations are divided among the other choices. There are also variables C, D, E, F, and G. Which of these variables do I evaluate first? What factors should I consider? Are there algorithms that will help me in determining this?

Ch-ch-ch changes

The previous year was a year of changes. I worked for three different companies, in three different cities, one of which required me to move 800 miles away. Now, I'm back where I began and I've learned an awful lot and met some truly amazing people.

One of the first things I learned is that there is nothing more important than your family. I missed being able to pick my kids up from school and taking care of them in the afternoons. I also missed the good friends I had made here. I didn't realize how much I had missed them until I moved back.

Second, I learned that going to church is an important part of your spiritual development. Seems obvious, right? Well, I've recently been going to the First United Methodist Church of Wynne and have really enjoyed both the service and the Sunday School. I've been going there with friends, which really helps.

Third, I have learned a lot, technically. I've learned the value of having business people drive functionality and I've learned the value of architecture that is planned over architecture that evolves. I've learned about J2EE, CORBA, networking, TIBCO, and a whole host of other things. I enjoy learning, so that was fantastic.

Fourth, I learned that to be a good team leader, you've got to be able to stand up for those who are working for you. I've had leaders that I really enjoyed at all the companies -- they were all really positive experiences. Each one had a different style, but when it came time to go to bat, they were there to stand up for me.

Fifth, I learned that if you want to stay challenged in a traditionally business IT shop, you need to challenge yourself. It seems the expectations of those places are low and you can easily "just get by." In fact, you can easily descend into madness. But, to truly excel, you have to challenge yourself to come up with innovative ideas to make their software cost less and be more stable. I think more and more the key to good programming is focusing on stability.

Sixth, I learned that I have to make tradeoffs and I'm thankful for the past year because it helped me understand what those tradeoffs are. Right now, I work from home. It allows me to pick up the kids and help out my wife, who drives an hour to work three days a week. However, it also isolates me from communication with others. I really like the office setting where I can talk to an adult when I run into a roadblock, instead of being isolated from everyone. Therefore, I have to realize the need exists and try to assuage it in a different manner. Eventually, I'll go into the office more (at least 2 days per week) and hope that helps. However, right now I'm not working with a team, so going into the office doesn't really help. Nevertheless, spending the last year in two different office environments helped remind me that it can be a lot of fun and that I need to remember to go into the office more when I feel isolated.

Finally, as I head back to my old job, I realize that the reasons I left it are still there. However, after spending a year studying myself and other companies, I'm am in a much better position to adapt so that those things will no longer bother me. Right now I just want to be appreciative of the great gifts that God has given me. I'll worry about the other things later :-)

Thursday, December 28, 2006

Similar Characters

I try to keep track of people who are similar to me: past, present, and fiction. I have three, currently, that I believe fit the bill.

1) House - I've blogged on this before, but Greg House from the TV show House is very similar to what I am like. I think he has let his pride control him more than I have and that accounts for a few differences. But, on the whole, we are very similar.
2) Isaac Newton - We have similar interests, religious beliefs, and personalities. He, of course, was far more brilliant that I ever could be, but we are very similar in our pursuits.
3) Brad Paisley - Yeah, the country singer. It's uncanny how many ideas in common we've had. In fact, before they released his "Two people fell in love song", I commented to my wife that everything that has ever been was because two people had sex. Mine was more crass, but that's beside the point. When my wife heard it on the radio, she immediately came to find me and show me my idea had become a song. It was funny. I still say he has spies that watch me...I need to go now...I have to elude them!

Tuesday, November 21, 2006

Aliens

I watched a history channel special the other night on UFOs. It had some interesting interviews on it. One thing that was eye opening was the replay of a nasa astronaut saying over the radio that he was looking at the alien spaceship out of his window. He said it very matter-of-factly and very clearly. He was neither surprised nor concerned. I have no problem believing in aliens. However, the Roswell stuff is a bit far fetched. How is it that the aliens can fly billions of miles across the universe and then crash in New Mexico? Doesn't make a whole lot of sense to me.

However, I propose the following. I think the next candidate for president of the United States should run on a platform of revealing all government knowledge of UFOs. I'd vote for that person, I think it'd be cool to know. What about you?

Monday, November 20, 2006

Unfair Infinities

We all know that the cardinality of the set of all integers is countably infinite. To me, it seems unfair that the cardinality of the set of real numbers is uncoutably infinite. It also seems odd that it is unintuitive to most (including me). However, I now understand why it was unintuitive to me. The key point to remember is that mathematicians don't allow integers to have an infinite number of digits. Instead, you can increase the integer by 1, but at any given time it has a fixed number of digits. However, with real numbers, there can, and often does, exist an infinity of digits. For instance, 1/3 is 0.333333.... So, not only do the real numbers extend countably to infinity, they also can have an infinite number of digits. In essence, they are infinite in 2 dimensions, whereas integers are only infinite in 1 dimension. I would imagine that complex numbers will one day be proven to be infinite in 3 dimensions, but who knows.

To be honest, I think it is a bit silly. We're not actually talking about different orders of infinities here, we're only talking about the existence of a function M that will map from one infinite set to another. If we can find that function, then we say the sets have an equivalent cardinality. Otherwise, the cardinality of one set is said to be greater than the cardinality of the other. If we could extract M from mathematics, and instead use an algorithm, then M could simply be, pick a real number, get next biggest integer, rinse, repeat. However, algorithms are not yet mathematical (unless you're Stephen Wolfram).

Tuesday, October 10, 2006

Readable Code

I think, to structure your code readably, you have to partition your code into different levels. Perhaps the different levels are written in different languages, perhaps not. The top level is the business level. At this level, you should be speaking a mostly declarative language and should be side effect free. Your code should be straightline. The second level is the logical level. At this level, you can have conditions and iterations and anything in the first level. The third level allows anything in the second level plus side effects. Finally, the fourth level allows comparative and arithmetic operations plus anything in the third level. The interesting thing to note, I believe, is that side effects are more readable than comparative and arithmetic code. Of course, it could be that comparative and arithmetic code is less useful than side effect code, so it is pushed to a lower level.

Let's look at an example:

extern int debug;

void foo()
{
int i;
for(i = 0; i < 100; ++i)
{
if(debug)
{
printf("%d\n", i);
}
}
}

This is a level 4 function. It has side effects, comparisons, etc... Let's try to turn it into a full 4-tiered function (yes, you wouldn't do that to something like this, but just for example.)

// level 1 - straightline
void foo()
{
print_numbers_to(100);
}

//level 2 - conditions
void print_numbers_to(int count)
{
if (in_debug())
loop_to(count, print_integer);
}

// level 3 - side effects
void print_integer(int i)
{
printf("%d\n", i);
}

// level 4 - comparisons
int in_debug()
{
return debug == 1;
}

// level 4 - side effects
void loop_to(int count, void (*func)(int))
{
int i;
for(i = 0; i < count; ++i)
{
func(i);
}
}

See how much better that is?

Tuesday, September 12, 2006

Responses

I put some responses in with comments. However, I would like to say that I was imprecise in a previous statement...it's a habit of mine. My statement should not have been that deficit spending causes war. Cos is entirely correct that war causes deficit spending, not the other way around. You can check the comments for the full scoop, but I was attempting to say that the ability to deficit spend leads to war. If you can't fund the war, you won't have the war (or you won't be successful at it). If you have easy access to funds, then you can easily create a war machine. If not, it is much harder. It is not deficit spending, but the ability to deficit spend. Check the comments ;-)

Response 3

Now, I'd like to respond to a few things I agree with from Cos's reply

1. I really like the idea of the states paying the Senator's salaries. This further reduces the federal stranglehold on our country. That is just pure brilliance!

2. My hope is that people who have a concern for our country would want to be electors. However, now that you mention it, I wouldn't be surprised if party-liners didn't run for electors just to be paid back at a later date by the party nominating them for an office position. I really don't want to enable a lot of restrictions, because that doesn't lead to a naturally balancing system. For instance, we could say that electors can never be publicly elected, but that is just silly in the long run. My hope is that the more educated, politically motivated people would become electors...however that is naive.

Nevertheless, what I realistically expect to happen is that entitlement spending drops dramatically, and socialistic programs begin to decline. When the voters don't elect you based on the entitlements you give them, you no longer give entitlements. Obviously, an elector could run on a platform of entitlements, but it doesn't really help him. He's not getting a huge salary and a plush office for life, so why does he care. His sole purpose is to do what is right for the country, because it is right, not because he's getting compensated. It could be a naive position, and I know it will eventually be corrupted, but I'm hoping that won't be for a number of years, when the system is stable enough to support a number of years of corruption before being reset again.

Response 2

I'd like to respond to a portion of a very well thought out comment from Cosbert Callis:

3) Frankly neither suggestion of the barter system or the gold standard, nor the idea of eliminating the Fed represents an understanding of economics 101.
(as a fiscal conservative, with a BA in Political Science and a Minor in Economics, you would have failed any econ class I was in with these ideas..)

Foreign trade (including deficit spending, which is a form of foreign trade) represents one of the most important natural forces for peace in the world. People just do not make WAR with trading partners. I don't where your original thought inevitably leads to war and inflation. comes from, there is NO, ZERO, NADA in the way of empirical evidence to suggest there is a single iota of truth in that statement.


First, deficit spending is not always a form of foreign trade. Deficits can come in many forms. First, you can sell bonds to your own country. Second, you can have the Fed print more money, thus causing inflation, but giving you the money you requested. Third, you can sell bonds to other countries. I would only classify the last as deficit spending.

Second, deficit spending MOST DEFINITELY causes war. You would have failed any history class I took with your ideas. War's have to be funded in some way. Without funding, you can't fight. There are many ways to get funding, but the easiest is through deficit spending. Do you think the war in Iraq would have happened if we would have had to raise taxes to pay for it? People understand taxes: they understand the effect that taxes have on their pocketbooks. With deficit spending, they don't understand the correlations and are less likely to complain. Therefore, deficit spending is VITAL in starting and maintaining a war machine. Without easily obtainable money, there are far fewer wars.

Third, we didn't always have a national bank. The current Federal Reserve system was established in 1913. Notice that it DIDN'T provide economic stability, as was its mandate. We still went through booms and busts (including the great depression). Before 1913, we had a myriad of different banking styles, including no central bank from 1837 to 1862. Do I think we need banking regulations? Sure. Do we need a central bank? No.

Fourth, a return to the gold standard is independent of the federal reserve system. The Gold standard was not revoked until 1971, almost 60 years after the establishment of the federal reserve. The goal of the gold standard is to reduce inflation. I would say that it did it's job pretty well considering the amount of inflation that has happened between 1971 and today.

Finally, trading partners most definitely go to war. The US traded with Iraq (oil and weaponry). Iraq traded with Kuwait. The US invested in Germany before WWII. The examples go on and on. In a global economy, there is more than one way to get at any resource, so it becomes much easier to bite the hand that feeds you. But, it becomes much harder to bite if you have no teeth. By eliminating easy access to money (via deficit spending through the Fed), we can take out a few rows of teeth.

Friday, September 08, 2006

Fanatics

I love this quote from Ron Jeffries. I found it on a thread in his Agile Forum.

A fanatic is anyone who believes differently from us, and just as strongly.

Thursday, September 07, 2006

Response

Mark made a very intelligent comment on my previous post and I'd like to respond to a few of his points.


1. Repeal the 17th amendment:
This sounds good on paper, but what guarantees that the state congress isn't going to simply choose their Senator soley on party lines for just the possibility of future political favors? Even the lowest of state politicians harbors hopes of being president.


I want the state congress to choose their Senator for future political favors. The point is that the people do what is in their best interest, but the state politicians will do what is in their best interest. What is in their best interest is greater state powers. The people's interest is different. If a Senator lessens the state powers, then the state will not vote him back in. Therefore, the Senator must pander to the state in order to be re-elected. Why is that important? Because state's don't care about gay rights, flag burning, and national health care. State's care about schools and roads and infrastructure. All of the stupidity that is pure popular pandering goes away.


2. I like the idea of taking away the presidential election from the idiot masses who decide solely on superficial traits of candidates ("He's a normal guy who I think I could have a beer with." Who the hell wants to have an average person in the most powerful position in the world??). Anyway, I digress. The problem with this is that you're assuming that the electoral college will be composed of people who are more intelligent than the masses as well as unaffiliated with any political party. That's a tough thing to find I suspect.


I do believe that the electorial college will be slightly more educated than the typical voter, but that's not the main benefit. The main benefit is that the electorial college has no constituents. They don't get paid, they shouldn't get bribes (having fewer people will allow more scrutiny on their financial affairs), and they have no real reason to want to get re-elected. Therefore, they won't pick a president because he promises more welfare or free healthcare. People pick a president because he promises entitlements or "No new taxes" or whatever. They vote in what they think is their best interest (unfortunately, they don't understand the consequences). The electorial college could safely ignore those issues and pick a president based on what is best for the country. Since they don't care about re-election, they are free to make more intelligent choices.


3. Let's go back to the barter system. Most politicians really have nothing to offer other than words, so that should sufficiently weaken the ever-growing hegemony those in power represent. :-)


I believe a return to the gold standard is very similar to going back to a barter system. By preventing the devaluation of the dollar because of deficit spending, we can force politicians to tax us explicitly instead of through inflation.

Waiting on the World to Change

I really like John Mayer. Especially his new song. However, I have to disagree with his approach. First, the song:

me and all my friends
we're all misunderstood
they say we stand for nothing and
there's no way we ever could
now we see everything that's going wrong
with the world and those who lead it
we just feel like we don't have the means
to rise above and beat it

so we keep waiting
waiting on the world to change
we keep on waiting
waiting on the world to change

it's hard to beat the system
when we're standing at a distance
so we keep waiting
waiting on the world to change
now if we had the power
to bring our neighbors home from war
they would have never missed a Christmas
no more ribbons on their door
and when you trust your television
what you get is what you got
cause when they own the information, oh
they can bend it all they want

that's why we're waiting
waiting on the world to change
we keep on waiting
waiting on the world to change

it's not that we don't care,
we just know that the fight ain't fair
so we keep on waiting
waiting on the world to change

and we're still waiting
waiting on the world to change
we keep on waiting waiting on the world to change
one day our generation
is gonna rule the population
so we keep on waiting
waiting on the world to change

we keep on waiting
waiting on the world to change


Now, the analysis
The sentiments are good, but he misunderstands a fundamental part of human nature. The elite that are corrupting our society now will hand their corruption to their chosen successors. Just because those successors are from our generation doesn't mean they will be any less corrupt. They will continue to attempt to plunge us into a feudalistic society with them at the helm. The fact that we're "waiting" just makes them smile broader. They know we're powerless to stop them so they're taking full advantage of it. Our freedoms have been tromped upon. Our civil liberties abolished. We have our own version of the SS (we call it Homeland Security - anyone notice how these guys go after child porn? No one complains because we all know child porn is bad, but how is that homeland security? It's not. It's the start of our own version of the secret police.) Why will our generation act any differently? The fringe, artistic types such as John Mayer have never supported war or abdication of civil liberties; however, their generations didn't set things right. Why does John think his will? He's completely wrong. Instead, he's promoting the attitude that suits the would-be dictator's perfectly. He's sitting back and watching it all happen. Eventually, it will be too late to stop it.

What can we do to stop it? We have to take away their power base. Of course, we can't rely on them to do it, because they like their power base. Instead, we have to use the one constitutional outlet that they haven't taken away. We need a 2/3 majority of the states to support certain constitutional amendments.

I propose three amendments that should erode the power base of those in charge.
1. Repeal the 17th amendment. This amendment took away states rights and consolidated power to the federal government by moving the election of the senators to the people. Previously, the states chose the senators that represented them. This made it far less likely that a media campaign could buy an election. It also ensured that the person would actually support the state instead of what made them electable. Now, it is all too easy to support a law because it is "popular". Since you only answer to the people, that is all that matters. The Senate is no more than a balanced House of Representatives. Instead, it is supposed to represent the states. Senators would have a much harder time revoking states rights if they had to answer to the state congress.
2. Make the party nomination of presidential candidates AFTER the election of the electorial college. Furthermore, electors should not be allowed to express allegiance to a political party. Currently, the population, not the electorial college, elects the president. This is NOT how it was designed to work. The writers of the consititution knew that the general population is too stupid to choose the correct presidential candidate. Therefore, the general population should choose a bunch of smart people, who choose the president. This has been lost on today's generation. We have to bring back that purpose. The popular vote for a president should never be taken.
3. We must erode their monetary control of our society. The last amendment should abolish the federal reserve and return us to a gold standard. For the reasoning behind this, I would recommend Griffin's The Creature from Jekyll Island. It is a very informative (and very large) book about the history of the federal reserve, fiat money, and fractional reserve banking. It shows how control of the money system directly leads to a controlling, manipulative, feudalistic government.

We have to stop waiting on the world to change and use what power we have as a people to ensure our future.

Saturday, September 02, 2006

Cultural Progression

How do we, as a culture, define our differences with previous cultures. Do we define it by our use of technology? I don't think we can. There are cultures that have advanced technology, yet have similar cultures to their ancestors who didn't have such technology. I think we have to find something that shows motion, not advancement. To that end, I propose we use music. It appears to me, that music is the definition of culture. In other words, by sharing music, you share your culture. By blending music, you blend your culture. Furthermore, music is always changing and adapting. It blends the past into something completely new and unexpected. Music seems to define our progression. Music defines our culture. Explore that concept and let me know what you think.

Saturday, August 12, 2006

Generics

Boo doesn't have generics. This makes me sad. Since I want a strongly typed matching solution, I _really_ want generics. I also want macros. ARRRRRGGGGG! Why won't people give me EVERYTHING. One option is to use arrays instead of lists. Another option is to add generics to the language. I think I'll start with the first and then migrate to the second. That should be sufficient for my needs.

Ok, now on to the O/R mapping.

First, since this is all about types, we're going to add the type information for (most) things in the database. Why, you ask? Shouldn't we know if it is a FirstName or a LastName because we have knowledge of the table? Yes, that is correct, but you're thinking too simplistically. In true OO fashion, FirstName and LastName will be nothing more than base classes. We will end up with an AsianFirstName, AngloFirstName, HispanicFirstName, etc... The same is true for last names. Then, when we consult our statistics, we'll be able to use statistics based on the frequency of the name within its ethnic culture (and also within the geographic location). Therefore, we want to be able to generate and store this extra type information. In addition, we'll want to use the type information when comparing names with the matching engine. We might use a completely different function when comparing a HispanicFirstName to an AngloFirstName as we would a HispanicFirstName to an AsianFirstName (auto-reject, anyone?). By emploring multi-method dispatch on the type information, we can quickly choose the right matching logic.

But enough skipping ahead, back to the O/R mapping.

Obviously, each Element will be stored in its own field. It will have an associated type information field. It may also have a pointer into a metadata table. I'm not sure on that one yet, we'll have to see what sort of metadata we will keep that is outside of the type system.

An entity becomes a little more complicated. Remember that an Entity is a collection of other entities, groups, and elements.

Let's look at two different Entities:

class Name(Entity):
first_name as FirstName
last_name as LastName
middle_initial as Nullable(MiddleInitial)
name_suffix as Nullable(NameSuffix)

Wow, what's that Nullable thing? Well, the type system should include whether or not the field can be blank, and Nullable is just as good a choice as any. I'm really starting to want to do this project in O'Caml. I'm getting very close to breaking open the docs on F#. Of course, now that I think about it, C# might be a good choice. I wouldn't need to modify the parser if I had introspection, which C# gives you. Plus it is strongly typed and has generics...hmmm...I had forgotten introspection...drat!

Ok, back to the task at hand.

In most cases, the Name entity will have a Name table. Each element within the entity will correspond to a field in the table. There will also be the associated type field (and perhaps metadata fields). Each record in the table will also have a unique, primary key.

That was easy...

Now, for a more complicated example:

class Person(Entity):
name as Name
address as Address
ssn as Nullable(SSN)
birthday as Nullable(Date)

For a person we will have a unique primary key, but we will also store the primary key of both the name and address information. The ssn and birthday elements will be stored "in-line" like the name elements were in the previous example.

Of course, we might want to force denormalization of the table...we could try something like

class Person(Entity):
[inline]
name as Name
address as Address
...

Now, name has an inline attribute and will not go in a separate table. Instead, the name fields will be placed in the Person table. However, when we extract the Person object from the table, we'll extract a Name object as well, so you can't tell the difference from the user side.

I'm hesitant to allow an [inline] attribute, because you get to the same point as with C++ and its inline modifier. The compiler can't inline without you telling it, so you're forced to make decisions that the compiler should be able to make. Therefore, if we have an [inline] modifier it will be more like "auto" in C++, a hint but the compiler can do what it wishes with regards to inlining. Hopefully, it's usage will vanish just like auto's.

Ok, next time we'll look at the O/R mapping of groups. I'm still not using the mailing list from SF because I sent things to it that I never got back, so I'm waiting until I have a successful test run before I move there for good.

Sourceforge Site Available

You can now go to The STARS Sourceforge Site and sign up on the see-stars-devl mailing list. I hope to make most of the technical discussions through that list so that they will be archived and open for discussion. I will also post previous blog entries to the mailing list for historical purposes.

Friday, August 11, 2006

SourceForge site

http://see-stars.sourceforge.net will be the sourceforge site. It has been created and I am in the process of getting someone to set it up ;-)

Secondly, I'd like to discuss how the type system will interact with the database. First, I said that we're going to be strongly typed. This is not just fancy terminology, this will affect how the database is structured. For instance, we might want to subdivide names for etnicities. John Doe might get the standard US ethnicity, but Wing Fe might get an asian ethnicity. This can be represented by using subtypes. Therefore, this type information must be stored in the database for fast access. In addition, we might have types for strongly cohesive groups, loosely cohesive groups, etc... (perhaps even a composite group that composes multiple strongly cohesive groups so you can see the heirarchy).

But wait! You say. If we're only doing updates then it won't take too long to create all this information, but our initial database population will take FOREVER! People will get tired of waiting! To that, I say, you're absolutely right. That is another beauty of callbacks. We're not going to do it for the initial population. Instead, we're going to use reasonable defaults, but we're going to allow for processes to continually improve the data. I think consumers expect this. They want their data fast, but they also want it right. They hope that over time their data gets better. To ensure that, we'll have reasonable defaults (so, they won't get the Asian/American matching function, they'll get the statistically based one), but when we correct that decision later, we'll let them know through the callbacks. So, they can make their business decisions quickly and then revise them when the situation merits it.

Of course, if they want to wait around, they definitely can, but we need to be adaptive and fast and continuously improving!

Thursday, August 10, 2006

Two quick things

1. All operations must be idempotent. I'm not sure (yet) how to enforce that...it may just be a really strong suggestion.

2. Versioning will be a big part of the system. We need to be able to add fields to an entity and remove fields from and entity, inline an entity (more on that later) and extract an entity. I'm sure there are a number of "refactoring" tools that come from this, but I want them FIRST, not "when there is time." For instance, if I want to add a Title to a Name, then that needs to be as easy as adding title as Title to the Name class and running an upgrade program with an optional map function to create the title given the name (the map could set them all to a default (blank) title or it could try to guess a title of Mr or Mrs based on a derived gender).

Right now, I know I want automated upgrades when I
a) add a field
b) remove a field
c) inline an entity
d) extract an entity
e) add/change the validation method
f) add/change the normalization method
g) add an implied attribute

Also, we better darn well be able to downgrade!!!!!

Creation

Now that we can define the objects (though not rigourously, that's coming later), we need to know how to create them. Obviously, object creation could possibly mean invoking the database for lookups and the like, so it needs to be well abstracted. In this case, we'll use the abstract factory pattern to create factory objects which will create our entities. For example, if you define the following entity:

class PersonAtAddress(Entity):
name as Name
ssn as SSN
birthday as Date
address as Address

Then you will end up with the following abstract factory

class AbstractFactory:
def getNameFactory:
...
def getAddressFactory:
...
def getPersonAtAddressFactory:
...

class PersonAtAddressFactory:
def Create(name_key as Name.ID, ssn as SSN, birthday as Date, addr_key as Address.ID):
...

Now, we have a consistent, programmatic way to create entities. We can wrap these calls in CORBA or SOAP or whatever, but the foundation is solid.

Next time, we'll start looking at how the objects map to a relational database.

Wednesday, August 09, 2006

Two Things

I'm pretty sure of two things at this point. First, the name of the system will be the Staticly Typed Advanced Recognition System (STARS). I hope to get a sourceforge site for it up soon.

Second, the system will be written primarily in Boo and run on mono. Boo has all the right things for this project.

1. Strongly Typed
2. Type Inference
3. Macros
4. User defined compile steps (this will be very useful when creating the database schema from the source files).
5. .NET/Mono compatible - at some point people will want to run their own code, there is a likelihood that the code will be .NET.
6. Clean syntax - based around python (I prefer Ruby, but...)
7. Duck typing
8. Multi-thread capable
9. Functional + Object Oriented

As soon as I get the SF project up, I'll post a link to it here. I'd like to write the specifications in the project's mailing list.

Callbacks

Callbacks are the key to a good recognition system. A typical batch recognition system forgets about the importance of letting consumers know as soon as information is available. However, this is the lynchpin to a good recognition system.

Let's assume that a typical use case is the following:

1. Run a large file through the system to create a repository
2. Run the files through the repository to ensure correct linkage
3. Rinse and repeate monthly

You have to run the file through twice because you don't know what might happen later in the system to change one of your records. This is because the system is not set up to tell you about events.

If, instead, we allowed the sytem to tell you about important things that are happening, you would be able to complete your run in one pass. So, what needs events. Well, first let's say that we'll use a publish/subscribe mechanism so that only those events that we're interested in will be delivered to us. Second, let's say make the rule that anything that could have an impact on the end result should have an event fired. That means that any time an Entity or Group is created or deleted as well as any time an Entity is moved from one Group to another. I would say that Element updates should be allowed to have events, but not forced to. It could be that updating the salary field doesn't affect anything and you don't need that information to be disseminated.

There are lots of optimizations you can do to make this fast and I don't want to get into those right now, but suffice it to say that the event/callback mechanism can make for an extremely flexible (and efficient!) system.

Obviously, the code for the callback won't be in the same file as the code defining the entities. However, we may want to augment the event with some information at event generation time; therefore, we allow the override of the OnX methods (where X is something like Consolidation).

For example:

class Consumer < Group
def Consolidation(Group other):
if ...:
consolidation_reason = ...
elseif ...:
consolidation_reason = ...

def OnConslidation(ConsolidationEvent event):
event.reason = conslidation_reason

I'm not sure, but you might even be able to suppress events...I don't necessarily like that, but it could come in handy.

Adding operations

First, for the sake of this post, let's change the syntax a bit. Instead of saying

Entity X

We're going to say

class X < Entity

Basic types, like FirstName, will become less basic and will inherit from Element (or in this case, a derivitive: StringElement)

class FirstName < StringElement

Groups will go from the generic Group(PersonAtAddress) to

class PersonAtAddress < Group

Now, this is not going to be the final syntax, but I want you to think of it as inheritance, because we're going to overload functions.

For instance, an Element should know how to validate itself. A simple example might be

class FirstName < Element
def validate:
return representation =~ /[a-zA-Z-]+/

A more complicated example might make a SOAP call to the validation server defined for that type (we'll see how to do that in a later post).

Elements also need to know how to normalize themselves. For instance, a name might wish to be represented in all upper case:

class FirstName < StringElement
def normalize:
representation.upper!

Another operation you're probably screaming for by now is creation. In this case, the StringElement does the right thing for you, it sets the internal representation to a passed in value. However, you might want to do more. For instance, you might want to keep a map of how often you see each first name to handle statistical based maching. Therefore, you want

class FirstName < StringElement
static Hash seen # would be @@seen in ruby
def initialize(String value):
super(value)
seen ||= new Hash(0)
seen[value]++

Other methods might include update and clear.

For Entities, we need the following operations:

initialize - pass in a value for each of the "member variables" and assign them if they are consistent. Otherwise, throw an exception.
validate - validate the state of an entity
update - update one of the fields of an entity

Update is the most interesting because we could update one or many of the fields. I think it might be interesting to use nil for fields we don't want to update, but I'd rather not. Perhaps a hash? But I don't want to miss a field on accident, and I'd like it to be "compile-time" checked. So, we're back to nils.

class Name < Entity
FirstName first_name
MiddleName middle_name
LastName last_name
NameSuffix name_suffix

def update(FirstName fn,
MiddleName mn,
LastName ln,
NameSuffix ns):
begin_transaction:
first_name.update(fn) unless fn.nil?
middle_name.update(mn) unless mn.nil?
last_name.update(ln) unless ln.nil?
name_suffix.update(ns) unless ns.nil?
rescue => { rollback } #undo all operations in the transaction (maybe a validate fails?)

I'm fairly convinced that a transaction is the right way to handle this situation, but I'm not convinced of the syntax, there are definitely other ways to handle it. For instance, you could have an updater that makes the determination based on the exit strategy of the function.
FirstName::Updater updater(first_name, fn)
It could also handle the nil? case. If the function exits normally, the updater commits - otherwise it rollsback. Regardless, a transaction is needed for exception safety.

You may also delete an entity. I imagine that this should return a boolean on whether or not the delete should succeed, but delete's probably shouldn't fail.

A Group has some additional operations. Since a group is just a collection of entities, it may have an entity added to it or removed from it. It addition, it may also have a list merged into it or part of another list sliced into it.

Let's look at each operation in the abstract:

Add:
original list -> [A,B,C] value -> D new list -> [A,B,C,D]

Remove:
original list -> [A,B,C] value -> B new list -> [A,C]

Merge:
original list -> [A,B,C] value -> [D,E,F] new list -> [A,B,C,D,E,F] new value -> []

The merge adds all of the elements of the value into the list and deletes them from the value.

Splice:
original list -> [A,B,C] value -> [D,E,F], 1, 1 new list -> [A,B,C,E] new value -> [D,F]

The splice takes an array and a begin and end offset and adds those elements to the new list while removing them from the old.

The merge we will call a consolidation and the splice we will refer to as a split.

An important property of a Group involves it's MetaGroup. The MetaGroup is the Set that consists of all the instances of the Group.

So, let's say we have a Consumer Group. We want to say that a consumer can not appear in more than one group. That means that the order of the union of all groups is equivalent to the sum of the order of all groups. To do this, we say that for every group, it's MetaGroup must be a true Set, and cannot include duplicate entities.

Now, back to the operations. Each one of these operations must take place in a transactional environment. For example, an update can fail because of an implied field mismatch.

What's an implied field? A field in a group may be declared strongly or weakly implied. I'm not sure how this declaration will take place (I hate the .NET attribute syntax). However, once it is so declared, it will enforce conformance for all the elements in a group. A weakly implied field will ensure a no conflict match for a field across all the entities in a group. So, if we say that NameSuffix is weakly implied for the Consumer Group, that means all Consumers in the same Consumer Group must have the same Name Suffix (or a blank one). A strongly implied field removes the ability for blanks to match.

Ok, this post has gone on long enough. In future posts, we'll return to the formal specification of these operations.

Type safe recognition

I believe that recognition (part of Customer Data Integration) can (and should!) be made strongly type safe. Basic types (in the US) might include FirstName, LastName, SSN, Street, CompositeStreet. More advanced types would include an Entity (a collection of basic types) and a Group (a collection of entities).

Notice how this mirrors a programming language. A basic type would be something like an int or float. An entity is like an object, and a Group would be a collection of objects (think an array).

This leads to a few interesting questions. Let's consider the following:

Entity Name:
FirstName first_name
MiddleName middle_name
LastName last_name
NameSuffix name_suffix

Entity Address:
CompositeStreet street_line_1
CompositeStreet street_line_2
City city
State state
Zip zip

Entity Person:
Name name
SSN ssn
Date birthday

Entity PersonAtAddress:
Person person
Address address

Entity Consumer:
Group(PersonAtAddress) occupancies
SSN preferred_ssn
Address preferred_address

At this point, we have types for our recognition system that can be manipulated and understood by both humans and computers. We can further augment them. I'll work on describing the augmentation/annotation of the types in the next blog.

Questions:

Should an entity be allowed to contain more than one group?
Should an entity be allowed to contain another entity that contains a group?
For example:

Entity Household:
Group(Consumer) members
Address preferred_address

More to come!

Saturday, June 10, 2006

What to do

My old job gave me many ideas on how a company should be run. The second job I took, at the beginning of this year, changed some of those. Basically, this blog was a running commentary on things I wanted changed at work. With my current job, I have no complaints. It makes it hard to write about. We do code reviews, unit testing and integration testing. We do some pair programming, when it makes sense. We have continuous integration and we hire smart people. We try to do more with less quantity and more quality. It is, in many respects, an enviable situation. So, I'm going to start posting more at Thoughts of He and less here, because I don't have a lot to gripe about, anymore. What a great feeling that is!

Tuesday, June 06, 2006

I'm Back!

I'm back with all new material! How cool is that!

My last employer blocked Blogger, so I stopped posting, since it was a pain to get around the firewall. However, my new employer is much more lenient in what they allow.

What have I learned? What haven't I learned is a better question. My previous job, which lasted all of 5 months, taught me TIBCO and Java/J2EE. However, what I really feel like I took from there is the value of logging and some new techniques on how to retain information. I also learned what management should really be like (and what it shouldn't!). I learned how to do more with less, and how to prioritize based on ROI instead of a manager's whim. All in all, it was a very positive experience and I hope we cross paths again someday.

In my current role, I'm learning how productive one can be when all distractions are taken away. I'm also learning how to test properly and ensure the software works correctly. This is also more "mission critical" software, so I'm getting my first exposure to that side of things.

This employer is also away from my home state, Arkansas, so my family is going through a transition period. I'm not expecting it to be easy, but I'm sure we'll pull through.

Until next time!

Wednesday, March 15, 2006

Smart

Last night on American Idol, Stevie Wonder referred to "Chicken Little"'s voice as "interesting". This was the kiss of death. In other words, Stevie was implying that his voice was horrible and he had nothing better to say. For me, in school, the worst thing to hear was not that I was interesting, but that I was smart.

When you're smart, it often happens that it is your only attribute. No one notices anything else. You're labeled as "the smart kid". Sometimes, you get worse monikors, but we won't disucss those :-) I spent most of my childhood and adult life hiding from that term. In school, I never wanted to tell others what I made on my tests. I never announced my grades like others did. My goal was to blend in completely. In my adult life, I never bragged on my degrees or how fast I got them. At my previous workplace, the only way most people knew I had a Ph.D. was that someone else told them. I never announced it or put it in my email signature or anything else. When I see people with a Ph.D. at the end of their name, I laugh inwardly. "Why would someone want to draw that attention to themselves?" I would ask. When I taught at the college level, I had my students call me by my first name.

Deep down, this was only to avoid the "smart" label. I hated it so much from my childhood that I would have done anything to avoid being labeled with it. But my new year's resolution is acceptance. Last week, I dug my Ph.D. degree out of a box in my office and put it up in my cube. It's a first step.

Sunday, March 05, 2006

ALAR Conference

Long time, no post. However, I'm back from the ALAR Conference and I have a little time, so I'll post. Hopefully, this won't be the last for a while. The ALAR Conference is a data engineering and grid computing conference put on by the Acxiom Laborator for Applied Research. There were a number of great presentations there. I had a paper accepted into the conference, so I got to speak as well.

I brought two important things away from the conference. The first is that I really miss research, especially applied research. I'm going to have to seriously consider how to inject research into my current job, or move on to something new. Research has been a part of my life for so long. In many ways, it defines who I am.

The second thing I took away from the conference was the need to examine distributed game programming. It appears to me that they have some insight on how to handle a massive number of incoming transactions. Sure, they cheat a little, but it might teach us how to cheat in similar or different ways. For instance, they use "realms" to limit the number of players on a single server. Data engineers can also mimic that by the use of a good, pre-defined partition key. They also allow lossy transactions. Since you're going to be sending another transaction in the next milisecond, it's ok if we drop one. I don't know how to replicate that ability with data, but I'm willing to investigate the possibility.

Lastly, it was a great chance to meet up with old friends both from the Corporate world and from the Academic world. I was amazed at how many people I actually knew there.

Well, until next time...

Friday, December 16, 2005

Big Changes

I don't talk about my employer on this blog, but I did want to mention I'm changing employers. It is a very scary and exciting time for me. I have confidence that my new employer will be a great fit for me and will give me a number of challenges and plenty of opportunity for advancement. However, I will be sad to leave my current employer, the company has been good to me and I have a ton of friends I am leaving behind.

Saturday, December 10, 2005

Crossroads

Two companies, in different industries, stand at the crossroads. Both have chosen to make a continual investment in IT in order to beat their competitors. Both believe strongly in their long term plan. I think one will succeed and one will fail. However, until now I didn't know why I thought that; it was just intuition. I don't mind basing a decision on intuition, but I want to explore the decision afterwards to find out what the answer should have been.

In the case of the two companies it was in the IT strategies themselves. One IT strategy requires the continual purchase of depreciating stock. The other IT strategy requires the continual creation of appreciating assets. Both require heavy, year over year investments. The first model, due to depreciation, is actually an anti-strategy. Since your assets are always depreciating, it becomes easy for your competitors to catch up and eventually pass you. However, the second model eliminates the ability for your competitors to catch up because your assets are always appreciating.

Therefore, when contemplating the effect of an IT investment in your company, do your best to ensure that the outcome of the investment will be an appreciating asset instead of a depreciating one.

Saturday, December 03, 2005

Comparing companies

I've recently been studying companies in an attempt to determine the best company to work for. One of the most interesting things that I discovered was that it varies depending on your risk/reward tolerance. To better represent these differences, I broke down companies into a number of different categories, or tiers. Companies across tiers cannot be compared because they have a different risk/reward structure that appeals to different people. Instead, only companies intra-tier can be compared to find a "best".

Tier A - These are start-ups. They are usually in someone's basement or spare room. They have a HUGE risk and can go boom or bust. Most often, they don't even last a year. They require long hours and the ability to live a few weeks without a paycheck.

Tier B - These are larger private companies. They either have not or do not want to go public. However, they are also more established so that the risk is not as large, but the reward is not as large either. They still require long hours, but can be very rewarding financially and socially.

Tier C - These are fairly new public companies; those that have gone public in the last 3 years or so. They still have the chance of doing great things, but much of the initial gain is gone. In addition, their beurocracy is starting to take shape and the system will disolve into process over the next few years. The risk is still there, as public companies can disolve or be bought out in their early, formative years. However, the risk is less than a Tier A and many Tier B companies.

Tier D - These are stable public companies. Usually these companies have a global name and produce hundreds of millions, if not billions, of dollars of revenue. The company is process laden and hierarchical. The ability to make money is diminished to a much smaller chance and the risk is very small as well compared to the other tiers.

Tier E - Government jobs. The reward is your salary and pension. The risk varies, but is often low. Typically, it is a process laden (burdensome) job and nothing more.

Each tier is attractive to different people. There is no point in comparing across tiers except to find the tier you want to work in. Once that is done, you should seek out companies in that tier and compare them. Are they market saturated? How is their health insurance? Is their stock going up or down? What is their pay rate? How many different things can you work on? What are the chances of movement, both with technology or people? Do they value the same things you value? Do you respect their leader?

All of these questions need to be answered in order to lead you to the right company for you, but don't get caught up in comparing cross tiers, for that way lies madness!

Thursday, November 24, 2005

Experiences

We are chained by our experiences. They bind us, control us, enslave us. They provide our definition of normality, by which we judge the world. We can obviously break from the bonds, but it is not easy. People who were abused as children have a higher chance of abusing their own children. Why? Because to them, abuse is normal. Notice that I didn't say right, but normal. I believe they still understand that it is wrong, and they still feel pain and remorse. However, it was what was done to them and therefore, they see it as normal behavior.

My father died when I was 20. To me, not having a father when you are an adult is normal. It seems weird to me when I hear my friends talking about thier parents. I don't have those experiences to draw on, so I have to extrapolate how it would have worked out, but it doesn't feel normal to me. Even as I look at my children's future, I often don't picture myself with them when they are grown; it wouldn't be normal.

How much do we judge people and situations based on our view of normality? How much could we prevent abuse by working to alter people's view of normality? How much do you try to extend your experiences to understand other people's view of normality?

For those who want to dispute my definition of normality, feel free. But, all things in life are tied to perceptions. What is normal is just another perception. In fact, what is right is just another perception - the only difference is who is doing the perceiving.

Tuesday, November 15, 2005

MIS Degree

Over on Code Craft, Kevin describes Three Theories on How to Use Developers Effectively. He describes theories by really smart techies, really smart business people, and the theory of interchangable parts. What does this have to do with MIS degrees you ask? Well, an MIS degree holds to the theory of interchangable parts. The person who seeks out the MIS degree doesn't have a specialty. He is not super technical, nor is he a business man. Instead, he spent one to two years studying different domains. While learning is always useful, I can't imagine how those two years wouldn't have been better spent learning the domain of the company for which you work, or learning the domain of the tools which you will use to solve those problems.

I must say that I understand the temptation. When I finished my bachelor's degree, I thought about getting a business degree because I wanted to help people solve their problems. However, I would have gotten an MBA because that would have given me enough details to understand the domain of the people whose problems I wish to solve. Paul Graham once said that a metaphore is a function applied to an argument of the wrong type. Specializing in technology, pursuing an MS or Ph.D. is designed to give you additional functions and additional types on which to build metaphores. Specializing in business, pursuing an MBA, is designed to do the same thing. Getting an MIS is an attempt to get you more comfortable with the functions and types you already know, which is not nearly as important to me.

If you're going to learn it, then specialize, you can always back up and be a generalist later, but your generalizations will be a lot more correct if you understand the specializations that determine them.

Tuesday, November 08, 2005

House pt 2

This week's house was great. I was just thinking today that they should show him more in a T-shirt and Jeans b/c that is what his personality type dictates he wear. Sure enough, that is what he was in tonight. I would love to meet the writer, he must be my twin. I think I'm going to start wearing a T-Shirt and Jeans to work and see how long I can get away with it. Even if someone complains, how long can I go without some kind of action...keep watching here for updates!

Supreme Nerd

I am nerdier than 94% of all people. Are you nerdier? Click here to find out!

Tuesday, October 25, 2005

Open Source and IBM

Martin Fowler's review of JAOO2005 includes this blurb:

"On panels 'Bedarra' Dave Thomas and Brian Barry said that they believed the current spate of Open Source development was unsustainable. Much support for open source is funded by large companies, IBM's support for Eclipse is a long one. They felt that this support wouldn't last, if so there's likely to be quite a collapse in open source activity. (I'll stay silent on this question, I prefer to avoid trying to predict the future.)"

All I can say is WHAT!?!? Someone doesn't understand IBM's business model. They are not a products company, they make their money on services. Therefore, they want to sell things based on thier services expertise. By making products open source, they eliminate the competitive advantages of their opposition. That allows them to compete on an even keel, services vs services. If anything, IBM wants MORE open source products so that they can sell more integration services. Sheesh!

The Value of Domain Knowledge

I recently gave a talk to the computer science department at my undergraduate university. I was trying to inform them about the state of the industry and give them a little heads up on what they were getting themselves into. One thing that I think they found interesting was my statement that we are merely problem solvers. We are like astronomers and computers are our telescopes. Obviously, being able to use a telescope is important, but our ability to solve problems is our most valuable asset. Computer Scientists have replaced mathematicians as the problem solvers of choice. While mathematicians have their formulas and proofs, computer scientists have the ability to solve problems with brute force. Being half academic, I pine more than wonder at our barbaric problem solving abilities, but my other half enjoys the "get it done" mentality.

As a problem solver, one of our most valuable assets is domain knowledge. Any newly minted B.S.C.S. can turn specs into code, but to be able to have empathy for the user and understand the domain comes only with years of experience. When you ask the user what he or she wants and then you come up with more items than the user does, you have achieved domain expertise. It is akin to the astronomer who can predict the number and size of planets based on the wobble of a star. To a company, this is the most valuable an employee can get. Not only can they contribute to a company by "tuning the telescope", they can also help analyze the results and even help construct a telescope that better measures what they wish to measure.

Where does this leave consultants? Well, there are two different kinds of consultants, and it effects each differently. Some consultants make thier living off of training others. Technology changes at such a rapid pace that schools cannot provide adequate training for everything that you might ever see. These consultants work hard to stay abreast of best practices and then transfer their knowledge to those who hire them. It is similar to teaching earth-bound astronomers how to operate the hubble. Obviously, they never envisioned the hubble when they were in school, so that new knowledge has to be imparted to them in some manner. This is a valuable service, and we should think of it as an extension of school rather than as a consultant.

The other form of consultant is an expert telescope user. This consultant is paid to be very good at using telescopes. He cannot tell what he is looking at, but the telescope is guaranteed to take the best pictures possible. Obviously, this consultant needs a lot of hand holding. In many cases, it is easier to just tune the telescope yourself; however, there are times when the person reading the pictures no longer knows how to use the telescope. This position has a value, but it is small compared to the other two positions I described.

So, as you evaluate where you fit in on the spectrum, see if you provide domain knowledge, education, or just an extra hand. If it is the latter, don't be surpised to see your hand replaced by a cheaper one. If it is the former, and you are being replaced, then you have to seriously wonder about the health of the company. And if you are in the middle, enjoy the moment, but keep learning for the future.

Dilbert Blog

Here it is, Scott Adam's Dilbert Blog.

Thursday, October 13, 2005

Reuse

Reuse is a good thing right? Give the option to reuse good code or write your own, you would always pick reuse. We've heard tales of horrible programmers and their NIH (Not Invented Here) syndrome and we're not like those guys. We want to leave our pride at the door and hold up the shining shield of reuse and the burning sword of standardization.

Not so fast.

Reuse, like ANYTHING ELSE, has its tradeoffs. On the one hand, you get "free code". On the other hand, its not free because you have to learn how to use it, debug it, and write integration code to get it into your project. On the one hand, you get "free upgrades" as a separate development team makes upgrades and enhancements. On the other hand, you suseptible to changing interfaces, newly introduced bugs, and silent but deadly logic changes. You are dependent on that code and its uncertain future.

Joel defends NIH for good reason. His basic premise is something I've been saying for quite some time: software development is the practice of managing dependencies and the dependencies that are easiest to manage are those that are not there. Just like in life, the more dependencies you have, the less agile you are (those with children: how easy is it to do something spontaneously?) - and in the tech industry agility is everything.

Often the development cost of the product increases when you eliminate dependencies because of some duplication; however, the ability to improve the product quickly also increases and that will gain enough income to overcome the incresed development costs. Now, don't get me wrong, you shouldn't go rewrite Windows in order to not be dependant on it; there is something to be said for interoperability as well. However, if it is core to what you do or it plays a vital role in your product then you should own it and it should have a one-to-one dependency chain between you and it. The cost will be well worth the rewards.

Large Company Dilemma

Once your tech company reaches a certain size, it has a dilemma on its hands. Usually, the problem is that it cannot increase its profitability without taking on more people. However, the people available to take on are not the same calibur as the people already on board. Therefore, the company can either refuse to take on additional hands and stagnate or take on average employees and eventually capsize. There appears to be no other outcome. The difficulty lies in taking average employees and making them great, when I'm not sure that is possible. Like turning lead into gold, it is just not within our abilities, today. Of the two options, capsizing seems to lead to the most profit and the capsizing doesn't necessarily mean the downfall of the business. Instead, it could just mean massive layoffs followed by restructuring as was the case with IBM and Apple. However, it seems to me that there should be a way to fill the traditional product units with the new mediocre guys, because their profit has been extracted and then spin off a new company with the masterminds, letting them focus on creating "The Next Big Thing." Just a thought...

Thursday, October 06, 2005

Double Entry Bookkeeping

I don't remember if I mentioned this on my blog, or someone else's, but I view TDD as double entry bookkeeping. Apparently, Uncle Bob does, too.

Update: it was on my blog.

Wednesday, October 05, 2005

The 4th Quarter

I've been recently wondering if always having to win the game in the 4th quarter makes you stronger or weaker. When you have to come from behind and fight for your survival, does it give you mental toughness or does it just drain you, emotionally? What happens when you finally lose, is it just another loss, or is it the proverbial straw that breaks the camel's back? How much can you take when it is always on the line? Interesting questions, no answers. Keep coming back for updates.

The trip

So, you're going on a trip from San Diego to Buffalo and you've decided to travel by automobile. You now have two popular options: drive your own car or ride in someone else's. For the moment, we're going to ignore driving someone else's car or riding while someone else drives yours.

If you decide to drive, you get to pick what car you take. You can take the gas guzzling SUV or the environmentally friendly hybrid. You also get to choose who goes with you. I hope you choose who goes with you for good reasons. You might pick Bob because he is friendly and good company. You might pick Ann because she is great with a map. You might pick Fred because you know he doesn't have to stop at a bathroom every 30 miles. Finally, you might pick Jill because she offered to pay for the gas if she could tag along. One thought is to pick people who want to go in the same direction as you. Perhaps they don't want to go to Buffalo, but going to Seattle or Tampa Bay would probably be a little out of the way.

If you decide to ride, your options are different. First, you need to find someone who can take you. It could be that you have to get on a greyhound bus: they are cheap, fast, fairly safe, and if it breaks down you don't have to worry about fixing it. Or, you might have a friend or acquaintance driving their own car and they invited you to come along. Often, the invitation process is stressful and intrusive, but once that is over with the ride can be quite nice. There are additional things to consider, things over which you have no control. For instance, who are they taking with them? Are they taking Reggie, the loud obnoxious guy you can't stand, or are they taking Gina, the cute secretary from across the street? These decisions can definitely affect whether or not you choose to ride. Of course it could be that you don't own a car and Reggie is your only hope, but we try not to be in that situation, don't we. We brush up on our map reading skills, learn how to control our bladder, and try to save enough money to help buy gas. Then again, it could be that your best friend is going to Columbia and has asked you to come along. Yes, he is your best friend, no you don't have your own car; however, going to Columbia (the country) does not help in your quest to get to Buffalo, so it is best to avoid that trip. It also helps to examine the car you will be riding in. Do you think it can get to Buffalo? Are there already too many people on board? Has the driver engaged in preventative maintenance? It could be that you just hop in the first car available and blindly wish for good fortune, but that inevitably leads to disaster. The best thing to do is to know yourself and choose the transportation method that is right for you.

Sunday, September 25, 2005

Paid Apprenticeships?

A minor rant. I hear now and again someone talking about the old craftsman ways and how an apprentice used to pay his master for services. Now, we expect our first job to pay us well and it is a terrible injustice. To that I can only say that we do pay for our apprenticeships, but it is now called college. Whether you agree or disagree that it is a valuable apprenticeship, it is one just the same and probably had the same value now as it did back in the olden days. So, the next time you think you ought to get paid for taking on an apprentice, go teach school.

Thursday, September 22, 2005

More on Company Death

I happen to like the beekeeper story, but I think it is incomplete. The author blames a company's demise on the leader leaving or changing, but I don't necessarily believe that is it. I think projects get larger and mature and they change in nature. Take Office. It doesn't have to get a ton of new whiz bang features out immediately. It has market share and it needs to maintain its dominance while not hurting quality. To do that, you add processes in place to control the change in the product and to ensure integration with other products. I'm curious if this is just natural product evolution. It doesn't have to be company wide. For instance, at MS the XBox, MSN Search, and V. Studio teams are still quite agile and seem to enjoy their jobs, where the larger, integrated systems people are miserable. Is it the leader? Is it integration? Is it complacency (wanting to maintain your product instead of revolutionize it)? Is it all 3? Why do some companies impose process in some areas and not in others whereas other companies impose process over the entire company at one time? What is the survival rate of each?

On complacency, I think that is a valid reason for a while. Office, in the past 5 years, has not had to revolutionize. I think that the open source movement and OpenOffice in particular has forced its hand and required Office 12 to be produced. However, therein lies the problem. When a new challenge faces a process laden product, it can't adapt. The top programmers have left to find projects with less process and processes have been inserted to maintain the quality in their absense. Now, the mediocre programmers are faced with a challenge that is too great for them, create a revolutionary project in Internet time. The processes don't allow it, the programmer's don't have the skills, and the result is a buggy product that is late and not close to what users expect or need.

Do I think MS is on its death throws? Most certainly not. It is too big to go down the tubes over night. However, many people thought the same thing about IBM and it lost huge amounts of market share and almost was divided into many separate, smaller companies. It took Lou Gerstner to turn the company around, eliminate the beurocracy, and get the company agile again. Coincidentally, Lou named his book, Who Says Elephants Can't Dance. It is a must read if you are in the technology sector as it directly addresses a company's shift from beurocracy to agility. Making a large company dance is an amazing feat. In Lou's opinion, it took a leadership change to make it happen. In fact, had he not come in time, they would have spun off many IBM divisions and it would have continued to decline. According to my interpretation of Lou, agility is not something that can come bottom up, it must be a leadership decision.

What about Oracle? How does it handle process? What are the other older and bigger tech companies: SAS, Apple, Sun - what do they do? What about larger tech companies that have failed: DEC, Compaq - what did they do wrong. I've studied Apple's problems and they seemed to stem from arrogance and a fundamental lack of marketplace economics, not process. It was more an unwillingness to change rather than an inability to change. Looks like it is time to hit the books...

The Other Side

I really don't want to promote this guy's blog, but in the interest of fairness, I'll show the other side of the coin. This post describes why a business man believes Microsoft needs more process and more beurocracy, not less.

They just don't get it. There is a known connection between process and productivity, one that until recently I forgot. When labor unions want to strike, but cannot for whatever reason, they often strike "by the book." What this means is that they follow the process manual to the letter. Basically, no work gets done because all of the process gets in the way. It is a very effective tactic. The same holds true for companies, as process is added, productivity drops. The blog talks about the problems with bugs in the old MS software and how process should fix that. WRONG. You can't "process" the bugs out of the system. The delayed release of Vista, Monad, and Office 12 because of bugs shows that process doesn't help that. Instead, it drives away your best developers, causing MORE bugs, not fewer. It just goes to show that business people don't understand the relationship between quality craftsman and quality craftsmanship. They like to think in assembly lines and mass production. I hate to tell them, but the software industry isn't there yet, and may never be.

Of course, many companies don't need the best. They can settle for a prefabricated home instead of a custom built mansion. However, if you're life's work is hosting parties, then you better opt for the mansion and you better make sure it is built well. The same goes with software companies. If you're life's work is making great software, then you better hire and retain the best in the business.

Wednesday, September 21, 2005

The Perils of Integration

Thinking more about mini-microsoft and what has been posted by him and his readers, I wonder how much a company's downfall has to do with integration. When a company decides to integrate its systems into one mammoth system, it seems to spell doom. I think the process goes like this:

First, some senior executive decides that everything needs to be "integrated." This could mean that everything needs to work together, run on the same box, or in the same environment, who knows the exact definition?

This decision spawns a massive project and a few reorgs. The massive project introduces hundreds, if not thousands, of dependencies between teams. Managing these dependencies becomes too much for the teams and therefore process has to take over. The system is fragile and buggy, but any change to the system has to be approved by everyone else, and may cause everyone else to recompile or recode their part. Therefore, changes are to be avoided at all cost. It begins to take longer and longer to put out a release.

At this point, the individual teams have lost all power. The power is centralized into a few leadership individuals who make decisions, but don't know the technical details of those decisions and therefore can't understand the ramifications. It's not their fault, there are too many dependencies for ANYONE to understand all the ramifications. The teams can no longer respond to customer needs; they'd like to, but their hands are tied because if they change the Foo widget then it will cause a recompile of 5 other libraries and 2 months of testing - and the testers are already running on empty.

The best programmer's get frustrated. They can see the problem, but they can't put their finger on what exactly is wrong, plus they spend more time filling out forms and "communicating" then they do programming. They get disgruntled and leave for greener pastures. Those who remain are left maintaining systems that they didn't build and changes to the system are slower and more painful than before.

Finally, one of two things can happen, either the company can fold, or some spark occurs and the old, integrated system is left for dead and a project is started to build a new system that relies on customer teams and independence. The best programmers are hired to create the system and it becomes a success. Then the cycle repeats itself again.

Integrated systems are the hardest things to do correctly. As I've mentioned before it is our job to manage dependencies. The best way to manage a dependency is to eliminate it. With an integrated system, you have the potential to create an exponential number of dependencies, thus ensuring the demise of your company. Now, the question remains: how do you create an integrated system correctly? I'm not sure, but I'm going to think on it, so keep checking back here for the/an answer.

Mini-Microsoft and the Death of a Corporation

Mini-Microsoft is an anonymous Microsoft employee that is trying to change his company by airing its dirty laundry, publicly. I don't know that I agree with his approach, but I certainly feel sorry for him. It is sad to see a corporation leave the expansion phase and go into the recession phase, but all the signs are there. They have reduced or stopped raises, their earnings are slumping, they are using reorgs as a way to increase productivity, they are using process over people, and their best people are abandoning ship while those that stay have low productivity due to low morale. Microsoft had one of the longest expansion periods I've seen. Now, the giant has peaked and it must move into a recession phase. This is not to say that sometime, in the future, it can't expand again, but it must fall hard before that is to occur. Some of its products will decline as well, some have already been in decline. Some of its products may even excel: I'm a huge fan of C# and I think it will do well, though lack of true multi-platform support may hinder it (sorry Mono, you're not there yet). The warning signs are clear and its sad for the 50,000 people that work there. So long and thanks for all the phish.

C++ Tip: lower_bound and functors

Here's an interesting thing I learned recently via necessity. Let's assume you have a vector of string/int combinations.


vector<pair<string, int> > myvec;


Now, lets assume we add data to the vector, in sorted order (based on the string).


myvec.push_back( make_pair( string("A"), 0 ) );
myvec.push_back( make_pair( string("B"), 1 ) );
myvec.push_back( make_pair( string("C"), 2 ) );
myvec.push_back( make_pair( string("D"), 3 ) );


Finally, we want to search for a string key in the sorted vector using lower_bound.
We can't do the following:


lower_bound( myvec.begin(), myvec.end(),
string( "C" ) );


It will complain that there is not an operator<(std::pair<std::string,int>, std::string). Well, that is annoying, but we can write one. However, it is usually better to not clutter the global namespace with such functions and instead we choose to create a comparator function.


bool pair_str_less(const pair<
string,int>& lhs,
const string& rhs)
{
return lhs.first < rhs;
}


Now we pass in the comparator function

lower_bound( myvec.begin(), myvec.end(),
string("C"), pair_str_less );


Ouch! Now it complains that there is no pair_str_less( std::string, std::pair<std::string, int> ). It needs both permutations of function arguments! Unfortunately, we can only pass in one comparator! Perhaps it is not meant to be? Maybe we should just write our own version of lower bound? Or perhaps we should create a dummy std::pair from the string? Or, perhaps we can use functors!


struct pair_str_less
{
bool operator()( const pair<
string, int>& lhs,
const string& rhs ) const
{ /*...*/ }
bool operator()( const string& rhs,
const pair<
string, int >& rhs ) const
{ /*...*/ }
};


Now, we can use the pair_str_less function like


lower_bound( myvec.begin(), myvec.end(),
string("C"), pair_str_less() );

and the right version of operator() is chosen each way. Yeah!

Tuesday, September 20, 2005

Process vs Best Practice

I'm tired of process. Yes, I know it is supposed to help. Yes, I know it encourages responsibility and increases quality and decreases entropy and blah blah blah, but it is WRONG.

If you hire the smartest people, give them the best tools, and let them work, they will do the right thing, intrinsically. You only need process when your people have let you down. In fact, when you institute a mandated process, you are slapping your employees in the face. You are saying, "I don't trust you to get this right, so I, your leader, have mandated that you must follow these steps." The best people will follow the best practices. They will use version control, bug tracking, and change management because it is the right thing to do, not because someone forced them to do it. Moreover, they will also be able to choose when not to use it. The problem with process is that it must be blindly followed. It will bog your people and your company down in its forced standards. It will lower morale due to trust issues. It will lower morale due to developers being forced into the "one right way" (developers naturally resist standards and blind obedience).

Get the right people on the bus and get the process off the bus. Let the people find the best practices and let them have a great time producing great code.

Sunday, September 18, 2005

C# 3.0

There was a recent post on slashdot sending people to view a video of Anders talking about C# 3.0. For those who don't know, Anders Hejlsberg was the creator of Delphi and the Object Pascal language. Microsoft hired him away from Borland to oversee their .NET languages. While I think C++.NET is an abomination, C# is an amazing staticly typed language. It has stayed away from all of Java's problems and has adopted a much more agile flavor: very ruby-ish in nature. I've been reading the O'Reilly book on C# 2.0 and I've been impressed, but 3.0 looks like it will blow all other static typed languages out of the water. Kudos to Anders for giving us a language worth using. Now, I wonder about performance...

Another thing I found interesting about this video was Anders continual discussion of the "shape" of data. I've heard this term before (I think it was in Code Complete), but it had been a while. Acutally using C# to show the difference between rectangular data and heirarchical data was very interesting to me. It is something I'll have to start thinking about on a regular basis.

Dependencies

One of the most important jobs a programmer has is to manage dependencies. In fact, many of the Code Smells that we learn to avoid have their roots in the removal of unnecessary dependencies.

Especially evil are implicit dependencies (i.e., dependencies that are not easily noticed/not explicitly written). For example, once and only once is about the removal of implicit dependencies. Why? Because all instances of duplicated code depend on each other. If you have the following lines

int i = 0;
for( ; i < 10; ++i )
{
...
}
i--;

and you decide that the last i-- is a bug, then you have to remove the last i-- in every instance of the duplicated code - they are all dependent on one another. What's worse is that there is no semantic information associated with this code. It could be that one instance of the duplicated code actually precedes something slightly different so that the i-- is actually needed. Because there is no semantic information tieing all the pieces together, there is no way to know which i-- should stay and which should go.

Magic numbers, being another form of code duplication, have exactly the same problems; however, the problems with lack of semantic information is exacerbated with magic numbers. In the code above, there is no knowledge of what the number 10 represents. When you realize that it should be 15 do you change all the instances of 10 to 15? If not, how do you know which ones to change and which ones not to?

Dave Chelimsky discusses how to Manage Dependencies Not Aesthetics in his blog. He uses the following example:

display.print( person.getAddress().getZipCode() );

and then compares it with

display.print( person.getZipCode() );

and

person.printZipCode(display);

Which one has its dependencies managed correctly? Let's examine where the dependencies are. In example 1, there is a dependency from the main code to display, person, address, and zip code. There is also a dependency between display and zip code. Even though zip code is just a string, there is a logical dependency between the concept of a zip code and display.
In example 2, there is a dependency from the main code to display, person, and zip code. The dependency on address has been eliminated. There is still a logical dependency from display to zip code. In example 3, there is a dependency from the main code to both person and display, the dependency to zip code has been removed (even though the function still says zip code, the zip code is not seen or used in the main code, so there is no dependency). There is an additional dependency between a person and a display object.

Now, elimination of dependencies is not the goal, only the effective management of dependencies; however, it is much easier to manage a dependency that doesn't exist, just like it is much easier to maintain code that doesn't exist. I think we can quickly eliminate example 1 as being the best because it has no advantages over example 2 and an additional dependency. Example 2 does have an additional dependency over Example 3, but there are some trade-offs involved. With example 2, if the zip code changes, then the display class must also change. With example 3, if you want to write to something other than a display, then the person class must change. So, the question becomes which is more likely to change. Obviously this will change on a design by design basis.

Interestingly enough, the comments section of David's blog dives off into a discussion of formatters and visitors and how to isolate the change, but each introduces more dependencies in an effort to find the best dependency combination that is least likely to change. In this case, I might stick to the TDD principle of doing the simplest thing possible and then refactoring when we actually see the change, at that time you can make a more informed decision about what needs to be changed and how the change needs to occur.

Friday, September 16, 2005

J - A - C - K - E - TS

My daughter just cheered at her first game. Kindergartners through fifth graders had the chance to be "mini-cheerleaders" at tonight's football game. My daughter was very excited. She had her cheerleading outfit and her yellowjacket face tatoo. She knew the cheers and she was ready to go. I'll post some pictures of it over the next day or two. She did a great job for only being in Kindergarten. I was very proud of her, as always.

I certainly don't encourage cheerleading. I've known very few cheerleaders that I actually respected (two). I tend to think the "sport" is for the flighty, energetic people that I typically don't associate with because they have way too much energy and way too little intelligence for me. However, I made a commitment many years ago to support her in whatever she wanted to do (as long as it was legal and moral). I'll stand by that even if she wants to be a cheerleader, but I'm sure she won't -- she's way to smart for that ;0)

Tuesday, September 13, 2005

DRY needlessly

DRY, don't repeat yourself, has become a battle cry to many in the programming profession. Wiping out all causes of spatial, temporal, and logical repetition is the aspiration of today's elite. However, zealotry has never suited me, and I tend to think that there exists good repitition as well as bad. Let's look at some examples:

First, Test-Driven Development is full of repetition. The tests and the code define the specification separately, but equally. Yes, you can make up all sorts of excuses for why there is duplication, but the fact remains that both the tests and the code are asserting the same thing.
Second, in Code Complete, Steve McConnell talks about the practice of using a function to check another function's work. For instance, if you are making some difficult calculations, you may wish to write a slow and steady function that you know works and then write a swift and dangerous function that you are unsure of. In debug mode you would run both functions and ensure the identity of the results.
Third, web programming often has client and server side validation of input parameters. Yes, I know you could write a control file and have both generated from that, but I seriously doubt that is a big enough win to justify the programming time involved. The duplication is accepted as useful.

In each of these cases, we're using repitition to enable a sort of "double-entry bookkeeping". We perform our work and then check our work to make sure everything balances out. The duplication is there for a reason, to ensure quality and correctness. The duplication that DRY is intended to eliminate is not there to support itself, but is instead needless. Think of it as a teepee. You have two sticks leaning different directions, supporting one another. There is repetition, you have two sticks instead of one, but they are working together and leaning on one another. This is the key, if you have duplication in your code, make sure the duplication is leaning in opposite directions.

Monday, September 12, 2005

Forty Things Wiki

I've created a wiki where I can post my thoughts and preliminary drafts of Forty Things Every Programmer Should Know...But Doesn't! I'm using phpWiki because it was a breeze to set up and should be just fine for what I intend to do. Please feel free to contribute to The Forty Things Wiki, but be sure to leave your name so I can give you credit!

Sunday, September 11, 2005

Rake

Martin Fowler has a post on how to use Rake (a Make/Ant alternative written in Ruby). I'm going to give it a try sometime this week or next. I'll let you know how it works out.

40 Things Every Programmer Should Know

On my website I have started an outline of 40 Things Every Programmer Should Know...But Doesn't. There are eight categories consisting of 5 topics each. Obviously, most programmers will be familiar with some of the topics, and some will be familiar with many, but the hope is that there will be a few nuggets of interest lying in wait for the curious programmer willing to read my "book". The goal is not to have an in depth treatment of each topic, but a high level overview of what it is, when you would use it, and how and why it should make its way into your everyday programming life. I hope to complete one topic each week (or at least a rough draft). Keep checking in for progress reports.

Tuesday, September 06, 2005

Football pt ?

I have no idea what part I'm up to, but today I heard Dick Vermeil talk about Herm Edwards as someone who was an expert at his craft. For all the talk about software development being a craft, we now have talk of football also being a craft...the similarities continue.

Monday, September 05, 2005

Two Books

I read two books over the weekend, The Pragmatic Programmer by Andy Hunt and Dave Thomas and Programming Pearls by Jon Bently. Both were excellent books, though entirely different. The Pragmatic Programmer was about how to write robust systems and really began the craftsman movement. It discusses automated builds, regression tests, prototypes, estimation, source code control, text manipulation, code generation, and much more. It gives gems such as the correct answer when your boss asks how long it will take (the answer: "I'll get back to you."). It is about producing great software at all levels and is very similar in spirit to Steve McConnell's Code Complete, though it deals with design decisions much more.

Programming Pearls is about design in the small. It discusses how to choose the correct data structures and algorithms, how to determine complexity, and how to write efficient, fast code. If you didn't have a CS background, it would be an excellent book to give you a crash course. If you do have a CS background, it is an excellent in-depth view of small design decisions and is a timeless classic.

I definitely recommend both books and give them ***** out of *****!

Automated Build

Since I couldn't do much physically this weekend, I decided to practice what I preach and create an automated build for one of my projects. First, I'm embarrased that I hadn't done it sooner. Automated builds are one of the top three things that make a project successful. It is rather pathetic of me that I hadn't yet done it. However, I took the time this weekend and did a decent job of it. I hope that next time I can get it done early in the project instead of procrastinating like I have. In all fairness to myself, I have been working on some necessary infrastructure components so that the automated build can be easy to create and maintain, but I still should have got it done sooner. Anyway, it is done now and I can breathe easy!

Saturday, September 03, 2005

Scooby dooby doo...

Where are you?

Well, I haven't been able to post too much as of late because I've had gall bladder surgery. It really isn't that bad anymore, it is an outpatient operation. They do it laproscopically, so you only have a few small incisions. I'm a little sore, but not too terribly. Hopefully, I will be back to blogging regularly again in the next day or two, at least by Monday. See you then!