Applied Research

At one point in my career, I had the experience of being on an applied research team. In fact, I was one of its founding members. At the time, I wasn't sure what applied research was; even now, I can't say for sure I know.

For our team, applied research was new product development. Honestly, I think that had disastrous effects. We were in limbo between delivering a new product and maintaining our "research" mantra. In the end, we wanted product teams to adopt and maintain our product. This, too, led to tension. Product teams want to feel comfortable with what they will eventually own. They want to use and support things they have used and supported in the past. Research products are wild-cards and not to be used unless nothing else like it exists. Even then, most product teams will choose to rewrite it. Take, for instance, Pig. Pig is a dataflow language created by Yahoo! Research to run on Hadoop. It was created to prove a point, but it was also created to be used. However, once it transfered to a product team in Yahoo!, it was scheduled for rewrite. I'm not sure the cause. It could be Not Invented Here syndrome. It could be that research teams are not focused on the long term, so they deliver a short term product. It could be that research teams don't know how to code or that product teams and research teams speak a different language, so they can't see eye to eye. Regardless, having a research team deliver to a product team is doomed to failure, if you count failure as an eventual rewrite of the product or technology.

To help guide my thoughts on this matter, I had the good fortune to speak to Hector Garcia-Molina of Stanford University. If you don't know who this guy is, you are missing out. Stop and go read the wikipedia entry linked to above. He recounted to me his description of what a research organization should do. I'll recount here and embelish a little.

First, a research organization should publish. The benefit of publishing is that it brings notariety/publicity to your organization. The company's sales force can go to battle with slides that reference conference proceedings. Of course, no sales person would read the proceedings, but it does make their presentation look legit. In addition, presenting at conferences allows the researches to make the aquantence of people like Dr. Garcia-Molina and bring new insight and innovations back. Finally, publishing creates an attractive employment environment. New graduates from top schools want to go to a place that has a publishing history so that they can continue their research. To get a graduate of MIT or Stanford, you need a publication record. I'll add one more to his list. I think a research department that publishes shows a commitment to innovation by the company; a commitment that roots itself in the culture and makes the company a hotbed of innovation.

Second, a research organization can be used as a SWAT force. Tackling a hard problem or subset of the problem. I think this can be the area where the most "good" can be done internally. Instead of creating a product, create an extension to an existing product. There are always "next version" features that never get created. These features either provide too little value or are too difficult. It is that latter segment where the research department can really shine. Since they are not constrained by time to market, etc..., the research department can think outside the box and take the time to create an "academic" solution. For example, PageRank was an academic solution to the problem of which web sites were more popular. It is a beautiful recursive algorithm that just happens to produce great results. Is it a perfect algorithm? No, of course not. Was it better than the "engineering" algorithms of the time? You betcha! It was what happened when two academics got together and had the time to think about the problem. They realized the probelm was similar to that of research paper citations, so they devised an algorithm that treated the problem thusly. Had they had to meet an arbitrary deadline so that their employer could make the trade show deadline, they would not have come up with PageRank. That is the benefit of a research team. Not to create some fancy product for the trade show, but to take the existing product the next step. To finish out the "next version" features and do so elegantly.

The final task for a research organization is to benefit the company at large. There are many ways to do this. One is to create best practices and spread them out to the other development teams. Another is to investigate various technologies and report on how they could/should be used throughout the company. An additional way could be to create the infrastructure to ensure developer productivity if no other team is responsible for it. Do developers have access to distributed key/value stores? If not, install MemcacheDB or CouchDB, and help developers to connect to it by creating modules in various languages. Do developers understand and have access to technologies like Hadoop and Pig? If not, create demos and give a roadshow. All of these things lay in the "best practices" umbrella and often organizations don't have the teams in place to create and distribute the "know-how" required to use them.

In all three cases, the applied research team stayed away from the product team's core product. Instead, the research team focuses on small, manageable pieces that fit nicely into the strategy already outlined by the product team. This creates trust and will foster future collaborations between the product and research teams.

Good luck and let me know of your experiences dealing with research teams!