Parallax : Calculating Technology's Future

Putting the Crowd Horse Before the Cart

In my recent post The Wisdom of Sales Crowds i talked, among other bigger things, about the idea of crowdsourcing your dating choices. It seems like the folks at SideTaker probably agree with me that crowdsourced dating is coming and have already prepared a way to crowdsource the inevitabnle arguments you'll have in those relationships. I see a merger in the future!

The Wisdom of Sales Crowds

I can’t stand reading business books. I think they’ve become an inherent oxymoron; many extolling bold new ways of creating personal or business efficiency while somehow justifying 270 pages of my valuable time to explain it. Bugger!

At first pass, James Surowiecki’s “The Wisdom of Crowds” falls right into this category. The book is split into two parts – the interesting part and the part that makes the book thick enough to charge $14.95. The best business books though are the ones that have psyche-burning memes; those annoying simplicities that stick in your head like the catchy chorus of an 80’s pop song, constantly bubbling into your conscious long after you’ve put the book down. The “Wisdom of the Crowds” is Wham!-like in its ability to do this.

The basic premise of the book is (shockingly) that crowds of people can actually be pretty smart. The unintuitive part of the argument is that in certain situations the crowd is actually smarter than any individual expert in the group. To some extent this is an interesting counter argument to one of my favorite thought pieces, Jaron Lanier’s Digital Maoism. The classic example given is the gumballs-in-a-jar game where a group of people is asked to pick the number of gumballs in a jar. While the individual estimates vary widely, systematically the averaged guess of each person in the crowd ends up approximating the actual number of gumballs more accurately than the best individual guesser. It’s a bit of a trite example as I’m not familiar with many gumball counting experts - but let’s not move too quickly here, we have 269 pages to go.

A more impactful example would be the long term success of fund managers versus market indexes (fund managers rarely outperform the market over a long period of time). Another example would be the Iowa Election Markets where the “crowd” of people buying and selling political outcome futures has historically been quite accurate. “Wisdom of Crowds” is littered with other examples covering politics, economics, gambling, social experiments, and beyond.

Surowiecki’s justification for this phenomenon is that in most groups of people you have a distribution of localized or limited knowledge (which can also be regarded as a bias). By aggregating localized knowledge and bias, the outliers tend to cancel each other out and the resulting aggregate wisdom approximates reality. This is a sort of crowdsourcing version of dropping the high and low score in the Olympics to approximate the real value of any 15-year-old Chinese gymnast’s floor routine. Another way to consider localize phenomena’s affect is to consider that sometimes localized (or biased) information is actually the most influential to the greater outcome. Take for example Palin’s VP nomination. Some people will have a bias that her political earmarking will have a large affect on people’s opinion of her while others will dismiss it as a triviality. The results will show that one of these answers is correct. If you expand this information to include the 100 other accusations that have and will come up, somewhere there is a permutation of specifics that will affect the outcome. 

Surowiecki further argues that in many situations if you add in even a meager financial incentive to be right and allow individuals to refine their guesses over time, you tend to get an even closer approximation of the actual results (bookies evening out bets on both sides of a football game would be a good example of this). In general this tends to work well in situations where new information and developments over time will affect the actual results (e.g. new information such as the weather on a day the football game will be played or the outcome of the 2008 presidential election).

Okay, so enough about the basics – you get it. I just saved you $14.95 so don’t say this blog never did anything for you.

CrowdMatch.com

So I was sitting around trying to think of all the situations where the wisdom of crowds could outperform the experts. And then it came to me as I saw a commercial for Free Communication Weekend on eHarmony: dating!

I have been on very few blind dates in my life (with paltry success). I think the reason is that I don’t trust any one individual in my “dating recommender group” to understand what ends up being a good match for me. Usually an individual’s recommendation is based on what they want for me (or their friend) and not necessarily what I will enjoy. So it dawned on me, why not throw this decision out to the crowd in some kind of crowdsourcing fashion?

Here’s an experiment you can all run. Get 15 of your friends, family members and perhaps even exes (yikes) to sign up for eHarmony’s next free communication weekend. Have them all peruse the dating selections in your local area. Each person in your “crowd” would pick who they think is the best match based on their own biases about you. Perform some sort of reduction in a few rounds of elimination where each person must again pick who they think is your best match, but from the increasingly limited pool of possible choices. Finally you’d whittle it down to one (or a few) people that should theoretically be your best match. Why not outsource your dating choices to the crowd as they should supposedly be smarter than you (the self anointed expert of the crowd)?

The reason this might work is based on Surowiecki’s argument for local bias. Some people in your recommender crowd are going to be more sensitive to a physical match for you (assuming they know what you like), some are going to focus on hobbies you have in common, and some may focus on beliefs or career choices. Through the aggregation of these biases a true pastiche of what might make you happy may emerge. Come to think of it, this would be a really interesting application to build on Facebook. People propose friends from their network that they think you should date and then your friends effectively narrow down the field to the one (or few) people you should go out with.  When CrowdMatch.com turns profitable please just remember you heard it here first!

But Seriously..

While I believe that the results of CrowdMatch.com would have some legs (all puns intended), it dawned on me that there was likely a more valuable business problem in wait of a crowd solution. I think I have found the perfect problem. This situation is one that many of us have found ourselves in countless times; a situation where traditionally the (supposed) expert is listened to over the crowd’s opinion and where the results have historically been very poor.

Sales forecasting in startups should absolutely be determined by the wisdom of the crowd.

Let’s review why:

1) Sales forecasting in startups is generally mediocre at best (so it’s a problem that needs a different solution than the status quo)

2) The forecast is usually compiled by one person who is supposedly the expert (the VP Sales)

3) Every person in the sales ecosystem has only a limited amount of information during the quarter (only certain people have directly interacted with prospects, sales people are regionally focused, there may be limited knowledge of the true deliverability of a product in the quarter, etc..)

4) The expert simply cannot know all the localized information that impacts the outcome

5) New information enters the sales ecosystem constantly throughout the quarter

6) Every person in the sales ecosystem has a bias based on personal experience, style, compensation plan, or management approach

For a long time now I have had the pleasure of working with Wendy Lea. Among many other things, Wendy ingrained in me a fundamental belief that sales is not art, it’s a science. When you find yourself in a pipeline debrief and you notice you’re starting to hear more stories than data you know you’re dealing with a VP of Sales that either has no good news, has little definitive information, or doesn’t fundamentally believe in the theory of sales being a science not art. As a side note, Wendy and I have also worked Seth Levine in a number of board situations and he wrote a fantastic piece cum plea to rationality on his blog about this exact subject. The point of this preamble is that I have come to believe strongly that sales forecasting should be a rational and predictable exercise. So why is it so damn hard to get right? I’ll attempt to argue Surowiecki-style that it’s because we have learned (poorly) to trust the expert and not the crowd.

As background, I have always been close sales but not directly responsible for it. That’s allowed me somewhat unimposingly to have a lot of private conversations with people in the sales ecosystem about their own predictions. Will a certain deal close or not? What will the eventual size of a specific deal be? Will we make the number? Usually their predictions are based on their personal experiences in the sale process either working directly with revenue sources (prospects, existing customers in the pipeline that you plan on upselling) or their own interactions with the sales team (if they are not directly a quota carrying sales person). In hindsight, inevitably somewhere in the cacophony of conversations there have almost always been pretty accurate predictions for my questions. I’ll also note that rarely is it the VP of Sales that, no matter how much he or she talks to their team, has the closest prediction of the final quarterly number during the first couple months of the quarter.

The last point is particularly interesting because it shows a weakness in the expert driven forecasting model. Depending on the dynamic between the VP Sales and your sales team, answers to various questions are going to vary widely depending on who is asking the question (a peer like me or a manager like the VP of Sales). I believe this is a trickle-down of bias phenomenon. CEOs are inclined to stay optimistic to the board and thus the VP of Sales is inclined to stay optimistic to the CEO (even many times in the face of abundant information to the contrary). And thus sales people are inclined to stay optimistic to the VP of Sales. When peers talk about the quarter they predict without this implicit bias.

The Wisdom of Sales Crowds

So this takes us back to the wisdom of the crowds. Why not leverage the crowd to get the best possible forecast of the quarter as it moves along. Theoretically the collective perspective of the crowd would almost always be the best guess at any point in time. That doesn’t mean that the collective guess of the crowd would be correct on the first day of the quarter but it is likely that it would be the most accurate possible prediction given all the information available at the time. As the quarter progresses this guesstimate should get closer and closer to the final number in theory much more accurately than the VP of Sales could individually give. How many board members have been told by a VP of Sales that the quarter will be X only 2 weeks before quarter end and the final number is something dramatically off X (unfortunately almost always lower).

What’s fantastically counter intuitive to the classic expert-driven model, per Surowieki’s analysis, is that the more diversity you have in the crowd of estimators the better. Why not let sales engineers participate in the prediction? What about professional services? What about other executives that are exposed to prospects during sales discovery, or product managers who might have the most current customer conversations at their disposal? This diversity, while it might seem to allow uneducated opinions into the mixture, tends to lead to more accurate results. To compound the problems of a crowd limited to just direct sales people, remember that sales folks are many times driven from a perspective of wanting to hear that there might be a sale. Someone like a sales engineer who might not be comped on making the sale directly might be more inclined to hear the few “not gonna happen this quarter” signals thrown out by the prospect. The diversity should balance out these situations where an overly optimistic sales person might be counterbalanced by an overly pessimistic sales engineer.

What’s interesting about this is that other than informally no one I know does this, or even does anything like it! To some extent this is the result of hierarchical structures in an organization. In the same way the VP of Engineering is looked to for the definitive reality of a release schedule, the sales VP is looked to for the definitive reality of the sales forecast. Come to think of it, the wisdom of crowds here could be as equally well applied to project deadlines as to sales (engineering, professional services implementation, etc..). One unique oddity of sales though is that the VP of Sales, in my experience, has a more inherent need to manage and control expectations (even if they are dead wrong in the end). Perhaps it’s the quarterly pressure of predictions (engineering teams don’t get lambasted if they deliver a product on October 4^th instead of September 28^th) or perhaps it’s just bad old school behavior that we’ve not unlearned to question. While crowd predictions might be broadly applicable, for now let’s just focus on solving all the problems of the sales world.

Sales and Marketplace

So how would you implement this? While poor man’s approaches might work (an email survey here an email survey there), Suroweicki provides a much more elegant approach. Remember that predictions sometimes have more accuracy if there is an incentive to be right. A simple way to engage the crowd would be to set up a sales forecasting prediction market just like the Iowa Election markets. This market would allow anyone in the sales ecosystem to buy and sell shares in “forecasting assets” about quarterly sales outcomes. Consider a marketplace with the following “assets” that could be bought and sold freely between market participants (for real money):

1) Final results of the quarter in increments (say $750,000-1M is an “asset”)

2) Whether a certain deal will close (e.g. “ECOLab will close this quarter” is the asset)

3) The size of a deal (e.g. “ECOLab will be greater than 250K”)

4) If the quarterly goal will be met (e.g. “Quarterly target of 2M will be met”)

In this forecasting market all the participants can constantly buy and sell shares of these assets amongst each other based on their local information. This would provide an incentive for the participants because they could make real money in the transactions. You could also set up the market to payout to the most accurate participant (e.g. the person holding the most shares of the right final answer would get a $1000 bonus).

What would be interesting about this forecasting market is that the current and historical share prices of each assets could be made available to anyone: CEO, VP sales, board members, engineers, etc.. As the quarter progressed you’d see these share prices shift as new information came into the business. One could imagine that you’d see a lot of volatility in the share prices when dramatic new information came in as well. Imagine what you’d see in the forecasting market after a disastrous customer demo? A plunge in share values for the “We will make the quarter” asset. Or a run up in share value when a blue bird deal came into the pipeline. It would also be interesting to note if trading volume was high on an asset which might imply an opinion disparity between buyers and sellers around a forecasted sales outcome. Another interesting scenario would be when there was high transaction volume on polarized assets classes in the market (a lot of people buying “The Quarterly Number Will be High” assets and a lot of people buying “The Quarterly Number Will be Low” assets).

Part of what would make this forecasting market work well would be the anonymity of the buyers and sellers (as it is, with some exceptions, in the general stock market). This anonymity would prevent peer influence (e.g. if you knew all the sales people were buying high revenue assets you might be inclined to buy some as well). This would also solve the practical human problem of sales people not wanting publicly to disagree with the VP of Sales whose inclination might be to manicure forecasting information to the CEO or the Board until better information became available. This market would inevitably drive transparency into the sales forecasting process which would also require all participants to curtail capricious actions (e.g. if a board member sees the high revenue assets in the market tank they need to not get on the phone and start rattling the CEO’s cage).

The goal here would be to transcend the emotional nature of forecasting volatility (the “art”) in favor of better information for all the participants (“the science”). If for example a specific transaction asset dips (e.g. “Company XYZ will close this quarter”) the VP of Sales would have a leading indicator to try and figure out what was going on – a leading indicator they may likely not get from talking to just the account manager for that prospect.

Putting the Force in Salesforce.com

The more I have thought about this the more I am a believer that this is a fundamentally better approach to sales forecasting. I’d love for someone to build a forecasting marketplace application for Salesforce.com. I’d love to see (as a starting point) all Salesforce user accounts for a company be able to buy and sell shares on the market anonymously. Perhaps in the beginning all the forecasting marketplace “assets” for the quarter are automatically created from SFdC data (you can see which opportunities in SFdC are scheduled to close this quarter and what the predicted aggregate quarterly number would be). Or perhaps the VP of Sales can set up the assets each quarter. Eventually you’d like to see anyone participating in the market be able to “float” an asset that they wanted a prediction on. To some extent this approaches the generality of some of the new prediction market players like Predictify. The key here is to structure it as an incented marketplace and not just an opinion poll (which much of Predictify is).

Either way this approach would drive a level of transparency into sales forecasting that is sorely lacking. It’s not that most startup VP of Sales are bad, it’s just that the expert model doesn’t work with such broad diversity of localized information. And hey, and if nothing else you can finally have a way to pacify anyone that  that complains that only sales people make money off of sales.

Editor 2.0 - The Emergence List

As i see (or people point me to) Editor 2.0 types of products, I will keep a running list of them. This announcement got me off my ass to start the list as its pretty much in the sweet spot of what Editor 2.0 predicts:

Aug 1, 2008: Google (YouTube) acquires Omisio

There will be more!

Ruby Developer Survey

To all Ruby developers:

I've been talking to the guys over at Devver recently. They are cooking up some pretty interesting testing tools for Ruby developers. If you code in Ruby they'd love your feedback about the tools you would find most interesting in the pre-check-in phase of your development. Take 10 seconds and fill out the survey if you can.

The survey is here.

Thanks. I'm getting back to blogging soon so don't count my commentary out yet :)

Editor 2.0

It’s not often that I have an epiphany in a book store. Today, however, was the exception. I was lazily wandering around Trident Bookstore (a sort of Boston version of the Tattered Cover but with more humus options on the menu), looking in vain for a present for my mother. Granted I was distracted by the idle bookish banter going back and forth between myself and the friend I was with, but I finally got tired of the hunt, said “forget it!” and walked back up to the front of the store. And there, neatly displayed in convenient impulse-buy format was the book shelf devoted to employees’ picks of the month. From a small set of choices I was quickly able to find something I thought she’d like (The Red Tent). And herein came my epiphany: what the world needs now is not more content but more editors. And to boot, it dawned on me that a lot of the interesting startup ideas I have seen recently are following this trend - reintroducing the editor back into our lives.

You Can’t Make Choices without Bias 

Let’s step back a second and understand what’s happened in the world of media consumption in the last 10 years. Pre-internet (which sort of sounds like “pre-modern-day” now) pretty much everything we consumed was controlled by a limited number of sources. Whether it was classic fourth estate newspapers, Sam Goody and the Wherehouse, or a small offering of cable channels, our access to content was limited. Of course not every newspaper or television channel that leveraged its distribution monopoly into our homes became successful. Those that rose to the top did it through excellent editorial oversight.

Editors essentially have two functions: pick the most important content from the various sources they have at their disposal and manage the brand or positioning that is projected through those choices. That said, the blessing, the curse, and the not so unobvious dirty little secret of said editorial function is that it introduces bias into the equation. In the end though, it’s this bias which leads to consistent editorial choices. 

Slouching Towards Authorgeddon

Take a newspaper editor’s choices of content for example. They have AP feeds, staff writers, freelance writers (stringers), op ed pieces, etc.. all at their disposal when putting together their dailies. On top of that, they have multiple choices about richer media with which to emotionally paint their pieces. Remember the bru-ha-ha [scroll down] over the differing OJ Simpson covers on Time and Newsweek? For every issue, an editor makes 1000s of choices to about what to print and what not to. 

A friend of mine who is the editor of the Atlantic Monthly recently told me he has over 12,000 non-fiction submissions (not including his own staff’s work) to choose from a year. And in the television world, through a friend of mine who has written for CSI, Vanished and Chuck, I have heard about the reams of spec pilot scripts that NBC, CBS, etc.. have to choose from each season. Any way you slice it, the editorial function has been busy at work for us over the last 10 years.

This all changed one day when someone inspirationally stuck the word “citizen” in front of “journalism” and  “media” in every 2002 VC presentation about the changing face of online content. And with that one small step, a giant leap for mediakind was taken. 

With the dawning of user generated content, we have cast aside the editorial function, and in the process migrated from a consumption generation to a participation generation. Blogs, YouTube, Wikipedia, Podcasting, Facebook, Flickr – the alphabet is littered with participation economy portmanteaus and poorly spelled company names that allowed anyone with an internet connection to become a content producer. The last time I checked, Technorati reported 57MM blogs in existence (its probably much more by now). As of 6:00pm EST, MySpace has had 75,195 videos uploaded TODAY. And last year’s 90,000 member strong National November Writing Month [www.nanowrimo] novel-in-a-month club generated 473,201,894 words all slouching further towards Authorgeddon in only 14 days!

None of this goes through an editor. It’s all headlines on the global front page that is the Internet. Unless I get to start using more than 10% of my brain, I think I’m going to have to pull a Gary Larson pretty soon: 

The Production Curve and Community Percolation

Okay, so after a few years generating all this stuff (insert appropriate George Carlin “stuff’ reference here) we clearly have the problem of too much stuff. How do people end up picking what to read and watch and what to ignore. Well, what we’ve seen so far is three emergent solutions to this problem. The first is what I would call the quality over quantity (QoQ) approach, the second I’m going to name “community percolation”, and the third is the authority approach. 

Quality over Quantity (QoQ)

QoQ occurs when content producers decide that the best way to attract consumers to their content is by adding a higher degree of production to it. By pushing up the production value, one can argue there is a direct correlation to moving from the right to the left in the classic long tail curve. The most obvious (and first) examples would be produced blogs. Sites like TMZ, Gawker, TechCrunch, Engadget, AskTheVC, etc.. are all what I would call lightly produced QoQ content. That means that someone (or more likely a team) with actual writing or journalistic skills puts some effort into silly little things like fact checking, correct punctuation and grammar (clearly I don’t strive to elevate myself to this level), editing, and generally well-written or thought out pieces. People go to these blogs because, on the whole, they are guaranteed a more valuable consumer experience (e.g. there are few posts about what movie Mike Arrington watched last night). But wait, you say, this is really just like a newspaper or magazine. Well, as I will explain in a bit, it’s not, because for the most part it’s all sourced by the staff bloggers themselves. Yes, there is an editorial choice about what to write, but the key is that it’s not culled from multiple sources. 

We see the same phenomenon on YouTube and MySpace with the emergence of higher quality content channels (MySpace’s Roommates excepted!) and also with a number of higher production quality independent pod/videocasts like the one from my good friend Dana Brunetti at TriggerStreet.com. In general “production” can mean higher core quality (e.g. using something more than a Motorola Razr to take video at a Tool concert) or thoughtful post-production editing like you see on TriggerStreet or another friend of mine’s Citizen Sports Network.

And to some extent the QoQ approach has worked. Feedburner quotes 768K (yes that’s 768,000) subscribers to TechCrunch. While that’s not quite the New Yorker’s 1M subscriber base, if you include what is probably a much higher secondary and tertiary readership with blogs (most people not using subscriber email or RSS feeds), we’re talking aggregate readership levels that the ABC would register firmly as a blip on their publication radar. 

Community Percolation

The second solution I mentioned is what I call community percolation. Community percolation can really be attributed in origin to Digg (and its multiple successors and automated derivatives like TechMeme and StumbleUpon). Many people call this community filtering but I think that moniker is a bit misleading. Filtering, to me, implies an intentional slicing of a data set using specific criteria. By definition, the masses have no common criteria other than  grunting “this blog good” while hitting a button. Semantics set aside, the idea here is that the wisdom of the masses shall help the cream of the blogosphere rise to the top. If you believe this will solve all of our problems let me remind you of two things: first, Jaron Lanier’s fantastic treatise on lowest common denominators, Digital Maoism and second, Bush’s second term (come on guys, that one was tee ball). While Digg does reduce the overall noise, it serves in reality as a band-pass filter not a high-pass filter (yes, I took one EE course at MIT – and all I got to show for it was one semi-intelligent sentence in a blarticle). 

Authority

The last solution is the authority approach. This has a few derivatives but generally follows the mantra “if X is an authority on Y and I am interested in Y and X recommends something about Y its worth my time to check it out”. This takes place in the form of recommendation engines (Amazon and NetFlix), the Technorati Authority system, and my friend Matt Cutler’s random indie music recommendations (most recent would be Bassnectar and Phantom Planet). In general this can work in limited circumstances (picking a horror movie to add to your NetFlix queue), the problem is that it’s hard to find out who is an authority on anything very quickly on the net. Go ahead, try and find out who is a real authority on Bangladesh in less than 2 minutes. I just tried this and it’s not so easy! 

So after a few years of user generated content, what we’re seeing is that the three solutions I mentioned do help a bit but not in the way that any of us really want. The metric for “really want” is simple – it still frigging annoying to find content you like on the net. And the more people hit the publish button the worse its getting.

The Return of the Editor 

If you consider a spectrum with the New York Times on one side and your average blogger on the other, it’s interesting to consider what the two end points really represent. Some might see that spectrum as simply the continuum from professional journalism to citizen journalism. But I’m sure most people could point out a blogger that writes better than an in-house journalist or a journalist that is more insightful and free of voice than some bloggers. So I don’t think that’s the scale. What I think the scale is really about the editorial function.

On the left we have a tightly controlled editorial function performing both a filtration and assembly mechanism pulling from a number of different sources and also introducing bias to the benefit of the readers. On the right we have an individual voice, someone who by definition has a bias, but generates all content and thus provides no filtering or assembly other than what they choose to write about. So what’s half way between Bill Keller and me? This is what I call Editor 2.0. 

Editor 2.0

If we can have citizen journalist (bloggers), citizen media (YouTube and people on the street uploading live coverage on UStream.tv) we can definitely have citizen Editors. This, in my opinion, is the next step in the future of media. 

Consider for a minute some of the trends that are pointing in this direction. First, let’s consider the editor function of filtration and assembly. Last time I checked, pretty much every single thing on the web was getting delivered by a feed or was designed to share. Flickr, blog entries, YouTube videos, del.icio.us tags, etc.. these are all input sources akin to the Reuters and AP newswire feeds that most daily editors cull from. Second, while blog entries are already published when you find them, such fragmentation in the market means they are probably relatively unread and assembling them into an aggregate format and republishing them to a broader audience probably encroaches little on their readership loyalty by the fact that they might exist elsewhere as well.

Second, consider that many people already play the editor role, if but for themselves. iGoogle, myYahoo, NetVibes, the Daily Me, etc.. they are all really just systems set up to let people become their own editors using the familiar section-based format of a paper with access to all the feeds underlying those sections. If you took this model and you turned it outwards, not inwards, this would be citizen editing in its simplest and roughest form. 

Tabula Rasa

But let’s get a bit more specific about what the next generation of media is going to look like. Let’s take the easy one – blogs. A while ago a friend of mine, Eric Herman, and I came up with an idea we called Tabula Rasa (TR). TR was in essence my first exploration into citizen editing. I was frustrated with the insane growth in blogs and more particularly the breadth of topics covered in most blogs I read (e.g. a blog purportedly about PeopleSoft sometimes has random entries about a vacation spot the blogger went to). I was so overwhelmed culling through the sheer volume of blog entries in my blog reader every day that I simply gave up and took the lazy blogosphere approach (a derivative of the more general Lazy Web). The lazy blogosphere approach relies on me staying in sparse communication with the smarter bloggers and entrepreneurs I know and relying on the fact that if anything really important comes up, someone will send me a link to it. Frankly I think this works pretty well as I just got two links to the same NYT article related to something I am working on within 5 minutes this morning. This, as a side note, was a subconscious default to the authority solution I mentioned above. 

What we really need is a new platform designed for the editorial function. This is what Eric and I envisioned TR to be. A sort of console for taking incoming feeds, finding related media, and cobbling together a front page of the best of the best. If you wanted to add your own blog entries to it, then by all means. Imagine that rather than write a blog about Tennis I was more interested in pulling together the best writing about Tennis on a daily basis. The TR platform we designed would let you do this in 15 minutes.

One of the beauties of the TR concept is that it would publish in a newspaper front page format. What I mean by this is that it disrupts the time based entry format of blogs. This is one of the biggest problems with blogs in my opinion. The best content is seldom at the top of the blog. When I scan my blarticle readership, seldom is the top entry the most read (due to inbound links from Google and emailed links to outside readers). With the front page format, citizen editors could choose to keep important articles and stories on the front page for longer, swapping out side stories that might be more current. In my own blog I’d love to keep some of my better and more timeless pieces prevalent on the blog but I am relegated to making a “best of the best” sidebar set of links. 

The other beauty of the TR concept is that different media types about a subject could be sourced from different places. Maybe there is a fantastic blog entry on Wimbledon but the best picture is from Flickr and a great video of the final match is on YouTube. TR would allow me to assemble these quickly into one seamlessly produced front page.

I know, I know, everyone will say “what about licensing issues”. Well there are a few simple half-solutions (like creative commons licensing) and blurbing (ala Google News) and such, but in general I think the benefits would outweigh the means and you could always include attribution and eventually advertising revenue share to all the content contributors. 

And the same thing is emerging in video. With live video platforms like Ustream.tv and Justin.tv becoming more popular, it won’t be far off when we can skip between multiple live video streams of events. An editor can easily insert themselves into this process and hand direct the best “camera” angles in real time just like TV stations do for sporting events and live TV programming.

Sooner or later someone will construct something like this and the citizen editor will emerge. And once the citizen editor has emerged, community editing will emerge as well. A few like minds can pretty easily keep on top of a topic or two on the net and edit it into one useful front page. 

Swinging the Pendulum

I am starting to think there is a palpable gestalt emerging around this right now. You can even see it in traditional media. Walk into any bookstore and you will note that some of the most popular titles on shelves are edited compendiums. A good example would be the “Best American Writing” series. These things are everywhere and they espouse the value of an editor. If you hunted around you could find all the articles yourself but I can tell you from personal experience it’s worth just buying the book. It’s scary to identify a media trend by pointing at something from the printed media world, but let’s not get too full of our internet selves here. 

Whether traditional media goes first or entrepreneurism comes to the rescue, it’s definitely time the pendulum starts swinging back towards the middle.

Why Enterprise Software is Not Dead

This is part 1 of a multi-part series on the virtualization revolutions no one is really talking about. If you enjoy my longer blarticles, make a cup of coffee; you won't be disappointed by this one.

And They Wonder Why They Go Bankrupt

Every once and a while you get a pretty good glimpse why certain airlines seem like they file Chapter 11 more than they file 10-K's. Here's a great one from today. I was trying to book a one way from from SFO to BOS on January 11th. I accidentally asked for a round trip on Expedia the first time I did my search. Realizing my mistake I went back and did the exact same route (SFO to BOS), conditions (non-stop) and date (Jan 11th, 2008 outbound) but only one way. Guess what, the plane ticket was substantively MORE for a one way. I can understand it being substantively more than 1/2 a complete roundtrip but more than the total roundtrip? I seriously wish Expedia would let me book a double or triple round trip fare. I am pretty sure I could get this puppy down to a buck or two using that pricing logic. I even took screen shots lest you skeptics not believe me. Screen shots taken within 10 seconds of each other.

One way (above) - $792.00.

Roundtrip - $628.00 (above).

If anyone actually understands this pricing logic from an airline perspective I would absolutely like some education on it (leave a comment please).

2008 (well at least 364 days of it) Predictions

I decided to gain a slight statistical advantage over my other Enterprise Irregular bloggers by hedging a few days into the New Year to publish my 2008 predictions. Fortunately (and to many chagrins, I am sure), waiting a mere two days has revalidated the premise of one of my predictions: 100 dollars a barrel anyone? Its gonna be a year to remember.

As a side note, and somewhat true to form, I have a number of posts related to these topics in the workshop, sitting on the easel, one coat of paint away from publishing. I would promise to try and get some regular pace to this mess in 2008 but why make a resolution you’re bound to break (I have not gone to gym yet either folks).

Lastly – I apologize that so many of these things are interconnected – I can’t help it as we’re in a major phase shift right now – a sort of tech will follow the dollars will follow the tech kind of Mobius strip.

Newmerix Crosses the Chasm

When I first started Newmerix in 2002, I had a few initial premises for why the business could be very large and a would end up being a "must have" and not just a "nice to have":

• #1 - Packaged applications were not going away anytime soon and, in fact, would become the central
  platform for IT because of their back office nature (as opposed to web sites or Exchange infrastructure).
• #2 - While customers who bought PeopleSoft, Oracle EBS, SAP, etc.. thought they were moving away from custom development, they were in fact just exchanging traditional custom development for a new kind of customization development on those platforms.
• #3 - Packaged applications, while different, had all evolved to have the same basic architecture; one based on metadata and the classic MVC design model. As a class they were more similar than they were different.
• #4 - Managing a packaged application was a change-centric activity while traditional development was a build centric activity.

There are clear examples of these items coming true (#1 - NetWeaver/Fusion/Dynamics, #2 - the patch nightmare of owning a packaged application, #3 - PeopleTools/ABAP Workbench/Oracle Forms and OAF, #4 - SOX compliance , patch and customization management, and ITIL). That said, the subtlety (especially built into #2 above) was that Newmerix could build a business (and product) in stages, where each successive expansion became cheaper and easier to build than the prior stage, while the overall value of what we were building would grow non-linearly to the customer.

Today we announced our complete suite of products for the SAP and Oracle EBS platforms. Not only am I proud of the amazing engineering, sales and field work that has gone into getting us here, when I look at the milestone timeline of the business, it confirms much of this subtle original premise. To business school aficionados this will strike you as a textbook case study in “Crossing the Chasm”. Here's the quick timeline to illustrate our growing expansion momentum:

2002

• Business started

2003

• PeopleSoft testing product (Automate!Test for PeopleSoft) developed and first version GA

2004

• Centralized change request management product (Automate!Control) developed and GA

2005

• PeopleSoft change management product (Automate!Change for PeopleSoft) developed and first version GA

2006

• Acquired Object Manager which became Automate!Change for SAP
• Extended Automate!Control to integrate with Automate!Change for SAP
• Extended Automate!Control to integrate with Automate!Test for PeopleSoft
• Added Business Process Documentation to Automate!Test for PeopleSoft

2007

• Extended Automate!Control to integrate with both Automate!Change for SAP and      Automate!Change for PeopleSoft for customers who owned both and wanted to centrally manage them
• Launched Automate!Test for SAP including Business Process Documentation
• Developed and launched Automate!Change for Oracle
• Developed and launched Automate!Test for Oracle including Business Process Documentation

2008

• (coming soon) Extending Automate!Control to integrate with SAP, PeopleSoft and EBS in one central location for cross-ISV change and lifecycle management
• (coming soon) Extending Automate!Control to integrate with all the Automate!Test products for one central location to manage all business process testing
• (and much more…)

Note that while the size of our engineer team has only grown a little bit over this time, we’ve been able to launch complete new products and ISVs built on the foundation of our other products in a fraction of the time our first versions took to build. The side effect of this (taking Automate!Change for EBS as the example) is that the first versions of our new products are really the third generation of our existing products and thus are extremely feature rich.

From the customers perspective, the more we deliver in our roadmap, the more they have been able to integrate all activities across the packaged application change lifecycle into one location (Automate!Control). In addition, many (I’d say 50%) of the customers we serve own more than one packaged application due to history, acquisitions, etc.. Being able to come to one vendor and buy the same product set for all your ISV management needs is incredibly valuable. We’ve already closed a number of multi-ISV customers and we’re seeing our pipeline fill up with people who have SAP and Oracle, PeopleSoft and EBS, and even all three.

We are now the only company in the world to support the complete lifecycle of change in PeopleSoft, Oracle EBS, and SAP, let alone integrate them all together in case you have more than one. That's a pretty amazing statement given this was nothing but a PowerPoint less than 5 years ago.

It’s exciting times around the office. Congratulations to the whole team at Newmerix who have worked very hard for a long time to get us here!

Click here for the complete Newmerix Automate!SAP and Automate!Oracle press release.

Do Programming Languages Matter Anymore

Not to long ago, SAP did something they had omitted to do for the last, say, 30 years or so. They appointed a CTO (Chief Technology Officer). Having been a CTO most of my professional life, I was pretty excited by this prospect and looked forward to eventually meeting the person who could break the 30 years of non-CTO-having SAP tradition. A few weeks ago at SAP TechEd, I had the privilege of finally talking with SAP’s new CTO, Vishal Sikka.

I was very intrigued to meet Vishal because when people talk about him his name is usually clustered up in the sentence pretty close to all those adjectives that most people would call compliments. Also, Vishal is not only the first CTO of SAP, but the most visible promotion of a technical figure after the departure of Shai Agassi (who I have written about numerous times). Like it or not, Vishal can not escape comparison with his closest predecessor and this leaves him some pretty big shoes to fill and a long way to run in them.

As a side note, over the last 10 years I have taken pride in crafting the role of the business technologist or as I sometimes refer to it, “external” CTO. This role is really a fusion of marketing and product management with a strong (some would say dangerous) amount of participation in engineering. And when Vishal starting seamlessly talking up the stack to business and down the stack to technology, I knew he was cut from this cloth too. 

What makes Vishal a very capable successor to Shai it that his logic is almost always coherent as he moves conversationally between the layers of the stack. He clearly spends a lot of time talking to the market, customers, and his internal engineering teams to craft the right story continuum. He is also actually able (or willing) to go the opposite direction in the stack too (in a way that Shai was not or did not want to) and actually dig deep into the technology with you. As the true litmus test, James Governor, my favorite new Enterprise Irregular press room curmudgeon, tried for 45 minutes to stump him with a notpron of technical TLAs, and only once had to step in and offer him a clue. The most fun and most challenging thing about talking to someone like this is that if you’re not listening closely, you can miss tons of tiny nuggets of wisdom or even full fledge projections into the future of SAP’s architecture (or architectures and now they have about 4 different platforms).

Okay, all CTO to CTO gushing aside, I was actually listening very carefully to what Vishal was saying in both the SAP TechEd executive press conference as well as our dedicated Enterprise Irregulars’ session. I have learned over time as a conference blogger that press rooms are not the place to spark debate or incite ideological riot. While it’s tempting to have the whole executive team on stage in front of the press, analysts, God (or your equivalent), and a few straight-to-YouTube cameras, you basically have to restrain yourself to opening salvos for the conversational battles you’d like to fight later. Sometimes you have time to pick up the topic and sometimes you move on. With this dynamic, there was one question that Vishal answered that I still am not sure I agree with and never got a chance to follow up on in depth. I especially listened to this specific answer because it was in response to a question from me. Here was my question: 

“What is the adoption curve of Java developers on the NetWeaver platform? How do you measure it and do you have any internal goals like 50% of new code development by 2010 will be in Java”.

My question stems from the fact that SAP’s new NetWeaver platform is a relatively new hybrid between Java and ABAP, SAP’s old proprietary language that has approximately 17 trillion lines of aggregated customer code out there in the field. While the use of Java is probably most prevalent today in SAP Portal and XI, SAP’s intergration framework, it is also slowly becoming a viable alternative for core SAP business logic customization. The opening salvo part of my question to Vishal was not really about the popularity of one language (ABAP) or another (Java), but about the real cost of retooling and retraining an SAP customer’s organization from an ABAP skill set to a Java skill set.

Just like mainframe COBOL programmers, ABAP programmers are still in business, hanging around making changes and modifications to SAP systems on a daily basis. Even though they are few and far between and quite expensive to hire, a SAP system simply can’t run without them. So if a SAP customer really wants to move towards the future of NetWeaver, not only do they have to adopt Java as a core development language along the way (and train everyone in the process), they need to consider the cost of upgrading (or abandoning) the existing legacy ABAP code base they have. This makes my analogy to COBOL even more ironic as many companies simply tossed out their COBOL code during the Y2K scare and moved over to SAP as the alternative instead. Now neither I nor SAP is recommending you dump your investment in ABAP – in fact their unique (as compared to Oracle’s Fusion) co-residency approach tells quite the opposite story. And here, with perhaps a better backdrop for you as to why I think my question is quite important, let’s pick back up on Vishal’s answer. 

Vishal basically said, “We don’t measure it [Java adoption] and we don’t care. We are not putting all of our eggs in the Java basket, that’s just what we’re offering for the time being.” And then the most interesting part, “Languages tend to have 10 year timelines - and then they are always outmoded by a new language. So in 10 years we won’t be talking about Java, we’ll be supporting something else.” If you step back and think about it, on the eve of asking a majority of the SAP customer base to embrace Java with open arms, saying in 10 years SAP won’t care (author’s choice of words) about Java anymore is a pretty interesting and ballsy statement. Again, these are paraphrases meant to capture the spirit of his answer (do not take them as verbatim quotes) - please keep reading for a bit more insight into what I think Vishal meant and why this is an important discussion.

The optimist in me hears Vishal’s answer and says, “Hurray!” Finally SAP is moving away from a proprietary-only framework (ABAP) towards a much more open language approach. Java is clearly the most open of the contemporary language choices and hopefully it’s just the first step in a longer string of language support. While this is good news, clearly it is going to take some time for SAP to fit another language comprehensively into their software stack.  

For example, SAP developers can only now, with CTS+ (which I think according to the new naming scheme is called NetWeaver 7.0 Stack 12), move both Java custom objects and ABAP objects at the same time as part of the a transport. With that said, many holes exist in SAP’s tools for the deployment and management of heterogeneous language environments. That’s why Newmerix is about to GA an extension to our very successful SAP change management suite to fill all of the existing holes that the ABAP/TMS/CTS tools and process have as well as all the new ones being introduced by moving Java around as well. To stay an optimist for a second here, while it may take a long time to do this, not locking yourself into another long term language choice is a strong decision.

That said, the pessimist in me says that I don’t believe Vishal’s implications for a second if you look at the realities of SAP’s history. Lets first verify that Vishal’s basic premise is correct, that languages, in general, emerge every decade or so. As a side note, I love Wikipedia. I just Googled “programming language timeline” and viola – everything I needed to know on a page called, surprise, “Timeline of Programming Languages”. I swear if I had to choose one, sometimes I’d take only Wikipedia over the whole internet!

Let’s start by looking at the emergence of some of the most notable and relevant business languages (e.g. code people build their own business applications with or code that is used to customize others’ business applications): 

• COBOL (1959)
• BASIC (1964)
• C (1972)
• SQL (1978)
• C++ (1983)
• ABAP (1983)
• Perl (1987)
• Visual Basic (1991)
• Java (1995)
• C# (2000)

While I am sure some will split hairs on minutia like the difference between when a language emerged and the notable year it attained mass adoption for use in enterprise software development, I think we’re actually looking at a shorter timeline than Vishal’s 10 year horizon. Across most data points it looks like 5-7 years is the average time between major advances in the field. This actually does not surprise me because, in general, IT cycles are almost always around 7 years long. What did surprise me was the consistency of this pace for almost 60 years now! And what is interesting is that the closest data point to Vishal’s estimate is the time between ABAP’s adoption and the start of Java. 

Whether it be 5, 7, or 10 years, here is my problem with Vishal’s logic and the implications he was making (that SAP doesn’t care about the adoption of one language or another as they all fade away). While programming languages come into fashion and then are inevitably replaced by newer and sexier ones, code bases are not subjected to the same popularity contest. In fact, it’s quite the contrary. The long tail of programming languages is actually not the classic Chris Anderson/Zipf power distribution graph. It’s actually quite the opposite with the total lines of code accumulating over time more quickly than they are being replaced. You can argue that new languages are adopted much faster and there are more programmers, but this, I would say simply increases the slope of the lead line and not the total accumulation.

Take for example various reports that triangulate that between 200 and 500 Billion (yes, that’s a “B”) lines of COBOL code are still running in production today [1, 2, 3]. As I mentioned above, the Y2K scare/phenomenon did a painful job of reminding us how true this was. Was the main surprise that 2 digits could bring down the computing world as we know it? No, it was how much frigging COBOL code was actually running the financial and transactive underbelly of the United States and the globe’s economy. And this long tail of computing is hard to fix because there is an inverse relationship between the number of knowledgeable programmers in a language and how many years ago it happened to emerge. In 1997, companies with massive mainframes were finding it pretty hard to find a COBOL programmer who, to Vishal’s point, right around 1972 had not picked up K&R and moved on to greener programming pastures.

Now let’s take the second and probably more poignant example of the long half life of a code base. According to our contributing friends at Wikipedia, ABAP was introduced in 1983. Counting on my fingers and toes (and yes, running out), its now about 24 years old. One more year and its going to be SAP and ABAPs silver anniversary. I for one will send them a lovely chaffing dish. But the real question is after so many fantastic years together, how many lines of ABAP code are running around in the customer base. Well, while I was being glib about 17 trillion lines before, let’s do a one of those McKinsey-interview-question quick calculations (like “How many haircuts are given each year in the US?”). If there are 35,000 SAP customers, and about 2.3 production instances per customer (I got this number a couple of years ago from SAP), that’s about 80,000 instances. Let’s ignore that each landscape has multiple systems and just focus on the production system. Let’s estimate that, on average, an SAP customer has owned SAP for 7 years (R/3 4.0 was release in 1998). Let’s also estimate that 100,000 net new lines of ABAP code are written per year per customer. So doing the math we get: 

80,000 x 7 x 100,000 = 56 Billion lines of SAP code

Okay – even if I am effectively off by an order of magnitude on net new code generated per year, that’s still 5.6 Billion lines of code in the customer base. Holy legacy code base, Batman! And that does not include the millions of lines of ABAP code that SAP has written, deploys, and still writes into their vanilla code base. This shipped code still gets poured over, integrated with (ABAP APIs) and modified by customers. My basic point here is that the 10 year cycle is not a flush and restart cycle as Vishal would sort of imply, it’s a compounding cycle that only gets more expensive to maintain over time. 

So, you might be sitting there asking yourself, “So what?” Well the problem is that you have a mismatch between the inherent desire of programmers to use what is new and the need to integrate the new in with the old. Over time this gap widens to a point where, as we saw in 1997, people literally make $100M reinvestments to start from scratch. How does SAP and their customers avoid this? Well, I’ll offer my readers the same idea that I offered Vishal. And lest anyone think I am picking on just SAP for this problem, our friends in Redwood Shores are facing the exact same challenge with Fusion. That said, my solution is one that either SAP or Oravcle could employ, but for various reasons you’ll see, SAP is much more likely to consider.

Let me first broaden the conversation slightly. I think there are realistically three pieces of the puzzle that are important when introducing a new language into an architecture like NetWeaver: tools, training and transports. The later (for those unfamiliar with SAPese) really means the process of deploying changes to the actual production infrastructure of the code base. In more general terms, how does your Java code get from development to production while taking a scenic drive through testingville?  

My proposal to Vishal actually covers all three of these with one fell swoop. It’s simple; integrate a CLR runtime engine native in the application server (or at least have a three legged stool of ABAP, Java, and CLR-based runtime). For those unfamiliar with the CLR (common language runtime), this is just Microsoft’s .NET equivalent of a Java virtual machine. When you program something in C# (or any other .NET language – more on this in a second), it gets compiled down into the rough equivalent of byte code (called IL – intermediate language in Microsoft terms). The IL is actually what the CLR then runs.

On another side note, I used to have this argument with Shai all the time. He’d say, “We do support .NET applications.” But what he really meant was you can run a parallel Microsoft application server and build services in .NET languages and connect them all with SAP through BPEL. While that’s architecturally a technical solution, the whole point here with NetWeaver is to drive all of your development (SAP and custom built apps) onto one platform over time. That’s the composite application strategy and stack that SAP and Oracle are currently battling for ownership of in the Global 5000 IT department. If you have any doubt, note Oracle’s current bid for BEA as indication that owning the application server is a key part of the strategy.

So why add yet another programming language execution engine to the stack? Well it has to do with the difference between Microsoft’s approach to .NET and Sun’s original approach to Java. Java came first, pioneering a renewed concept of the virtual machine and cross-platform independence. They built in network security, generic user interface harnesses across platforms, etc.. Part of this was visionary brilliance, part of it was because Sun original though Java would live primarily in the browser, and also because Sun was not dominant in owning the OS and thus the interface around it (e.g. Windows). When Microsoft’s showed up, they took all the best things from Java and added some game changing pieces of the puzzle. The main invention (in my humble opinion) with the CLR is not OS neutrality, its language neutrality. Any.NET programming language gets compiled down to the same IL and can be run by the same CLR. Rather than move neutrality to the end point (running the app) they moved it backwards to the starting point (writing the app). That means programmers using Visual Basic.NET and C# can co-mingle their code bases and use the right programming language for the right task.  

Let’s face it - part of why languages evolve over time is that the underlying tasks people are trying to accomplish change. Relational databases drove the need for SQL, the massive number of UNIX platforms in the 80s and 90s created the need for OS portability (which is why ANSI C emerged), code reuse brought C++ and modularity, and the list goes on. One language simply does not fit them all. More contemporarily, check out the emergence of Flash in the browser and now on the desktop with AIR (another post on this coming soon) showing the need for a language designed for lightweight mixed media presentation. Microsoft realized this fact when they were designing .NET I suspect because of their long relationship with developers.

Unlike Java, Microsoft has aggressively supported language designers in porting their design time language syntaxes to runtime CLR compatibility. Now there are over 40 languages supported in the .NET world. When a new language comes out, whatever that will be and for whatever reason, a .NET version of it could be made available and no infrastructure change would be needed to run it.

Let’s review our three legs of the language platform stool again. First is training. Well there are two sides to this coin. While I agree with Vishal that as time and necessity marches on, new languages will emerge, we cannot discount that people (developers and others) need to be trained. What’s beautiful about the language-neutral CLR approach is that it allows companies to harness all the existing resources in their organization that might know any .NET supported language to come over and work on the SAP system. If you have an army of Visual Basic form designers and you need some work done on the SAP portal, let them come over, use VisualBasic.NET (which is only a small learning curve from classic VB) and start developing interfaces and code for your SAP team. 

The second item is tools. While Microsoft has done wonders in supporting many languages in their Team Studio environment, the bigger problem is being solved by the open source community. Whether Borland likes it or not, unless you’re doing solely Microsoft development, everyone is s moving to Eclipse as the standard framework for IDE based code creation. While people tend to think of this as a Java programmers environment, it is quickly outstripping those perceptive chains and becoming a world class environment for developing in most languages. For example, C# plugins exists and it would not surprise me if an ABAP plugin was in the works somewhere.

The third item, transports, is simple. If you have the CLR deployed (and you keep it up to date with the right .NET framework upgrades), then you can run whatever language comes along. As long as the framework upgrades are backwards compatible (which they generally have been to date through three releases) - 5 year cycles, 10 year cycles, or whatever, it doesn’t matter at all. In this world, the older code bases which seem never to die do not become deprecated and the new code base can be supported on the same system. It’s a win-win all around. 

I think the lynch pin to the potential success of my proposal to Vishal is based on the fact that SAP seems to be getting very cozy with Microsoft right now. With their continued work on Duet and the new OBA integrations with SAP, this seems like the right time and the right partner to solve a very long term problem they have. Let’s face it, Microsoft needs this as much as SAP. They are slowly loosing their dominance in terms of enterprise software development. Less code means fewer developers, and as I have written about repeatedly, the long term battle always has been and always will be about owning the hearts and minds of developers. While I think Vishal usually gets it spot on, perhaps he will realize the blessing and the curse of his programming language timeline theory and put a framework, not a language, into the stack that will stand the test of time.