Nostradoofus

Market power (scale, network effects, or other) allows an operationally inferior firm to sustain a position. To see see how powerful this effect is, consider that people are actually still using Hotmail.

Since April, everything I send to Hotmail is classified as spam. Everything. Even mail to myself. Hotmail then deletes all spam after only 5 days.

So, when I email any client at Hotmail, it goes to spam, 100%, even if we’ve corresponded for months. If he doesn’t check within 5 days, Hotmail deletes, and I can never prove delivery. This renders Hotmail unusable for business.

This problem occurs only at Hotmail. My domains and IPs are not blacklisted. I have never sent bulk mail. I send fewer than 20 messages per month to Hotmail, all to clients who have paid me in advance to deliver to them.

Is this really a problem with Microsoft? Consider what happens when you try to fix the problem.

First, set up Sender Policy Framework. This is good practice — I should have done so long ago. However, at Microsoft, due to the principal-agent problem in the marketing department, SPF is called “Sender ID.” It has no discernible difference from SPF other than the name.

Once Sender ID is configured, postmaster.msn.com instructs you to notify senderid@microsoft.com. Oops:

senderid@microsoft.com:
Remote host said: 550 5.1.1 User unknown

So they’ve shut down the Sender ID cache update address, and failed to document it on their own instructions page. Ominous.

Undaunted, I found the next noodle in this strategic spaghetti: MSN Smart Network Data Services. To proceed requires signing up for Windows Live (formerly MSN), which yields this:

From the user’s point of view, it looks like Windows Live is offline in the middle of a weekday. Actually, it’s yet another case of silent failure on incompatible browsers, a widespread Microsoft problem mentioned in previous posts.

No problem. Changed browsers. Got a Windows Live account. Went back to Smart Network Data Services, and was greeted with this:

So Microsoft’s email security site has a bad SSL certificate. It’s a security site. With a bad security certificate. Ah, what they hey, I’ll risk it. Set up SNDS, and all was done.

After all that, everything sent to Hotmail is still classified as spam. No change.

Just one option left, according to the MSN postmaster: “Sender Score Certified,” formerly “Bonded Sender,” formerly “Return Path.” Under this scheme, you can rent the right to send anything to Hotmail, for a fee of US$1,400. To send 10 messages a month. Right.

Fortunately, there is a much cheaper and simpler solution, for some reason not mentioned at the Hotmail support site: I can refuse to support Hotmail, and persuade clients to use a different email provider. This has been effective.

The Orphans of the Sky scenario at Microsoft is not limited to the engineering department. With no overarching plan to guide them, various hives of Microsoft product marketers buzz with pointless activity, endlessly renaming myriad half-finished products and services.

Sample paragraph from a recent .NET propaganda piece:

“At MIX07 in Las Vegas, Microsoft detailed planned .NET programming support for Silverlight, formerly known as Windows Presentation Foundation/Everywhere … also disclosed was the Dynamic Languages Platform, itself based on the Dynamic Language Runtime.”

In 2 sentences, two “initiatives” are renamed for no discernible reason.

Another good example is “Sender Score Certified,” the new name for “Bonded Sender,” which was itself the new name for “Return Path.” (This service lets spammers tunnel into Hotmail, with MSFT approval, for a fee of $1400).

What does this marketing sound and fury imply? High turnover. How’s that? Well, it works like this.

You’ve just graduated from a major business school. You want to get into tech, but have no experience. No problem: Microsoft will take you, based on your pedigree. But the only position open to you is entry-level product marketing.

You feel this job is beneath you. You are an ambitious Harvard MBA. You want out, as fast as possible. But how best to do it? Answer: put easy-to-identify accomplishments on your resume, purpose-designed talking points for a job interview.

Just one problem: you inherited “Windows Presentation Foundation/Anywhere,” a 3-year-old project, as your first gig. How can you turn this into a bullet point? Simply rename it. Suddenly your resume says “Founding project manager, Silverlight.” Fantastic! Call the recruiters! You land a COO position at a VC-backed startup, and you’re off to the races.

In business school, we call this the principal-agent problem.

Why is Steve Jobs so good at making products people want? An Apple engineer I know had this to say on September 25, 2005:

“Say what you will about SJ, but he (and/or the people he surrounds himself with) are genius. I feel part of the secret is simply having built a culture where everyone cares about things down to the smallest detail — we sure do that with some of the stuff I’m
working on now — but there’s also the quality of what I’ve always thought of as ‘yummy.’ You make a yummy little morsel of a couple of different kinds of already-existing technology and mash them together, make sure it is (deceptively) simple, and make sure the fit+finish is perfect, and use ultra-clean modern design. Still, it’s that ‘yummy’ quality that’s paramount.”

Can this be copied? Possibly not. Why?

Aesthetics and integration are organizationally thankless. They require heavy up-front investment in things that are not identifiable product features, and thus carry benefits that are hard to communicate. Integration is particularly thankless, because it requires iterative design, causing unpredictable delays and blown budgets, again in return for no specific feature. Middle managers are typically punished for such things, and learn to avoid them.

Most companies are unable to measure the benefit of aesthetics and integration. So even when these ideas can be communicated, their benefit (price premium, increased sales, increased customer satisfaction) cannot be assessed internally before release, and so there is no way to decide how much to invest. So no one else does it.

Simplicity is worth more, but appears less.Ease of use often takes the form of superficial simplicity, creating the appearance of fewer features, and thus is potentially risky to an organization unable to assess its true market impact.

Apple has invested decades in a corporate culture that values aesthetics.This is singular. Apple has been indoctrinating new hires with its aesthetic “rules” for at least twenty years, giving them a big advantage at producing products where such things are valued. Since everyone in a software company imagines himself a creative genius, it’s a huge advantage for Apple to get everyone pointed in the same direction from the very beginning. That said, it’s probably safe to say Apple’s vision is all Steve, and wouldn’t survive his departure. Similar to Akio Morita at Sony: when he passed away in 1999, the company ran down like an old battery.

To the extent Jobs’ Apple enforces a unified, simple, aesthetic design sense, it’s unreasonable to expect any other company to copy. Thus it constitutes a sustainable competitive advantage.

—

This was originally an email to Doug. He liked it, so I posted.

Woot is designed around simplifying customer service, which is an underappreciated tactic.

To simplify doesn’t mean to skimp, but instead to do away with service entirely by redesigning what they sell and how they sell it. Rather than use dorky buzzwords like “reengineering,” I’ll illustrate by example.

I loved a take-out lunch place that used to operate on the Stanford campus. The grumpy Thai lady who ran it had a key insight: they do their entire business from noon to 1, so service speed is the scarce resource. She responded by rounding off all prices to whole numbers of dollars. By not handling coins, they double their throughput, I estimate. They sling chow and pocket greenbacks at an unbelievable pace, serving hundreds from a single window in only an hour. Wonderful.

I was so impressed by that restaurant that I adapted the idea years later to Firepad. From 2003 to 2006, the entire online software industry was giving away free samples, except me. I found that by offering no free trial, but an unlimited 30-day money back guarantee, I eliminated 90% of my support load, with almost no effect on sales. The free-trial guys rarely buy, it turns out. They just consume resources. My Firepad income per work hour (the key measure) went up by an order of magnitude.

Like the grumpy Thai lady, Woot seems to do nothing but sling chow and pocket greenbacks.

—

MSN Search (or Live Search, or whatever they call it this month) now accounts for under 6% of my search engine traffic. In fact, even this is overstated, because it includes all links from “unknown search engines.

Google accounts for 68% to 91% of my search engine traffic, depending upon the site.

My sites are all #1-ranked for their primary search phrases. My site content is highly diversified (video games, Palm image software, and investment newsletters). The implication is that my results are representative of search traffic generally. It implies strongly that Microsoft’s share of search traffic has fallen dramatically (more than half) in the past year.

This may explain their desperation to attract advertisers.

—

A popular theme of science fiction writers is a sort of high-tech cargo cult, in which ignorant descendants of an advanced culture live, dependent and uncomprehending, on the towering technical achievements of their forefathers. This idea turns up in everything from Heinlein (”Orphans of the Sky”) to Star Trek (#38, “The Apple” #63, “For the World Is Hollow and I Have Touched the Sky” — thanks to Eric for the correction).

There is historical precedent. Radiocarbon dating of the Egyptian pyramids yielded an archaeological surprise: the biggest and most advanced ones came first. Thereafter, they became progressively more primitive, as successive generations essentially made imperfect, poorly understood copies of the original.

One might argue the Enlightenment was similar: a new reasoning methodology fomented by Descartes, Galileo and Bacon triggered a blinding flash of collective insight, fading ever since, even as we discover ever more.

A more immediate example appears to be Windows Vista and, more generally, Microsoft product development. Here are some forgotten technologies of the Forefathers of Microsoft

Compatiblity engineering. This is the primary complaint about both Vista and AdCenter. These two products are arguably the top two strategic priorities for the company, so if both suffer the same failing, the problem is systemic to the company.

Sustained initiatives. Traditionally, companies feared Microsoft in part because it could sustain a strategic focus for years at a time, chipping away at a market until it took over. This characterized Excel in the 80s, Windows in the 90s, and XBox in the 00s. This seems less true today. What looks like sustained focus today is merely a group of insular departments just doing what they did last year, over and over again.

Speed. Say what you want about MSFT, but in the 80s and 90s, by and large, they wrote fast code. They were early commercial adopters of C/C++, they knew downcoding, they knew about designing data structures for speed, etc. It’s hard to see those strengths in the newer products. Adcenter is slow compared to lightning-quick alternatives (Google). Vista is slow compared to alternatives (Intel Macs). In both cases, no advantage is offered in return for this slowness.

The overall impression is that Microsoft is now a pilotless, decaying spaceship, carrying the childlike descendants of its brilliant forebears to a distant, forgotten destination.

—

Free advertising offers I’ve received from AdCenter:

July 2006: $30
Feb 2007: $100
Mar 2007: $200

Tables don’t communicate the drama as well as a graph:

On current trends, within a few months, it will be cheaper for MSFT to just buy out my business than to convince me to try their ad system.

At any rate, $200 is a lot of money, so I visited AdCenter and found the front end has been completely overhauled, but contains the same amateur-hour website errors.

1. Incompatible with Safari browser. Safari is inconsequential for the general population (probably 2%), but widely used by web designers (probably 30%) — AdCenter’s primary target audience. But OK, I can use Firefox.
2. Fails silently on incompatible browsers. AdCenter doesn’t bother to detect and warn of incompatible browsers — it simply fails in various comical/nonsensical ways, halfway through whatever task you were attempting. Browser detection (”sorry, this site is incompatible with Safari”) is vanishingly easy to implement, so it’s utterly amateur for an organization MSFT’s size to ignore it.
3. Credit card entry page is 1200 pixels wide. Good thing I upgraded my monitor last month. Clearly no one outside the ivory tower tested this app, or they would have noticed most people don’t have a monitor this big.
4. On Firefox, the voluminous “Terms and Conditions” legalese appears in a tiny text box only 2 lines high by 20 characters wide. It’s like reading Tolstoy through a keyhole.
5. On credit card validation, received mysterious “Alert: www.contentmedianetwork.com has sent an incorrect or unexpected message. Error Code: -12263.” I click “OK,” and am greeted with “Thanks for signing up!” So everything is OK now? Who was contentmedianetwork.com? What was error -12263? The mysterious unexplained errors are nothing like a normal Web experience. They remind me more of… Windows.
6. Inside the app’s campaign center, web pages load forever, evidently to keep a live connection with the server. How can that scale? They keep a connection open for every user simultaneously?

It is absolutely shocking how far behind Microsoft is in this arena.

—

Microsoft says its biggest strategic threat is Google. If true, then their most critical tactical initiative is MSN AdCenter, which competes directly for Google’s revenue. Yet my experience with AdCenter was utterly amateur-hour, unbecoming a Fortune 500 firm in any industry. The disconnect between top-down strategic initiatives and bottom-up execution suggest basic operational problems at the firm. Some highlights…

—

October 2005 through April 2006

• Frequent 500 errors, forced logouts, all progress lost.
• All browsers except IE6 failed silently, no error message. IE 7 crashed.
• Support (800-852-3568) connects to some poor sap’s personal cell. No calls returned.
• Ads only run in Singapore. Can’t they scale up to run in a major market? Maybe the poor guy with the cellphone will call back and explain.

—

July 5, 2006:

MSN today sent me two $30 gift certificates to try AdCenter again. There followed a comedy of errors:

• Account funded, ads approved, but no ads appear. This has been true for months. “Support” does not respond.
• Account thinks I’m in Singapore, and doesn’t let me change the setting.
• Started over today with a new account. Explicitly entered USA, yet my location is again recorded as “en-sg” (Singapore).
• New account system forwards me to a blank page at MSN. No buttons, text, instructions — nothing. Had to browse away, return, log out, log back in.
• Waited 20 minutes for a “credit check” on my MasterCard. Huh? Any one-man shop can instantly determine validity of U.S. credit cards.
• Unable to save new ads: browser shows “missing object reference” error. Googled an ASPX programming forum and was able to guess the workaround: log out and back in. Nontechnical users would stall here.
• Long delays on each page load. While waiting, I was able to log into Google Adwords, check status, and log out — all before the MSN AdCenter page had finished loading.
• Entered my ad text. On hold again while the ad is approved. Stay tuned…

My overall sense is that this system was not designed to handle the pace of the modern Web. It feels like something from five years ago. Today, customers expect credit cards to be approved instantly, as with Adwords. They expect advertisements to go live instantly, as with Adwords.

And above all, there should be no “A” bugs in the signup and login paths. For a startup, this would be egregious. For a market leader, it’s unconscionable.

—

September 12, 2006

Microsoft emails me a product launch advertisement for MSN Live Search. Email was sent with the wrong mime type, arriving as an unreadable pile of naked HTML code.

Astonishing. Fly-by-night solo spammers in emerging countries send me hundreds of emails per day, and none have formatting problems. Microsoft is a huge multinational with a reputation for quality execution at all levels. How can their bulk mail performance be two standard deviations worse than the average Russian mobster’s?

I’d ignore it as a onetime screwup, were it not for all the rookie moves in Adcenter — errors routinely avoided by
twentysomething web service development teams of just a few people, on shoestring budgets and GPL platforms.

The only way adCenter could make so many mistakes is if someone in the management chain just doesn’t care. Is the team leader a wealthy, bored MSFT lifer?

—

October 12, 2006

My credit card expired, visited AdCenter to update it, and the dance began again…

• AdCenter’s entirely static home page takes 6 seconds to load over my dedicated T1.
• AdCenter uses a login name distinct from email address (a previous-century design rule).
• Requested my login name by email. Response didn’t arrive from MSFT for two hours.
• Now I have my login name, but need my password. Another two hour wait.
• This 4-hour retrieval process takes 5 minutes at Adwords. No hyperbole.

—

October 20, 2006

Finally received adCenter password reset info. It requires copying a 512-bit validation code (five hundred and twelve bits, no kidding) back to the browser, along with the new password.

The password reset function in AdCenter failed silently, no error message. Second try, it worked. Went to log in with the new password. Again failed silently on the first try, worked on the second. Tried this on my other AdCenter account, and the same thing happened: fails once, works the second time. I.e., this is a 100% bug in the login path of a shipping product. This could ONLY be shipped by a project manager that does not care about quality.

Finally, I’m all logged in and ready to go and… nothing. Contrary to PR announcements, adCenter still doesn’t support my Mozilla-based browser, and still fails silently: no error messages and half-working pages.

Forget this. Every other piece of my workflow works fine. I’m not firing up IE just to get an extra 3 clicks a month from this also-ran. Sheesh.

—

The Zune is Microsoft’s recent entry into the handheld music player business. I explain this for posterity, as the product will likely be little remembered 5 years from now.

Currently, the Zune PR whisper machine likes to say the hardware was done by the xBox designers, somehow implying that’s an advantage. That’s mistaken. The xBox is a strategic and software success, but absolutely not a great hardware design.

xBox beat PlayStation because Sony failed to understand what everyone else in IT has understood for 25 years: it’s extremely valuable to create a stable software layer to abstract programming from hardware. This “abstraction layer” makes it inexpensive for both hardware maker and software maker to preserve backward compatibility across major hardware revisions. It also allows programmers to re-use code and knowledge for a longer period of time, which reduces software investment cost. Since game software is a high fixed cost, low variable cost business, a relatively small reduction in development cost can turn a big loss maker into a big profit maker. Multiply this across the hundreds of software titles available for a game platform, and you start to see the value of the abstraction layer.

Microsoft’s core business for over 20 years has been to build that sort of abstraction layer for PCs. For xBox, they simply built a similar layer atop a competently executed video game machine, and the battle was over. Microsoft didn’t win, so much as Sony fumbled. A competent software roadmap for Playstation 1/2/3 would have made it impossible for Microsoft to get a foot in the door.

Now, back to the ill-fated Zune. In music players, architectural control does not depend upon a client-side abstraction layer, because the media is not software, but simply music data. There’s no advantage for Microsoft in being expert at operating system development.

Instead, the key feature in music players is ease of use, which means software design and seamless integration. Say what you want about the roller coaster that is Apple Computer, but if there is one thing they have consistently done for over 20 years, it’s simple, elegant software design and seamless integration. It’s their hallmark.

Unlike Sony, Apple does not lack insight or execution. Their reputation is profoundly the opposite. Since 2001, they have become a rare combination: a design-driven firm that appreciates the value of architectural control. They have pursued that control relentlessly with iTunes, using superior integration and ease of use to establish their position, and then network effects to remain there.

Apple still lacks scale. If they’re beaten in the music business, that will have been the reason. It will have nothing to do with product quality, because there is no meaningful way to outperform what Apple already offers. It’s good enough. Their 70% share is proof of this.

Built into that compliment, however, is a bigger risk for Apple: the iPod is maturing rapidly. It may turn out to be a onetime sale with no followup, rather like the PalmPilot. 99% of the music buying public already can fit every song they own on a $300 iPod. Why ever buy another? To continue to grow, or even to sustain sales, Apple is almost forced to move up to cellular phones. That is a very big, very risky step.

The safest way to tackle this might be to turn the iPod into a WiFi phone. There is no organized competition, the system can plug directly into Apple’s existing iChat system, and it then becomes a network effects business, in which 30 million iPod owners can all phone each other. In certain settings, e.g. high schools and colleges, practically everyone has an iPod, so there is critical mass. Also, students are price sensitive to cellular airtime, while a WiFi phone would be free.

Still risky, but perhaps a logical next step.

—

A big advantage of open source software is to align the interests of developer and end user.

To see why that’s true, notice how it is not true in commercial software. The competitive dynamics of commercial software push developers to make decisions that are “piecewise competitive” with rivals, even if such decisions create inflexibility, incompatibility or increased maintenance costs in the long run.

For example, and most commonly, a commercial developer may optimize for performance by writing code in ways (e.g. choice of language or data type) that cause long-term inflexibility with respect to future hardware platforms or customer needs. This is in the developer’s best interest, because it’s the only way his firm can survive short-term competitive pressures. However, it is against the user’s interest, because it forces upgrades or incompatibility in the future.

Another example is that a developer may add features that are useful to only a small fraction of customers; this creates a temporary competitive advantage for the developer, because it adds a bullet point to the box in the store, influencing a customer’s purchase. It also raises the barriers to entry in that software segment, which benefits all leading developers. However, this decision often works against the average user’s best long-term interest, because it makes software unnecessarily complex for the vast majority of users, and raises the long-term cost of the software, as barriers to entry rise and competitive pressures fall in the segment.

By contrast, many GPL projects make design and implementation decisions that seek to maximize coder productivity and device independence, at the expense of peformance and features. This is in the project developer’s interest, because it is the only way such a project can exist: without high productivity, maintainability and succinctness, even a successful project would eventually drown in its own bug reports. The coding team is too small to support such problems, and generally cannot expand over time.

This is generally in the end user’s best interest, because it allows simpler, more invisible upgrades to future hardware platforms; reduces confusion over features; increases modularity and stability of behavior over time.

You can extend the argument to cover other noncommercial projects like the original Unix, i.e. it survived into modern times because of some developer-centric design choices (performance-inefficient in the short run but maintainable in the long run) that happened to be compatible with the end-user’s best interest.

Thanks to Jeff Cohen of Genezzo for sparking this thought with the following email:

—

From: Jeff Cohen
Date: Nov 29, 2005 10:47 PM
Subject: Re: Genezzo version 0.53 is now available

If you asked them, they’d probably say it’s impractical, and nobody could design such a schema. Of course, given the precedent of computer-generated queries of insane complexity past human comprehension, I can easily envision computer-generated schemas. Generally, the issue is that languages that C/C++ tend to encourage certain types of design rigidity and premature optimization. For example, to encode the number of tables in a join, they could use an int, or maybe an unsigned int, or even a byte if they want to shave some space. And they might pre-allocate some fixed data structures inside the join row source for efficiency. Oracle was literally riddled with these little”optimizations”, and it caused all kinds of headaches when we needed to expand an existing data structure. I believe I told you about the COUNT(*) overflow at WalMart, where ancient coders couldn’t conceive of a table with over 2^32 rows. With languages like Perl, Ruby, or Python, the native data structures are dynamic, flexible and extensible, so coders are less inclined to construct fixed-size structures for practical and philosophical reasons. And the native numeric type is a generic number –the languages automatically switch representations internally for proper efficiency and precision. For example, Python might start using an int internally, then switch to float, and finally to bignum for arbitrary precision. And finally, since I’ve been bitten by these issues so many times, I am very vigilant in order to avoid getting trapped. For example, even the basic block encoding on disk uses variable-length BER encoding for lengths, so I can use a single-byte length for strings under 128 bytes, and up to 128 bytes of length forBLOBs over 2^1022 bytes. BER is more expensive than a fixed-length encoding, but in the future, if someone wants to shove a 3-D, holographic movie into a column inGenezzo, we should be able to handle it.

Jeff

—