SUNDAY, NOVEMBER 27, 2005

Enough about me...

POSTED BY DOUG AT 7:19 AM

 

In my first post here, I invited other Xooglers to join me in reminiscing about life back in the day. Ron Garret has accepted that invitation. Ron was the lead engineer on the first release of AdWords, and the experience affected him so fundamentally that even his name changed.

 

It was great having Ron on the staff for many reasons, not the least of which was that he nicely filled out our hyperbole portfolio. We'd been talking about how smart our technology was and after Ron joined us from the Jet Propulsion Lab, we could truthfully say that, "Yes, Google is so complex that we have both a brain surgeon and a rocket scientist working on it."

 

Welcome Ron...

 

 

1 COMMENTS:

 Ron said...

Thanks Doug! I'll try to get a first post up later today.

10:22 AM

 

SUNDAY, NOVEMBER 27, 2005

Hello, world

POSTED BY RON AT 1:13 PM

 

The first post is always the hardest.

 

I've been debating with myself whether or not to write about my Google experience at all. It seems like such a self-indulgent thing to do, and, while in retrospect it has certainly turned out quite well, there were some bumps along the way (to put it mildly) and I did some things that I'm not altogether proud of.

 

On the other hand, I've always enjoyed being on the receiving end of a good inside scoop, so now that I'm in a position to share an experience that others seem to care about I feel like I ought to do so. Besides, they say confession is good for the soul.

 

A very brief summary to set the stage: I worked as a robotics and artificial intelligence researcher at the Jet Propulsion Lab from 1988 until 2000 when I went to work for Google as a software engineer. I was there for just over a year. I worked on two main projects, the first release of AdWords, and a little known widget called the Translation Console. If you go to Google's preferences page you will see that Google is available in well over 100 languages, including Klingon and Elmer Fudd. (There used to be a Swedish Chef option, but that seems to have succumbed to political correctness.) All those translations are provided by volunteers around the world. The interface they use to do the translating is the translation console (officially called the Google In Your Language Program). As far as I can tell it hasn't changed much since I wrote it. I'm pretty proud of that.

 

I guess the #1 FAQ for people who have left Google is why did you leave. My main reason for leaving was that I was commuting from Los Angeles. I'd fly up on Southwest early Monday morning, fly back on Thursday evening, and telecommute on Fridays and weekends. That regimen was pretty stressful even under the best of circumstances, but when 9/11 happened it became completely untenable. I had already given my notice before 9/11, but I don't think I could have stayed on after that even if I had wanted to. But I'm getting ahead of myself.

 

As I've said, my story is not entirely rosy, so I want to make a sort of blanket disclaimer, and to do that I have to indulge in a little bit of tooting of my own horn. I have a Ph.D. in computer science. I was a senior scientist at JPL when I went to Google, and when I returned to JPL I was promoted to Principal, the highest rung on their technical career ladder. (Actually, it turns out there are higher rungs, but their existence in not publicly known.) I am generally considered to be a pretty bright guy.

 

I am tooting my horn to put the following assessment in perspective: at Googe, if I were to rate people on general smartness I would have put myself in the bottom 25%. It was pretty much the first time in my life that I found myself not at the top of the intellectual pecking order. It was not an easy adjustment for me. But I'm getting ahead of myself again.

 

I wanted to say this up front because since the IPO there has been a steady chorus of criticism along the lines of, "Google has little real value and it's only a matter of time before it all comes crashing down like a bad flashback to February 2000." In my humble but better-informed-than-most opinion this is all sour grapes. Google is a valuable company because the people who built it are incredibly smart and they work incredibly hard. I feel priveleged to have been a (small) part of it. There are many valid criticisms of Google (and I expect I'll be making some of my own), but that they have built little of real value is not among them.

 

I'm not saying this because I want to kiss up to Sergey (as one commentor suggested might be the motivation for one of Doug's postings). I have no need to kiss up to anyone any more (and, though I don't have any firsthand knowledge, I strongly suspect that Doug doesn't either). I'm saying it because Google has taken a lot of bashing Some of it was well deserved IMO (like when Google blacklisted CNet for doing a story that included personal information on CEO Eric Schmidt which was obtained by doing a Google search), but most of it was (and is) not, and I just wanted to stand up and say so.

 

With that out of the way, I'll get on with the story.

 

 

9 COMMENTS:

 http://search-engines-web.com/ said...

... if I were to rate people on general smartness I would have put myself in the bottom 25%.

Hmmm,...how are you defining & rating Intelligence ???????

What is of grave concern is - if everyone there is so-o esoteric, and SEARCH has now become a "common persons" resource, Can they really empathize and understand the needs and habits of the so-called less extreme intelligent members of the Human Race who dominate their customers (and the world)?

The interview where you were asked for your SAT and GPA scores makes one wonder if they are only focussing a very LIMITED portion of Human Intelligence.

Homo Sapiens are so varied and verstile, hopefully the channels of communication will open up to include and understand the masses in what is becoming a universal source of information retrieval!

Hopefully the masses will ALSO be allowed to contribute to Google's evolution ;-)

3:55 PM  

 Ron said...

Hmmm,...how are you defining & rating Intelligence ?

A good question, and I'm actually planning a whole post to answer it.

4:30 PM  

 Ron said...

Oh, I guess I should mention too that they didn't ask me my GPA or my SAT score. (I suspect they reserved that for the non-technical folks.)

4:34 PM  

 Milly said...

Ron, the Swedish Chef lives! It's under "Bork, bork, bork!" in that prefs page dropdown. It even had some utility for a year or so: selecting that language brought back the old blue and white tabs (and less tricky Groups) design. But they've fixed that now (hey, some security holes take 'em longer) : 
www.imilly.com/google-cookie.htm#bork

Once there's enough of you xooglers gathered, maybe you could do an exit reasons survey. I wonder if many would cite the growth from jetski to supertanker as a reason?

In July Marissa Mayer said "We are planning on releasing some alternatives in the next one to two months that make our current captchas more compatible with screen readers ...".

Yet here we are with most Google services, Blogger comments included, still using evil visually-oriented captchas: www.w3.org/TR/turingtest.

Apparently it takes time for a supertanker to change course ...

5:58 PM  

 Milly said...

Oh, and Doug - how about an RSS feed for the comments?

6:04 PM  

 Anonymous said...

I'm looking forward to seeing the comments on intelligence as I am very interested in how it manifests itself in terms of how Google makes decisions as well as how well the technology works.

Here is an example. A long time ago, some Google employees claimed that it was not possible for their index to be spammed because their algorithms were powerful enough to defeat any spam attempts. I think they did not take into account the economics of the Internet -- in particular, that web sites (and therefore links) could be built rather cheaply. What I want to understand is what aspect of intelligence does reaching this type of conclusion fall into and to what extent it factors in the hiring process.

Disclaimers: I have been turned down for employment by Google. No sour grapes -- I realize it is like getting into Harvard; some very intelligent people don't get in. Also, I used to work for another search engine that ran into trouble and got sold.

7:31 PM  

 Steve said...


What is of grave concern is - if everyone there is so-o esoteric, and SEARCH has now become a "common persons" resource, Can they really empathize and understand the needs and habits of the so-called less extreme intelligent members of the Human Race who dominate their customers (and the world)?


Well, maybe not..

http://www.infoworld.com/article/05/12/06/HNgoogleuserstudy_1.html

4:00 PM  

 cayblood said...

On the other hand, there are so many obvious problems out there that don't necessarily require UI skills that they can just work on those and hire a few good UI people to write the interface to them.

8:45 AM  

 Anonymous said...

I also worked at Google and I understand the feeling of being in the bottom 25%. Everyone there was smart, but I didn't feel that the top 25% were that smart, and I didn't feel that the management was that smart either.

Basically, it felt like a graduate research group. Everyone was good or they wouldn't be there. There was a definite pecking order based on sucking up to the prof.

12:16 PM

 

SUNDAY, NOVEMBER 27, 2005

In the beginning...

POSTED BY RON AT 3:34 PM

 

The memory of my first encounter with Google is seared into my memory like JFK's assassination is supposed to be seared into the minds of many baby boomers.

 

As an aside, It's always seemed odd to me that the baby boom generation is supposed to include anyone born through 1964. Technically I'm a boomer, but JFK was long dead when I entered first grade. On the other hand, I do clearly remember what the world was like before there was Google, before there was the World Wide Web, and even before there were personal computers. I am a member of the last generation to know what that world was like. God, do I feel old now.

 

But I digress.

 

I was reading a usenet newsgroup (I'll tell you which one later -- it bears on a peculiar personality trait of mine that has some bearing on my Google experience later on) when someone answered a particularly obscure question and followed up with "Thank God for Google" or some such comment. (When Google acquired Deja and brought their database of old usenet postings on line I went looking for the post that started the whole thing and couldn't find it. This was just the first of many Twilight-Zonish (or X-Filesish for you younger readers) events that happened to me of the course of the next few years.)

 

Google? What's Google, I wondered. So I pulled up a web browser and took a wild guess (which was as good a way as any of finding things on the web in those days): www.google.com. Oh, it's a search engine, kind of like Alta Vista. But, holy shit, it's fast! And it has this uncanny way of putting just what you're looking for right at the top of the results list.

 

To understand what happened next I have to give you a little background about my professional life up to that point. I had moved to California from Virginia in 1988 along with my Ph.D. thesis advisor, David Miller, who had just been offered a job at JPL to do research for the Mars Rover program. Over the course of the next twelve years my career had a lot of ups and downs, and I had a lot of opportunities to leave JPL, but somehow the opportunities always seemed to come along at the "up" times when life seemed good and I wasn't much inclined to rip up my roots, which were growing deeper as the years went by.

 

But the day I found Google just happened to be during a "down" time.

 

My first thought was, "How the hell do they do that?" Alta Vista was astonishing enough in its day, but this took speed and accuracy to a whole new level. I had always been idly curious about how Alta Vista worked, and now I just had to know. In a fit of what Alan Greenspan would have called "irrational exuberance", I dashed off a resume.

 

Google got back to me with astonishing speed. It was early 2000, the dotcom bubble was just reaching its peak, it was a seller's market when it came to any kind of technical talent, and I looked pretty good on paper. If memory serves, it was only about two or three hours before my phone rang. A week later I was flying up for an interview.

 

I don't have nearly as many colorful interview stories to tell as Doug did. I met with half a dozen people. (Neither Larry nor Sergey interviewed me.) They grilled me on the usual things -- caches, hash tables, virtual functions, etc. It was a pretty standard technical interview as best I can recall, with a few Googley twists (how would you write a program that could identify news sites on the web?) I guess I must have hornswoggled them pretty good because they made me an offer. (That may sound like a bit of self-deprecating humor, but it isn't. The truth is I really wasn't qualified for the job. But that didn't become apparent until later.)

 

So now I'm in a pickle. On the one hand I've got this job offer and an opportunity to learn how this cool technology works, and maybe even make a few bucks on the stock options (though that was never the main motivation for going. It was pretty clear even in early 2000 that the internet bubble was gong to burst sooner or later, and besides, how was a search engine ever going to make money?) More to the point, I was worried that if I didn't get away from JPL now I never would, and I didn't really want to retire without ever having experienced anything but working for one organization.

 

On the other hand, I had a pretty cushy situation. My seniority at JPL made it possible for me to work on pretty much anything I wanted to. I was well paid (by my standards at the time). My job was (or seemed) secure. My wife and I had just bought a nice new house and gotten a dog and a cat. (The Southern California real estate market was just starting to pull out of a slump and we were able to buy for what now seems dirt cheap.) To take this job we would either have to move or I would be doing the commute from hell. Neither of those prospects seemed very appealing.

 

I made my decision while visiting some friends in upstate New York. One of my friends had just had her father die unexpectedly. He was an orthopedic surgeon. Very wealthy, or so it seemed to me at the time. Full of life. Commuted from his farm in Virginia to his job in his own Bell Jet Ranger. (I remember he flew in one night while I was visiting my friend on the farm. He was a real regular down-to-earth guy. Asked me if I'd like to join him tomorrow -- he was going to the Pittsburg Steeler's summer training camp. I'm not that much of a football fan, but it was pretty damn cool anyway. There were fans everywhere wondering who the hell we were that we got to go into all the VIP areas.)

 

One day he just keeled over while skiing. Heart attack. He was dead before they got him off the mountain.

 

It was during that reunion that I had an epiphany: life is short, and I had everything I ever wanted: a nice house, a secure job that I (mostly) enjoyed, why in the world would I want to put myself through hell just to hang out at some dotcom that would probably be bankrupt in a year or two?

 

When we got back from New York I called Google and told them that I had decided to decline their offer.

 

 

13 COMMENTS:

 Loughlan said...

Must have been a tricky decision. I wonder if knowing what you had known now, would you have still done the same thing?

I realise it is probably a faux pas to post here. But I wanted to mention what a wonderful piece of writing this is.

Good luck with whatever you choose to do in life.

Kindest Regards
Loughlan Burnett

5:01 PM  

 Anonymous said...

Fascinating!

6:40 PM  

 Leah said...

Hi. Please don't remove the anonymous option for posting comments -- it allows for more interesting people to feel comfortable posting. I see that you've already reserved the right to remove comments; I think that's just, but please do so sparingly. Just my HO.

Great blog, btw. The exaggerated lovefest that people have had with Google gets tiresome.

6:56 PM  

 Lucian said...

I'm kinda facing the exact same situation, though probably not at the same scale or level as you.

The thrill of working on something everyone uses excites me to no end. I've a pretty cushy job at the moment; pretty inconsequential, but cushy.

I wonder about family life should I decide to take on the new, more exciting job. Whether I should just coast, and whether I'll regret that I never took the plunge. That is the question.

7:04 PM  

 Anonymous said...

Does anyone at Google ever stop to think that 99.99% of the company's revenue is the result of a cold hard reality-

The majority of websurfers out there don't have the experience to understand that those adsense ads or google search result ads aren't organic links?

There was a poll done by the BBC about 6 months ago that showed 68% of surfers were unable to point out what links on a page were ads and which were part of the content.

Banner ads had a huge clickthrough rate when they were introduced, then plummeted to almost nothing after a few years when even the newest of the newbies realized they were just advertisements. Text ads right now seem to be enjoying the same "novelty" in that plenty of people arent aware they are ads either. 

Since almost every website using adsense matches the colors and fonts and background of the ad panel to look like the rest of the content, and google's engine puts ads that seem related to the rest of the text on the page, is it any surprise that the clickthrough rate is so high? 10 percent of Google's revenue comes from AOL users, who have a clickthrough rate that's more than 5x higher any other demographic on the net. That should tell you something.

Anyway, I guess my question is, how long until even the AOL crowd catches on and clickthroughs plummet?

Btw- why does it say ads by "Goooooooogle" ? Are you trying to be cute or is there some reason dealing with spiders that you want to be able to identify casual mention of Google from adsense ad mentions?

8:16 PM  

 Doug said...

I'll let Ron address the issue of ad clickthroughs, though it seems pretty evident at this point that the difference between banners and Google ads is relevance to the user. While there's undoubtedly confusion in some instances about what's an ad and what is not, the utility of the ads is unmistakable.

I can also state without hesitation that I worked very closely with Google's partner team to ensure that search partners followed our guidelines for clearly identifying what was a sponsored link and what was a search result. I'll likely post about those wars in the future.

As to "Ads by Goooooogle," we needed a way to make Google ads unique and thus protectable under the law. This was not a universally beloved way of doing that, but it seemed to be the best of the options available.

8:58 PM  

 Ron said...

I'll let Ron address the issue of ad clickthroughs

I'm not quite sure what I'm supposed to address here. At least two issues were raised: 1) the prediction that text ad clickthroughs are going to plummet the way banner ad clickthrough did, and 2) the possibility that people don't know that the ads are ads.

I don't have much to say about issue #2. The ads are pretty clearly labelled, but I will not dispute that there are some pretty dim bulbs out there.

As to issue #1 I'll defer to Carl Sagan who observed that prophecy is a lost art.

10:12 PM  

 Jorg Brown said...

Turned 'em down? Heh. For exactly the same reasons, I stayed as far away from dotcoms as I could. Especially in 2000, when some of the stupider ventures had already disappeared.

I eventually interviewed at Google in 2004, but only after two friends of mine went, and I could grill them on whether or not it was really a place they liked to work. Since Google had just gone IPO, and the stock had shot from 85 to 120, I figured the easy money was already gone from stock options. But my career wasn't going anywhere at work, and I didn't have to move, so why not?

I still want to know what Google told you when you said no. For me in 1998, I told Microsoft no, and they responded with a better offer. Given the job market in 2000, I'm surprised that Google didn't do the same to you.

11:06 PM  

 Anonymous said...

IMO, banner ad clickthroughs fell because they were (are) intrusive to the search engine user's experience. Intrusive ads seem to work on sites like ESPN (at least, I have not heard that they feel the need to abandon them). It would be interesting to learn what % of their regular users also use Google.

I think at some point the rate of "legitimate" clickthroughs will fall off (if it hasn't already) as the medium matures and users become more selective in their surfing and buying behavior.

11:51 PM  

 Red said...

I think the defining difference between Google ads and the more 'traditional' forms is utility, as Doug suggests. Google ads can be so darned interesting that you want to click on some, especially if you're searching hurriedly for product or services. I find myself scanning the page, then automatically scanning the Google ads alongside if I haven't found what I'm looking for. It's an interesting twist on the ad form.

2:37 AM  

 Stefan said...

"Some pretty dim bulbs"?

http://www.pewinternet.org/PPF/r/146/report_display.asp

1:27 PM  

 Anonymous said...

WHAT IS THE CONTINUATION OF THE STORY?????

DID U WORK FOR GOOGLE OR NOT?

11:54 PM  

 nick said...

great piece of writing!
sorryabout your friend...
i'm wondering uif you could include some pieces of writing for your spanish-speaking fans... specially the ones from mexico, like me...
Kudos and good luck in life.

6:13 PM

 

 

SUNDAY, NOVEMBER 27, 2005

The wormgear turns

POSTED BY RON AT 5:07 PM

 

Ron sez:

 

It ocurred to me that by putting up my story piecemeal like this most people are gong to be reading it in reverse order. Hm. I wonder if there's a way to reorder postings -- or if people are so used to this that they tend to read blogs backwards anyway. Wow, the things you start thinking of.

 

Anyway...this is part three of my ongoing saga of my year at Google.

 

Of all the random shit that has affected my life surely none has had more impact than the fact that Google would not take no for an answer. They asked if they could fly me up again to try to change my mind. And they wanted me to bring my wife along as well. Well, heck, a free dinner for two, how could I turn that down?

 

The person assigned to win me over was Urs Hoelzle, at the time VP of Engineering. Three things impressed me about him. First, he told a really good story about how Google was planning to make money (one that turned out to be pretty much right on the money, so to speak). Second, his technical background was in programming languges, which was a big plus for me for reasons that I will explain shortly. And third, he had a really huge dog that he brought to work with him. (Yoshka, the Leonberger, has since become the stuff of legend.)

 

(Another interesting bit of trivia about Urs: he always wore red socks. Something to do with him being Swiss, I believe, but I never fully grokked it.)

 

To fully appreciate the horrible irony of what happened next, I have to tell you about this peculiar affectation (or perhaps the word I'm lookin for is affliction?) of mine: I am a fan of Lisp (the programming language, not the speech impediment).

 

For those of you reading this who are not programmers I have to digress even further and explain a little about programming langauges. Programmers are craftsmen, and craftsmen can sometimes get a little weird about their tools. It's funny to watch Tim Allen wax rhapsodic about the relative merits of the Binford model 2330 frabnobulator because, like all good comedy, it contains more than a grain of truth. Programmers are craftsmen, and programming languages are their primary tools, and so it is not uncommon (understatement of the day) to see spirited discussion of the relative merits of this or that language.

 

One of the reasons that I had chosen to stay safely cloistered in the ivory tower of JPL for over a decade was that my favorite tool/language had definitively lost the language wars. I got into Lisp in the 80's because it was the language used for artificial intelligence (AI) work, which was the field I wanted to get into. Unfortunately, AI failed to deliver on lofty promises, and so it fell out of favor with its (mostly government) sponsors. As a result, industry abandoned Lisp in favor of C, and later, C++, Java and Perl.

 

I don't want to rehash the language wars here. Suffice it to say that I attributed a significant portion of my own professional success to my use of Lisp over the years, and I was loath to give it up. But the industry had moved definitively away from Lisp and towards C++ and, in the Web world, Java. I found both of these languages very difficult and frustrating to master, and that was a large factor in my decision to stay at JPL during most of the dotcom boom. Why should I submit myself to all that pain when I had a perfectly good job where I could do as I pleased?

 

But in Urs Hoeltzle I saw a beacon of hope. I had done my homework on Urs and found that he had made a major portion of his career in the development of a language called Smalltalk, which was very similar to Lisp in many important ways. In particular, it was not a mainstream language, so I thought that in Urs I would find a kindred spirit, an engineer who knew what it was like to work on and perhaps even fall in love with a programming language that was not part of the mainstream.

 

My wife and I have different recollections about what happened next. I have very strong memories of my telling her about starting to change my mind about Google, and her responding that she wasn't too enthusiastic about the prospect, but that if it was what I wanted to do that she would support me. She insists that she never said any such thing.

 

It seemed like a good time to leave JPL. Things were changing there and I was beginning to get somewhat marginalized on the project I was working on. (That's a whole 'nuther long story. If you're interested you can read it here. Be aware that it contains some spoilers for the Google story, so if you're hanging on to the edge of your seat waiting for the next installment here at xooglers (yeah, right) you might want to give it a miss.) So I called Google and we set a start date in June of 2000, and gave notice at JPL.

 

To my astonishment, people started coming out of the woodwork to express their shock and dismay that I was leaving JPL. Several people who I was sure would be happy to see me leave actually asked me to reconsider. It did my ego a lot of good to hear it, but it was too late. I was off to Mountain View.

 

 

13 COMMENTS:

 Anonymous said...

But Erann, in the end you didn't get to use Lisp (or Scheme, I presume). That's too bad! Is it still the case that a lisper has no chance at Google? They have Norvig there. Inquiring minds want to know.

11:54 PM  

 Anonymous said...

Ron, 

Sorry -- I got Erann from your JPL/Lisp/Google piece, and didn't see yet you'd changed your name. 

I'm really curious about languages at Google. E.g. is ML or CAML used? As I'm sure you know, there is neat and powerful software that runs in those languages. And Haskell too, I guess. 

Thanks for any info -- and aren't you afraid of getting sued by Google? I heard they are paranoid -- but not psycho like Intel.

12:08 AM  

 TraderEyal said...

Hey Doug, Cool blog. I really enjoy reading your story. As for reading the blog backwards, you can consider putting a 'sticky' on top by way of a future dated post recommending readers to start from the oldest post..

All the best
TraderEyal

12:09 AM  

 Ron said...

Is it still the case that a lisper has no chance at Google? They have Norvig there.

Peter Norvig came to Google from NASA's Ames Research Center to be Director of Search Quality. Peter was well known in the Lisp community, having written am excellent textbook on the topic but he, to my dismay, was already abandoning Lisp in favor of Python and Java long before he came to Google.

As far as whether a Lisp programmer has "no chance" at Google (by which I assume you mean no chance of being allowed to use Lisp), I have no idea. It's been nearly five years since I left; perhaps things have changed. But I wouldn't count on it. BTW, it is not at all clear to me in retrospect that using Java was not the right thing to do from the company's point of view. More on that in a later post.

aren't you afraid of getting sued by Google?

No. Fear that the First Amendment no longer applies is a self-fulfilling prophecy.

12:42 AM  

 Philipp said...

Yep, I'm currently reading the last few days' posts backwards... but once you're up to speed you can read the most recent post everyday and that means it'll be in chronologically correct order. That being said, I believe there's an option over at blogger to order things chronologically, but I'd rather turn this into a book later on (via Lulu.com or BlogBinders.com or even better, a publisher).

8:00 AM  

 Phil said...

Nat Friedman's blog has the first page with newest-first, and all previous months in chronological order. It's a bit disorienting at first, but it makes a whole lot more sense. I haven't gotten around to doing that on my own blog, but I really should.

9:45 PM  

 David said...

I actually prefer having the blog with most recent posts first. Otherwise when I came, I'd have to page through old entries. 

And yes, I do start at the bottom and work my way up. ;)

6:11 PM  

 Charles said...

I hope you don't mind but I created a new RSS feed which will spill out the posts in the opposite order (one per day), ie. the oldest posts first then moving forward. Details here:

http://www.surfarama.com/?p=266

It sort of serializes the story. I'll be adding new posts they come.

6:40 PM  

 Doug said...

Hey, thanks! Neither Ron nor I know anything about RSS, and we had it way down the list of things to figure out. And to prove I'm clueless, let me ask you this: can I just add a link under the Atom feed link on Xooglers to the page you've set up? Is that what people would expect if it the link were labeled "RSS feed"?

7:01 PM  

 Charles said...

Just for everyone elses interest, I have sent code to Doug to display the serialized RSS feed link (as opposed to using the link above. Reason being it requires some javascript to generate unique urls so it can deliver feed content in a serialized fashion.

Or you can visit my blog to subscribe here.

10:02 PM  

 Anonymous said...

Reading the blog backwards on a computer screen ain't that big of a deal. However, I don't read it on a screen, I actually print it out every other day (in order to read it at home or/and on the bus), and this is where things start getting really complicated. So yes, I would love to be able to re-sort the entries prior to printing.

Ah, and by the way: this is a darn good blog, let me tellya that.

4:02 AM  

 Isaack said...

I was recommended your blog from another blogger. Once i got here, I saw that I had to take some time getting it from beginning, so I had the same browser with your page on opened for two days before I started it... now, an hour ago. I've gotten to to this point and really enjoying myself!

With the backwards posting... I just started reading cronologically, scrolled all the way down and started from there.

4:07 AM  

 Anonymous said...

Ron, another admirer of Heinlein. Hopefully science fiction is not dismissed as BS at Google

10:10 AM

 

 

SUNDAY, NOVEMBER 27, 2005

Day One

POSTED BY RON AT 11:34 PM

 

Ron sez:

 

My first day at Google went like this:

 

5:30 AM, wake up.

6:00 AM drive to Burbank airport.

6:30 AM stand in line at the Southwest airlines gate to get one of those now defunct plastic boarding passes

7:30 AM board the plane

9:00 AM or so land in San Jose

9:30 AM get the rental car

10:00 roll up to the Googleplex

 

It was a murderous commute. Four hours, which I eventually whittled down to three after a year of optimizing. I went through various permutations of cabs, rental cars, having my wife drive me to and from the airport (that didn't go over well at all) before I finally settled into something that vaguely resembled a routine. I ultimately ended up driving two different cars back and forth between LA and San Jose, and on a couple of occasions I did my commute in a Cessna 182RG (I'm a pilot) which was just too cool for words.

 

But getting to San Jose wasn't nearly as bad as trying to get back home. Originally I took Monday to be my telecommute day so that I could be at Google on Friday afternoons for the weekly TGIF meeting. TGIF is, as far as I know, still a Google tradition where everyone gets together to hear the latest developments from upper management (which back then meant hearing it directly from Larry and Sergey. Maybe it still does, I don't know.)

 

Trying to get on a Southwest flight at the San Jose airport on a Friday evening in mid-2000 was a serious nightmare. Heck, just trying to get to the airport was a nightmare. (I remember on one occasion the traffic was so gridlocked that I could not get to the rental car return, which back in those days was still in the main parking structure. I just left the car in a remote parking lot and walked from there to the terminal to make my flight. I guess the rental car company eventually found the car because only the normal rental charges ever showed up on my credit card.)

 

I found a place to stay, renting a room from Susan Wojcicki (now Google's director of product management) and her husband Dennis. Really cool people, and coincidentally, their house is also the one where Google first set up shop, so I really felt like I was in the bosom of history.

 

In the back of my mind there was one niggling little worry: I had neglected to ask Urs what I would actually be doing. Having decided to embark on this adventure I guess I decided it didn't matter. I was going to do something new and different, and, more to the point, I was going to learn how this incredible search technology worked. And I was going to knock Urs's red socks off with all the cool ideas I had for new features.

 

Things started off really well. The work environment at Google was everything that it has since been reputed to be. I learned to ride a unicycle while I was there, and my pool game improved considerably. My office had a spectacular view. Chef Charlie's cooking was delicious. (I still make some the recipes he gave me, but I can never get them to come out nearly as good as he did. I hear he's planning to open up his own restaurant soon. I'm looking forward to patronizing it.) The conference rooms were all color coded, except for the main one which was called the Lucky Lounge. Meetings, in stark contrast to my experience at JPL, started on time. It was all very refreshing and energizing. I was brimming with enthusiasm.

 

So I was a little disappointed when I found out on day 1 that I had been assigned to the ads group. But that disappointment turned to dismay when I learned what my assignment was to be: I was the lead engineer on a new advertising system code named "adstoo", what eventually became AdWords. That part wasn't so bad. The bad part was, this was going to be the inaugural Java project at Google. Google, which had until now been a Java-free zone (which was one of the reasons I took the job) was going for Java in a big way, and I, the consummate Java hater, was supposed to be its chief evangelist.

 

Just peachy.

 

On top of that there was trouble at home. My wife was having a really tough time trying to keep everything together back on the home front, what with the new house and four-legged kids and all. Our finances were starting to look a little rocky because I had taken a pay cut and taken on extra expenses for weekly plane tickets, rent, and a second car. (Google did give me a travel allowance, one of the factors in my decision to take the job, but my salary and travel allowance together were still less than my old salary had been at JPL.)

 

And then there were the stock options.

 

Yes, stock options can make you rich, but getting to that point can be pretty damned annoying. When you sign on to a company that has stock options as part of the compensation plan you get an option grant up front, but you don't actually vest any of those options until you've been at the company for a year. After that you vest a few more shares every month, but if you quit -- or get fired -- after eleven months and twenty nine days you're screwed.

 

Notwithstanding all that, you can actually choose to excercise your options up front even though they aren't vested, and it's often wise to do so because if you wait and the stock price goes up then when you do exercise the options then the difference between what you paid and what they were worth at the time counts as income for purposes of the alternative minimum tax despite the fact that you might be restricted from selling your shares because, for example, your company hadn't gone public yet. A lot of people got badly burned during the dotcom crash because they exercised their options when they couldn't sell the stock, generating a huge amount of AMT income on which they owed taxes. Then they watched helplessly as the stock price crashed, often to levels below what they had paid for the stock. So not only did they lose some or all of the actual cash they had paid to exercise the options, they also owed huge tax bills on paper "profits" that not only had they never actually earned, they could not have earned even if they wanted to because their stock was restricted. It was horribly unfair.

 

On the other hand, if you vest in your options but don't exercise them and then leave the company then the options expire and become worthless. Since I wasn't really planning to stay at Google for the long haul I figured it would be wise to exercise early to avoid the AMT trap. But to do that I had to come up with the cash to actually pay for the stock. And to add insult to injury, the day I joined Google they announced that they had finalized their deal with Yahoo, and the board raised the stock price by a factor of nearly 5! So not only did I have to come up with the cash, I had to come up with five times as much cash as I was expecting to have to come up with! My wife was not pleased.

 

In retrospect of course it all turned out OK, but at the time it was nerve-wracking. The dotcom crash was gathering steam, and Google's long-term prospects were far from clear. Still, growth was steady and there was electricity in the air, so I was fairly optimistic, even at the time. But writing that check was painful.

 

All in all, it was a stressful week.

 

 

7 COMMENTS:

 Uri L. said...

A great post! thanks for sharing.

What a manic commuting routine...

6:40 AM  

 Kiltak said...

And I tought that 3 hours of car each day was bad enough. Personally, travel time is killing me, so taking the plane each week to go to work would be out of the question. Especially with a demanding wife and a baby.

You were very courageous my friend ;)

12:31 PM  

 Tuor said...

tuor sez:

the doug / ron thing is not working very well, and its sort of confusing to read things the way they are (though things are, of course, extremely intersting, otherwise I wouldn't be here) As one of the chief complainers on blogger, I note that MovableType has had a category function for the longest time now, so that you can only view things by category (e.g. click on "Ron" link and see only Ron's posts). Maybe you guys should go off Google (or companies that have been bought by Google, though I suppose its pretty much the same thing) and try some open-source fun!

Of course, despite my complaining I'm still here using blogger, so I'm not sure what right I have to make suggestions. Take it all with a grain of salt, if you will.

1:22 PM  

 Anonymous said...

Thanks for sharing all those "ancient" moments of Google with us.

"TGIF is, as far as I know, still a Google tradition where everyone gets together to hear the latest developments from upper management (which back then meant hearing it directly from Larry and Sergey. Maybe it still does, I don't know.)"

Yes, it still does. :-)

5:09 PM  

 Anonymous said...

Surprised you didn't take Caltrain or the light rail to get to/from SJC.

11:29 PM  

 Taco said...

iTs a great post! you should change the way the comments are viewed. allowing comments on a pop up window make me read both, all other post and comments because the main page takes a hwile to load because the text is too long!

3:35 PM  

 Chris Marino said...

Ron, not surprised that you were at Google only a year. As important as AdWords is, I don't think it can provide the ongoing technical challenge that is necessary to keep the best people engaged.

1:48 PM

 

MONDAY, NOVEMBER 28, 2005

Into the breach

POSTED BY RON AT 12:36 PM

 

Ron sez:

 

I dove into the adstoo project with as much enthusiasm as I could muster, which I'm ashamed to say wasn't much. The situation was exacerbated by the fact that we had no Java development infrastructure. We were writing bareback, so to speak. We had no debugger. We were using JSP, but had no editor support for JSP syntax. (That turned into a real debugging nightmare. It could take many tens of minutes to find a single typographical error because the only indication that there was a problem was that the output just looked all wrong, but the actual problem could be very far away from the apparent source of the problem.)

 

Fortunately for me, I was assigned a junior engineer to work with/for me, and he actually knew what he was doing. While I struggled to learn the Java libraries and debugging techniques (I knew the basic language, but I had never done any serious development in it before) this guy just took the bull by the horns and pretty much just wrote the whole damn thing in a matter of weeks. I sometimes pull this old joke out of the dustbin, that in the ancient tradition of senior-junior relationships, he did all the work and I took all the credit.

 

That's not quite true. I did end up writing the credit card billing and accounting system, which is a nontrivial thing to get right. Fortunately for me, just before coming to Google I had taken some time to study computer security and cryptography, so I was actually well prepared for that particular task. Back in those days internal security was more or less nonexistent. All the engineers had root access to all of the servers. But I believe in planning ahead, and I anticipated the day when Google was not going to be a small company any more, and so I designed the billing system to be secure against even a dishonest employee with root access (which is not such an easy thing to do). I have no idea if they are still using my system, but if they are then I'd feel pretty confident that my credit card number was not going to get stolen.

 

But on the whole I was struggling, not just technically, but personally as well. The situation was exacerbated by my manager, who thought that the answer to my falling behind schedule was to start micromanaging me. That just made things worse. Much worse. I had been a senior scientist at a national research lab, essentially the equivalent of a tenured professor (but without the teaching responsibilities). I was used to being my own boss more or less, and I really resented being asked to make detailed lists of everything was going to do every single day. After all, Google had come asking me to join them, and so I felt I ought to be accorded more respect. But on the other hand I had to admit that I was not really performing, and so all the micromanagement was in some sense justified. I responded, I'm ashamed to admit, with some passive-aggressive head games, and so the situation deteriorated rapidly to the point where my manager and I were barely on speaking terms.

 

Things were made worse by the fact that I had been assigned an office mate who was also new to Google, and who was not part of the ads group. Most of the other ads group members were sharing offices (or cubicles) with other ads group members, and so I felt I wasn't really part of the club. On top of that, I was away from home and didn't really have a life up there in Northern California. The stress mounted. I started to get paranoid that I would get fired before reaching the one-year mark. I started experiencing stress-related health problems, some of which are still with me today. On more than one occasion I came that close to quitting. To this day I have no idea why I didn't.

 

It was about this time that I had my one and only meeting with Larry Page. It was to discuss the progress of the adstoo project and to set a launch date. My manager was there along with a couple of other people (including Doug I think). Things went smoothly until Larry suggested changing the way billing was handled. I don't remember the details, but my response was that this would be significant work. No one challenged me, but I found out later that the reaction of people in the room was something along the lines of, "Is he crazy? This ought to be a trivial change." This little incident turned out to have very far ranging repurcussions later, but that will have to wait for the next blog entry.

 

Somehow we actually managed to launch AdWords on schedule, in September of 2000. It still seems like a bloody miracle. Most of the credit goes to Jeremy, Ed and Schwim. It could not have been done without them.

 

I can still remember watching the very first ad roll in. It was for a company called Lively Lobsters. Two months ago, after five years of intending to do so, I finally bought myself a little toy stuffed lobster to commemorate the occasion. (Update on 12/9/2005: It appears that Lively Lobsters has gone out of business. There's some irony for you.)

 

About two weeks later all hell broke loose.

 

 

13 COMMENTS:

 Philipp said...

You got me really curious now...

2:40 PM  

 Zoli Erdos said...

Ron, this should be illagal! Like in soap operas, at the most exiting line...stay tuned .. ahhhhhhhh :-)))

3:03 PM  

 http://search-engines-web.com/ said...

One bit of information that would really be helpful to clarify

Who came up with the idea of SPONSOR LINKS
The (two, now three) listings ABOVE the SERPs with the pastel background color..

Was there any concern/debate about ethics. 

Whose idea was it to create the ADWORDS BOXES -then years later take them away.

The Google Sandbox (suggestion) tool - why did it NOT include numbers - like the inventory.overture tool?

How did the transition from Pay per View - to pay per click happen?

3:16 PM  

 AP said...

It appears that the boys at google didn't realize Ron's talent and lost out eh?

5:01 PM  

 John said...

Man, I am hooked.. very interesting.. keep going...:-)

9:09 PM  

 Anonymous said...

I have a question about the 20% time to pursue one's own interests. Must the project be a totally different project or may someone just do more work on their current projects?

11:40 PM  

 netron said...

aah man - this is like one of those cliffhangers that you see in soap operas...

compulsive reading!

3:40 AM  

 Anonymous said...

Ron,

The lisper in me wonders why you didn't write a lisp program to generate your Java program -- at least then, you'd get to use all your lisp tools/skills, and the stuff would be syntactically correct (as long as it didn't "blow out" the girly-man Java development tools).

But then again, you'd probably have only gotten into big trouble for that sort of smartass behavior.

Also, I sympathize with your AMT/options/salary woes. A lot of people can't deal well with making monetary sacrifices for risky businesses -- it keeps them up at night. 

If you own risky securities, you can always sell them and buy less-risky ones (but perhaps get whacked by the 365-day capital gains tax) -- but if the risky securities come as part of your job, you are really stuck. And you got screwd with the 5x revaluation! What a shame!

8:15 AM  

 Anonymous said...

Your blog is interesting. The Google persona I read about on tech sites (being super friendly and laid back) seems to be at odds with your experiences. There is a lot more business than engineering involved in your work there.

-Ivan

9:18 PM  

 Kevin said...

I liked the lobster part

2:03 AM  

 markg said...

Doug, When do we get our Xooglers Tee Shirts. Love the stories can't wait until you start telling about the Google Ski trips to Squaw Valley!!

3:00 PM  

 Ryan said...

Ron,

Here is some more irony for ya... I founded Lively Lobsters in 1998 and was the guy who placed the first AdWords ad you speak of. I am not sure if you are aware, but a nice young lady named Allegra emailed me requesting a high-res version of my company logo for use on some t-shirts being made for the engineers to celebrate. (I never received one... I have asked around and heard mixed news on whether they were ever made, and if they were, I would love to have one now!) You correctly stated that Lively Lobsters went out of business, but I am reviving it this fall. The only reason I shut it down in the first place was because I became so involved with other profitable AdWords advertising that it no longer made sense to focus on Lively Lobsters. These other AdWords ventures have kept me a faithful advertiser ever since, and I am one of Google's top spenders all time, having purchased literally many millions of dollars worth of clicks through my various companies. The power of AdWords itself is responsible for my success, and those very fruits are being used to resurrect Lively Lobsters. In a nutshell... your system (AdWords) made Lively Lobsters, then me, a success and now that success is coming back to revive Lively Lobsters! I am not sure if this is interesting to you or anyone else, but I thought I'd share my story when I came across this blog via a Google search for "Lively Lobsters" while satisfying my curiosity about references to Lively Lobsters still floating around. Thanks for everything you did for Google and AdWords... I owe ya some long-overdue lobsters. Get in touch with me and they are yours!

10:42 PM  

 Ron said...

Hi Ryan,

Wow, what a great story! Glad to hear everything worked out so well. If you are spending millions on AdWords then you must be doing well for yourself indeed. Congratulations! And glad to hear the Lively Lobsters is rising from the broiler. Maybe I can return the favor by placing the first order on the new site :-)

I owe ya some long-overdue lobsters. Get in touch with me and they are yours!

Thanks! We'd gladly take you up on your offer, but we don't know how to reach you! Would you please send a clue to xooglers at gmail? Doug has some things he wants to ask you privately as well.

8:33 PM

 

MONDAY, NOVEMBER 28, 2005

The billing disaster

POSTED BY RON AT 7:48 PM

 

The AdWords launch went fairly smoothly, and I spent most of the next two weeks just monitoring the system, fixing miscellaneous bugs, and answering emails from users. (Yes, I was front-line AdWords support for the first month or so.)

 

The billing system that I had written ran as a cron job (for you non-programmers, that means that it ran automatically on a set schedule) and the output scrolled by in a window on my screen. Everything was working so well I didn't really pay much attention to it any more, until out of the corner of my eye I noticed that something didn't look quite right.

 

I pulled up the biller window and saw that a whole bunch of credit card charges were being declined one after another. The reason was immediately obvious: the amounts being charged were outrageous, tens of thousands, hundreds of thousands, millions of dollars. Basically random numbers, most of which no doubt exceeded people's credit limits by orders of magnitude.

 

But a few didn't. Some charges, for hundreds or thousands of dollars, were getting through. Either way it was bad. For the charges that weren't getting through the biller was automatically shutting down the accounts, suspending all their ads, and sending out nasty emails telling people that their credit cards had been rejected.

 

I got a sick feeling in the pit of my stomach, killed the biller, and started trying to figure out what the fsck was going on. (For you non-programmers out there, that's a little geek insider joke. Fsck is a unix command. It's short for File System ChecK.)

 

It quickly became evident that the root cause of the problem was some database corruption. The ad servers which actually served up the the ads would keep track of how many times a particular ad had been served and periodically dump those counts into a database. The biller would then come along and periodically collect all those counts, roll them up into an invoice, and bill the credit cards. The database was filled with entries containing essentially random numbers. No one had a clue how they got there.

 

I began the process of manually going through the database to clean up the bad entries, roll back the erroneous transactions, and send out apologetic emails to all the people who had been affected. Fortunately, there weren't a huge number of users back then, and I had caught the problem early enough that only a small number of them were affected. Still, it took several days to finally clean up the mess.

 

Now, it's a complete no-brainer that when something like that happens you add some code to detect the problem if it ever happens again, especially when you don't know why the problem happened in the first place. But I didn't. It's probably the single biggest professional mistake I've ever made. In my defense I can only say that I was under a lot of stress (more than I even realized at the time), but that's no excuse. I dropped the ball. And it was just pure dumb luck that the consequences were not more severe. If the problem had waited a year to crop up instead of a couple of weeks, or if I hadn't just happened to be there watching the biller window (both times!) when the problem cropped up Google could have had a serious public relations problem on its hands. As it happened, only a few dozen people were affected and we were able to undo the damage fairly easily.

 

You can probably guess what happened next. Yep. One week later. Same problem. This time I added a sanity check to the billing code and kicked myself black and blue for not thinking to do it earlier. At least the cleanup went a little faster this time because by now I had a lot of practice in what to do.

 

Stress.

 

And we still didn't know where the random numbers were coming from despite the fact that everyone on the ads team was trying to figure it out.

 

 

6 COMMENTS:

 Anonymous said...

There's no need to apologise for your geekiness -- as if we don't all know what chron and fsck are. 

I'd save the apologies and explanations for the "Steven Levy" style book. I'd guess that most know to go look at things like this for definitions.

12:20 PM  

 Jan said...

There are still problems with creditcards being declined without any reason - perhaps its all your fault :)

PS: support-team still can't explain the problem

12:48 PM  

 Philipp said...

Uhm, just to contrast the first comment, yeah thanks for letting us known what Fsck means right here and now without us having to jump around websites too much :)
Who says you can't read a Steven Levy book online? Plus, I've got a suspicion this will turn into a book one day or another, and then it will come in handy to not rely on links for crucial points :)

2:26 PM  

 Greg Linden said...

I'm curious, what database were you using at that time?

3:16 PM  

 Anonymous said...

The problem is still there , they are supposed to bill every other day or so but sometimes they will let it accumulate and the bill the card - But many times the transaction get declined (as many cards have a 10k limit for a single transaction)

6:02 PM  

 Anonymous said...

PS. fsck is file system consistency checker.

7:50 AM

 

 

WEDNESDAY, NOVEMBER 30, 2005

Racing to the finish

POSTED BY RON AT 8:26 PM

 

Ron sez... oh wait, don't need that any more.

 

OK, time to wrap up this little soap opera.

 

The problem turned out to be something called a race condition, which is one of the most pernicious and difficult kinds of bugs to find. (Those of you who are technically savvy can skip to the end.)

 

Most modern server code is multi-threaded, which means that it does more than one computation at once. This is important because computers do more than just compute. They also store and retrieve information from hard disks, which are much, much slower than the computers. Every time the computer has to access the disk things come to a screeching halt. To give you some idea, most modern computers run at clock speed measured in gigahertz, or billions of cycles per second. The fastest hard disks have seek times (that is, the time it takes the drive to move the read/write head into the proper position) of several milliseconds. So a computer can perform tens of millions of computations in the time it takes a hard disk just to get into position to read or write data.

 

In order to keep things from bogging down, when one computation has to access the disk, it suspends itself, and another computation takes over. This way, one computer sort of "pretends" that it is really multiple computers all running at the same time, even though in reality what is happening is that one computer is just time-slicing lots of simultaneous computations.

 

The ad server, the machine that actually served up ads in response to search terms, ran multi-threaded code written in C++, which is more or less the industry standard nowadays for high-performance applications. C++ is byzantine, one of the most complex programming languages ever invented. I've been studying C++ off and on for ten years and I'm still far from being an expert. Its designers didn't really set out to make it that complicated, it just sort of accreted more and more cruft over the years until it turned into this hulking behemoth.

 

C++ has a lot of features, but one feature that it lacks that Lisp and Java have is automatic memory management. Lisp and Java (and most other modern programming langauges) use a technique called garbage collection to automatically figure out when a piece of memory is no longer being used and put it back in the pool of available memory. In C++ you have to do this manually.

 

Memory management in multi-threaded applications is one of the biggest challenges C++ programmers face. It's a nightmare. All kinds of techniques and protocols have been developed to help make the task easier, but none of them work very well. At the very least they all require a certain discipline on the part of the programmer that is very difficult to maintain. And for complex pieces of code that are being worked on by more than one person it is very, very hard to get it right.

 

What happened, it turned out, was this: the ad server kept a count of all the ads that it served, which it periodically wrote out to the database. (For those of you wondering what database we were using, it was MySQL, which leads to another story, but that will have to wait for another post.) It also had a feature where, if it was shut down for any reason, it would write out the final served ads count before it actually quit. The ad counts were stored in a block of memory that was stack allocated by one thread. The final ad counts were written out by code running in a different thread. So when the ad server was shut down, the first thread would exit and free up the memory holding the ad counts, which would then be reused by some other process, which would write essentially random data there. In the meantime, the thread writing out the final ad counts would still be reading that memory. This is why it's called a race condition, because the two threads were racing each other, with the ad-count-writer trying to finish before the main thread freed up the memory it was using to get those counts. And because the ad-count-writer was writing those counts to a database, which is to say, to disk, it always lost the race.

 

Now, here is the supreme irony: remember the meeting with Larry where he wanted to make a change to the billing model that I said would be hard and everyone else in the room thought would be easy? The bug was introduced when the ad server code was changed to accommodate that new billing model. On top of that, this kind of bug is actually impossible to introduce except in a language with manual memory management like C++. In a language with automatic memory management like Java or Lisp the system automatically notices that the memory is still in use and prevent it from being reused until all threads were actually done with it.

 

By the time this bug was found and fixed (by Ed) I was a mental wreck, and well on my way to becoming a physical wreck as well. My relationship with my wife was beginning to strain. My manager and I were barely on speaking terms. And I was getting a crick in my neck from the chip I was carrying around on my shoulder from feeling that I had been vindicated in my assessment of the potential difficulties of changing the billing model.

 

So I went to my manager and offered to resign from the ads group. To my utter astonishment, she did not accept.

 

 

18 COMMENTS:

 Anonymous said...

Nice story. But I wish you did not end most of your posts with dramatic phrases like 'utter astonishment'. It is a really sophomoric style for a narrative.In fact, it reeks of the Dan Brown style that I totally detest.

12:04 AM  

 Anonymous said...

Dude, it's either a nice story or not. Stop nit-picking to show off and just enjoy!

2:12 AM  

 justin flavin said...

yeah - stop nit picking. just sit back and enjoy the cliff hangers.

2:54 AM  

 Anonymous said...

Actually, I doubt Ron is exaggerating at all when he says "utter astonishment."

Lispers are perfectionists -- and they drive lesser programmers nuts with their well-deserved superior attitude.

It doesn't surprise me that after Ron screwed up ("failed") because they made him use a lousy language, he felt disgusted with himself and the job, and figured that they wanted him out.

Indeed, if he'd been working with a lisper and the guy made an error with similar ramifications on a NASA mission, Ron would probably be disgusted with the loser.

But most normal programmers use lousy tools, and they tolerate errors more than the perfectionists. They were probably just psyched that Ron's stuff worked and was making them some real money. 

But at no point did they ever decide that they needed to start using Lisp instead of their second-rate Fortran-derived languages. [Ron -- you can see I've suffered at the hands of management too.]

5:01 AM  

 Anonymous said...

Just curios whether it is on Microsoft or Linux.

9:16 AM  

 Anonymous said...

I'm surprised Larry and the others didn't realize this was a difficult problem requiring a careful design, testing, etc.

10:55 AM  

 Brinke said...

great story, but i totally have a headache after reding the C++ nightmare part. Time for a, uh, nap.

3:37 PM  

 Adi Oltean said...

Nice story.

Multithreaded programming is not for everyone. Finding bugs in MT code, or designing some new MT code require experience, a certain state of mind, and an ability to analyze methodically a large set of possible cases. Whenever I have to write C++/C# code that uses shared data structures by two or more threads, I am 10x more careful than usual.

I can certainly say that it takes years to become somewhat familiar with multithreaded programming, especially in things like finding common MT bugs, writing efficient MT code that won't trash the L1 cache or introduce false sharing in multi-proc systems, dealing in a defensive way with shared data (life-management, ownership issues of global objects on the heap, etc), and even more advanced topics like lock convoy issues, lock-free MT programming, etc.

And, no, don't even think that a high-level language like Java is going to automatically solve all your problems here. You can have can be lots of suble MT bugs in Java/C# code as well.

11:11 PM  

 Anonymous said...

Ugh, the nigthmare of race conditions. The story gives me shivers just thinking about when I had to track down similar bugs.

10:22 AM  

 Anonymous said...

Reminds me of my RA during my MS for some reason... Great posts, I'm addicted, keep it up.

3:23 PM  

 Anonymous said...

Definitely a great post. Carefully choosing language/platform is very difficult in business environment. I had an internship at of the giant software companies this summer, and was quite surprised by some of the executive decision made by managers. Essentially I had to design a test framework for a system that set up a certain environment for the production code. I wanted to do it in C# or perl, but everyone on the team favored C++ (with no STL libraries!!). As a result, I spent 3/4 of my time writing string manipulation functions, implementing various storage containers, and looking for memory leaks. At the end of the internship I got most of the things to work, but my life literally went to hell.

7:54 PM  

 Anonymous said...

Definitely a great post. Carefully choosing language/platform is very difficult in business environment. I had an internship at of the giant software companies this summer, and was quite surprised by some of the executive decision made by managers. Essentially I had to design a test framework for a system that set up a certain environment for the production code. I wanted to do it in C# or perl, but everyone on the team favored C++ (with no STL libraries!!). As a result, I spent 3/4 of my time writing string manipulation functions, implementing various storage containers, and looking for memory leaks. At the end of the internship I got most of the things to work, but my life literally went to hell.

7:54 PM  

 Anonymous said...

Dude, you worked for a woman?!!! Do they have those in geekdom?

10:57 PM  

 roschler said...

It is nice that Java has automatic memory management and for many, many apps it is a better choice for fast development without memory management headaches.

But it's lack of C++ style destructor brings in a host of problems of its own, especially when it comes to managing lockable resources. Destructors and the intelligent use of smart pointers (or shared pointers) can go a long way to making resource management much easier. Last I checked Java's finalizer wasn't much help here and in fact introduces problems of its own.

I agree that C++, especially when people try to turn STL into a borderline functional programming language, can be a nightmare. If someone could create a language with automatic memory management and C++ style destructors, that would be great.

Multi-threaded programming will always be a huge precarious chore until the Einstein of operating systems builds something we haven't seen yet into the next big O/S.

1:16 AM  

 Anonymous said...

use PERL

2:33 AM  

 saurav said...

This kind of reminds me of the pain i went thru writing a multithreaded external sorter for a dual cpu machine.
Race conditions in multithreaded programs can be nightmarish. I wonder if anyone knows of a modelchecker
for C++ which can allow us to express and validate multithreaded protocols to avoid such problems.

4:15 AM  

 Joshua Smith said...

"But at no point did they ever decide that they needed to start using Lisp instead of their second-rate Fortran-derived languages."

Of course, C++ is not Fortran-derived. It's ALGOL-derived: ALGOL begat Pascal begat C begat C++.

ALGOL is fundamentally different from Fortran in that ALGOL was the first computer language defined in BNF.

1:59 PM  

 Anonymous said...

"I wonder if anyone knows of a 
modelchecker for C++ which can 
allow us to express and validate 
multithreaded protocols to avoid 
such problems."

I think Coverity Prevent can 
detect some of the problems 
discussed in this thread

coverity.com/concurrency/index.html

1:51 PM

 

 

 

THURSDAY, DECEMBER 01, 2005

Interlude

POSTED BY RON AT 10:07 AM

 

I'm going to be away for a couple of days so no new posts from me before the weekend. I did want to say before I went, though, that it may seem that I am, as Doug put it, a disgruntled ex-employee. I know it's not apparent from the story so far, but I am in fact quite gruntled (now there's a word for you), and not just because everything turned out well in the end. Even at the time (well, more towards the end) I thought that my experience at Google was on balance very positive, though it won't be apparent why until I get a little further along in the story.

 

I wanted to say this up front because I got an email today from my old manager, the one I've been kind of harshing on in previous posts, and that made me realize that I may have some people worried that I'm writing all this because I have an axe to grind. I don't. It's just coming out negative now because it started out that way, and because, frankly, negative experiences make much more interesting stories. I'll have more to say about that later too.

 

Thanks for bearing with me. Hasta luego!

 

 

10 COMMENTS:

 http://search-engines-web.com/ said...

It is vital that this information be publicized - not only for historical research, but so that Managers and Administrators can discover what imperfections exists, and tweak the atmosphere.

Most would be less likely to risk offending some hierarchy by confronting them while you are a subordinate.

Whan a company is constantly seeking the creme' de la creme' of Engineers and Programmers - they have an obligation to be as concerned with the emotions and frustrations of this elite workforce - as much as they are concerned with perfecting the Search Engine Softwares.

From a historical perspective, it will be another piece of the puzzle in the history of one of the most globaly influential companies in the history of the World Wide Web.

The motives do not automatically invalidate the validity of these concerns being expressed, and archived for future generations.

12:25 PM  

 John Vanhara said...

I cannot even imagine someone would say that you are disgruntled ex-employee. It doesn't feel like that at all. Maybe that's because I am European, but I am used to much harsher criticism and it's still considered OK. Nothing is perfect in this world. Everybody is doing mistakes. Bad stuff and good stuff happens all the time. So there is no reason to be too careful about how to explain what happened. I have very positive feeling from the way you explain everything. It is really good story.

1:50 PM  

 wfzimmerman said...

P.G. Wodehouse's Bertie Wooster may have been the first modern figure to use "gruntled" as a verb. You are following in heroic footsteps.

1:56 PM  

 Anonymous said...

The flood is over,
the land is dry,
so why do you wear,
such big balls to write all this stuff.

10:23 PM  

 amar said...

just wanted to let you guys know that i'm enjoying reading your recollections. you don't come across as disgruntled employees at all. looking forward to hearing more.

10:29 PM  

 KenP said...

"Do no Harm."

Interesting post that isn't on topic but what the heck...

Seems the bad guys over in Iraq have a GPS and a laptop with Google Earth with them most of the time. They know where they are and they know the land's appearance they are attacking.
--from a young Marine just back from Iraq

On the good side the laptop often provides a lot of good intel.

12:28 PM  

 Anonymous said...

"I could see that, if not actually disgruntled, he was far from being gruntled."
-- P. G. Wodehouse

I think Bertie Wooster was referring to D'Arcy CheeseWright :-)

12:47 AM  

 Purnell said...

That's so very true exspecially in America negative stories gains the most publicity and attention.

11:32 AM  

 doug in Seattle said...

Choices, decisions, consequences. If one consequence is a slam on /., I don't do it.

9:10 AM  

 r_wolfcastle said...

Whan a company is constantly seeking the creme' de la creme' of Engineers and Programmers - they have an obligation to be as concerned with the emotions and frustrations of this elite workforce - as much as they are concerned with perfecting the Search Engine Softwares.

'Tis true. And as someone who spent a combined 16+ years at Bell Labs and Adobe, both places where you couldn't swing a dead cat without hitting a PhD, I can tell you that managing herds of brilliant people is an art. Furthermore, although the best practitioners of that art are usually quite bright, they are not necessarily brilliant PhDs themselves. And that is something that I am not sure Google has yet been forced to grok, but they will have to eventually.

Google does have several distinct advantages in this arena, all of which Doug has pointed out. Among those are their egalitarian idea culture and their continual striving for excellence while doing no evil. All these things make even the worst prima donna geniuses happier, more productive, and much easier to manage. Google also seems to decidedly not be a place that hires the cream of the crop, burns them out before they are 30, and throws them away. That would be Oracle or, back in the day, Texas Instruments.

11:47 AM

 

 

SUNDAY, DECEMBER 04, 2005

Just do it!

POSTED BY RON AT 5:17 PM

 

Doug wrote:

 

The upside of this philosophy is that Google did many things quickly, most of which turned out to be positive. The downside is that Google sometimes did things with unforeseen effects and individuals occasionally misinterpreted exactly how much power they possessed. The tradeoff appears to have been worth it.

 

This reminds me of two anecdotes.

 

I didn't really appreciate it at the time, having never experienced anything different, but except for having to do it in Java and some basic functional requirements, I pretty much had absolute technical freedom to design and implement AdWords however I saw fit. I wasn't really able to take full advantage of this freedom with all the problems I was having, but no one ever second-guessed any of the decisions I made within the constraints that were given to me up front.

 

The second story is something that happened in my (one) meeting with Larry Page to discuss AdWords. The issue came up about whether or not ads should be screened before appearing on the site. There was almost universal consensus that they should. The fear was that neo-nazis or some such group would post an offensive ad and Google would get bad PR for publishing it. The lone dissenter in the room was Larry. He was adamant that he wanted to give users the instant gratification of seeing their ads live right away. And being the boss his view prevailed. But we all left the room thinking he was making a big mistake.

 

Turns out Larry was right and we were all wrong. The instant liveness turned out to be a major feature distinguishing AdWords from the competition, and the neo-nazis never came. Most organizations worry about the negative consequences of the things they do. Google, thanks in no small measure to Larry and Sergey's leadership, tends to worry more about missing out on the potential benefits of things they don't do. (Some of this approach was visible to the public for the first time during the IPO.) The net result is that they do more. Occasionally they hit a bump in the PR road (like during the IPO, or when GMail launched), but on balance, as Doug says, the tradeoff seems to have been worth it.

 

(There's an interesting book by the same person who wrote "The Tipping Point", which examines the benefits of not over-analyzing decisions. I'm about halfway through it. So far it's been a worthwhile read.)

 

 

22 COMMENTS:

 Noam Samuel said...

How analytical is the book?

6:01 PM  

 http://search-engines-web.com/ said...

Spontaneouty was indeed a likely factor in AdWords early success during its "Pay Per Impression" days.

What is not clear is..
"why did they NOT use some keyword filtering software technology - to decrease the probability that these potentially objectionable words would get online?

it would be interesting to elaborate on whose idea it was to ultimately CHANGE to the current "Pay Per Click" model :?

8:45 PM  

 Anonymous said...

I liked the book. Malcolm Gladwell came and gave a tech talk a few months ago and I was lucky enough to snatch a copy while he was there. Very inspirtional guy.

9:44 PM  

 Anonymous said...

Can any of you guys talk about the new compensation item called the "Google Unit" ?

Is it like a stock grant that vests over time?

Why did they start issuing those instead of regular options or grants?

12:22 AM  

 Robert Carnegie rja.carnegie@excite.com said...

Is this the same technology that decorated news stories about the Asian tsunami with Google ads for surfing equipment, until someone figured out how to stop that? Genuinely. I saw 'em.

3:29 AM  

 charlie_zoo said...

I just found your site Doug - really interesting stuff - and I wonder if any of you out there are UK-based Xooglers? Can you show yourselves? I'd be interested in having a conversation on a professional basis, either here or at Zoo Time.

5:14 AM  

 Matisse Enzer said...

I've often wondered how conciously Google set about creating the huge real-time auction system that drives adwords.

By now it must be one of largest auction systems in the world. Can you say something about the the various degrees of concious planning and accident that went into it?

7:50 AM  

 Technicator said...

Google's so different internally that it's probably why their stock is doing so well. I've covered that on my website and it's always so bullish. I give them credit. Is it true that Google has more pHDs hired than a typical clinic?

2:00 AM  

 Anonymous said...

why does everyone think that a PHD is a magic bullet to making infinite profits?

if so, half the faculties at universities around the country would be millionaires.

they arent.

all a PHD proves is that you can follow instructions.

1:24 PM  

 R2K said...

Isnt is a conflict of interest for you to host this on a google blog?

R2K..Bathrooms..Rocketry

5:01 PM  

 Ddot the King said...

Oh ok cool....good day.

6:11 PM  

 Anonymous said...

well..ok, had to comment. OK i am dumb founded on this sunject...what exactly is adwords? I am lost..ahumm..sorry, clue me in?

7:41 PM  

 Anonymous said...

lol..OOPS not sunject...,haha subject*

7:42 PM  

 Andrew Goodman said...

I got a lot out of Blink. Beyond your post, which is great to learn from: we get into debates with clients about how much to analyze campaign data, etc.

Although Gladwell probably doesn't have all that much first-hand knowledge, his journalistic account touches a chord: experienced professionals see the benefit of reaching decisions based on "quick analysis" or "thin slicing" rather than a full rundown of all relevant data. The more your job involves seeing the same complex data patterns every day, the more impatient you become with the comprehensive approach.

This is not the same as acting on instinct.

Anyway...

I am not entirely convinced this relates to Larry's conviction on the instant ads.

Our impression has always been that "instant on" was and continues to this day to be a wonderful differentiator for AdWords. Larry is a marketer par excellence (without knowing it?). People still assume that "instant on" is a big unique feature of AdWords, even if this isn't entirely true and isn't particularly important.

11:33 PM  

 David Towers said...

Doug,

This blog is a goldmine, thank you so much. I'm currently writing my dissertation on the organisational culture of Google (I'm an undergraduate in International business commerce). Your posts have been so helpful in helping me understanding things from inside Google.

Also, as I will be applying to work with Google in Paris it makes it even more interesting to hear about these things.

Please keep the posts coming, thanks again,

David Towers

8:46 AM  

 Anonymous said...

That "someone" would be Malcolm Gladwell, author of "The Tipping Point" and "Blink," and a frequent contributor to prestigious magazines. But that's a small aside. Although Google has prevailed with AdSense, I wish it wouldn't run ads for paper mills. It doesn't need the money and it encourages scummy behavior.

2:10 PM  

 Anonymous said...

Re: overanalyzing decisions, fair enough, but let's not forget that a lot of companies fell apart because they didn't analyze their decisions closely enough.

12:40 AM  

 doug in Seattle said...

Sounds to me like the whole premise of Blink is:

Go with your gut instinct

Or as is often dismissed "woman's intuition".

9:05 AM  

 Anonymous said...

No neo-nazis? The way I found out the ads weren't screened was when my search returned one for a *child porn* site. Naturally I contacted Google about it immediately. I didn't save my original email to them, but here's their response. The ad was removed, but reappeared a couple of times before they apparently decided to block it somehow...

9:41 AM  

 Anonymous said...

Offensive ads were definitely a real problem in the early days of Adwords. I understand the marketing reasons for instant approval, but, it let some really nasty stuff get through onto Google result pages. Always they would be "unapproved" (no one had reviewed them yet), but, not infrequently, unapproved ads would run for many hours and generate lots of impressions.

2:47 PM  

 Bob Aman said...

I'd have probably just used a bayesian filter to identify bad vs. good ads. If the filter let it through, it'd run until someone complained, at which point the filter could just be retrained with the new information. If it failed the filter, it would sit there and wait for approval or denial.

8:08 AM  

 R2K said...

Wow several clicks from this very old post. I wonder what happened?

5:12 AM

 

 

FRIDAY, DECEMBER 09, 2005

They say it's darkest before the dawn

POSTED BY RON AT 1:37 PM

 

The billing bug was the only major problem we ever had with AdWords. It actually astonished me how few major disasters there were given that the potential for them was so high. The Google system is mind-bogglingly complex and it grew up very fast, which is usually a recipe for building a huge collection of intractable bugs (look at all the problems Microsoft has with Windows and IE). But it didn't happen. In fact, other than the billing disaster, the worst problem I can recall that was visible to users was that they once pushed a version of the home page that caused it to appear without a logo. Other than that things seemed to go very smoothly. Before coming to Google I would have given long odds against that being possible for a development of this magnitude done in C++. I was beginning to have serious doubts about my professional worldview.

 

A lot of the credit goes to three of the most brilliant hackers I have ever known, Jeff and Sanjay who shared an office so I dubbed them the dynamic duo, and Craig Silverstein, director of technology (I'm making an exception to the xooglers policy of referring to people only by their first names because Craig has already been outed in the press). I didn't have many occasions to interact with them since I was working on things that weren't very technologically demanding, but on a few occasions I did go to ask their advice about one thing or another and they always blew me away with the sheer quantity of abstruse technical information that they had stashed away in their heads -- to say nothing of their ability to type fast. Craig especially was really something to see. He used this funky ergonomic keyboard and when he typed it sounded like one of those ratcheting noisemakers you spin over your head at new years. I once came to ask him a question that I thought would take me half an hour to figure out on my own and he replied, "Well, let's find out" and in less than a minute he whipped up a piece of code that answered my question. My jaw was on the floor.

 

With the billing problem fixed, AdWords settled into a nice steady growth pattern, and everyone involved got an engraved lava lamp. We hired some people to man the customer support lines and do ad screening, and I was left to relax a bit and build some internal tools for monitoring the system's performance. Google was a very open company in those days, and the results were visible for all to see, so anyone could pull up a web page on the internal server and see just how much money AdWords had made on any given day. It was very exciting to watch the numbers go up. And up. And up.

 

It was becoming clear that Google was on a serious roll, and one of the things that would soon have to be done was to internationalize AdWords so that it could serve non-English-speaking audiences. This presented a major headache. Because AdWords had been written in such a hurry, there was English text scattered all over the system. A lot of it was in HTML pages (actually JSP pages, which is like HTML with embedded Java code), and a lot of it (like error messages) was embedded in the Java code. Just finding all the English text would be a significant challenge.

 

Here again I hit a major frustration as a result of being forced to use Java and JSP. Just before I had come to Google I had written a bunch of infrastructure to write web servers in Lisp. One of the cool features of Lisp is that you can very easily change the way the language is parsed, so finding all the English text is easy. You can more or less have the system do it for you automatically. But in Java (or any other existing language for that matter) you can't do that. The way that the system processes programs is carved in stone and you can't change it. The best you can do is write another parser for your program and use that to do whatever automatic processing you want to do. But that's a lot of work. In Lisp it's literally five lines of code (maybe ten for a really industrial-strength version).

 

But, of course, that avenue was now closed to us, and because we had to deal not only with Java code but also with JSP, we pretty much had to do it manually. Fortunately for me I was able to weasel out of doing this incredibly tedious work, which was fortunate because if I had been forced to do it at that point in my life I very well might have thrown myself off the Golden Gate Bridge.

 

It was becoming very clear to me that I was not a good fit for the ads group. There was actually some very interesting work going on there. A fellow named Eric, who had hired on just a few weeks after I did, had come up with a brilliant idea for increasing clickthrough rates. It was so brilliant that it didn't even make me slap myself on the forehead and say, "Why didn't I think of that?" because I knew that I never could have thought of it. I didn't have the mathematical background. I was really beginning to have serious doubts about my ability to contribute anything meaningful in this company. But on the other hand I had already invested four months, and it was becoming clear that Google was not going to be a casualty of the dotcom crash any time soon. Besides, I had offered to resign from the ads group and my manager refused, so I figured that put the ball in her court to figure out what to do with me.

 

I was assigned a series of small projects and I pretty much screwed them all up. As this is not one of the proudest episodes in my life I hope you will forgive me if I don't go into too much detail (actually I think I've repressed a lot of the details). I can remember two incidents in particular: Jane (my manager) had asked for some statistics to be produced. She, unlike many managers, was actually quite a competent hacker in her own right (despite (or perhaps because of) the fact that her degree was in philosophy) and had taken a preliminary cut at the problem, but the results were not what she wanted. She handed the project over to me. It was just on the heels of a series of screwups on my part and I really wanted to redeem myself, so I threw myself into the work and got it done in an impressively short amount of time. Too short as it turned out. I sent her the results I had produced and didn't even notice until she pointed it out that they were identical to the (incorrect) results she had produced herself. I really wanted to crawl into a hole.

 

That was pretty much how November and December went. Anyone who watched me during those two months would have justifiably concluded that I was the world's biggest moron. I was beginning to believe it myself. I have no idea why Google didn't fire me. And given that they didn't fire me, I have no idea why they didn't try harder to help me find a place where I fit into the company. It still baffles me to this day. For a company that does so many things right, and in particular that takes such pains to make sure its technical employees are happy, why they chose to dump me in purgatory and leave me there is a great mystery.

 

UPDATE 12/11/2005: I'm putting this here rather than in the comments section because blogger won't let you put a <pre> tag in a comment so you can't format code properly.

 

Some anonymous coward wrote:

 

So it's trivial to write a piece of code that walks a program, replacing string constants with function calls that pull the correct string for the current language out of a database (for instance).

 

It's actually even easier than that. I'd do it with a read macro, something like this:

 

 

(defun my-string-reader (stream char)

  (let* ( (s (ccl::read-string stream char))

          (translations (rest (assoc s *translations* :test 'string-equal))) )

    (if translations

      `(or (getf ',translations *current-language*) ,s)

      s)))

 

(set-macro-character #\" #'my-string-reader)

 

 

Here's an example of how it would work:

 

 

(defvar *translations* '(("Hello" :spanish "Hola" :german "Guten tag")

                         ("Goodbye" :spanish "Adios" :german "Auf wiedersehen")))

(defvar *current-language* :english)

 

(defun test ()

  (print "Hello")

  (print "Goodbye")

  (print "This string should not be translated")

  (values))

 

 

? (test)

"Hello"

"Goodbye"

"This string should not be translated"

 

? (let ( (*current-language* :spanish) ) (test))

"Hola"

"Adios"

"This string should not be translated"

 

? (let ( (*current-language* :german) ) (test))

"Guten tag"

"Auf wiedersehen"

"This string should not be translated"

 

 

This is actual working code, and it took me about fifteen minutes to write.

 

 

19 COMMENTS:

 Derrick said...

Ron, I completely admire you for your ability to write about this. You are a better man than I to be able to share what sounds like a really wrenching time in this forum.

I think you've said that it turned out OK. I'm really cheering for you!

4:59 PM  

 Anonymous said...

Your honesty is an asset. Google's willingness to allow failure is their asset. Both of you will work out fine on this planet of dishonesty and required perfection.

5:32 PM  

 Anonymous said...

My guess is that they assumed you would figure out something useful to do...

7:18 PM  

 Anonymous said...

xxxx

10:07 PM  

 Anonymous said...

This sort of gybes with my experience as a recent interview candidate. They never really explained what the job was, although I knew roughly the role. In retrospect, I believe this is because they expect smart people will find and essentially define the job, and if the current one isn't working out, then its just a matter of time before self-adjustment occurs ;-)

I am also learning that there's a pattern of low-balling candidates on pay. I think there are several motivations for this:

1. The total benefits package has to be taken into consideration, and the benefits are substantial, when compared to other companies.

2. It's like heaven. Once you're in, the living is good. And if you're smart, you'll only excel and advance.

3. They can get away with it!

10:23 PM  

 Ron said...

My guess is that they assumed you would figure out something useful to do...

Maybe. I actually did try to take the initiative several times. Twice I was shot down (remember, I tried to resign from the ads group so that I could go somewhere else, but I wasn't allowed to quit), and the other times my proposals were ignored (maybe I should have pushed harder, but I was getting more than a little gunshy by then). This was not a systemic problem, because other people were getting their ideas off the ground (like Eric with his clickthrough enhancement idea). I don't know, maybe all my ideas just sucked, but then I'm back to wondering why they didn't just fire me.

12:15 AM  

 amit Dixit said...

First of all, that’s very honest of you...What I would like to suggest is that, you should try to learn more about yr self (your -+) rather than google in first place....Always remember to start from grass root level of a problem and use your common sense to look for its solution.....and when you put forward your idea in front of your boss/colleges make sure you give ample justification for it. Question your own ideas, in your mind before putting it forward to avoid criticism. Above all always present yourself in such a way that you are looking for solutions, rather than proving your intellect and try to involve every1....as I think you must be surrounded by the mastermind of their field…………………………..And Maaan you are da best ….Cheeer up NOW!!

3:53 AM  

 Brian Slesinsky said...

Could you say a little more about how you would do i18n in Lisp? If I understand this correctly, it sounds like some kind of structured search-and-replace, which some Java IDE's do have now. (But not every string needs to be internationalized, so it seems like there'd still be some cleanup.)

10:26 AM  

 Anonymous said...

Programs are data in Lisp; or, more clearly: the structures built by the parser are part of the language specification, and are quite simple and easy to understand besides. So it's trivial to write a piece of code that walks a program, replacing string constants with function calls that pull the correct string for the current language out of a database (for instance). In fact, with a Lisp macro, this can be made to happen transparently as part of compilation, so you never have to see those calls at all.

7:29 PM  

 Anonymous said...

So, what was the great idea to help with clickthroughs!?!

10:42 PM  

 Ron said...

This post has been removed by a blog administrator.

12:40 AM  

 Ron said...

So it's trivial to write a piece of code that walks a program, replacing string constants with function calls that pull the correct string for the current language out of a database (for instance).

I've posted a response to this in the main body of the post (along with an explanation of why I put it there).

12:52 AM  

 Anonymous said...

Re: Lisp read macro. Very cool, but what if "Hello" happens to be a command sent to an internal server via RPC? When you change languages, your RPC fails. It seems dangerous to mark ALL strings to be translated automatically. I think it'd be better to explicitly mark strings that are for "external" use to distinguish them from internal strings (logging, RPC, etc.).

8:11 AM  

 DIE said...

fascinating, the lisp references reminded me of the yahoo store articles. I wish more webbased companies would look at lisp as an alternative you but tend to be stonewalled with zealots. Sysadmins preach Perl, and managers tout Microsoft dotnet. 

I can work magic in any language but I'd could do so much more if given the right tool. 

Word to the wise!!
Math isn't much of a problem man, I was one of the ones that didn't sale the math books from college. When in doubt I use them as reference. Stats books are cheap. peace out and keep it real yo

10:50 AM  

 Matisse Enzer said...

OK, I need to get around to learning Lisp :-)

I sympathize very much with what you describe about being sort of "abandoned" an Google - I've had that happen to me, and for my particular personality, it is very tough. Very cool of you to post about it so thoughtfully.

2:09 PM  

 Ron said...

what if "Hello" happens to be a command sent to an internal server via RPC? 

Then I'd suggest calling that code without rebinding *current-language*.

4:52 PM  

 Radha said...

Hi Ron,
Sorry for taking off the veil of Jeff and Sanjay. I believe they are Jeffrey Dean and Sanjay Ghemawat. I read their papers on MapReduce and GFS.. Truly astounding..

One trend I noted is that a lot of early systems folks inside Google came from DEC(later Compaq , now HP labs) Western Reasearch lab in Palo Alto. Is it possible to shed some light on how Google(or Sergey/Larry) could have convinced these great minds to work for Google. Or was it Carly Fiorina that really drove them away??

9:05 AM  

 Joshua Daniel Franklin said...

That example looks a lot like how it's done in C with the poorly-named gettext.

10:06 AM  

 Anonymous said...

Python has some limited introspection abilities, but it would still be difficult to output "fixed" programs.

I am familiar with scheme, it was used in S&IoCP, great book, used for MIT freshmen to teach programming concepts. However, I've never run across a program where I thought "this is a perfect reason to use scheme!". So in reality, I've never used it. Which is to say, I've hardly ever needed to manipulate source code.

7:26 PM

 

FRIDAY, DECEMBER 09, 2005

Let's get a real database

POSTED BY RON AT 3:09 PM

 

This story is a bit of a tangent, but I think it's worth telling because there are some useful lessons to be learned.

 

AdWords was built using the MySQL database, which is open-source and therefore available for free. It is by now also nearly as full-featured as the best commercial databases, but back in 2000 this was not the case. MySQL was quite a capable system, but missing a few (what some would consider basic) features. These missing features were obviously not a show-stopper, as we managed to get AdWords to work without them, but in a few cases it did take some extra programming to work around one of these missing features. On the plus side, MySQL was fast and reliable and, as I have already noted, free.

 

After AdWords launched, Jane, the ads group manager, decided that now would be a good time to switch over to a "real" database. "Real" is one of those words that Doug ought to add to his list of words. It means "expensive". Many managers seem to have this idea that it is invariably true that you get what you pay for, and that therefore nothing that is available for free can possibly be any good. Using MySQL was acceptable as an expedient to get things up and running quickly and with a minimal of capital outlay, but now that things were settling down it was time to recognize that this was really, fundamentally, a mistake, and it should be fixed sooner rather than later.

 

The flip side of this philosophy is the one more commonly espoused by engineers, which is nicely summed up by the old aphorism, "If it ain't broke, don't fix it." Yes, MySQL was missing some features, but it wasn't broke(n). We had spirited debates in ads group meetings over what to do.

 

We finally decided to go with a commercial database (I won't say which one) over the objections of a number of engineers, including myself. To ease the transition it was decided to convert AdWords over to the new system first, and to do the main ads system later. It was a project on a par with the internationalzation effort in terms of the tedious work required to comb over nearly all of the AdWords code and change all of the database queries. (Databases are supposed to all be compatible with one another, but in reality they pretty much aren't.)

 

To make a long story short, it was an unmitigated disaster. The new system was slower than molasses in February. Some heroic optimization efforts eventually produced acceptable performance, but it was never as good as the old MySQL-based system had been. For a long time we were stuck with the worst of all possible worlds, with the two ads systems running on two different databases. It was still that way when I left Google in October of 2001, but I have heard through the grapevine that they eventually went back to MySQL. (Since then, MySQL has added many of the features that had been missing at the time.)

 

The moral of the story is that sometimes, and in particular with free software, you get more than what you pay for. There are a lot of companies out there paying dearly for commercial databases (and operating systems for that matter). As far as I'm concerned they might as well be flushing that money down the toilet. Actually, they might be better off. We certainly would have been.

 

As an aside, there is a raging debate in the hacker community about the overall economic merit of the open source model. (Making money producing free software is quite a challenge.) I am not taking sides in that debate here. All I am saying is that from the end user's point of view free software is often much better than the producers of commercial software would like people to think.

 

 

69 COMMENTS:

 James A. Warholic said...

I am actually somewhat surprised that there was a push by some at Google to pay for another program in which the original was working quite well to begin with. After all, I thought the main search servers were built with Open Source Software such as Python. Anyways, I have thoroughly enjoyed reading the posts back from the beginning. I look forward to the next segments.

5:36 PM  

 Jonathan Ellis said...

holy crap, you ran adwords on a transactionless database?

you have balls. and you're lucky as hell nothing (else) went wrong.

9:48 PM  

 Ron said...

holy crap, you ran adwords on a transactionless database?

Yep.

you have balls. and you're lucky as hell nothing (else) went wrong.

Nope. Luck had nothing to do with it. If you don't have transactions you just roll your own. It's actually not hard at all.

10:21 PM  

 Anonymous said...

Aren't there issues with the GPL? PostgreSQL for example offers a BSD license, which is why many commercial users prefer it to MySQL.

1:45 AM  

 Rob said...

This neatly encapsulates my employer's experiences (and I was one of the tech guys who encouraged the company to move to *cough* Oracle *cough*!). You can get an elephant to tapdance, but the results will never be graceful. Asking one to do so in your living room is twice a mistake.

6:39 AM  

 Anonymous said...

For the life of me I don't understand why more folks don't use Postgres. The BSD license seems to be the way to go. Similarly, I can't comprehend why Google uses a Linux kernel when the BSD kernel is completely free of any GPL b.s.

6:47 AM  

 Ron said...

I can't comprehend why Google uses a Linux kernel when the BSD kernel is completely free of any GPL b.s.

Google isn't distributing software based on GPL code, so the GPL "b.s." doesn't come into play.

9:06 AM  

 seo black &amp; white said...

Btw, MySQL supports transaction rather well since they added InnoDB storage engine. In the last three years it works for me like a charm...

9:23 AM  

 Steve said...

This story says a lot about the culture at Google--especially in those early days. While LAMP (er non commercial platform based) architectures are today given the respect they deserve--it was a much more aggressive thing to do back then (and still be taken seriously). I recently left a moldy old company doing bank automation systems which collapsed under the weight of WebSphere, DB/2 and Oracle. Unfortunately it takes an engineering culture to cut through the hype and distill the issues to the essence of a particular tool or system's purpose--then to select the right tool for the job. Google's culture embraces this and is a fundamental reason for their success.

10:18 AM  

 Anonymous said...

In 1999 I gave a presentation to the bankers taking MP3.com public (July 1999, $370m offering). We used MySQL (wish I'd used Postgres and today, that's all I use) and the bankers were agast when I told them we did not use Oracle. Today your bankers would be agast if you did use Oracle in your startup. MP3.com also funded Hans Reiser to add journaling to Reiserfs so as to reduce dependencies on VxFS. An estimated savings of $8m over five years. Open source definately has its place and you've got to give back to the community to keep it fresh and moving forward.

6:36 PM  

 Anonymous said...

People who don't like the GPL must really like the reach around Microsoft and Apple give you when they are fucking you for your code.

6:51 PM  

 Anonymous said...

A former employer of mine got the "let's get a real language" fever. They decided to ditch Python for Java because of a whole bunch of pointy-haired boss reasons. The mentality seems the same as that of the "real database" story here...why does management so often disregard the opinions of the people that have actually been in the trenches doing the work? Why do they decide to throw away or overhaul working systems because of some marketing gobbledygook? 

This is an apect of business I really don't understand. Even when the decision precipatates horrible consequences (unbearably slow systems, unmanageable hulking codebases), some folks just can't recognize that they made a mistake by not listening to the engineers.

7:02 PM  

 Jonathan Ellis said...

"If you don't have transactions you just roll your own. It's actually not hard at all."

You're scaring me.

7:05 PM  

 Jonathan Ellis said...

To elaborate: some transaction benefits can be simulated "easily," such as logging each update to make sure Bad Things don't happen when you, say, charge a customer and the db goes down before the ad gets inserted. Although if single updates aren't even atomic -- and if your db doesn't support transactions, they probably aren't, which is the case for MySQL's in non-innodb tables -- you're still in trouble if you're running something like "update users set balance = balance - delta" when it goes down.

The harder ones, though, such as isolation, I've never seen done outside the database, and I wouldn't trust the code of anyone who claimed to do it. Much safer to use a database that's already debugged their code many times over.

7:14 PM  

 Mike Hillyer said...

Jonathan Ellis said...

holy crap, you ran adwords on a transactionless database?

you have balls. and you're lucky as hell nothing (else) went wrong.

Actually, MySQL has had ACID transactions for many years now, including savepoints and four levels of transaction isolation. Best get current before criticizing.

7:24 PM  

 Anonymous said...

Erm, the article is about yr 2000. MySQL got transactions about 2001 if i'm correct, but don't really count on that :) However, we can presume that MySQL had no transactions back then :D

7:47 PM  

 Anonymous said...

Single MySQL updates have been atomic for a looong time (1999 at worst), maybe always. With those you can roll your own transactions.

It's funny when people trot out disaster scenarios that derive from premises that are incorrect. "Well, yeah, if not for Gravity, things might be different like you said..."

7:48 PM  

 Jonathan Ellis said...

MySQL does claim that updates are atomic even on MyISAM tables, but I've seen many people report that this is only true if nothing goes wrong, i.e., power loss, simple crashing, etc. Two such people have commented on the relevant doc page: http://dev.mysql.com/doc/refman/5.0/en/ansi-diff-transactions.html

I'm no expert on MySQL source but I've seen enough non-idiots report on MySQL corruption that I find toeing-the-party-line claims to the contrary difficult to believe.

In any case, my main point stands: atomic updates alone aren't going to help you improvise isolation support, etc., if your db doesn't have a concept of transactions.

8:22 PM  

 Hank said...

Actually Ron, "rolling your own" is not a smart way to go. Trying to emulate transactions in your app means that you don't actually have the guarentee of a transaction, since your app and the DB are seperate, and one or the other can go down in the middle of a pseudo-transaction, leaving you with inconsistant data.

Using such a crippled and broken database is even more crazy when you consider that it corrupts data (intentionally) and that there was a free alternative without these problems, and that handles many concurrent queries better.

And anon, they can't use BSD, they're from Stanford.

8:33 PM  

 Anonymous said...

MySQL is rock solid.

10:16 PM  

 Online Degree said...

We have been using MySQL on our site (www.degreeme.com) since its inception and have never had a single problem. Impressed? Just kidding... I know that isn't at all impressive :) But I am still a huge supporter of MySQL although I also have heard of instances where MySQL has had corruption problems. Hopefully the new version 5 has fixed that.

11:35 PM  

 Anonymous said...

i love this story... you do get what you pay for... of course how you pay is relative... because some of the best things in life are free... fun stuff

11:38 PM  

 Anonymous said...

I'd dare to guess that the reason they had such a big problem porting over to some other database was that their code base was full of "hand rolled transactions", including a huge number of calls to "lock table".

If they had gome with a database with transaction support from the get go, like PostgreSQL, the porting work would have been much simpler.

The fact that MySQL makes you code a lot of non-standard stuff which makes it difficult to port the code over to other databases is being presented as a good thing in this article. If the author had opened his eyes a bit he'd learn to use something else from the get go the next time.

And yes, PostgreSQL is a "real database", and it's not expensive, it's free.

11:53 PM  

 Ron said...

Actually Ron, "rolling your own" is not a smart way to go.

What can I say? It worked for us. Worked for this guy too.

12:33 AM  

 Anonymous said...

As some previous people said, I would be surprised if the problems porting to the 'other' database from MySQL where really any different from the problems developers generally get when porting from database A to B. Without a really good understanding of how things work in a given DBMS, especially locking and transaction logging, it is very difficult to get something to work as well when ported. I would wager than even the high calibre googleplex folk would find it a challenge to get that experience quickly. Tom Kyte's Oracle books contain numerous stories which illustrate this point from his experiences of going from Sybase to SQL Server to Oracle.

3:08 AM  

 sf said...

It's not "free" vs "proprietary". It's some features against other features and tradeoffs against other tradeoffs. MySQL has the single unique feature of being a speed demon amongst the databases. The tradeoffs are quite dire in the domain of functionality, and need to use specific MySQLisms instead of the "high way" to have things running fast, not slower than all the rest. Other databases are more feature-packed, but they are simply slower. Sometimes WAY slower. The change of the database should have considered this.

3:48 AM  

 Samuel Cochran said...

It is quite possible to make money from Open Source software. Rather than charging for software, one must charge for services. Installation and configuration of software, customisation of software and solutions, etc. can all be marketed rather than software. This is my approach.

By doing this one can contribute any changes and improvments to the Open Source community whilst providing solutions for companies, and make a living, too. :)

5:38 AM  

 Anonymous said...

I went through similar issues at a biotech startup. I was the only programmer there for the the first 1.5 years and had built everything with a postgres back end. I caught a lot of heat resisting managements wish to convert to Oracle, but persist I did, because there was no technical justification for the effort. I left for a much better opportunity six months later and my successor apparently got the job by promising the conversion. After 1.5 man-years, they still did not have the system working as well as it had been. Eventually the company was sold for pennies on the dollar.

8:09 AM  

 Anonymous said...

You have a team of engineers that are resisting the move to another database system pushed by management.
These engineers are all happy with their current database, which is probably the one they use any chance they get, so they are really in tune with how it works.
The new database has to be used and tuned in a completely different way, requiring its own years of experience to master.
Of course the second solution using the new database is going to be worse, how could it not? It says nothing about the database products tbemselves.

9:01 AM  

 Anonymous said...

postgres advocates are delusional -- millions of mysql users have no time for crashgreslow.

Back in 2000, postgres didn't support the LIMIT keyword. I got hoodwinked by the FUD they spread about MySQL and tried using it for three projects: each time I got crashes and data corruption with small data sets. You could find very few sites that were using Postgres on the web, and the operators of many of those sites regretted that choice.

Somewhere around postgres 7.4, postgres became stable enough that you actually ~could~ use it in a production system. Still, it's pretty common for people to try a few queries on Postgres and mysql, notice that some very common queries run 60 times faster on mysql, and come to an obvious conclusion.

People who push the "no transactions" FUD also forget that transaction support often reduces the reliability of applications -- yes, it's the stupidity of the application programmers to blame, but an organization I've worked for spent $250,000 on some commercial junkware which doesn't do correct error handling with transactions, and this leads to strange, glitchy, sporadic problems... And these guys aren't using an "off brand database", but they're using Oracle.

The truth is that it's not the end of the world if you mess up a row or two in most databases: if you consider all the other ways an e-commerce site can screw up because of user error, human error in fulfillment, problems in shipping, other programming errors, losing a row because of the lack of transaction support is a minor problem... As the CEO of one company told me, "if we mess up one order in six months, we can just let the customer have it for free" -- that's a lot cheaper than the cost of a DBA and the extra hardware to run Oracle, never mind the cost to license Oracle.

9:22 AM  

 Anonymous said...

What I don't understand is why modern huge database applications are written using the database engines that run on top of some universal OS? I mean, what else can be on that server except the database? So the ideal modern database should be the OS, possibly even without the file system as we know it. He hard-drive itself can be structured to suit the database needs so one would never have to rewrite any files, just the segment of the disk that was assigned to the particular data. The improvement is speed and reliability would be enormous.

9:26 AM  

 Lauri said...

Ron, how much need did you have for transactions when working AdSense anyway? 

It looks to me like it is mostly INSERT operations and lots of them. They get written to separate servers based on the user hash and the balance changes get calculated at X intervals offline. 

Am I close? Or did you have some other setup.

10:42 AM  

 Anonymous said...

I'm not a developer, i'm a financier so please excuse my ignorance. I found this post quite interesting -- where would you suggest I go to read the best open source debates? I'm interested in what the technology community pitches as the pro's and con's. Thanks.

4:58 PM  

 Ron said...

Ron, how much need did you have for transactions when working AdSense anyway? 

Well, that depends on what you mean. In some sense we had no need for them at all, obviously, because we built the system without them. If you want to know how many times we had to write extra code that we wouldn't have had to write had we had transactions, it's hard for me to recall exactly (it's been five years) but it wasn't very much. It certainly was not a major issue. And, like I said, in those situations where you need real transactions (fairly rare in most real world applications I believe), emulating them yourself isn't particularly hard.

Of course, it's all a moot point now that MySQL has transactions.

5:11 PM  

 Anonymous said...

The MySQL doc page says an update might not update ALL of the rows if the thread is killed during the update. That still does not stop single-row updates from working, as mentioned. Furthermore, if you are using your own transactions, you are writing some extra code to make sure multi-row updates work as well.

6:15 PM  

 Matisse Enzer said...

One way of setting up MySQL these days is to do all the INSERT/UPDATE/DELETE to an InnoDb table and then to have a pool of "query slaves" where the table is MyISAM.

The query-slaves are behind a load-balancer and are used by applications that need to do SELECTs, and this way they never have to wait for the locks that the INSERT/UPDATE/DELETE create.

The Master can be a MyISAM table if transaction-safety isn't as important as write-speed - INSERT into a MyISAM table is very very fast.

10:32 PM  

 Sugree Phatanapherom said...

Did they develop any lock server to simulate transaction in global scale?

1:53 AM  

 Anonymous said...

All I am saying is that from the end user's point of view free software is often much better than the producers of commercial software would like people to think.

"End user" being the developer or the manager who actually gets to pick and pay the product?

2:55 AM  

 Anonymous said...

People prefer MySQL over PostgreSQL for the same reason people prefer GNOME over KDE: Because the developers are arrogant bastards, and the supporters act like dicks.

5:16 AM  

 Jayson Vantuyl said...

In all seriousness, I don't see what the GPL vs BSD comments are about.

Google developed an app on MySQL on Linux. The GPL didn't come into play unless they tweaked and then distributed a modified Linux kernel. Even then, they CAN STILL LEGALLY SELL IT!

The only thing that someone else can't do is improve the source without providing a little quid pro quo. This is not unreasonable, it's a LICENSE choice. For the small developer, it's a damn sensible one as well.

That said, I prefer Postgres for reasons absolutely unassociated with the license. I feel it implements much "purer" SQL than MySQL. I like its administration better. I like how it plans queries better.

As for rolling your own transactions, you should have covered this if you took a CS degree. Between lockless techniques and a simple test_and_set routine, rolling your own locks (and thus everything Atomicity, Concurrency, and Isolation) is really childs play. As for Durability, a log table (or fsync'd file even) is really not that difficult either.

7:15 AM  

 Anonymous said...

Ron, I don't see anything at all about "that guy" trying to create safe transactions on top of year 2000 mysql. He says very clearly that he didn't use a database at all, he used files. Because he not using tables and rows accessed through SQL, he can ensure that his updates are completely atomic. You can't do that with year 2000 mysql. There is a very big difference.

And Jayson, I realize its easy to say you can make safe transactions, but its not easy to do. The only way to make it safe is to rely on a log, and if a query fails, take the database down and restore it from a known good point, then replay the log. This is not transactions, its recovery. There is simply no way with year 2000 mysql to run a bunch of queries in a single atomic transaction and guarentee that they either all succeed, or they all do nothing.

10:56 AM  

 Arqueaopterix said...

They were right in that MySQL is a toy database. It only got popularity because Slashdot uses it.

However, unless you're using Oracle, PostgreSQL is better than your database.

12:07 PM  

 Ron said...

There is simply no way with year 2000 mysql to run a bunch of queries in a single atomic transaction and guarentee that they either all succeed, or they all do nothing.

That is true, but irrelevant. What is important is not to guarantee that they do nothing, but merely to guarantee that whatever they do they do not affect the results of subseqent queries unless they all succeed. That is possible, even with y2k MySQL. Figuring out how is left as an exercise.

11:17 PM  

 Rob said...

What I don't understand is why modern huge database applications are written using the database engines that run on top of some universal OS? I mean, what else can be on that server except the database? So the ideal modern database should be the OS, possibly even without the file system as we know it. He hard-drive itself can be structured to suit the database needs so one would never have to rewrite any files, just the segment of the disk that was assigned to the particular data. The improvement is speed and reliability would be enormous.

This exists presently. It is called Microsoft SQL Server (presumably, as usual for MS products, its hooks dig deep beneath the OS into the kernel). It is of necessity a security nightmare, as are all MS products.

You have a team of engineers that are resisting the move to another database system pushed by management.
These engineers are all happy with their current database, which is probably the one they use any chance they get, so they are really in tune with how it works.
The new database has to be used and tuned in a completely different way, requiring its own years of experience to master.
Of course the second solution using the new database is going to be worse, how could it not? It says nothing about the database products tbemselves.

Certainly there are elements of this, but then, if the DB requires an army of expensive high priests to keep it running smoothly, why bother? The design flaw of failing to self-tune becomes a business feature in that the customer now has to sign up for hours and hours of expensive consultant hours. I don't see this as a positive.

6:31 PM  

 r_wolfcastle said...

I have worked at 2 companies that rolled their own transaction logic in the application and used MySQL MyISAM tables -- one starting in 1999 and the other starting in 2001. Anyone remember AvantGo? That entire thing was a single MySQL instance with a load balancer and multiple query servers.

At the second job, the one starting in 2001, as jack-of-all-trades I got put in charge of database stuff. We wrote our own layer between the application logic and the database for ease of porting. Our app was essentially 100% database-centric, but we were able to port from MySQL to MS SQL Server in a couple of weeks because of the way we had written the layer. It turned out that none of our customers wanted MS SQL Server (actually, they didn't care what we used as long as it worked and it was cheap), so we let support for that lapse and just stuck with MySQL.

If you are selling a product that is substantially smaller in scope than, say, a comprehensive SAP solution or e-commerce infrastructure for eBay or Amazon, you cannot use Oracle for economic reasons. You can't use the customer's existing Oracle installation for your app, because that installation is already highly-tuned by expensive folks for SAP or CRM or whatever. They won't let you and your app and its processing and bandwidth requirements anywhere near it. So to use Oracle the customer has to have an Oracle instance just for your app, and it is pretty damned difficult to sell a customer an $80,000 product when you have to say, "Oh, and by the way, you'll need to fork over about $30,000 for Oracle and hire another admin to tune and babysit it." That is where open source databases like MySQL really shine.

Finally, someone said that INSERT into MyISAM tables is very fast, and that is true. But there are special circumstances (and they're not that rare) in which you can go even faster using LOAD DATA INFILE. MySQL claims is "up to 20 times faster" (last time I looked) than using MyISAM INSERTs, but which in practice is actually 5-8 times faster in apps I've measured.

3:18 PM  

 Anonymous said...

This has just got to be a wind-up, right?

BTW Oracle bought up InnoDB a while back. You stick to your, um, MySQL disaster-in-waiting. I'll stick with PostGres, or Firebird rather than a toy.

6:55 AM  

 Tyler said...

Just wanted to chime in and comment on some of the "Oracle is expensive" comments. There are really 4 versions of Oracle to consider:
- Enterprise Edition = $1x
- Standard Edition = $.38x
- Standard Edition One = $.13x
- Express Edition = FREE! (just released this month)

The .38 and .13 numbers are from the current price list.

I use x, since government pricing is typically MUCH lower than what you see on our price list.

Most people have seen the price for EE, but don't know about the other versions. MS has a VERY similar pricing structure.

Full Disclosure: I am an Oracle employee.

6:05 AM  

 Anonymous said...

What a sad commentary about how little so many in the IT industry know about data.

I've read that mysql was free, but commercial databases are expensive. Then the author admits that they had to write their own transaction logic. Where did this free labor come from?

I've read that "it's not the end of the world if you mess up a row or two in most databases". The issue is that the results are *unpredictable*. You don't know what will break when you violate your contract between components. 

I've read that mysql is a speed demon. Ah, right. With no parallelism, no partitioning capability, inability to properly optimize queries with 4 joins, no memory tuning. Please. The only queries that it runs fast fall into a very small niche of read-only retrievels of highly selective data (<1%) via btree index. That's good - but only covers a subset of possibilites that you should be looking at. 

I've read that mysql has all the features you really need. Back in 2000 it didn't support transactions, pk/fk constraints, views, unions, subselects, triggers, stored procedures, etc. And it still doesn't support query parallelism or partitioning. If you're writing database apps without any of these features you might as well stick to MS Access.

I've read that they wrote their own transaction logic (since mysql didn't support transactions well) and it worked fine. Really? How do you really know? How did you empiracly measure data accuracy? Hmmm, you didn't? Really, I wonder what kinds of problems (loss of accuracy of adsense) were caused by data quality problems. Note: I've never encountered a database that didn't have data quality problems, the lack of constraints and transactions almost guarantees their widespread existance - whether or not it was shoved in the face of a programmer.

I've read that conversion from mysql to the other database was tough. As though that was caused by the other database. FYI, mysql is the least ANSI-SQL compatible database in the market (especially so in 2000). Having to roll your own transactions, and getting by with mysql deliberately converting invalid data arbitrarily to work and then going to a standard solution would of course, be a nightmare.

Undeniably, oracle was (and still is) overpriced. Though it isn't tough to get discounts if you know what you're doing. DB2 and Informix are both cheaper and easier to work with, though they weren't VC darlings in 2000. And undeniably, you can write working software with mysql. But there are compatibility problems, data quality problems, and scalability problems. Maybe it's unfair to assume that this is obvious to junior programmers unfamiliar with database technology. Especially given the current hype around mysql.

9:35 PM  

 Anonymous said...

I've been using MySQL since 2000 to implement a community website (bicycle touring journals, now about 800 journals and over 60,000 pictures). The site gets somewhere in the region of 20-50,000 page requests (150,000 - 300,000 http reqs) per day. So it's not exactly a super busy site, but it does have quite a few active users. I've had the site up and running 24/7 (with some breaks for moves and hardware failure) for about six years now. Not once have I had any integrity or dataloss problems caused by MySQL. It's been really rock solid. All these people who say that using MySQL for anything significant is a "disaster in waiting" are simply blindly repeating religious dogma. That's what it is - religious. They believe that MySQL is crap, so no matter what other people demonstrate to the contrary (that it, uh, actually works, rather well in fact) they will still continue to twist reality around to suit their existing viewpoint. I don't really understand the outright hatred that exists for MySQL in the minds of some people. It's been working for me for six years now... how much evidence do you need before you accept that it really does work well? And as for the data integrity issues, lack of transactions etc... all I can say is this: Despite all the wailing and gnashing of teeth, the fact is that fixing data problems is really not all that hard. It's very occasional, and actually I haven't even seen any problems that were caused by MySQL itself - the biggest problems were caused by user error (mine, generally). This involves stuff like dropping the wrong table or field, or mistakenly deleting entries wholesale. Sorry, but arguing that "the database should stop you from making that kind of mistake" isn't valid at all. Any database that lets you change data will have the potential for user error, and that's the biggest source of data loss in my experience. Only on a couple of occasions has MySQL itself had a problem - and then it was just a corrupted index file, which was easily fixed. I'll take that kind of rock solid stability any day. Sure, MySQL could be better - I have wished for a while that it was able to utilize more than one index in a query. I think that is starting to happen. I'll be using MySQL for a while yet, I think.

9:23 AM  

 Neil Gunton said...

Sorry, for some reason the post preview system reverted my identity on the previous msg to "anonymous" even though I had previously selected "other" along with my home page. Anyway, the community website I was referring to is crazyguyonabike.

9:26 AM  

 b7j0c said...

This post has been removed by a blog administrator.

10:11 PM  

 Anonymous said...

Rob said:
What I don't understand is why modern huge database applications are written using the database engines that run on top of some universal OS? I mean, what else can be on that server except the database? So the ideal modern database should be the OS, possibly even without the file system as we know it. He hard-drive itself can be structured to suit the database needs so one would never have to rewrite any files, just the segment of the disk that was assigned to the particular data. The improvement is speed and reliability would be enormous.

The ability to create database "files" on raw disk slices/partitions has existed in Oracle for as long as I can remember (at least 13 years). This was a highly recommended way of installing Oracle some years back but modern Oracle documentation now downplays this method in favour of cooked file systems.

7:00 AM  

 Aminorex said...

I've been running dozens of websites offering commercial services, all backed by mysql, since 2001. I can attest that it is good enough and fast enough and plenty plenty cheap enough for everything I've ever needed in a database system. It scales on demand, with no incremental cost. Unless you have a very specialized need, anything else should draw the intense ire of your CFO, and maybe a shareholder lawsuit.

4:46 PM  

 Anonymous said...

Someone asked where the hatred of MySQL comes from? It started with the... I want to call them lies... that they continually repeated in their presentations, community websites, and documentation; like how transactions are not important, that ACID doesn't really matter, and that being strict about data integrity and features like foreign keys and views were only things that "purists" cared about. And the hatred grew as people watched them implement all these things over the last 5 years. 

Sure you can make MySQL work, you can make GOTO work as well, but that doesn't change that 90%+ of MySQL users would be better served by choosing a different database.

12:35 PM  

 Paul Reilly said...

At the end of the day, MySQL WAS adopted by Google. OK, so additional transactional development was required but it worked, and more importantly, it worked more efficiently that the commercial counterpart! I see this attitude towards the adoption of commercial software Vs. the adoption of Open Source software. 

Here within the corporate culture, where I work, I often see the same pro-commercial attitude. The key point here is that by adopting the commercial alternative, it provides a scape goat surely every corporate top director who answers to shareholders, surely needs to protect himself by having someone to sue in the event of a mission critical failure, a fine motivation to adopt a commercial database.

8:26 AM  

 Anonymous said...

A few thoughts about recent posts:

1. Why the hatred towards mysql? I don't think there is hatred, I think a few issues have really raised people's hackles:
- hype vs reality
- history of company for spreading worst-practices (don't use transactions or foreign keys, etc)
As proof of the above - note that you almost never see so many criticisms of Postgresql - a database that is both freer and more standard than mysql.

2. It works great on my hobby website, it should be good enough for everyone: Ah, no. Go do a dozen business critical database projects - and for god's sake, use a few other products besides mysql to get a perspective. *Then* you're in a much better position to talk about mysql vs the rest of the options. In the meanwhile, you really have zero credibility on the subject of databases.

3. I've used it for years on my site and have never seen any data quality problems, therefore that just isn't a big deal: Really? How do you know you don't have data quality problems? Are you explicitly testing for them? Almost nobody does - they take it for granted that things are working right. The fact that they are only looking at a report with 95% of the data instead of 100% is quite difficult for them to know!

4. mysql is good enough for everyone, it's just the out of touch purists that don't like it: the people who say this are typically those with very little experience with databases, perhaps only with mysql. And they are complaining about the people with 10+ years of experience across a half-dozen products. These aren't purists - they're just experienced professionals.

5. I've been running dozens of mysql websites offering commercial services. Anything else is just a specialized application: if you're just supporting e-commerce packages for smallish companies at your isp you've touched on about 1% of what people use databases for. You haven't yet handled large volumes, 100% failover environments, reporting, complex queries, etc. It's great if it worked for you, but you really probably wouldn't notice the data corruption caused by silent data truncation/conversion, orphaned data allowed by lack of fk constraints, etc. 

Sigh. The proponents of mysql really need to broaden their perspective by trying out a few other products and developing a few applications that have higher standards (not just performance, but also reliability & quality). Additional recommendation: work with people who have been doing this for a while and who can show you the many ways that data quality problems occur, how they can have unpredictable impacts on systems, and how to best prevent them.

10:36 AM  

 Anonymous said...

Sigh. The ignorance and lack of objectivity from commercial hucksters is disheartening. The commercial bigotry persists. The truth is many of those closed minds have their careers tied to a commercial product and have little or no knowledge of alternative products or solutions. The folk pushing commercial products need to ask themselves what features of a commercial product are 'really' needed in the majority of real world applications and STOP preaching vendor product marketing jargon and inventing FUD of open source.

5:53 PM  

 Neil Gunton said...

I thought I'd come back here to see what other people had posted since my prior comment. And, of course, there it is... the typical attitude of the PostgreSQL pusher, which can roughly be summarized as this: "You don't know what a database is, even if you think you've been using it for years to do non-trivial tasks, you don't really know what a real database is. There are certain things that any database MUST do, and if it doesn't do them then it is NOT a database and you should run along now and run your little hobby website. Leave the REAL database stuff to the professionals who know what they're doing". Wow! Talk about arrogance, not to mention obnoxious. It really shouldn't surprise anybody if I actually avoid PostgreSQL because of people like this. Also, I do hear from the odd person that actually tried shifting over to something like PostgreSQL that it was *very* slow. That little fact gets conveniently swept under the carpet by the fundamentalists. If you look at any discussion involving MySQL vs PostgreSQL over on slashdot (and it seems that any discussion that has to do with MySQL will bring out the PostgreSQL fanboys), then you'll see comments tucked away in the oblivion of "unmoderated hell" which talk about how difficult or slow PostgreSQL is in reality. I have learned to take the Open Source zealots with a huge pinch of salt these days. What you tend to see are comments that just keep repeating the same thing over and over - PostgreSQL is the only thing anybody should be using (even though it's actually still really slow)... or Mozilla/Firefox is the best thing since sliced bread (even though it was basically unusable on my old 450 MHz workstation, it was so slow, and moreover has had major bugs unfixed in the rendering engine for literally years)... seems like the whole Open Source crowd has a tendency toward the religious in terms of closed thinking. Something to do with a mixture of "us vs them" tribal attitude, with an added dose of dogma and hatred of anything that threatens their cosy little world view. You'll notice that these people won't acknowledge that MySQL is pretty good for many applications (the majority of database applications, actually). In fact, many of them won't even acknowledge that MySQL deserves to be called an RDBMS. Any post praising or supporting MySQL is rebuffed with a link to a "Gotchas" page listing irrelevant or out of date bugs or anomalies. Any post defending MySQL is picked apart, point by point - you can tell a zealot when they think you are *totally* wrong on *every* point, each of which has to be individually addressed, simply to show you just how *wrong* you are. No middle ground, no no no. Just "mysql totally sucks, PostgreSQL is reasonable and sane, anybody who says otherwise is delusional". Whatever. I'll just keep developing my "hobby site" quite happily, scaling it up as needed using MySQL. It works, thanks MySQL! Oh, and for the record: I'm sure PostgreSQL works just fine too. I'm sure you can tune it just so, so that it is ok speedwise. But a note for the PG fanboys: You don't win people over with overbearing, arrogant posts that say anybody who thinks that you can do quite a lot with MySQL is somehow less professional or just a hobbyist. I'm sorry to go on, but this kinda gets on my nerves after a while. It's like swatting annoying flies - every time someone mentions MySQL, these fanatics show up claiming that it can't possibly be used for anything. To which I reply: GET A LIFE MATE!
Thanks.

10:41 PM  

 GoodBytes said...

I am coming a little late in the debate...

I wanted to congratulate you on this wonderful article and two excellent points:

"with free software, you get more than what you pay for", and

"from the end user's point of view free software is often much better than the producers of commercial software would like people to think".

I have used both MySQL and PostgreSQL, and have good opinion about them both. They have their good and bad sides, but you get to learn them in time and find workarounds. What's best, they are free and they are constantly improving with new versions.

Also, it is good to mention that not everyone needs a database for heavy-lifting applications. For simpler tasks, having one of these databases is priceless.

Thanks.

3:28 PM  

 Anonymous said...

I think someone made a comment a bit like that one. I've used both MySQL and Oracle and I must say that i'd rather work with an Oracle db that with MySQL. It has many advantages like having the "data integrity logic" directly in the database. This is priceless when two or more apps are accessing/writing to the database.

But, i would never use Oracle for an high number of concurrent sessions. By high number I mean as low as 1000. IMHO, Oracle for a complex and/or high data volume database and where transactions matters; MySQL for a simple but heavely queried database. Like the frontend-backend thing.

I still believe that you can achieve the same performance with Oracle or with MySQL. But with Oracle you need the knowledge and the resources to do so. And why would you need to put your hand in your pocket, MySQL is there to do just that (the performance thing). But I fear that while on the road of adding more and more "basic" features, MySQL will release a light version of itself.

js

8:22 PM  

 uri said...

Just wondering, when you talk about the "AdWords program" is this the one external customers work with, or something used for internal processing of ads etc.?

Thanks for the insightful story :)

6:06 AM  

 Andrei Lopatenko said...

Hard to believe.
I certainly agree that MySQL may outperform untuned Oracle (sorry, I talk ONLY about database I am truly experienced with fort many years) for transaction-free database under 10 million records. I believe that AdWords, AdSense needs much large database to operate.
Tuning of Oracle to real environment which depends on both hardware and software - RA memory, HDD, network, operating system, RAD is hard and time consuming but results maybe surpising. Performanse may increase scores times comparing to "default" untuned configuration. Proper programming like more functionality to PL/SQL then to Python may improve functionality up to 10 times in data intensive applications even for plain select statements without any database update functionality.
I do understand that select processing is 99.9999% of time of functionality required from Google AdSense database, but remaining 0.0001% is not less important for such applications.
Besides performance issues -- what is about fault recovery which includes transaction loging, backup and restore. If hardware and software fails, could Google restore it to the point just before fail or some users, customers, will loss their money, settngs etc
What is about updates? MySQL is pretty efficient for SELECT, but not for updates especially for updates of indexed tables. How many update transactions (new AdWord customers? users? settings changes? clicks?) per day ar eperformed? Can MySQL handle them? If it can handle them will it handle in one year? Are updates done in parallel?

Again, let mt emphasize a point about perfortmance. It is hard to believe that good Oracle engineer can not tune Oracle to be faster then MySQL in case if you have a database more then 10 million tuples.

I am not again Open source and I do not sell Oracle Database :). I like Open Source, but my around 12-year experience proves that for large databases of mission critical data Open Source databases can not compete with Oracle.

I do not say anything about MS SQL or IBM DB2 for only the reason that I do not know how to tune them or how efficient they are for large databases.

12:04 PM  

 Andrei Lopatenko said...

But, i would never use Oracle for an high number of concurrent sessions. By high number I mean as low as 1000.

What do you mean "concurrent sessions"?
User connections? You definitly do not need to use 1000 connection to run applications like AdSense
1000 parallel. My experience - Oracle is pretty efficient for parallel queries, especially if you try to apply cluster architecturw

about PostgresQL. It is reaLLy slow

12:08 PM  

 Anonymous said...

"To elaborate: some transaction benefits can be simulated "easily," such as logging each update to make sure Bad Things don't happen when you, say, charge a customer and the db goes down before the ad gets inserted. Although if single updates aren't even atomic -- and if your db doesn't support transactions, they probably aren't, which is the case for MySQL's in non-innodb tables -- you're still in trouble if you're running something like "update users set balance = balance - delta" when it goes down.

The harder ones, though, such as isolation, I've never seen done outside the database, and I wouldn't trust the code of anyone who claimed to do it. Much safer to use a database that's already debugged their code many times over."

....

It's all code that makes it all work in the database. If the base goes out at the wrong time, no matter how many hundreds of safeguards happen, you're going to have a loss of data. All this advanced junk to prevent it is pretty meaningless, because all it does is move the point where a failure will cause a loss to a different spot in the program.

9:32 PM  

 Administrator said...

Thanks for your blog...
http://best-casino-online.t35.com
Best Online Casino

11:15 AM  

 Anonymous said...

Hmm,

After reading the mixed opinions, and very stern beliefs I found it rather difficult to not post a response.
All in all the artical touts the success of Mysql Over a commercial database. This onto itself is a benchmark
for success. I have been involved in a large number of commercial projects, including one of the worlds largest
databases which happens to run on Mysql "EdgeHealth".

I have worked side by side with developers and dba's who have scratched their head at how well Mysql performs.
Oracle, DB2, Postgres all seem to perform rather well until you compair them to Mysql. Although I do often enjoy
the ability to extend postges with perl modules, and create objects for oracle in java.. The speed and reliability of
Mysql cannot be denyed.

Consider the following, Cox, Google, IBM, Sun, Oracle, Microsoft, and a hoard of other companies deploy
Mysql as thier database of choice for web and e-commerce applications, it speaks volumes when a vendor
does not trust thier own products for production enviorments.

Mysql Postgres and Oracle will be around for years to come because they are good products. As for which database is the best..
thats just a matter of oppion. For the moment Mysql is the king of spead, and step child of features. My experiance says of the
3 Mysql wins out in deployment for the reasons, 1 performance, 2 scalability, 3 cost. 

Don't forget products evovlve or die, so if all 3 of these products are still around in the next couple of years, I would expect them
to evolve in the dirrection of what they don't have now.

Oracle and Postgres simply lack through put, Mysql lacks external scripting langauges.
All of these products offer transactions, clustering, and high avalibility.

If Mysql ever adopts external scripting languages, will it still keep its speed? Only time will tell.

2:04 PM  

 Anonymous said...

Re: Database as OS

You are absolutely correct, the database server typically runs nothing else, and using the file system to store tables rather than a raw partition is inappropriate. The file system is tuned for small files; in the 1-8kB range. That's just what Unix normally holds.

OTOH, by keeping the databases as normal files, backup and manipulation don't require new tools; you just use mv and cp, like you would any other file (with the database turned off, natch).

5:58 PM  

 Anonymous said...

Speaking of SQL, I heard that no database actually implements the full feature set that was defined back in the 1970s.

I personally tried to write a python layer that implemented views for MySQL. It was quite a bit of work and I never finished it.

Does anyone else have their sense of aesthetics offended when they see SQL embedded in another language? Reminds me of inline assembler or something. There has to be a better way. I tried to invent an OO layer over MySQL in the views project, but it was very ambitious.

7:04 PM  

 Anonymous said...

"Real" simply doesn't mean expensive in the database case. PostgreSQL and Firebird are both just as free as MySQL, but are actually robust and reliable.

7:42 AM

 

SATURDAY, DECEMBER 10, 2005

Oops!

POSTED BY RON AT 8:57 AM

 

I accidentally posted a political rant on xooglers that I had meant to post to my personal blog. I have a BlogThis! button in my bookmarks, and I forgot that now that I have posting access to more than one blog I have to actually tell it which one I want to use. Sorry. It won't happen again.

 

(Interesting how this happened the morning after I wrote a xooglers enry about how I kept screwing up at Google.)

 

 

3 COMMENTS:

 Tuor said...

actually, this may be a bug in blogger and not your own fault. I have observed it myself using the "blogthis" feature with multiple blogs. Sometimes it posts to my primary blog even when I have selected another blog in the select option. 

I have probably filled out a bug report for this, but I have never heard back from blogger.com staff about any bug reports I have filed, so I don't know if they look at them.

(TIP: I have noticed that if the BlogThis window doesn't *blink* briefly, it has not updated and will post on the default blog instead of the blog you have selected)

11:29 PM  

 13TonGimp said...

I used to have tons of blogger related errors, so I switched to wordpress.

I have fewer problems now....

3:28 AM  

 Anonymous said...

My sites work fine


 

SATURDAY, DECEMBER 10, 2005

Good things

POSTED BY RON AT 9:07 AM

 

Since my last posts painted a pretty gloomy picture I thought I'd devote a post to some of the good things that were happening at the time.

 

Despite all the stress I actually looked forward to my time at the 'plex. I didn't really have a life up there so I could really immerse myself in work, which I actually enjoy. I am an engineer at heart. I get a real kick out of watching something that I've built work. And the 'plex was the perfect environment, filled with smart, friendly people, state-of-the-art equipement, and half a dozen different kinds of coffee machines. (I actually had some interesting conversations about the finer points of espresso making, and at one point someone even brought in a book that was entirely devoted to the topic.) I learned a lot, more I think in my one year there than in the previous five.

 

As I said, I didn't really have a life, so I almost always stayed late (except when I felt like making the odd pilgrimage to Fry's). Charlie served dinner every evening around 5:00 or so, and there were always about twenty or thirty people who showed up for that. Late at night when there weren't people running around I'd try to learn to ride Sergey's unicycle (I'm pleased to report that neither I nor any of my colleagues sustained any permanent injuries), or work off some stress by riding one of the Zappys around the 'plex at full speed. (Nowadays they have Segways!)

 

One week I got to go on the annual Google ski trip. The year before it had been only a couple of dozen people. The year I went we had 150 or so, and filled three or four busses. It was a really strange week. I flew up from LA, got on a bus, rode to Tahoe, went skiing, got bussed back to Mountain View, and got back on a plane to LA. And I got paid for it!

 

But the best part of being at Google was the sense of being part of history. At the time no one had any idea how big Google would get, but I had been a computer geek all my life, and the Silicon Valley was always the Promised Land. I had wanted to be a part of it ever since I touched my first computer, a Sol-20. I had done a couple of short stints as a co-op employee at IBM (the Cottle road plant), and visited a lot (my first wife's family lived in Los Gatos), but this was the first time I was really a part of it. I even got a certain perverse thrill out of the insane commute. I would look around at my fellow passengers and think to myself that these were the movers and shakers. There were always Cool Things going on in the Valley, and these were the people making them happen. And I was one of them. I was finally living my lifelong dream. It felt great.

 

In the main, it felt great a lot more than it felt crappy. But the crappy parts make much better stories.

 

 

21 COMMENTS:

 Tavarius J. Taylor said...

Wow, so you really had no idea of the sucecss to come for Goog? By the way I really enjoyed your posts.

9:44 AM  

 Anonymous said...

Hi,

I'm a mother of a Googler. I just wanted to say that I love your blog. Keep it up!

7:02 PM  

 http://search-engines-web.com/ said...

HERE IS ANOTHER "GOOD THING"

Just been added to Slashdot today - your will get thousands of visitors during the next hours

http://slashdot.org/article.pl?sid=05/12/10/2125204&tid=217&tid=95

7:36 PM  

 Anonymous said...

I think its an awesome right. Interesting perspective but I disagree that all computer geeks dream of working in Silicon Valley. These days with the cost of living I think that is less true then ever. Especially since computers are so mainstream now and companies are moving about the country like other industries.

8:39 PM  

 Anonymous said...

Keep up the great blogging! I love these stories!

8:47 PM  

 Anonymous said...

This place reminds me of my time at Disney. Good luck with the blog. ^_^

9:19 PM  

 Anonymous said...

This blog is shit. You complain too much. Get over it, move on with your life.

9:24 PM  

 Anonymous said...

your post is shit, you criticize too much. Stop criticizing other peoples blogs and get on with your life.

9:28 PM  

 Anonymous said...

Why do people leave Google? It seems that some people after just one year. I would stay for at least 10. Are you working contractually?

9:39 PM  

 sickboy said...

These are an awesome inside view of google, I have always wanted to work for a company like google, kinda helps you see through the stuff everyone talks about to the reality.

http://redsoxunixgeek.blogspot.com

9:40 PM  

 Anonymous said...

Psst...Getting a job at Google isn't so hard if you don't mind working as a contractor for awhile and thinking outside of the valley. Trust me!

10:36 PM  

 Anonymous said...

I worked for the IBM on Cottle Rd for most of 2000. When were you there?

10:36 PM  

 Ron said...

I worked for the IBM on Cottle Rd for most of 2000. When were you there?

The summers of 1983 and 1984.

12:35 AM  

 Anonymous said...

"equipEment"?

Oh my, Google hires illiterate people. yuk.

2:18 AM  

 Anonymous said...

In general, and not a comment about the poster. But why does every web hack call themselves an engineer? I remember having to take a state PE cert then be required to work in the field for several years before taking the engineering exam. Fields had to be recognized engineering fields. Maybe the word has been regulated *joke* to the same professions as rocket scientist and brain surgeon.

7:15 AM  

 Anonymous said...

this is very funny

Google BADSENSE!

http://www.worldbankinginfo.com/credit.htm

http://img506.imageshack.us/img506/1133/badsense9ga.jpg

7:19 AM  

 Luke said...

I am a professional barista by day so of course I have to ask: which machines did you like using best and which book were you handed? 
Thanks for the interesting read!

12:47 PM  

 Anonymous said...

Ron,

You are a true Lisper! Thanks for the awesome read-macro! That's awesome.

I hope you can find work where you can hack lisp and be happy. 

And thanks for putting the transactions-must-be-in-the-db fool in his place. As if a DB guy, working in C/C++, is going to do manage the concurrency better than you guys. That Paul Graham reference was awesome!

1:06 PM  

 Orodemniades said...

I'm merely a fellow blogger (with a BA in Eng Lit!) but this is without doubt one of the best blogs I've come across. 

Funny, I feel like I'm a part of history merely reading this blog.

3:47 PM  

 Anonymous said...

Hey Doug and Ron,

Just wanted to ;et you know that I found your blog on SlashDot Google feed.

I use gmail, Adwords, Adsence, Site Map and a few others. My home page has been google for years.

I have been using the Personalization feature and have added your Blog to 'Create a section'. However if you guys have any friends on this project tell them there is an anyoying bug. Everytime the page updates it forces 'always on top' to the browser I have it in. Seeing I am an avid user of the right click and open in new window this always seems to happen when I am typing in a form or something and screws me up. I am almost to the point of just going back to the old google home page but I figured someone would figure this out sooner or later.

Lastly, I work in the internet industry and work smack dab in the middle of Marketing and R&D so I can relate to both of your stories. I also always wondered how the "big boys" did things and it is nice to see my shop has a lot of the same things going on and the grass isn't greener...

9:09 PM  

 SJ said...

Nice post. I would like wokr for Google or bettre yet start the next Google. I am tryign to see if I can pick any lessons from here.

12:21 AM

 

 

THURSDAY, DECEMBER 15, 2005

What he said

POSTED BY RON AT 12:06 AM

 

Funny, I was finally getting around to writing a sorry-I've-been-away-it's-these-damned-holidays-will-probably-be-gone-a-while-longer entry but Doug beat me to it. I'm probably going to be out of commission until Jan 1.

 

I'd like to send a great big thank you to all of you who have left supportive comments. Aw heck, it's Christmas -- thanks to everyone whether you've been supportive or not. Just knowing that someone is reading all this drivel that I'm writing makes it all worthwhile.

 

I wish you all a happy holiday season and a healthy and peaceful new year. Merry Christmas, happy Hanukka, splendid Solstice, stupendous Saturnalia, kick-ass Kwanzaa, a fun fabulous fortuitous and fortune-filled Festivus, and from the better-late-than-never department, a delightful Diwali.

 

Now stop surfing the net and go drink some egg nog!

 

 

42 COMMENTS:

 AG said...

Excellent start guys. You both blog in an entertaining way which keeps me coming back for more.
Please keep up the good work as you have a lot of fans now.
AG

12:54 AM  

 la lara said...

Then call when you're back

3:48 AM  

 Matthew said...

Ya Salams!

here here, I too have been irrisitably drawn to return to glean more & more details about the conciousnesses involved in the big G juggernaut..... !

& don't forget 'eid al adha' for all Muslims just after Xmas!

God Bless!

MT

5:32 AM  

 Dave said...

and a happy Agnostica.

5:43 AM  

 Chrissie said...

I hope you have a happy holidays as well.

11:15 AM  

 Anonymous said...

No shout out to all us Muslims and Eid ul-Adha? Sure its in early January this year but still its almost the holiday season!

5:18 PM  

 Kiran said...

been going thro ur posts for some time now. its been interesting.

and abt Diwali it happened in Nov itself but anyways thanks and wish u all a very happy X'mas and New year.

Kiran

8:37 AM  

 jeffysspot said...

yah .....what you said

10:03 AM  

 Anonymous said...

Thanks for great blogg.

1:34 PM  

 Ron said...

No shout out to all us Muslims and Eid ul-Adha?

An unintentional oversight. My bad.

An excellent Eid ul-Adha to all!

3:30 PM  

 Noctis said...

This post has been removed by a blog administrator.

7:09 PM  

 Sports Bettor said...

Keep writing. Happy Holidays.

9:06 PM  

 Anonymous said...

i'll be watching my RSS =)

have a good holiday!

11:54 PM  

 Samer said...

great stuff! Happy holidays...

1:03 PM  

 Sam said...

Nice job... wonder why you call your blog, Xooglers? But I am just starting to type my own ideas and stuff on mine...

6:15 PM  

 omm said...

I am one of your big fans too. Merry Christmas to you too and thanks for the nice thoughts :-)

1:17 AM  

 Archosidf said...

one of the most fascinating things I've read on the web in the last months

9:19 AM  

 Dunzo said...

I'll stop drinking egg nog and start surfind the web right now.

10:16 AM  

 Anonymous said...

Here is a blurb from the 12/20 WSJ,

"Mr. Kordestani oversaw an important shift in advertising models at Google in 2003. Google moved from charging advertisers based on how many consumers saw their ads to its model of billing them only for users who clicked on their ads. That involved combining several sales databases -- an initiative known as "Project Drano" -- which Mr. Schmidt says "nearly killed" Google. Phasing out the old ad system entailed changing the incentive and commission structure for Google's sales staff -- a treacherous task, which Mr. Kordestani supervised."

Any juicy tidbits about this project or Mr K?

10:49 PM  

 Anonymous said...

Happy Holidays regardless what you celebrate!

10:37 AM  

 Anonymous said...

I'm curios about the relationship between Engineering and Product Managment. Was it cooperative? How was the PM group viewed internally? Did they fall under the 70/20/10 rule? What did they spend their time doing? How big was the team?

4:58 PM  

 JK &amp; PK said...

Dude, you deserve some eggnog (you can have mine too, i think it tastes like crap)

4:59 PM  

 Anonymous said...

Diwali is already over in November in India .

Happy X mas to you.

6:38 AM  

 Rufus the Great said...

I love your blog! Please check out mine, http://placeofrufus.blogspot.com/

8:56 AM  

 Jennas Blog said...

I'm new to blogging, and internet business. So far, I've found google to be difficult and Yahoo much more user friendly.

7:04 PM  

 chuck said...

IT'S OFF TO EGGNOG I GO...

9:38 PM  

 Stenar said...

G-ZUS LOVES YOU

11:27 PM  

 rafiq said...

The New Google doodle page has an error on it. 

http://www.rafiq.za.net/2005/12/24/google-holiday-doodle-page-has-an-error-on-it/

12:52 AM  

 rafiq said...

Wow! that was fast. Luckily I took a screenshot as proof (available at my blog)

1:19 AM  

 underpanter said...

you are hot, wanna link to or from my site?

3:55 PM  

 localudal said...

Merry Googlemas!

Even as the blog is dedicated to the Google past, here's something for the ominous Google future (just one excerpt from my blog http://goolocalizations.blogspot.com):

GOOGLEVISION

One Google opening for i18n specialist requires a candidate to possess a "vision". You want a vision, here you go.

In 1993-1994 I worked on Nostradamus TV script translation, and it was me who said that the scene of a global war of West against Islam must be dated 2001, not 2000, as faulty translation of Nostradamus' quatrains suggested. Check it out in Mosfilm archives, or in some Hollywood script hangars.

Here's what happens in the next decade from now:

1. Google buys Australia, Starbucks, Moffett Field, Antarctica, some parts of California -- from a weakening grip of Governator. Google finances the digging of the tunnel and the pipeline between Eurasian Empire of Czar Putin I and Alaskan Dominion.

2. On Moffett Field, Google builds the Babelfish Tower out of millions of nanocarbon tethers delivering food, water, replacement nuts and bolts to ISS by dozen molecules at a time. Ancient flotilla of space shuttles is grounded and converted into Starbucks restaurants.

3. Google buys CNN, FoxNews, replaces their field force by 360 deg Webcams put on every phone pole in US and in major critical points around the globe. The new service is called Google News, and its content, zoom, direction and choice of commenting blogs can be tailored by user's preferences. Hollywood movies can be watched in raw footage, or pre-director's cuts. Effects, actors and scripts can be edited by a viewer.

4. Cash is virtually abolished and used only in illegitimate trade of drugs, firearms, in presidential campains and in live prostitution. Numismatists grumble over cash substitute called AdCents: an ancient $2 bill is valued at 200 AdCents at GoogleBay: 1 AdCent is equal to 1 million of AdSense clicks. As the result, Google Cache replaces Fort Knox. Google's motto 'Don't Be Evil' now reads: 'Don't Be!" Accordingly, the final solution to private information and its theft is brought about: except for the name, approximate date of birth, and fingerprints there's nothing could be found on any individual. Legacy data like account numbers, PINs, SSN, Apt./House Numbers, etc., are known only to Google Index and corresponding agencies.

5. Google trades McDonalds' back to Canada for Québec. Resulting huge reserves of frying oil, with Persian Gulf's oil being almost finished, are used in mix with corn schnapps to fuel 4-wheeled segways, or SUVs, as in Segway Utility Vehicle. Ford leads the pack of auto manufacturers with a nostalgically smelling Segway-T model capable of 1 mpg.

6. Googlers (see section below) are routinely shot off the skies over China and Northern Korea by ancient DoS ACKACKers. Google retaliates by using Mandarin over the Continental China. Meanwhile, with Google's backing Al Gore, the old Internet inventor, buys EU Internet back from European rebels. A new resulting Internet is called Google Index.

7. Google buys LOC, NYPL, SJPL, Stanford Library, and copyright debates cease with an acceptance of Copygoogle Law. Google buys Wikipedia and Britannica, publishes the resulting Brikiwikipedia in 100 set in Katull paper volumes of 5000 pages each. To enhance readability of such a tome, Googlese it is printed in (see below) has lost many useless letters and signs like 'c', 'j', 'q', 'v', 'w', 'y', 'z', '$', comma, colon, semicolon, ellipsis, dashes, etc. Sites insisting on using old, or so called correct English, are progressively excluded from Google Index. 

8. Google launches Goobble, a space telescope armed with hurricane battling laser. First fire test was partially successful: the Omega hurricane was dissipated over Cancún, Transtexas, which was severely fried in the process. Goops!

9. Google's AdSense program is challenged by AbSence ideology by Teoma, the most used search engine of 2015. Teoma's success is based on abolishing spamming and sponsored links in its search results.

10. At Googleplex and next to Vint Cerf's, Google unveils a bronze monument of me, the best Google Borshcht Chef whose after-borshcht burp renders a better "Goooooooogle" sound out of majority of contest jury for 10th year in a row.

11:28 AM  

 Shaun said...

Awesome blog :)

"Happy HOlidays"

Shaun - ohpunk.blogspot.com

3:59 PM  

 JR said...

Merry 2 miutes afta christmas!!! 


comment to me at hairston.blogspot.com

12:03 AM  

 JR said...

u guyz no how to make a good blog!

12:05 AM  

 Rufus the Great said...

jr,
I cannot understand you. And with my superior intellect, you would think I could understand anyone. What is wrong with you?

6:49 AM  

 Mel Logan said...

happy holidays, and thanks for the good reading material.

12:46 PM  

 Miyagi said...

Great blog dude!

10:18 PM  

 Anonymous said...

Egg Nog rocks

Signature:
Get $25 free on first deposit at the world's largest poker room with 70,000+ players
http://www.partypoker.com use code: shefa

10:27 AM  

 todd said...

wow you have a lot of fans i just read this blog. i like it. i'm a fan of courtesy. your a writer thats cool i want to be a writer. i noticed you did the whole naming of every holiday to be politicaly correct. just say happy holidays. or merry christmas. reading your blog i can tell your smart. i didn't get bored reading your blog like i do with others. you kept me interested.

10:35 AM  

 frank white said...

hey hey now. i like your blog. please explain to a google browser search user what xoogler is all about? what does the word mean? is there a site? maybe a program? hey kiran where are u from dude? you need to wear slightly loose brand of jeans. those balls need to breath bro. by the way i can see that you have exactly 2 dimes and 4 pennies in your front right pant pocket

8:10 PM  

 JC said...

You left out Agnostica!

2:46 PM  

 Harry said...

And may the force be with you :p

4:51 PM

 

MONDAY, DECEMBER 26, 2005

We interrupt this vacation...

POSTED BY RON AT 4:13 PM

 

... to bring you an important message.

 

I had a little down time today and so I decided to hook up my laptop and check my email.

 

Big mistake.

 

It seems that Xooglers has attracted the attention of all sorts of people who think that because I used to work at Google I now know Everything there is to know about Anything. I try to be helpful when I can, but listen up folks: it's been nearly five years since I worked there. Everything I know about the current state of the company I know from reading the papers (or Bob Cringely). I cannot offer you any particular insight about how to get a job at Google, or how to prepare for an interview, or how to get better tech support for your blogger account.

 

But as long as I've gone to the trouble of getting on my soap box, here's a bit of advice I can offer: when sending a cold-call email, a little common courtesy can go a long way. Introduce yourself. (A simple "Hello, my name is..." is fine.) Tell the person you are writing to how you found them. Just because you found someone through (say) their blog doesn't mean everyone has. Tell them a little bit about yourself and what you are doing that leads you to ask whatever it is that you are asking. And finally, you can get a surprising amount of leverage out of the odd "please" and "thank you." (But don't overdo it. Being demanding isn't cool, but neither is groveling.)

 

And for God's sake, never ever send a cold-call email from a mail account with a self-aggrandizing alias like "Lord and Master of the Known Universe." You may just as well tattoo the word "dweeb" on your forehead if you're going to do [expletive deleted] like that.

 

We now return you to our regularly scheduled intra-holiday lull.

 

 

34 COMMENTS:

 Anonymous said...

More eggnog!

5:09 PM  

 Captured Intrigue said...

Hahaha....

Signed,

Princess of The Dark Realm Of Pixies That Hunt People Who Worked For Google But Are Expected To Know It All Still.

That is like asking somebody who used to work for McDonald's 7 years ago what this week's prices are for a chicken nugget meal.

5:38 PM  

 SJ said...

Hey you, get me a job at Google ok. And make it fast. 

:))

Happy holidays!!

7:30 PM  

 NN said...

Hi,
i'm am errr... a student of business. i read you blog because it gives a lot of insight on how google works/worked.
i don't remember where i came from.
but i come here often :D
-me

PS: can please make my professors stop talkign about google!

2:16 AM  

 Pasamio the Geek said...

Hello, my name is Pasamio. I found your blog when it was on slashdot and had a slow and gentle read through the posts that the two of you have made here. 

I'm a geek and I'm wondering if calling myself 'Pasamio the Geek' is too dweeb like. 

I await your answers! :p

On a more serious note, very interesting historic read from a time that I often wish I was a part of, but didn't get the chance. Season Greetings guys, thanks for providing more information for the world to digest!

8:12 AM  

 Anonymous said...

/haha

can u teach how hack? source code plz.

send to lordofall@hotmail.com

8:15 AM  

 Anonymous said...

umm ok why is whereever I go, google is there!! ?? !! ah

9:45 AM  

 Anonymous said...

HOW CAN I GET A JOB AT GOOGLE??? I WANT A JOB AT GOOGLE> GET ME ONE NOW> hahahahaha

9:46 AM  

 Anonymous said...

A little courtesy goes a long way. Great Blog...Thanks

Signature:
Get $25 free on first deposit at the world's largest poker room with 70,000+ players
http://www.partypoker.com use code: shefa

10:26 AM  

 Dr. Ægustüs Ŗiviŋiallli said...

What a glorious place the World Wide Internet is! In this vast "web" of information, we find expertise in the most mundane of areas, such as searching for jobs! Oh, the joy of this cornucopia of knowledge!

12:33 PM  

 Talha Masood said...

:)

interesting

thank You!

5:03 PM  

 Anonymous said...

Can you blog about the growth currently at Google? Your thoughts? Why does this company function as a institution rather a business? Is Google the replica of Stanford but in Search form?

9:22 PM  

 Trish said...

Hello lord and master my name is nobody, any idea how many hits you get on google when you type in buffalo tack glass and why?

6:52 AM  

 khaled said...

SJ 
me too!!!!!!

7:21 AM  

 Steve said...

Surely if you were the best and the brightest and worked at el Goog you knew everything at that time, and therefore still know everything now? And I DEMAND you tell me EVERYTHING! (As I understand, CAPITALISING some WORDS is likely to get you a FASTER response).

10:02 AM  

 Writer on Board said...

Trish says her name is Nobody. I'm Nobody too. Will the real NOBODY please stand up. What happens when you type NOBODY into GOOGLE? 

Tell me Jason. 

Tell me Mr. Google-Man, Mr. Googl-ator, Senor Goog-alito! Tell me! 

Sorry. I'm having a really hard day. Good Blog.

-Jason Nobody

3:37 PM  

 Silyth said...

Sometimes I think a hint is lost on others since you state clearly your response on some things and people just come back asking the same things. Oh well, how tedious it must be for you.
As you mentioned before introductions are nice so...hello I'm Silyth. It's a pleasure reading about what you want type. Now I want to get back to reading and I wish you a good holiday.
Thanks for the good read!

Silyth
http://silyth.blogspot.com

6:28 PM  

 W. said...

I am a Berkeley educated Social Scientist. I can tell you how to get a job at Google. I have cold emailed Google in the past to evaluate how they respond to various personalities. As well, I have been inside the mind of some of their stars. LOL

Ask yourself if you want to have some guy who is socially dysfuctional say things like "shit" to you because he got some false confidence after his equity in the company grew. I would rather work as a park ranger with people that actually are not so insecure.

1:15 PM  

 Anonymous said...

I don't mind reading interesting stuff from someone even if they are potentially socially dysfunctional or use words like "shit". Not to imply that the former is the case with xooglers, I can't see why someone would even care.

1:45 PM  

 submergent said...

I know what you mean. Once people find out I once bagged groceries for a couple years out of high schools, it's non stop questions about spacial relationships and weight ratios. It's a curse.

4:49 PM  

 Sarah Khokhar said...

My neighbours think I can solve all their artistic needs just cuz I learned most of the Adobe stuff in class. All I really know is how to crop an image.

9:49 PM  

 samuel said...

I kept reading, waiting for the guy with the problem with a word. That shit is fucking weak! It was actually kind of a let down when I did finally read it. I was ready for moralizing and I got some dimwit that ran out of real things to bitch about.
So is it true that you know the HR guy at Google? email me at drunkanddisorderly.com Dude, totally hook me up!

12:26 AM  

 speed_demon said...

Hi there,

No offense, but how do you feel when google went IPO?

Man, working at google must be great. It must have felt like "walking among the giants"!

Happy New Year!

3:22 AM  

 The Gradster said...

This is really stupid...
http://toypalace.blogspot.com/
go there!!!!!!

7:00 AM  

 remcgregor said...

Just wanted to comment on what a great Blog you have here. Google as a company has always fascinated me. Looking forward to when you guys are back and blogging again! :)

4:24 AM  

 IanShorten said...

Google is a sweet company. I've used google for so fucking long and so naturally when Gmail comes out I flipped. Wish they would make a win 98 compatible google talk.

1:18 AM  

 Anonymous said...

Isn't blogger owned by google?

1:24 AM  

 The Urban Kunoichi said...

Hello there,

I began to read your blog since it was in the featured list at blogger.com. 

Great blog!

But I have to admit, I get as much enjoyment reading your comments as I do from your own words!

Thanks for the entertainment!

5:26 PM  

 Anonymous said...

To the "Berkley educated Social Scientist" who could probably tell everyone a little bit about everything: A really big ego is often a sign of insecurity. So, you seem to have earned your false confidence at Berkley.

You probably couldn't fit your ego into a park ranger's shirt.

10:12 PM  

 Kimberly said...

Lord and Master,

Please please please please please please please please please please please please please please please please ...ignore this post.

Thank you, thank you thank you thank you thank you thank you thank you thank you thank you thank you.

4:03 AM  

 b said...

please can u give us ignorant folks some fast fast search advice.thank u in advance

1:13 PM  

 Anonymous said...

Jokes.

2:00 PM  

 Harsha said...

That is funny..

4:52 PM  

 Adbloggers.com said...

I love this blog. I check it often for updates. Keep blogging!

Ron Wilson

www.adbloggers.com

6:38 PM

 

WEDNESDAY, JANUARY 04, 2006

We are experiencing technical difficulties

POSTED BY RON AT 9:47 AM

 

Happy new year everyone!

 

If you've been watching my personal blog you will have seen that I've actually been back for a while now. The reason I've been quiet on xooglers is that Google legal has asked to talk to me about it (no big deal I am assured) and we've been having trouble making contact.

 

[UPDATE: I was hoping this would just pass, but I guess I'd better clarify before it spins wildly out of control: Google has not asked me to stop blogging. They have not even remotely hinted anything of the sort. All they have done is ask to talk to me, and I have entirely on my own initiative stopped writing Google stories as a courtesy to them until we're able to make contact and I hear what they have to say. So I'm sorry to disappoint all you conspiracy theorists looking for a juicy tidbit, but there is nothing more sinister going on here than a game of phone tag.]

 

[UPDATE2: We finally made contact. The conversation lasted all of five minutes, and the message was basically, "We think Xooglers is great, but please be careful not to reveal any confidential information, and if you have any doubts about what might be confidential please feel free to call us and ask." No big deal. So I should be picking up the story again later today.]

 

So in the interim I thought I'd tell a story that Google can't possibly object to because it happened in 1997, a year before there was a Google.

 

Doubtless what fascinates many people about Google is the vast amount of money it has generated. Larry and Sergey are now billionaires. They've bought a 767 (which is not quite the splurge it might at first glance appear to be). What is it like to have all that money?

 

Well, I have no idea what it is like to have that kind of money, though I can report from firsthand experience that having some money is awfully nice. But it's no panacea. Money can let you eliminate some of life's headaches, but it comes with some of its own. Don't get me wrong, I definitely place these in the category of "nice problems to have", but they are nonetheless problems.

 

My most direct encounter with "that" kind of money came in 1997 when I attended a technical conference in Santa Barbara. Because SB is pretty close to where I live and I'm a pilot, I decided to fly myself there. At the time I was flying an old Cessna 182RG, which is a pre-dark-days four-seater with retractable gear (hence the "RG". No, it's just a coincidence that my new initials are RG. The plane I'm flying now is a Cirrus SR22, which has fixed gear.)

 

(As long as I'm digressing I guess I should explain that the "dark days" were the period in the late eighties to early nineties when no small planes were being manufactured in the U.S. because of liability concerns. At the time, common law held that an airplane manufacturer was liable for manufacturing defects in the airframe no matter how old the airplane was. In 1994 Congress passed the General Aviation Revitalization Act which limits the liability of aircraft and parts to 17 years after the date of manufacture. 17 years still seems like a long time to me, but apparently it's enough to let you make a business case, and since then the U.S. aircraft industry has undergone a renaissance.)

 

Anyway, it turned out that I was not the only one who had flown their own plane to the conference. There was a very highly ranked executive of a major software company there as well. I won't say who it was or what company he was with. I'll just call him "Chuck." I had actually met Chuck several years earlier when I interviewed for a job (which I didn't get) so I re-introduced myself and we started chatting. It turned out that both our planes were parked at the same place at Santa Barbara airport, so I offered to give him a ride to the airport when the conference was over, an offer which he declined.

 

So the conference ends, I get in my little rental car and start to drive out of the parking lot when here comes Chuck running after me waving his arms. It turns out that his ride to the airport hasn't shown up and can he take me up on my offer after all? Sure, I say, hop on in, which he does. He then goes on to make himself comfortable by taking off his shoes and putting his feet up on the dash. OK, whatever.

 

We get to the airport and we both walk into the FBO. My plane is off in the boonies, but his is parked right out front. It's a Falcon 2000, pretty much top of the line in its day (about $30M worth of plane I believe). The red carpet is rolled out (literally) and a little army of crew people are running around getting the thing ready. I pick my jaw up off the tarmac and wish Chuck a pleasant journey.

 

I walked over to my plane, which seemed very, er, cozy at this point, and started preflighting. In the distance I can hear the Falcon spooling up its engines, and then spooling them down again. Then once more, up and down, and again. And again.

 

Hm, that's odd, I think. Why don't they just leave?

 

Then they shut the engines down and open the door. Hm. Doesn't look good.

 

I go back inside the FBO and there's Chuck sitting on the couch looking very unhappy. "Disaster," he says to me. One of the engines has a broken part. They can't take off. And he has to be in Europe in eighteen hours for a wedding.

 

Since he had no where to go, Chuck invited me for a tour of his plane. It was a pretty sweet ride. Overstuffed leather upholstery, a nicely stocked bar, I think there was even a shower in back. But the part that really got my attention was in front: it was the first time I'd ever seen a glass cockpit up close.

 

Being in a plane like that without being able to go flying is kind of like being in a candy store without being able to eat anything, so I hung out for a few minutes, then thanked him for the tour, and took my leave. I offered to fly him to LAX where he could catch a commercial flight to Europe but he declined, which was probably the right thing to do. Flying in to LAX is theoretically possible in a small plane (I've actually flown to JFK in a four-seater once), but it's not something you'd want to do lightly, and at the time I was probably not experienced enough to do it right.

 

So I flew home. It was a perfectly clear night, and the city lights were spectacular. I marveled at the fact that I was going home while behind me a man worth many billions of dollars was stuck at the airport like an ordinary shmoe.

 

Postscript: thinking back on it now, I have no idea why he didn't just call a charter company and arrange for another plane to come pick him up. For a few hundred thousand (which should have been chump change by this guy's standards) you can have your pick of the fleet at your beck and call. Maybe if I ever run into Chuck again I'll have a chance to ask him.

 

 

22 COMMENTS:

 EmailHosting.com said...

What is a xoogler?

6:25 PM  

 http://search-engines-web.com/ said...

That is sad that there are potential concerns about the blog express by Google legal - they must understand the effect Google is and has had on the planet - and Google history should be more than press releases and PR speeches given at keynotes....Good Luck

By the way, read Matts blog about the new BigDaddy

6:29 PM  

 Penguin said...

are Xooglers , former employees of google?

7:25 PM  

 Anonymous said...

I wonder - was that Larry Ellison, or Paul Allen ? Hard to think of many people with that kind of money and interest.

7:48 PM  

 tby said...

In my experience, people that have achieved wealth (as opposed to being born into it) sweat the little things. $30M plane, no problem. $12K for last-minute first-class airfare to Europe, problem.

I knew a guy that sold the company he co-founded and walked away with nine figures. He got the Ferrari, the sailboat, a small island with a log cabin of McMansion proportions in Canada. He made his wife go back to dial-up because the ISDN bill was a couple hundred bucks one month.

8:43 PM  

 Anonymous said...

the glegal stuff is lame! this blog has a following. are they stalling on purpose? anyway, i say give them another few days to say whatever they need to say, and then start writing again.

9:09 PM  

 Meme chose said...

Ha! My friends, who owned two planes, used to say: "If you really have more than enough time, take your own plane".

10:27 PM  

 Anonymous said...

I hope the "little talk" with Google isn't going to disrupt this blog. 

Keep it up, I love reading it :)

TF

12:05 AM  

 rafiq said...

A few more clicks and reading would tell you what xooglers are. 

The Xooglers blog is one of the best blogs I've ever come accross. Legal Shmegal, I think we should have the right to know what the guys over at the big G are up to seeing that they know what we are up to if we use their services, gmail, personalised homapges blah blah. Who know our lives online are are up to. Welcome back & Keep it up.

Instead of asking I will google for your personal blog or just check this one if it is somehwere here.

off topic: If friends or family ever use you notebook/workstation/tablet or pc be sure to be logged into your google account :) I wonder if there is a study comparing your personality to how & what you search for?

12:28 AM  

 Anonymous said...

My guess would be Charles Simonyi

1:06 AM  

 Tim said...

Yeah, I'm going for Charles Simonyi too. Doubt it was Paul Allen - I've met both and Charles is more sociable than Paul. It didn't sound like Paul. Well, he said something, for a start :)

3:44 AM  

 Anonymous said...

Keep up the great posts! Please keep us updated on the legal stuff. We may not be lawyers, but I am guessing your dedicated readers can help make some noise if you need it.

8:12 AM  

 TwistedNoggin said...

The little planes are SO much more fun to ride in. I wish I could pilot.
Sorry about the Google complications. Hope that all works out well. :)

9:39 AM  

 Anonymous said...

Interesting that you'd choose to call me "Chuck". Heh.

10:10 AM  

 Ron said...

Interesting that you'd choose to call me "Chuck".

Well, I didn't want to have to keep referring to you as "this person" so I had to call you something, and I figured you wouldn't want me to use your real name. If you prefer I used a different name contact me by email and let me know and I'll change it.

(For the record, I have no reason to believe that this particular anonymous really is Chuck, but you never know.)

10:22 AM  

 Ron said...

This post has been removed by a blog administrator.

10:42 AM  

 Ron said...

This post has been removed by a blog administrator.

10:43 AM  

 Ron said...

OK, I can hear the conspiracy nuts going to work on the two deleted comments above. What happened was that I was trying to respond to twistednoggin who wrote "I wish I could [be a] pilot" by saying but for some reason the link disappeared and the result looked really weird. Hopefully this more verbose version will work...

10:48 AM  

 Ron said...

Wow, something really seems to be broken here!

10:49 AM  

 Anonymous said...

Are you implying that the owner of the plane was a Google employee? And that the fact that the plane did not work is your way of suggesting that Google does not work? You better be careful there Mr. Blogger.

2:07 PM  

 Bernoulli said...

Kind of off topic but...... does anyone know about the new IPv6 and do you think Google is looking at this? Heres the link that made me curious.
http://www.imakenews.com/signal/e_article000497585.cfm?x=b11,0,w

12:02 PM  

 Anonymous said...

I do have the best guess of who Chuck is. It was Elon Musk former paypal's largest shareholder right? I know he owns a falcon 2000. His spacex rocket's name is also falcon.. He is not a billionaire though, I think

10:24 PM

 

 

 

THURSDAY, JANUARY 05, 2006

Lost in translations

POSTED BY RON AT 9:38 PM

 

In December of 2000 I had my first performance review. I don't really remember it all that well. Both my manager and I were keenly aware that on the one hand I wasn't performing very well, but on the other hand the project that I was nominally in charge of had launched on time and was a huge success. AdWords was starting to bring in a significant portion of the company's revenue.

 

There was something else had started to notice in the weeks leading up to the performance review: I wasn't being asked to do interviews any more. Interviewing was widely viewed as a chore. It was time-consuming and disruptive, but the company was growing, and someone had to vet the new hires. Personally I enjoyed doing interviews, probably because they afforded be a break from writing Java code. No one ever said anything to me about it, but after about September or October I noticed that I wasn't being put on the interview roster any more.

 

I remember only two significant things from that first review. First, I didn't get a raise, which is customary (or at least so I'm told) unless you've really screwed up. I suppose that was fair enough; I would be the first to admit that I'd really screwed up. But the second thing floored me: I was being kicked out of the ads group.

 

I was furious, though I tried hard to cling to the tattered remains of my professionalism and not to show it. Three months earlier I had offered to resign from the ads group and my offer had been refused, and now they were kicking me out.

 

I spent that winter vacation in a deep funk not sure at all what to do. I was getting so many mixed signals. On the one hand I had twice tried to walk away from the situation with some honor intact and twice I'd been refused. But on the other hand I was getting so many signals that the company really wanted to be rid of me: no raise, no interviews, and to top it off, getting kicked ignominiously out of the group.

 

I cannot for the life of me remember where I was between Christmas and New Years of 2000.

 

I remember my wife and I having an earnest discussion about whether I should resign. The situation seemed to be getting untenable, but I'd already invested six months, and it was starting to look as if Google was going to do reasonably well, and that the stock might actually be worth something some day. I didn't really want to walk away from that. So in January of 2001 I once again got up at 5 AM to catch the 7:25 Southwest flight from Burbank to San Jose.

 

Oddly, one of the things I remember most clearly from this time was this big patch of impatiens that were growing in a flower bed in one of the parking lots adjacent to the Googleplex. I had hooked up with a ridesharing partner who was doing the same commute I was (you'd be amazed how many people do insane commutes in California!) and he would drop me off at the corner of Shorline Boulevard and Amphitheater Parkway and I would walk the rest of the way to the 'plex. It's a beautiful area, right next to a wildlife preserve.

 

Imaptiens are annnuals, and I thought that they were supposed to die, and yet here was this batch of flowers just growing like gangbusters. The colors were brilliant. And as the winter wore on they just seemed to grow and grow and grow. It's amazing how a little color can cheer you up.

 

To mark the start of the new year and to really put the cap on my ignominy, the manager with whom I had had all the friction went on maternity leave shortly after the new year (as best I can recall -- I could be a little off on the timing of events here. My memory of this period is not the best.) She was replaced with a fellow named Ross, with whom I ended up getting along much better. At about the same time there was this big reorganization of the engineering department, and I ended up -- wait for it -- back in the ads group! Or at least I was reporting to Ross along with all the rest of the old ads group members. I wasn't actually working on ads stuff any more.

 

My new project was the Translation Console. At the time, Google was available in about a dozen languages. The translations were provided by a professional translating service which was very expensive and (I was told) not very reliable. The translations were also not particularly good. The solution was typically audacious: build an interface to allow Google's users to translate the site contents into their own languages for free. The result would be higher quality translations in more languages (over 100 at last count, including Klingon!) for free.

 

The problem with this approach is, how do you do quality control? If nobody at the company happens to speak, say, Serbo-Croatian, how can you tell if the ostensible Serbo-Croatian translation of "I'm feeling lucky" really means "I'm feeling lucky" instead of "Visit my hot new porn site"?

 

Again, Google audaciousness: let the users do the quality control too. The translation interface allows you to not only translate new content, but also to correct errors that you find in already translated content. As long as there are more honest translators than there are people trying to introduce spam, the spam will quickly get corrected away. Just as the Google search engine computes a "Page rank" based on how many links a page has pointing to it, the translation console computes a "translator rank" based on how many translations you've submitted and how many of your translations have been corrected by other translators. It works like a charm. To my knowledge, there has not been even a single instance of translation spam that has ever gone live onto the Google site.

 

I wish I could take credit for all this brilliance but I can't. I wrote the code, but the idea was conceived by Marissa Meyer (whose name I feel safe mentioning because her role in Google has been well documented in the press) and Deb (whose last name I'll withhold because her's hasn't). Marissa did most of the design, and I did the actual implementation. It was not the most challenging thing I've ever done, but at least I got to do it in Python instead of Java. If you can't have Lisp, Python is the next best thing. I also got to do some hacking on the Google Web Server code so that it could use the new translations, which was very cool and a little bit scary, because if you screw that up you will definitely hear about it.

 

It was probably good for me to have something a little less demanding to do at that point in my life. I got to decompress a little, and I had some free time to explore other parts of the company. In particular, I got to know people in the research group, which is where I thought I probably would be the best fit, but it soon became clear to me that I was way out of my league here. The sad fact of the matter is that my own field, autonomous robots, is (or at least was -- things may have changed) rather badly screwed up, and there was an awful lot of bogosity that passed for scientific research. (I actually published a paper about this at one point, which went pretty much unnoticed.) But the people in the Google research group were actually doing real stuff. I would have had to work for months, maybe years, to catch up to them, and it was becoming more and more obvious that my days at Google were numbered, so it seemed pointless to try.

 

I set my sights on my one-year anniversary in June when I would finally vest my first (and only) batch of stock options, gritted my teeth, and settled into the routine.

 

I can hardly remember any of it any more.

 

 

21 COMMENTS:

 Anonymous said...

MORE ABOUT GOOGLE :

Unofficial Google BlogGoogle news

11:09 PM  

 Steve said...

You surely had a lot more strength than I did. I aimed for doing at least two years with my previous employer, but I just could not hang on. I made it to my 18th month with the company, and although I promised my peers that I would try my best to stick around, I just couldn't.

The people who were working there did not drive me away from the company. It was the environment. I honestly could not give a feasible prediction on what I would be doing there over the next five years, and if it was me doing the same thing I was doing at the time, that future is simply unacceptable.

I am not even sure if I still have any stock options, but obviously the number of stock options and their potential value could not keep me from leaving. The promised quantity of options was also low enough not to create incentive for me to push for the advancement of the company.

So I left.

11:45 PM  

 Veky said...

Yes, your translation console worked pretty well. It's interesting that you mention "I'm feeling lucky" translation to Serbo-Croatian, because there was some controversy about it (in Croatian at least -- Serbian and Croatian are two "different" languages now). However, it wasn't about spam, but about clumsy translations. I guess you English-spaeking people don't really get that feeling: when something just stabs you in the eye as a bad translation. Yes, I know that "I'm Feeling Lucky" is very hard to translate (there was even a whole entry in your FAQ about that particular phrase, IIRC), but I thought I had a translation that sounded better -- at least, it evaded problems with gender and cases, which are so common in automatic translation to Croatian. 

However, I couldn't communicate it to Google well. The problem in your ranking scheme, as I perceived, was that for many "small" languages, one person had translated mostly everything in the beginning, and if just one his/her (you see? gender problem:) translation was bad (and he/she was convinced it was good), there wasn't much you could do -- because his/her "rank" was too high compared to your one.

But, I'd like to tell it was overall a pleasant experience (much the same as your work at Google -- a human mostly talks about the bad side of things). And it was surely much better than what we have today (e.g. for Gmail) -- where you can't even enter the circle of translators except if you are "chosen". :-(

2:39 AM  

 Danny said...

There was at least one translation problem I know of that, if not spam, certainly wasn't meant to help the program. The search button used a slang word in Malta that means "penis in a can." More on that here:
http://blog.searchenginewatch.com/blog/050412-160229 cover

4:01 AM  

 Greg Linden said...

Minor correction. "Marissa Meyer" should be Marissa Mayer, I suspect.

8:23 AM  

 Anonymous said...

You write: "I got to know people in the research group..." and "...the people in the Google research group were actually doing real stuff." This is what gets me about Google. Four years ago I talked to some senior folks at Google, about working there. I wanted to do research. I was told, in uneqivocal terms, that "Google does not do Research and Development.. it does Engineering and Development". One senior fellow told me that Google does not hire researchers, it hires engineers. That was enough for me to hear, and so I have never bothered applying for a job. 

So what is this "research group" you're talking about? Development, yes. Engineering, yes. But research? Hadn't I been told that "Google doesn't do that"?

Pshaw. Google makes no sense.

9:13 AM  

 Bernardo said...

Hey. I've been reading this blog with much interest since it got started. After working at Google for about 9 months, I quit in November (on the 17th, the first day of this blog, coincidentally). While the people I worked with were really awesome, there were things about how Google is run that I could not stand. (No, I will not talk about those things - I'm too afraid of Google's Legal dept, despite (or maybe because of) having done some work with them). Because of my less-than-great experience working there, I am naturally curious for any information about how Google evolved from a small start-up to a secretive, not-too-flexible, not-too-friendly large company. Hence my interest in your blog. Beyond this, I should say you guys' writing is very good indeed - the blog would be almost as interesting if it were about some other company. So keep it up! (And Ron, through this blog I discovered your personal blog. Your views on lots of things are very similar to mine (least of which is the wonder of flight), so I'm enjoying that a lot).

The reason I finally wrote just now was because I did some work with Google involving translation, so I thought I'd throw in my two cents. Most of that work involved revising the Portuguese translations made by that "very expensive and not very reliable" service of translators. The translations were indeed sub-optimal. I had trouble believing that someone was being paid a lot of money to translate this poorly, inconsistently, and uncreatively. (When I started working at Google, Portuguese was moving from being a Console-translated language to an officially-translated language, but instead of having everything re-translated, we inherited some material from the Translation Console days, which was also (predictably) not expertly translated, but (surprisingly) not much worse than the professionally translated material).

I was also one of the few people doing end-user support in Portuguese (and actually the only person doing end-user support in Portuguese for some products), so I was the first one to hear complaints from users about our bad translations. There were so many examples of poor translation, we (the Brazilian folks at Google) could not catch all of them. But we did try.

I liked Veky's comment about how "you English-speaking people don't really get that feeling: when something just stabs you in the eye as a bad translation". I think enough of the world speaks good-enough English that, when something needs to be translated into English, it's usually done pretty well (which is not so true of other languages) so English speakers just aren't exposed to really bad translation (with a few humorous exceptions, compiled in a handful of websites like Engrish.com). English vocabulary is also flexible enough that, if a slightly-wrong word is used, it doesn't look too bad. But I would CRINGE looking at some of the Portuguese material on Google's pages and products. I would then write somewhat angry emails to the people in charge of this material, saying "This is TERRIBLE. You're sending a message that says 'International users don't really matter to us'." I knew that they were busy, and that having the Portuguese material be perfect was not their highest priority, but I had to give them a hard time. =]

However, later in Veky's comment, there's a part about how "you can't even enter the circle of translators except if you are 'chosen'." I disagree. Well, the REAL reason that I'm writing is in the defense of the people in charge of the international side of the products. When I wrote them an email pointing out a mistake in translation, it usually got fixed. So it would be incorrect to paint an image of a Google that is too lazy, too skeptical, or for some other reason too reluctant to correct its language mistakes. International users matter a lot to Google. If it doesn't look like it, then this is just because there is just TOO MUCH to be translated and revised, and the few people at Google who speak (insert your not-super-common language here - like Portuguese) have things to do other than revise mountains of dubiously-translated material. Until that material gets revised, it's better to have that poor translation on our products and pages than no translation at all, right? But my point is, if you see a mistake, write Google about it (using the "contact us" on the website of the product/service where the mistake is), and they'll usually do something about it. As more and more international people are hired, and as more and more material is revised and updated, I am sure Google's foreign-language material will eventually be extremely well written. That's the current tendency anyways.

Google is working very hard right now to be as international as they can be - doing things like being sensitive to each region's culture while adapting a product to new regions and languages. The responsibility to be sensitive to local language and culture is not one Google takes lightly - there are a lot of very smart people dedicated to this. However, they are only human and have a LOT of work to do.

While I have some perspective into how the company operates, what it wants, how it makes decisions, etc, I would not share those insights publicly (I congratulate Ron and Doug on their courage). So I don't think I'd be comfortable sharing any more than I just did. Besides, almost all the work I did at Google was not very interesting at all. So don't bother asking me about it...

All right. Keep up this great blog!

11:28 AM  

 Bernardo said...

Ah, one more thing. Ron, you mentioned that the Translation Console languages numbered at "over 100 at last count, including Klingon!". But I don't think a user can ADD a new language to the console - only the Googlers in charge of the console can. I heard that Klingon, Elmer Fudd, Bork Bork Bork (Swedish Chef), Pig Latin, etc, were added while the console was being tested so that the people doing the testing and the debugging could understand the "languages" they were working on. I'm not sure whether or not this is true. (Is it?)

Some of the languages on the console are obscure enough that it would be hard for me to imagine they were suggested by the Googlers in charge of the console. Guarani, for example, is one of the languages spoken by Native Brazilian tribes. I wonder if a Brazilian person wrote in, saying they were willing to translate everything into Guarani, at which point the Googlers in charge of the console said "Why not" and added it. That to me seems to be the most likely explanation. (While working at end-user support, I once received an email from someone offering to translate the Google pages and interface into Qenya (Tolkien's Elvish language). I see this has not been implemented, probably because most computers do not have the Tengwar alphabet language pack...

11:40 AM  

 localudal said...

You say 'To my knowledge, there has not been even a single instance of translation spam that has ever gone live onto the Google site.'

Oh really? Who presented you with that 'priceless' knowledge? My estimate (see http://goolocalizations.blogspot.com, sections 'Goolocalizations' and 'Googlese') there's tons of 'translational spam' and 'junk translations' in Russian, Ukarainian, Japanese, Simplified Chinese. Someone with inflated 'page rank' and on a local competition assignment produced translational frauds, so, say, Google in Russian is just a silly joke compared to native Yandex.ru. Simplified Chinese of Google by the same scheme is either laughable, or illiterate, compared to native Baidu.com. Now, just for a second: Chinese market is no way Serbian, or Croatian, to believe so easily to any crook 'experts'.
This is the price of un-lingustic approach to translations and localizations, so even outstanding Python, Java, and whatever Lisp knowledges can't repair the damage done.

Besides, who said that professional (OK, maybe expensive, but never so costly as the loss of Chinese market) translators did poor job? Your miserable page rank of their Web site, lack of links to and from them? Who are you kidding with that?

Measurements of text qualities based on authors' claims/appearances/page ranks is a dangerous illusion. Say, I presume that page rank of the Google.com itself goes through the roof, right? In your scheme, it makes next to impossible then to really copyedit the junk American English used, say, in originals of AdSense/AdWords descriptions.

2:00 PM  

 Anonymous said...

all your base are belong to us

6:26 PM  

 http://search-engines-web.com/ said...

These scenarios appear to have been a result Poor Communication and lack of individual attention....

There HAD to have been a place at Google for you - but perhaps the Management style was just too solipsistic, perhaps everyone was just too focussed on SELF to see the entire perspective of things...

The poor communicationm, may have also been a result of being too individually self focussed...

It takes a superhuman to be Extremely technically skilled and also be a people-oriented generalist Manager....

This excerpt is really confusing...

I remember only two significant things from that first review. First, I didn't get a raise, which is customary (or at least so I'm told) unless you've really screwed up.

8:30 PM  

 Ron said...

I heard that Klingon, Elmer Fudd, Bork Bork Bork (Swedish Chef), Pig Latin, etc, were added while the console was being tested so that the people doing the testing and the debugging could understand the "languages" they were working on. I'm not sure whether or not this is true. (Is it?)

It certainly helped with testing, but that was not the main reason those "languages" were added to the list. They were always intended to go live, and indeed they were among the earliest languages to go live.

Some of the languages on the console are obscure enough that it would be hard for me to imagine they were suggested by the Googlers in charge of the console.

As best I can recall, we got a list of languages from some official-seeming website somewhere (maybe the UN or something like that?) I really don't remember. I just know that the list came from outside Google.

So what is this "research group" you're talking about? 

[Shrug] maybe they disbanded it. Don't forget this was five years ago. Or maybe they want to discourage academic-style researchers who think that publishing papers is a good measure of productivity. I don't know. All I know is that when I was there, there was a group called the research group and they did researchy kinds of things.

This excerpt is really confusing...

What I meant was this: Companies generally don't like to fire people except in the most egregious of circumstances because that exposes them to potential wrongful termination lawsuits. Instead they send subtle signals that you aren't working out and hope that you leave on your own initiative. Not giving someone a raise, even a small one, at their first performance review is one such signal.

9:45 PM  

 Anonymous said...

Google has hired a number of "research scientists," many of them belonging to the research team. (Try googling for "research scientist at google" with the quotes.) It's an impressive set of people.

9:51 AM  

 Anonymous said...

Oh, I know Google has hired a number people who are top research scientists. It's just that they don't call them researchers. They call them engineers. 

Check out the Google Jobs listing: Where is the "Research Department" on this list? 

Google Jobs

I see no research department. I see an Engineering department, yes. But no research. 

Maybe there really is a research department, too, but they're just not hiring right now? Is that why the "research lab" doesn't appear on the list? Maybe they've solved all the tough information retrieval problems with all the researchers they already have, and don't need any more? Uh huh.

Look, admittedly this is a minor point in the grand scheme of things. But it's one that gets under my skin. It's mild schizophrenia on Google's part.. to continually talk about how they only hire engineers, and then also talk about their fantastic research lab. Just call it for what it is, is all I'm saying. If a researcher has to do more coding at Google than in a comparable position at AT&T, IBM, or academia, just make that part of the job requirements. But you're still hiring a researcher, not an engineer.

10:39 AM  

 Anonymous said...

Whoever posted that it would take $100 million to keep Mountain View's Slater Elementary open didn't know what he was talking about. I have this from a VERY reliable source:

The cost of keeping Slater opened is estimated by the savings from reduced expenditures that would otherwise exist in keeping the school open. This savings in reduced expenditures is approximately $360,000. Additionally, the cost of keeping Slater open is the loss of the potential revenue of leasing the site. The potential revenue of leasing the site is estimated at $500,000. Therefore, the savings in expenditures and the estimated increase in revenue adds up to $830,000. This “cost” is ongoing; meaning, the district would need to find this amount not just for the 2006-07, but for every year thereafter.

Surely that is within reason and even easily doable by a small group of Googlers and/or Google corporate right? 

Who is the right person to ask about this?

10:49 AM  

 Anonymous said...

Ah, sorry, I'm the anonymous guy above, and I think I need to eat my words. A little bit more scratching on this particular itch led me to find, buried in a non-immediately obvious location in the engineering department, the following opening for "Research Scientist"

So I guess it does exist. I do stand corrected.

However, I also stand by the fact that some top Google tech folks I've spoken with really have said "Google does not do research and development; it does engineering and development". I still wish they would call it for what it is. But I suppose that's just part of the whole Google corporate personality. Anyway, back to this fascinating blog :-)

10:57 AM  

 The Irrational Investor said...

What about the "bio-info-nano" initiative? That's engineering too? It actually sounds scary.

10:32 PM  

 Greg said...

Regarding the "research" department at Google: back when Ron was at Google, there was a group of people we called "Research". These days, I don't know that most people think of there being an official research group any more, but researchers are still employed at Google (at least in the Mountain View and New York offices.) There are actually a couple of different groups that these people belong to (including, for example, the Systems Lab, which is where a lot of the Bell Labs people ended up.) And at least a few of the people I know from research have moved on (for example, the director of engineering in New York used to be in the research group.)

As far as the comment that Google doesn't hire researchers, only engineers, I believe that you could have been told that. But to clarify it a bit, Google doesn't want people who come up with ideas and then don't implement them: a researcher has to also implement at least a prototype of their system. So Google didn't want people who weren't willing to actually build something, and that's all that's meant by "engineering and development as opposed to research and development".

Maybe the research the anonymous poster posted about above was not concrete enough. Or maybe he/she was talking to the wrong people. But there certainly was a research group at Google, and without it, we wouldn't be seeing Froogle or Google News, among other contributions that might be less visible.

10:00 AM  

 Steve said...

I think this is a great blog. I'm very interested in the inner workings of Google, and somewhat angry with them right now since they kicked me out of Adsense because my friends and family clicked on my ads without my knowing it. Next time I will keep my mouth shut. But I digress...

It has been interesting to see Google rise from a little-known search engine to some kind of Internet... conglomerate, for lack of a better term... I don't really know what to call them now. Keep up the good work guys.

gonights.blogspot.com

6:42 PM  

 Anonymous said...

Greetings again.. I'm the anonymous "research" guy from above. I'd posted three times in a single day, so I decided to give it a rest for a few days, and let the majority of the readership move on, lest I come off as a troll. 

Anyway, in response to both Ron and Greg, above, I very much understood at the time, and still understand today, that Google is not interested in hiring someone who thinks productivity is measured in the number of papers published. 

My own area of research is multimedia information retrieval. In order to evaluate the research, I have had to build, over the years, a number of functioning prototype systems. With no system, there is no evaluation. With no evaluation, there are no results. With no results, you really don't have any research. So the prototyping component has always been a part of the research. That wasn't an issue.

So again, my main agitation was just that Google really just called everyone "engineers", and everything they did "engineering", even though I clearly knew that Google had hired researchers. 

If that were the only issue, I wouldn't be posting a comment again. However, in the past two days my mind has really started to change and clarify on this whole matter. Something clicked in me, when Google released Video. In just the past two days, I have begun to think that Google really does not do research. That they really only spend their efforts on engineering. Even with that 20% time. Let me explain.

See, when I talked to these senior Google managers back in 2002, the thing they told me about the research/engineering schedule is that the idea-to-prototyping schedule lasts 5 months. (I've since hear the same thing repeated from other sources, so I don't think I'm giving away any NDA here by talking about the 5-month cycle.)

In 5 months, Google expects you to come up with a new idea, work the kinks out of your idea, and then implement the idea in a working prototype, so carefully crafted that it could go live on the Google main page one or two months later, after being passed off to the hardcore coders. 

So whether or not Googles calls them "researchers" or "engineers", they basically only have a month or two to really be able to come up with new ideas, before you have to start hacking away, debugging, etc. to come up with this "soon-to-go-live" prototype. 

I think this is too little time to really think up and do the exploratory data analysis that you need to do, to come up with really great new ideas. Real research is not just about the prototyping. Real research is about having the time to do the exploration and bounce all the ideas around, to come up with something good. If you're worried about going live in four months, you just don't have -time- to do real research. You will be forever stuck in this shallow mode of slightly incremental improvement, but never a real breakthrough.

So I've been thinking about this in the context of the new Google Video store that was released just a few days ago. I've watched as the blogosphere has greeted the offering with a yawn. Everything from "We Sat Around Waiting For Google Video And All We Got Was This?" to "Google and CBS release embarrassment of a video store". 

And it makes me sad. 

I was there, in 2002, with my multimedia information retrieval experience, ready to buckle down and get some real ideas flowing, get some new, working notions going about how to index, retrieve, and present/display multimedia data. I was ready, with all that Google idealism, to help change the world. 

But even if I had applied to Google (which I didn't) and gotten an offer, it wouldn't have worked. If the expectation was that all the "research" someone in my position was doing had to go live to the Google main site every half year, then any research at all that was multimedia-related would not have been allowed... because Google Video itself didn't go live until just a few days ago. There would have been no place on the Google page to integrate those working prototypes, back in 2002, in 2003, 2004, and 2005. 

If Google did research, -really- did research, the kind that requires a dedication window larger than 5 months, they might actually have a video offering that wasn't so universally greeted as lackluster. 

And search, which I presume means all kinds of search, video included, is at the core of Google's mission statement. That is where the research had been needed. Not in developing interfaces for Google Talk. Not in incremental text search hacks such as letting you type in a Fedex tracking number. But in really developing the ideas (research) that will allow us to overcome the semantic gap in multimedia search.

And it makes me sad because what attracted me to Google in the first place, not just as a researcher but as a user, was this wonderful dedicated focus on search. Now, I feel like nothing has really happened in that area (esp multimedia search), when it should have been researched for years now. Maybe it has been. I just get the feeling it wasn't. 

Maybe there is a lot going on under the scenes. But.. with all that, where is my ability to do relevance feedback? This is a well-known, extremely useful IR technique. Why do we get GTalk before we get relevance feedback mechanisms, for example?

So Google.. if anyone is actually reading this post by this point.. you're a black hole. If you really are doing wonderful stuff, you gotta let us know somehow. If you're doing all this fantastic research, you gotta help us users see it, so that we get excited about it. As an outside observer, I see what you've been offering in the last 2+ years, and I don't see that you've really got anything that has pushed information retrieval forward.

10:39 PM  

 Eric Gometz said...

I read the comment above about Guarani being an "obscure language" spoken only by the native tribes of Brazil. Actually, it is the second official language of the country of Paraguay. In addition to Spanish, it is spoken by over 80% of all the people there. I am sure that the 6.3 million Paraguayans are happy that Guarani was included by the Googlers in charge of the console. Just FYI.

12:02 PM

 

THURSDAY, JANUARY 12, 2006

All your T-shirt are belong to us

POSTED BY RON AT 12:21 AM

 

Worldplay wrote:

 

Google: All your searches are belong to us!

 

which reminded me of something. It was at the height of the "All your base are belong to us" madness (and if you think that was weird you should take a look at this) when Doug held a little seminar to explain branding and marketing to us technodweebs over in engineering. The seminar was advertised with an incentive: everyone who attended would get "a special bonus prize" or something like that. The prize turned out to be a Google T-shirt with the slogan "All your search are belong to us" on the back. (Note that "search" has to be singular to remain true to the spirit of the thing.) We were admonished not to wear them in public.

 

Hm, I probably still have mine somewhere. I wonder how much I could get for it on eBay? ;-)

 

 

9 COMMENTS:

 Cole Blak Pascal said...

I dun have a lot of cash but I'd give you my bike for that shirt. Seriously.

2:50 AM  

 Anonymous said...

I think the ??? guy should get a shirt just for that comment.

4:27 AM  

 Ryan said...

Ron doesn't like when someone doesn't speak or write correctly. I think that "dun" disqualifies you.

Ryan

8:46 AM  

 Michael said...

The "All your base" phenomenon was certainly intriguing. My favorite sighting was in the index of Scott Meyers' Effective STL. It's on the second page of the index, between "algorithms" and "allocations". I think the indexer just slipped it in at the last minute.

11:51 AM  

 Doug said...

Yeah, that was one of the first non-standard logo shirts we made. I was surprised by the turnout for that Marketing 101 session and we quickly went through the allocated amount, pissing off some engineers who had been unable to attend. The whole point of the seminar was the power of marketing and having a collectible premium was intended to be part of the lesson. It worked pretty well. Btw, the first non-standard logo shirt was the one announcing the Yahoo deal. Susan put them together and they were given out at the party celebrating the signing. Jerry Yang was there, I think, but I was off on a family trip and missed the big event. I did manage to get a shirt though and have it stashed away somewhere with the rest of my collection. I'm off on another family trip, but if I get ambitious when I return, I'll post photos of all the t-shirts we made in the first few years. I tried to keep at least one of each. If nothing else, it should help future traders in Google logo gear establish provenance...

4:56 PM  

 Anonymous said...

Your 'you should take a look at this' link is one of the funniest things I have ever read. A cure for depression, indeed!

I rather suspect it would actually work, too.

11:16 PM  

 Ron said...

I rather suspect it would actually work, too.

It worked for me. (Reading the book, that is. I was laughing so hard that I couldn't be depressed!) ;-)

8:59 AM  

 Wordplay said...

You're kidding me.

You actually used that dumb phrase on a T-shirt? 

I want royalties baby!

7:31 PM  

 sarahintampa.com said...

Thanks a lot for the Amazon link. Fabulous. Now Amazon thinks I like books about my *^#@. Ugh.

5:03 PM

 

THURSDAY, JANUARY 12, 2006

2001 in a nutshell

POSTED BY RON AT 11:00 AM

 

After I finished the translation console I was quietly (and, as far as I can recall, unofficially) transferred back to the ads group. Like I said, my memory of this time (early 2001) is pretty fuzzy, and I seem to remember that official groups had been abolished by then, but I went back through my old notebooks and from February through June I was working on day-to-day operations of AdWords, with an occasional tweak on the Translation Console (though it never really needed a whole lot of intervention).

 

Google was growing at a frantic pace even as the dotcom world was melting down all around us. We were hiring like crazy and space started to be a real problem. People were being rearranged and stuffed three and four to an office or cubicle. By early summer it was, for the first time, pretty much impossible to know everyone else who worked at the company. Google was not a small company any more. The transformation had happened with breathtaking speed.

 

It was a very exciting time for the company, but a pretty dull time for me, and to be honest I was not unhappy to have a little dull spell after my first six months. I had settled into a nice comfortable if not particularly challenging groove, and I was worrying less about making it to my one-year anniversary when I could finally stop this crazy commute.

 

Urs Hoelzle, the guy who had hired me who had been VP of engineering, stepped down (or maybe it was up?) from that position and became the first "Google Fellow". He was replaced by Wayne Rosing, a Silicon Valley legend who, among other things, led the (alas, ill-fated) development of the Apple Lisa. In March, Eric Schmidt was hired as CEO, and for a couple of months his office was two doors down from mine. (I think he actually was sharing an office himself for a while, space was that tight.) I never got to know either Wayne or Eric very well because they were extremely busy, and I knew I was going to be leaving the company soon and didn't really want to waste their time.

 

June finally rolled around and I had my second performance review the week after my one-year anniversary, where I told Ross I was giving my two weeks notice. Once more, to my great surprise, Ross seemed dismayed. It seemed to me that the company had been sending me pretty clear signals that it wanted me out, and by then I was only too happy to oblige. But no. They wanted me to stay, despite the fact that at least by my own self-assessment I was contributing very little by then. I shrugged my shoulders and said, OK, if you want me to stay I'll stay, but I can't do the commute any more. You have to let me work from LA.

 

They agreed.

 

Well, actually we compromised. Instead of coming up to Mountain View four days a week I'd only have to come up one day a week and telecommute the other four, which seemed reasonable. Some things really require face to face contact. Even my wife was OK with the new arrangement since it meant I would no longer be away from home overnight.

 

I was pretty sure that the new arrangment wouldn't last long because Wayne Rosing was an big fan of colocation, even if it meant stuffing people into cubicles like sardines (which was pretty much what was happening by then). They moved my desk (which consisted of a door on top of two yellow sawhorses) out into the hallway, which suited me fine. I actualy had more privacy out in the hallway than I would have had in an office at that point. At least I could lean back in my chair without physically bumping into another person.

 

In early June I was also contacted by the high-level manager at JPL who had made me promise to come back there after a year asking me to make good on that promise. I had some reservations about going back. Things had not been going terribly well for me there before I left (that's a whole 'nuther story which I might write about some day) and things didn't seem to have changed a whole lot while I had been away. Since I still had the Google job I felt like I had a little leverage, so I told him I'd come back if I got a promotion.

 

I have to digress a bit here and explain a little about the technical career ladder at JPL. There were at the time four rungs on that ladder: Associate, Staff, Senior and Principal. Climbing the first three rungs was not particularly difficult (I was a Senior when I left) but getting promoted to Principal was a Big Deal because once you became a Principal you automatically got the ultimate JPL fringe benefit: on-lab parking.

 

JPL is at once the smallest and largest NASA center. It has the largest number of employees and the smallest physical space. Parking has been a serious problem pretty much throughout the lab's history. Only the privileged few (Principals, very senior managers, and people who have worked there for twenty-five years or more) get to park on-lab. The rest have to park in two remote lots and hike in to work or take a shuttle bus.

 

Even among on-lab parkers there is a pecking order. The parking lots are color-coded. At the bottom of the hierarchy is the green parking pass, which lets you park in the least desirable spaces. Then there is the blue lot, which is centrally located next to the main plaza. Then there are the legendary red and silver passes, which give you access to the only covered parking spaces in all of JPL. There are only about forty of these space in total. Less than one person out of a hundred gets to park there.

 

The upshot of all this is that getting promoted to Principal is really really hard. It's the equivalent of getting tenure at a university or making partner at a law firm. You have to be approved by multiple committees, and finally, by the JPL Executive Council, and the number of slots is extremely limited. Many highly qualified people get passed over year after year.

 

My insisting on getting my Principal promotion as a condition of returning had the effect of delaying JPL's ability to make me an offer as it shepherded the paperwork through the system, which suited me just fine as my Google situation had gotten fairly cushy, at least for the time being, and I was more than happy to wait and vest in a few more stock options.

 

You'd think that working from home would be an ideal situation, but the fact of the matter is that it gets pretty lonely. Face time is really important to building relationships, and email is a very poor substitute. June turned into July and July turned into August. There was one delay after another at JPL. Funding was tight. They wanted to promote me, but couldn't find the money to pay my salary. And Google was starting to hint that my days there were, finally, numbered. I was beginning to worry that the return to JPL could fall through and I'd end up unemployed, which would be financially disastrous. Coming up with the cash to pay for exercising the stock options on top of my commuting expenses had left us with virtually no cash cushion. We were living paycheck to paycheck.

 

Then, at about 8 AM Pacific time on September 11, 2001, I got this email:

 

 

To: ...@google.com

Subject: Is NY Alive?

 

In case you aren't watching... one trade center collapsed.

The pentagon is on fire.

turn on your tv.

 

 

WTF? I turned on the TV. I stared at it for a while in utter disbelief at what I was seeing, then went downstairs where my wife was making coffee.

 

She could tell just by looking at me that something was up. "What's wrong?" she asked.

 

I said, "It's going to be a bad day."

 

 

15 COMMENTS:

 Anonymous said...

Ron, Did you ever find out why they rejected your resignation (twice)?

3:23 PM  

 Ron said...

Did you ever find out why they rejected your resignation (twice)?

No, but stay tuned. The story is not quite done yet.

4:34 PM  

 http://search-engines-web.com/ said...

There are statements being made constantly about "feeling" that days were numbered - but there is no illustration of exactly Why those feeling were valid and what "specifically they were based on - besides not getting certain anticpated reactions - were your expectations too high about how explicitely you should be reinforced that you were a valuable commodity.

If nothing was adversely said - it could have just been everyone being - justifiably - extremely busy with themselves to constantly reinforce the more advanced people.
It may have been assumed that advanced people are not in need of great reassurance.

If better quality people were available - wouldn't they have hired and use them - with all the quality resumes they were getting???
~~~~~~~~~~~~~~~~~~~~~~
Could there also be posts about any initial concerns about Click Fraud becoming a problem - and what proactively was done to help prevent it in AdWords - and any Politics?

7:48 PM  

 Haridas Dave said...

Ron, yours posts are very engrossing, hope you write more than your normal quota for a day. Waiting eagerly for whats next !!!

7:53 PM  

 Ralf said...

Just wanted to stop by and say: I read this all the time and i really like it. Keep up the good work! Now, back to my code....

1:36 AM  

 Isaack said...

What a cliffhanger, you left us with!

6:06 AM  

 Ben said...

I kept reading, and the moment I looked at the scrollbar and realize it's coming to an end, I screamed internally that I wanted more. Keep writing. 

I'd also be interested to see how Google's been hinting to you that your days at google are numbered. what kinds of hints are they?

7:50 AM  

 Ron said...

I'd also be interested to see how Google's been hinting to you that your days at google are numbered. what kinds of hints are they?

Before June the main concrete indications were (as I mentioned in earlier posts) that I wasn't asked to do interviews any more, and that I wasn't given the customary token raise at my first performance review. On top of that was my own personal sense that I wasn't performing well or contributing much. After AdWords, most of the coding that needed to be done was in C++, and I just wasn't very proficient in it at the time. (To prepare for the performance review we had to fill out a form that included questions like, "List three areas where you need to improve." My answer for my second performance review was "1. C++ 2. C++ 3. C++".)

I don't know, maybe I was just being paranoid, and everything really was hunky dory. I'll probably never know.

After June, they were explicitly saying: Wayne wants everyone co-located, and so if you can't be here we will have to let you go before long, or words to that effect.

But like I said, stay tuned, there's more to the story.

8:47 AM  

 Ron said...

Oh, two more things:

1. Thanks to everyone for the kind words!

2. There was one other reason I thought my days were numbered as June approached: I knew I could no longer do the commute after June, and I wasn't expecting Google to allow me to telecommute full time because I knew that Wayne wanted everyone colocated. (That's why they were stuffing people in to the 'plex like sardines.)

8:51 AM  

 Anonymous said...

I always want to pat you on the back and tell you keep your chin up and things will get better. So far as your ending of this entry, we all remember where we were that day. I look forward to your future posts. Both of you have a comfortable style.

2:48 PM  

 Anonymous said...

Surprised you didn't just try to find some temp housing. During your time at Google apartment rent fell and there were many vacancies not far from the 'plex.

These posts have really been an eye-opener. It's good to see someone being frank and honest about what goes on within the company.

8:46 PM  

 Anonymous said...

hopefully, and more than likely googlers@google.com isnt a real distribution list. Otherwise, well - you can imagine

9:09 PM  

 Anonymous said...

hopefully, and more than likely googlers@google.com isnt a real distribution list. Otherwise, well - you can imagine 

Hey Anonymous, that address sounds like the real deal, or something close to it. I'd wager that most new tech companies have a list like that. But you should know that mailing list software for internal lists is usually set up to only allow subscribers to post. (As a side note, at the company where I worked at the time, we had an internal IRC channel that everyone used. As people logged in, we'd tell them personally...)

5:30 PM  

 Kunal said...

Hey - you work at JPL? Finally, a person to sympathize with about on-lab parking. We should do lunch sometime.

11:36 PM  

 Anonymous said...


But you should know that mailing list software for internal lists is usually set up to only allow subscribers to post.


Yes - but any one can fake a from address. I guess either way it would be easy to spam the entire googlers group if that is the case.

6:48 AM

 

FRIDAY, JANUARY 13, 2006

The real surreal life

POSTED BY RON AT 1:22 PM

 

9/11 was the start of the strangest five years of my life. It ranged from subtly weird to overtly bizarre. Example: at about 10 AM, an hour or so after I'd first turned on the TV and the magnitude of what was happening was beginning to sink in, one of our next door neighbors called -- I swear to God this is true -- to complain that our cat was defecating in his flower beds. Much later we learned that this person was having serious problems with chemical dependency, but at the time we had no idea.

 

Traumatic events weren't new to me. In February of 1991 I came home one night and surprised a burglar who took a pot-shot at me with a .38 on his way out the window. The bullet missed my left knee by about a foot, punched through one of the interior walls, ricocheted off a heating duct, and embedded itself in the living room wall. The odd thing about that kind of experience, I have found, is that it takes a long time -- like days or weeks -- to become consciously aware of the extent to which you've been traumatized. Immediately after it happened my heart was beating pretty fast, but I didn't feel panicked or anything like that. I crawled over to the phone and called 9-1-1, told the cops what had happened, etc. etc. The panic came later, with the nightmares. It was several years before I could really get a good night's sleep again. But the really insidious effect of post-traumatic stress was that it really shortened my fuse. I had never been particularly easy to get along with, but after that night and for a very long time I would say things to people that in my more lucid moments I would be amazed at. It was almost as if I -- or at least a piece of me -- had turned into a different person over which the rest of me had no control, although at the time I just felt like me. It's a very weird and difficult to describe sensation. Jekyll and Hyde were not entirely a product of Robert Louis Stevenson's imagination.

 

9/11 did that to me again, and again I wasn't aware of it. (And in retrospect I expect that's what happened to my neighbor too, although he probably helped himself along with a couple of pills.) The thing I did that I regret most was arguing to remove the Arabic translations from the Google site. Fortunately, cooler heads prevailed and the translations weren't touched, but there were a couple of hours when taking them down really seemed to me like the right thing to do.

 

The days immediately following 9/11 were oddly peaceful in LA. There were no planes in the air and less traffic than usual on the roads, and the weather was gorgeous. I like to think that this might have been a preview of what life will be like in a few decades when the world really starts to run out of oil, but that's a topic for another day.

 

A few days after 9/11 my wife and I got an invitation from a wealthy acquaintance of ours to spend the day at their house. Actually, calling it a house doesn't even begin to do it justice. It's an estate. I have been there several times and only ever seen about half of it. He was a bond trader and the markets were all closed, so he had some time on his hands. I can remember hanging out in the jacuzzi with his trophy wife (who has since left him -- money really can't buy happiness) thinking to myself how surreal this situation was. Two thousand miles away in New York people were digging themselves out of the rubble and here I was sitting in a hot tub with two gorgeous women drinking a glass of very expensive wine. It seemed horribly self-indulgent on the one hand, but on the other I didn't see how sitting at home moping was going to help anyone in New York.

 

(Fast-forward briefly to June of last year. We were vacationing in Europe, and landed at London Heathrow about an hour after the subway bombs went off. We were really struck by how little overall impact it seemed to have on the city. Our hotel was in central London, and when we drove in to town it was deserted, but by evening the busses were running again and things seemed pretty much back to normal by the following day. I don't know whether it's rational or rationalizing, but it does seem to me that the best way to thumb our noses at the terrorists is to live our lives. It worked for London.)

 

When they finally reopened the airports I did a few more commutes, but it was completely untenable. What before had been merely horribly inconvenient was now a serious nightmare. Fortunately, I was flying out of Burbank instead of LAX, so things could have been even worse. I shudder to think.

 

I don't know how long Google might have kept me on if 9/11 hadn't happened, but it was obviously the last little nudge that Google needed to finally let me go. My tenure there ended at the end of September. At the same time the process of getting me re-hired at JPL came to a dead stop. October rolled around with no word, and for the first time in my life I was unemployed.

 

I don't want to be overly dramatic about it because JPL did come through just a few weeks later, but let me tell you, if you've never been out of work, count your blessings (and if you have, I know how you feel). I had been working more or less continuously since high school (I got my first job programming computers when I was a sophomore) and I never in a million years imagined that I would end up on the dole. (I actually went to the umemployment office and picked up the forms to file for unemployment benefits, but I'm happy to report that I never turned them in.) It felt as if life had become a game of musical chairs and I was in the wrong place when the music stopped.

 

I finally got the word that my promotion had been approved. I breathed about a dozen sighs of relief before I got up the nerve to ask The Question, the one that a year and half earlier was the beginning of the end of my Google career: what would I be working on?

 

The answer came back: they wanted me to work on -- wait for it! -- search engines!

 

Aaarrrgghhh!

 

Fortunately, working on search engines at JPL doesn't mean the same thing as working on search engines at Google. Working on search engines at Google means you're working on a search engine. Working on search engines at JPL means you're working on procuring a search engine, and that was actually a topic I knew something about: call Google, place an order. (They had just come out with their search appliance.) You don't want to know how many of your tax dollars went to pay me to help shepherd purchase orders through JPL's procurement system.

 

Which brings me to one of the very few bits of juicy Google inside information that I have to share. (I'm able to share it because I learned it as a Google customer and not a Google employee. Furthermore, Google has been aware of this for four years now, and if they haven't fixed this problem by now then they deserve to be outed.)

 

We got one of the very first Google Search Appliances to evaluate. As everyone by now surely knows, the thing works by crawling your organization's intranet just like www.google.com does. But because this was an evaluation copy we would have to send it back to Google when we were done.

 

To address the obvious security problem (there were files available on the JPL intranet that contained sensitive information) the search appliance had a self-destruct feature that could be invoked to erase all the files on the device's internal hard drive. The problem was that self-destruct was a little too thorough. It erased everything, including the operating system, making the device unbootable. On top of that, the box was physically sealed with rivets. This made it impossible for us to verify that the information had in fact been erased.

 

I had the very devil of a time explaining to Google why this was a problem and why we could not just take their word for it that the self-destruct feature did what they said it did. We could not send the box back without actually verifying that the data had really been erased. That meant either getting root access to the machine, or physically opening the box and extracting the hard drive. Google suggested that we run a magnet over the box, and it took me a long time to explain to them why that wasn't good enough. (Just in case you don't understand the problem, it's this: it is very easy to make it appear that the data is gone without actually erasing it. For example, just erasing the boot sector on the drive will make the disk unbootable without actually erasing any of the data on the disk. Erasing the directory will make it appear that the files are gone to a casual inspection, but in fact all the data is still there.)

 

Google didn't want to give us root access because they didn't want us to be able to access their sensitive information (like their code). The solution seemed obvious to me: self-destruct mode should erase everything, including the code, but leave the machine bootable with no root password. That they didn't do this indicated to me that Google might be starting to lose some of their technical edge.

 

We finally reached the following compromise: they would send a technician down who would witness our running self-destruct mode. After this the tech would open up the box, take out the hard drive and leave it with us, and take the rest of the appliance back up to Mountain View. This seemed fine.

 

The tech arrived and together he and I pushed the Big Red Button, as it were. After launching self-destruct mode the machine put up a screen saying "Self-destruct in progress..." or words to that effect, with the colorful Google trademark bouncing ball animation showing that us that the machine was still alive.

 

Half and hour later the machine was still self-destructing. I asked the tech how we would know when it was done. He said that it should have already finished, and he didn't know what was going on.

 

After another fifteen minutes or so he pulled the plug on the machine. He plugged it back in, and it booted right up.

 

So much for self-destruct mode.

 

By now over an hour had gone by and the tech and I were both starting to feel a little impatient, so the tech logged in as root and typed "rm -rf /*'. (For those of you who don't know unix, that is the command that erases all the files -- except that it doesn't really erase them, it just erases the directory.) When that was done he opened up the box, took out the hard drive, and handed it to me.

 

All the data, including all of Google's code, was still on it. But we had already spent nearly two hours on this (or several weeks depending on how you count) and I really didn't want to drag it out any longer, so I just thanked him and walked him and the box back to the visitor center.

 

But I was starting to have some real doubts about the future viability of the company.

 

 

41 COMMENTS:

 Anonymous said...

Delightful, thanks Ron.

4:46 PM  

 Anonymous said...

well... what did you do with the code??? ;)

5:25 PM  

 Larry + Sergey said...

Another killer cliffhanger. Damed if this isn't the best blog on the whole stinking net.

5:48 PM  

 Anonymous said...

dd if=/dev/zero of=/dev/hda

Or better yet, plug the drive into another system running knoppix and do a few alternating writes with 0, 1, and rand.

Is anything stopping a customer from buying a unit and disassembling it?

6:28 PM  

 Andrew Hitchcock said...

the search appliance had a self-destruct feature

Oh :( . I read this line and was hoping it was a physical self-destruct. Perhaps in the next version...

6:34 PM  

 Dov said...

Amazing blog. I think it should be made into a thriller movie : )

Regarding this latest post about Google losing its edge, isn't a small incident like Google's refusal to change their appliance erase specifications a little too trivial to back up a "foreshadowing" statement like that?

Thanks for your story.

8:04 PM  

 http://search-engines-web.com/ said...

A colleague walking to a 9 to 5 job in the World Trade Center at
8:43 a.m., on that 70 degree, bright, sunny Septemeber Morning -
(just one block away from the office 17 minutes, I'll grab a cup of coffee to go)

then, all the pedestrians suddenly looking up the sky at that Low Cruising, Noisy Jet along lower Manhattan - minutes later crashing into the first Trade tower, people screaming and running out and all along the avenue...

then ultimately, dozens of office workers jumping out of windows many floors up with no other hope,

then the second crash ...

then the buildings collapsing...

the effect was a realization of how easily life and hard work can suddenly disappear, and the need to savor life's minutes and family more.

9:34 PM  

 Anonymous said...

Hmmm, quite a bind with the hard disk. Did anyone think about taking the hard disk out to the JPL carpark and smashing in with a hammer? Hard to recover data from that!

1:10 AM  

 Martin said...

"Damed if this isn't the best blog on the whole stinking net."

Sure is. Keep those updates coming!

3:37 AM  

 Anonymous said...

If Google already agreed that JPL keeps the HD, why did you guys insist on the the self-destruct? You could just take out the HD, send the HD-less appliance back to Google, and destroy the HD in whatever methods JPL deems fit without tying up a Google tech.

6:17 AM  

 Anonymous said...

hey, anonymous above me:
did you read the thing about google wanting to be sure not to give the harddisk with their secret code to the customer, hence the sealing of the psysical box?

6:21 AM  

 Eater said...

writing zeros with dd isn't really sufficent. 'shred -v /dev/{s,h}da' would be more appropriate, writing multiple patterns of pseudo-random data to the disk. increase the iterations with '-n' to satisfy your level of paranoia :)

6:49 AM  

 pBay said...

Nice blog guys, continue.

7:44 AM  

 Matt Bland said...

Why don't they use two drives. One locked inside the machine with their code on it. And another removable drive with your search data on it. Then all you need to do is remove the data drive and send the application box back. It's completely obvious to me that would be the solution for the next revision. Or alternatively put a SATA, USB, FireWire or SCSI port on the machine for using an external storage device for the search data as an option in the setup stage.

8:14 AM  

 Anonymous said...

The non-functioning self-distruct and shoddy tech work aside, this is a very interesting problem.

It seems like most evaluation appliance boxes would have a similar delimma. How do you protect your data and give the customer verifiable control over their data?

The dual hard drive idea makes some sense, but how does the customer know that you haven't pulled some of their data to the other drive?

Putting your code on a ROM might work, but then you can't easily patch it and you would constrained by size.

If your appliance is in customer hands, you can never guarantee 100% secrecy. Even if you decrypt it as you run, at some point the code is unencrypted. All you can do is increase the difficulty level.

8:38 AM  

 Brandon said...

Sounds to me like this's a time when a physical self-destruct would be helpful. Something which would physically destroy the HD and its contents, if not the entire machine.

8:38 AM  

 Anonymous said...

Realistically from your perspective keeping the harddrive is the only solution available to you.

Let's say their self-destruct feature followed the procedure you advocate, and keeps a root login. How do you know their OS hasn't been modified to report to you that the data has been deleted when, in fact, it still exists?

Nothing!

You either have to physically keep the harddrive, trust Google, or remove the harddrive, put it in a computer you trust and wipe it from there.

Doug

8:38 AM  

 Ron said...

why did you guys insist on the the self-destruct?

We didn't. That was Google's insistence.

9:01 AM  

 Ron said...

How do you know their OS hasn't been modified to report to you that the data has been deleted when, in fact, it still exists?

You always have to have a baseline level of trust or you quickly fall down a rabbit hole of paranoia. In the case of the self-destruct feature we had actual evidence up front that at least some people at Google "didn't get it", and that it was therefore plausible that the feature was not properly implemented, particularly since really erasing a hard drive properly is a decidedly non-trivial thing to do.

To modify the OS in the way you suggest would have required a significant covert effort on Google's part, one which, if discovered (which sooner or later it likely would have been), would cause significant damage to the company. At the very least it would have made a hash of their "don't be evil" slogan. So it would be a significant risk with very little potential benefit. It seemed safe to assume that Google would not put special effort into doing something that stupid. (I'll grant, though, that this is not always a safe assumption, c.f. the Sony rootkit.)

As an aside, I really believe that Google does do its best to live by "don't be evil". I think Larry and Sergey are idealists who in their heart of hearts really want to change the world for the better. (By way of contrast, I think Bill Gates is a ruthless businessman who just wants to maximize Microsoft's power, influence, and profit by whatever means he can get away with.) So I trust Google at least to try to do the right thing, but not necessarily always to succeed.

Is anything stopping a customer from buying a unit and disassembling it?

No, but this was an evaluation unit and did not belong to us. (We actually offered to buy it, but it was already an out of date model and Google didn't want to have to support it.) We later bought a box and someone at JPL did take it apart after the software license expired and it stopped working. It's just a generic rack-mount PC as far as we could tell.

9:26 AM  

 Rob L. said...

I think Bill Gates is a ruthless businessman who just wants to maximize Microsoft's power, influence, and profit by whatever means he can get away with

A common enough viewpoint, Ron, but in light of that what are your thoughts on the billions of dollars the Gateses putting towards world health through their foundation? The man is becoming or already is the biggest philanthropist in the history of humankind, isn't he? Is that nothing but a PR stunt?

Is it nothing compared to what we're going to see from Larry + Sergey?

10:38 AM  

 Ron said...

what are your thoughts on the billions of dollars the Gateses putting towards world health through their foundation?

I think it's completely irrelevant to the issue of whether or not Microsoft is trustworthy. I trust Google not because they are doing good things with their money, but because they made their money (and continue to make their money) by playing straight. Microsoft made their money playing crooked, and no amount of generosity can paper over that fact. (And I think that to the extent they're playing straight now it's not because they think it's the right thing to do but because they think they have no choice. I think they'd play crooked again in a minute if they thought they had half a chance of getting away with it.)

I also think that a person's generosity ought to be measured not by how much they give but by how much they have left after they give. A billion dollars is a lot of money, but if your net worth is tens of billions then giving away a billion here and there doesn't really call for tremendous sacrifice. To be fair, Bill is still a pretty young guy and I'm sure he's not done yet, but I'm not quite ready to nominate him for sainthood just because the numbers on the checks he writes are big enough to make most people's eyes bug out.

11:43 AM  

 SM Guy said...

The part about the being unemployed hit home.

Between Dec of 2002 and Sep of 2004 I was out of a job for a total of 18 months. I have a BS and MS in Computer Science (but I never wrote code for money, my expertise was elsewhere). 18 months is a mind boggling time to be out of work for someone that never even imagined he could be without a job. I still am shocked when I look at that number: 18 months.

I ran a small software company and we had to essentially shut down. So I had to fired my brother, cousin and 10 other employees (oh, that was fun) and tell 10 investors, their stock was worthless. And I got to watch every single penny I had to my name (in the high six figures) evaporate overnight because I had all my skin in the game.

Then I spent 8 months looking for a job. Got hired. My wife and I thought we could actually start to live a real life. Boy were we mistaken. 3 months later I was laid off from a well paying job and no severance pay. 10 months of sickening, mind-numbing agony followed. I sent out over 3,500 resumes over those two periods of being laid off (I am SERIOUS. three thousand five hundred). I contacted everyone I