The title of my talk is, “You and Your Research.” It is not about managing research, it is about how you individually do your research. I could give a talk on the other subject– but it’s not, it’s about you. I’m not talking about ordinary run-of-the-mill research; I’m talking about great research. And for the sake of describing great research I’ll occasionally say Nobel-Prize type of work. It doesn’t have to gain the Nobel Prize, but I mean those kinds of things which we perceive are significant things. Relativity, if you want, Shannon’s information theory, any number of outstanding theories– that’s the kind of thing I’m talking about.

Now, how did I come to do this study? At Los Alamos I was brought in to run the computing machines which other people had got going, so those scientists and physicists could get back to business. I saw I was a stooge. I saw that although physically I was the same, they were different. And to put the thing bluntly, I was envious. I wanted to know why they were so different from me. I saw Feynman up close. I saw Fermi and Teller. I saw Oppenheimer. I saw Hans Bethe: he was my boss. I saw quite a few very capable people. I became very interested in the difference between those who do and those who might have done.

When I came to Bell Labs, I came into a very productive department. Bode was the department head at the time; Shannon was there, and there were other people. I continued examining the questions, “Why?” and “What is the difference?” I continued subsequently by reading biographies, autobiographies, asking people questions such as: “How did you come to do this?” I tried to find out what are the differences. And that’s what this talk is about.

Now, why is this talk important? I think it is important because, as far as I know, each of you has one life to live. Even if you believe in reincarnation it doesn’t do you any good from one life to the next! Why shouldn’t you do significant things in this one life, however you define significant? I’m not going to define it – you know what I mean. I will talk mainly about science because that is what I have studied. But so far as I know, and I’ve been told by others, much of what I say applies to many fields. Outstanding work is characterized very much the same way in most fields, but I will confine myself to science.

In order to get at you individually, I must talk in the first person. I have to get you to drop modesty and say to yourself, “Yes, I would like to do first-class work.” Our society frowns on people who set out to do really good work. You’re not supposed to; luck is supposed to descend on you and you do great things by chance. Well, that’s a kind of dumb thing to say. I say, why shouldn’t you set out to do something significant. You don’t have to tell other people, but shouldn’t you say to yourself, “Yes, I would like to do something significant.”

In order to get to the second stage, I have to drop modesty and talk in the first person about what I’ve seen, what I’ve done, and what I’ve heard. I’m going to talk about people, some of whom you know, and I trust that when we leave, you won’t quote me as saying some of the things I said.

Let me start not logically, but psychologically. I find that the major objection is that people think great science is done by luck. It’s all a matter of luck. Well, consider Einstein. Note how many different things he did that were good. Was it all luck? Wasn’t it a little too repetitive? Consider Shannon. He didn’t do just information theory. Several years before, he did some other good things and some which are still locked up in the security of cryptography. He did many good things.

You see again and again, that it is more than one thing from a good person. Once in a while a person does only one thing in his whole life, and we’ll talk about that later, but a lot of times there is repetition. I claim that luck will not cover everything. And I will cite Pasteur who said, “Luck favors the prepared mind.” And I think that says it the way I believe it. There is indeed an element of luck, and no, there isn’t. The prepared mind sooner or later finds something important and does it. So yes, it is luck. The particular thing you do is luck, but that you do something is not.

For example, when I came to Bell Labs, I shared an office for a while with Shannon. At the same time he was doing information theory, I was doing coding theory. It is suspicious that the two of us did it at the same place and at the same time – it was in the atmosphere. And you can say, “Yes, it was luck.” On the other hand you can say, “But why of all the people in Bell Labs then were those the two who did it?” Yes, it is partly luck, and partly it is the prepared mind; but `partly’ is the other thing I’m going to talk about. So, although I’ll come back several more times to luck, I want to dispose of this matter of luck as being the sole criterion whether you do great work or not. I claim you have some, but not total, control over it. And I will quote, finally, Newton on the matter. Newton said, “If others would think as hard as I did, then they would get similar results.”

One of the characteristics you see, and many people have it including great scientists, is that usually when they were young they had independent thoughts and had the courage to pursue them. For example, Einstein, somewhere around 12 or 14, asked himself the question, “What would a light wave look like if I went with the velocity of light to look at it?” Now he knew that electromagnetic theory says you cannot have a stationary local maximum. But if he moved along with the velocity of light, he would see a local maximum. He could see a contradiction at the age of 12, 14, or somewhere around there, that everything was not right and that the velocity of light had something peculiar. Is it luck that he finally created special relativity? Early on, he had laid down some of the pieces by thinking of the fragments. Now that’s the necessary but not sufficient condition. All of these items I will talk about are both luck and not luck.

How about having lots of `brains?’ It sounds good. Most of you in this room probably have more than enough brains to do first-class work. But great work is something else than mere brains. Brains are measured in various ways. In mathematics, theoretical physics, astrophysics, typically brains correlates to a great extent with the ability to manipulate symbols. And so the typical IQ test is apt to score them fairly high. On the other hand, in other fields it is something different. For example, Bill Pfann, the fellow who did zone melting, came into my office one day. He had this idea dimly in his mind about what he wanted and he had some equations. It was pretty clear to me that this man didn’t know much mathematics and he wasn’t really articulate. His problem seemed interesting so I took it home and did a little work. I finally showed him how to run computers so he could compute his own answers. I gave him the power to compute. He went ahead, with negligible recognition from his own department, but ultimately he has collected all the prizes in the field. Once he got well started, his shyness, his awkwardness, his inarticulateness, fell away and he became much more productive in many other ways. Certainly he became much more articulate.

And I can cite another person in the same way. I trust he isn’t in the audience, i.e. a fellow named Clogston. I met him when I was working on a problem with John Pierce’s group and I didn’t think he had much. I asked my friends who had been with him at school, “Was he like that in graduate school?” “Yes,” they replied. Well I would have fired the fellow, but J. R. Pierce was smart and kept him on. Clogston finally did the Clogston cable. After that there was a steady stream of good ideas. One success brought him confidence and courage.

One of the characteristics of successful scientists is having courage. Once you get your courage up and believe that you can do important problems, then you can. If you think you can’t, almost surely you are not going to. Courage is one of the things that Shannon had supremely. You have only to think of his major theorem. He wants to create a method of coding, but he doesn’t know what to do so he makes a random code. Then he is stuck. And then he asks the impossible question, “What would the average random code do?” He then proves that the average code is arbitrarily good, and that therefore there must be at least one good code. Who but a man of infinite courage could have dared to think those thoughts? That is the characteristic of great scientists; they have courage. They will go forward under incredible circumstances; they think and continue to think.

Age is another factor which the physicists particularly worry about. They always are saying that you have got to do it when you are young or you will never do it. Einstein did things very early, and all the quantum mechanic fellows were disgustingly young when they did their best work. Most mathematicians, theoretical physicists, and astrophysicists do what we consider their best work when they are young. It is not that they don’t do good work in their old age but what we value most is often what they did early. On the other hand, in music, politics and literature, often what we consider their best work was done late. I don’t know how whatever field you are in fits this scale, but age has some effect.

But let me say why age seems to have the effect it does. In the first place if you do some good work you will find yourself on all kinds of committees and unable to do any more work. You may find yourself as I saw Brattain when he got a Nobel Prize. The day the prize was announced we all assembled in Arnold Auditorium; all three winners got up and made speeches. The third one, Brattain, practically with tears in his eyes, said, “I know about this Nobel-Prize effect and I am not going to let it affect me; I am going to remain good old Walter Brattain.” Well I said to myself, “That is nice.” But in a few weeks I saw it was affecting him. Now he could only work on great problems.

When you are famous it is hard to work on small problems. This is what did Shannon in. After information theory, what do you do for an encore? The great scientists often make this error. They fail to continue to plant the little acorns from which the mighty oak trees grow. They try to get the big thing right off. And that isn’t the way things go. So that is another reason why you find that when you get early recognition it seems to sterilize you. In fact I will give you my favorite quotation of many years. The Institute for Advanced Study in Princeton, in my opinion, has ruined more good scientists than any institution has created, judged by what they did before they came and judged by what they did after. Not that they weren’t good afterwards, but they were superb before they got there and were only good afterwards.

This brings up the subject, out of order perhaps, of working conditions. What most people think are the best working conditions, are not. Very clearly they are not because people are often most productive when working conditions are bad. One of the better times of the Cambridge Physical Laboratories was when they had practically shacks – they did some of the best physics ever.

I give you a story from my own private life. Early on it became evident to me that Bell Laboratories was not going to give me the conventional acre of programming people to program computing machines in absolute binary. It was clear they weren’t going to. But that was the way everybody did it. I could go to the West Coast and get a job with the airplane companies without any trouble, but the exciting people were at Bell Labs and the fellows out there in the airplane companies were not. I thought for a long while about, “Did I want to go or not?” and I wondered how I could get the best of two possible worlds. I finally said to myself, “Hamming, you think the machines can do practically everything. Why can’t you make them write programs?” What appeared at first to me as a defect forced me into automatic programming very early. What appears to be a fault, often, by a change of viewpoint, turns out to be one of the greatest assets you can have. But you are not likely to think that when you first look the thing and say, “Gee, I’m never going to get enough programmers, so how can I ever do any great programming?”

And there are many other stories of the same kind; Grace Hopper has similar ones. I think that if you look carefully you will see that often the great scientists, by turning the problem around a bit, changed a defect to an asset. For example, many scientists when they found they couldn’t do a problem finally began to study why not. They then turned it around the other way and said, “But of course, this is what it is” and got an important result. So ideal working conditions are very strange. The ones you want aren’t always the best ones for you.

Now for the matter of drive. You observe that most great scientists have tremendous drive. I worked for ten years with John Tukey at Bell Labs. He had tremendous drive. One day about three or four years after I joined, I discovered that John Tukey was slightly younger than I was. John was a genius and I clearly was not. Well I went storming into Bode’s office and said, “How can anybody my age know as much as John Tukey does?” He leaned back in his chair, put his hands behind his head, grinned slightly, and said, “You would be surprised Hamming, how much you would know if you worked as hard as he did that many years.” I simply slunk out of the office!

What Bode was saying was this: “Knowledge and productivity are like compound interest.” Given two people of approximately the same ability and one person who works ten percent more than the other, the latter will more than twice outproduce the former. The more you know, the more you learn; the more you learn, the more you can do; the more you can do, the more the opportunity – it is very much like compound interest. I don’t want to give you a rate, but it is a very high rate. Given two people with exactly the same ability, the one person who manages day in and day out to get in one more hour of thinking will be tremendously more productive over a lifetime. I took Bode’s remark to heart; I spent a good deal more of my time for some years trying to work a bit harder and I found, in fact, I could get more work done. I don’t like to say it in front of my wife, but I did sort of neglect her sometimes; I needed to study. You have to neglect things if you intend to get what you want done. There’s no question about this.

On this matter of drive Edison says, “Genius is 99% perspiration and 1% inspiration.” He may have been exaggerating, but the idea is that solid work, steadily applied, gets you surprisingly far. The steady application of effort with a little bit more work, intelligently applied is what does it. That’s the trouble; drive, misapplied, doesn’t get you anywhere. I’ve often wondered why so many of my good friends at Bell Labs who worked as hard or harder than I did, didn’t have so much to show for it. The misapplication of effort is a very serious matter. Just hard work is not enough – it must be applied sensibly.

There’s another trait on the side which I want to talk about; that trait is ambiguity. It took me a while to discover its importance. Most people like to believe something is or is not true. Great scientists tolerate ambiguity very well. They believe the theory enough to go ahead; they doubt it enough to notice the errors and faults so they can step forward and create the new replacement theory. If you believe too much you’ll never notice the flaws; if you doubt too much you won’t get started. It requires a lovely balance. But most great scientists are well aware of why their theories are true and they are also well aware of some slight misfits which don’t quite fit and they don’t forget it. Darwin writes in his autobiography that he found it necessary to write down every piece of evidence which appeared to contradict his beliefs because otherwise they would disappear from his mind. When you find apparent flaws you’ve got to be sensitive and keep track of those things, and keep an eye out for how they can be explained or how the theory can be changed to fit them. Those are often the great contributions. Great contributions are rarely done by adding another decimal place. It comes down to an emotional commitment. Most great scientists are completely committed to their problem. Those who don’t become committed seldom produce outstanding, first-class work.

Now again, emotional commitment is not enough. It is a necessary condition apparently. And I think I can tell you the reason why. Everybody who has studied creativity is driven finally to saying, “creativity comes out of your subconscious.” Somehow, suddenly, there it is. It just appears. Well, we know very little about the subconscious; but one thing you are pretty well aware of is that your dreams also come out of your subconscious. And you’re aware your dreams are, to a fair extent, a reworking of the experiences of the day. If you are deeply immersed and committed to a topic, day after day after day, your subconscious has nothing to do but work on your problem. And so you wake up one morning, or on some afternoon, and there’s the answer. For those who don’t get committed to their current problem, the subconscious goofs off on other things and doesn’t produce the big result. So the way to manage yourself is that when you have a real important problem you don’t let anything else get the center of your attention – you keep your thoughts on the problem. Keep your subconscious starved so it has to work on your problem, so you can sleep peacefully and get the answer in the morning, free.

Now Alan Chynoweth mentioned that I used to eat at the physics table. I had been eating with the mathematicians and I found out that I already knew a fair amount of mathematics; in fact, I wasn’t learning much. The physics table was, as he said, an exciting place, but I think he exaggerated on how much I contributed. It was very interesting to listen to Shockley, Brattain, Bardeen, J. B. Johnson, Ken McKay and other people, and I was learning a lot. But unfortunately a Nobel Prize came, and a promotion came, and what was left was the dregs. Nobody wanted what was left. Well, there was no use eating with them!

Over on the other side of the dining hall was a chemistry table. I had worked with one of the fellows, Dave McCall; furthermore he was courting our secretary at the time. I went over and said, “Do you mind if I join you?” They can’t say no, so I started eating with them for a while. And I started asking, “What are the important problems of your field?” And after a week or so, “What important problems are you working on?” And after some more time I came in one day and said, “If what you are doing is not important, and if you don’t think it is going to lead to something important, why are you at Bell Labs working on it?” I wasn’t welcomed after that; I had to find somebody else to eat with! That was in the spring.

In the fall, Dave McCall stopped me in the hall and said, “Hamming, that remark of yours got underneath my skin. I thought about it all summer, i.e. what were the important problems in my field. I haven’t changed my research,” he says, “but I think it was well worthwhile.” And I said, “Thank you Dave,” and went on. I noticed a couple of months later he was made the head of the department. I noticed the other day he was a Member of the National Academy of Engineering. I noticed he has succeeded. I have never heard the names of any of the other fellows at that table mentioned in science and scientific circles. They were unable to ask themselves, “What are the important problems in my field?”

If you do not work on an important problem, it’s unlikely you’ll do important work. It’s perfectly obvious. Great scientists have thought through, in a careful way, a number of important problems in their field, and they keep an eye on wondering how to attack them. Let me warn you, `important problem’ must be phrased carefully. The three outstanding problems in physics, in a certain sense, were never worked on while I was at Bell Labs. By important I mean guaranteed a Nobel Prize and any sum of money you want to mention. We didn’t work on (1) time travel, (2) teleportation, and (3) antigravity. They are not important problems because we do not have an attack. It’s not the consequence that makes a problem important, it is that you have a reasonable attack. That is what makes a problem important. When I say that most scientists don’t work on important problems, I mean it in that sense. The average scientist, so far as I can make out, spends almost all his time working on problems which he believes will not be important and he also doesn’t believe that they will lead to important problems.

I spoke earlier about planting acorns so that oaks will grow. You can’t always know exactly where to be, but you can keep active in places where something might happen. And even if you believe that great science is a matter of luck, you can stand on a mountain top where lightning strikes; you don’t have to hide in the valley where you’re safe. But the average scientist does routine safe work almost all the time and so he (or she) doesn’t produce much. It’s that simple. If you want to do great work, you clearly must work on important problems, and you should have an idea.

Along those lines at some urging from John Tukey and others, I finally adopted what I called “Great Thoughts Time.” When I went to lunch Friday noon, I would only discuss great thoughts after that. By great thoughts I mean ones like: “What will be the role of computers in all of AT&T?”, “How will computers change science?” For example, I came up with the observation at that time that nine out of ten experiments were done in the lab and one in ten on the computer. I made a remark to the vice presidents one time, that it would be reversed, i.e. nine out of ten experiments would be done on the computer and one in ten in the lab. They knew I was a crazy mathematician and had no sense of reality. I knew they were wrong and they’ve been proved wrong while I have been proved right. They built laboratories when they didn’t need them. I saw that computers were transforming science because I spent a lot of time asking “What will be the impact of computers on science and how can I change it?” I asked myself, “How is it going to change Bell Labs?” I remarked one time, in the same address, that more than one-half of the people at Bell Labs will be interacting closely with computing machines before I leave. Well, you all have terminals now. I thought hard about where was my field going, where were the opportunities, and what were the important things to do. Let me go there so there is a chance I can do important things.

Most great scientists know many important problems. They have something between 10 and 20 important problems for which they are looking for an attack. And when they see a new idea come up, one hears them say “Well that bears on this problem.” They drop all the other things and get after it. Now I can tell you a horror story that was told to me but I can’t vouch for the truth of it. I was sitting in an airport talking to a friend of mine from Los Alamos about how it was lucky that the fission experiment occurred over in Europe when it did because that got us working on the atomic bomb here in the US. He said “No; at Berkeley we had gathered a bunch of data; we didn’t get around to reducing it because we were building some more equipment, but if we had reduced that data we would have found fission.” They had it in their hands and they didn’t pursue it. They came in second!

The great scientists, when an opportunity opens up, get after it and they pursue it. They drop all other things. They get rid of other things and they get after an idea because they had already thought the thing through. Their minds are prepared; they see the opportunity and they go after it. Now of course lots of times it doesn’t work out, but you don’t have to hit many of them to do some great science. It’s kind of easy. One of the chief tricks is to live a long time!

Another trait, it took me a while to notice. I noticed the following facts about people who work with the door open or the door closed. I notice that if you have the door to your office closed, you get more work done today and tomorrow, and you are more productive than most. But 10 years later somehow you don’t know quite know what problems are worth working on; all the hard work you do is sort of tangential in importance. He who works with the door open gets all kinds of interruptions, but he also occasionally gets clues as to what the world is and what might be important. Now I cannot prove the cause and effect sequence because you might say, “The closed door is symbolic of a closed mind.” I don’t know. But I can say there is a pretty good correlation between those who work with the doors open and those who ultimately do important things, although people who work with doors closed often work harder. Somehow they seem to work on slightly the wrong thing – not much, but enough that they miss fame.

I want to talk on another topic. It is based on the song which I think many of you know, “It ain’t what you do, it’s the way that you do it.” I’ll start with an example of my own. I was conned into doing on a digital computer, in the absolute binary days, a problem which the best analog computers couldn’t do. And I was getting an answer. When I thought carefully and said to myself, “You know, Hamming, you’re going to have to file a report on this military job; after you spend a lot of money you’re going to have to account for it and every analog installation is going to want the report to see if they can’t find flaws in it.” I was doing the required integration by a rather crummy method, to say the least, but I was getting the answer. And I realized that in truth the problem was not just to get the answer; it was to demonstrate for the first time, and beyond question, that I could beat the analog computer on its own ground with a digital machine. I reworked the method of solution, created a theory which was nice and elegant, and changed the way we computed the answer; the results were no different. The published report had an elegant method which was later known for years as “Hamming’s Method of Integrating Differential Equations.” It is somewhat obsolete now, but for a while it was a very good method. By changing the problem slightly, I did important work rather than trivial work.

In the same way, when using the machine up in the attic in the early days, I was solving one problem after another after another; a fair number were successful and there were a few failures. I went home one Friday after finishing a problem, and curiously enough I wasn’t happy; I was depressed. I could see life being a long sequence of one problem after another after another. After quite a while of thinking I decided, “No, I should be in the mass production of a variable product. I should be concerned with all of next year’s problems, not just the one in front of my face.” By changing the question I still got the same kind of results or better, but I changed things and did important work. I attacked the major problem – How do I conquer machines and do all of next year’s problems when I don’t know what they are going to be? How do I prepare for it? How do I do this one so I’ll be on top of it? How do I obey Newton’s rule? He said, “If I have seen further than others, it is because I’ve stood on the shoulders of giants.” These days we stand on each other’s feet!

You should do your job in such a fashion that others can build on top of it, so they will indeed say, “Yes, I’ve stood on so and so’s shoulders and I saw further.” The essence of science is cumulative. By changing a problem slightly you can often do great work rather than merely good work. Instead of attacking isolated problems, I made the resolution that I would never again solve an isolated problem except as characteristic of a class.

Now if you are much of a mathematician you know that the effort to generalize often means that the solution is simple. Often by stopping and saying, “This is the problem he wants but this is characteristic of so and so. Yes, I can attack the whole class with a far superior method than the particular one because I was earlier embedded in needless detail.” The business of abstraction frequently makes things simple. Furthermore, I filed away the methods and prepared for the future problems.

To end this part, I’ll remind you, “It is a poor workman who blames his tools – the good man gets on with the job, given what he’s got, and gets the best answer he can.” And I suggest that by altering the problem, by looking at the thing differently, you can make a great deal of difference in your final productivity because you can either do it in such a fashion that people can indeed build on what you’ve done, or you can do it in such a fashion that the next person has to essentially duplicate again what you’ve done. It isn’t just a matter of the job, it’s the way you write the report, the way you write the paper, the whole attitude. It’s just as easy to do a broad, general job as one very special case. And it’s much more satisfying and rewarding!

I have now come down to a topic which is very distasteful; it is not sufficient to do a job, you have to sell it. `Selling’ to a scientist is an awkward thing to do. It’s very ugly; you shouldn’t have to do it. The world is supposed to be waiting, and when you do something great, they should rush out and welcome it. But the fact is everyone is busy with their own work. You must present it so well that they will set aside what they are doing, look at what you’ve done, read it, and come back and say, “Yes, that was good.” I suggest that when you open a journal, as you turn the pages, you ask why you read some articles and not others. You had better write your report so when it is published in the Physical Review, or wherever else you want it, as the readers are turning the pages they won’t just turn your pages but they will stop and read yours. If they don’t stop and read it, you won’t get credit.

There are three things you have to do in selling. You have to learn to write clearly and well so that people will read it, you must learn to give reasonably formal talks, and you also must learn to give informal talks. We had a lot of so-called `back room scientists.’ In a conference, they would keep quiet. Three weeks later after a decision was made they filed a report saying why you should do so and so. Well, it was too late. They would not stand up right in the middle of a hot conference, in the middle of activity, and say, “We should do this for these reasons.” You need to master that form of communication as well as prepared speeches.

When I first started, I got practically physically ill while giving a speech, and I was very, very nervous. I realized I either had to learn to give speeches smoothly or I would essentially partially cripple my whole career. The first time IBM asked me to give a speech in New York one evening, I decided I was going to give a really good speech, a speech that was wanted, not a technical one but a broad one, and at the end if they liked it, I’d quietly say, “Any time you want one I’ll come in and give you one.” As a result, I got a great deal of practice giving speeches to a limited audience and I got over being afraid. Furthermore, I could also then study what methods were effective and what were ineffective.

While going to meetings I had already been studying why some papers are remembered and most are not. The technical person wants to give a highly limited technical talk. Most of the time the audience wants a broad general talk and wants much more survey and background than the speaker is willing to give. As a result, many talks are ineffective. The speaker names a topic and suddenly plunges into the details he’s solved. Few people in the audience may follow. You should paint a general picture to say why it’s important, and then slowly give a sketch of what was done. Then a larger number of people will say, “Yes, Joe has done that,” or “Mary has done that; I really see where it is; yes, Mary really gave a good talk; I understand what Mary has done.” The tendency is to give a highly restricted, safe talk; this is usually ineffective. Furthermore, many talks are filled with far too much information. So I say this idea of selling is obvious.

Let me summarize. You’ve got to work on important problems. I deny that it is all luck, but I admit there is a fair element of luck. I subscribe to Pasteur’s “Luck favors the prepared mind.” I favor heavily what I did. Friday afternoons for years – great thoughts only – means that I committed 10% of my time trying to understand the bigger problems in the field, i.e. what was and what was not important. I found in the early days I had believed this' and yet had spent all week marching inthat’ direction. It was kind of foolish. If I really believe the action is over there, why do I march in this direction? I either had to change my goal or change what I did. So I changed something I did and I marched in the direction I thought was important. It’s that easy.

Now you might tell me you haven’t got control over what you have to work on. Well, when you first begin, you may not. But once you’re moderately successful, there are more people asking for results than you can deliver and you have some power of choice, but not completely. I’ll tell you a story about that, and it bears on the subject of educating your boss. I had a boss named Schelkunoff; he was, and still is, a very good friend of mine. Some military person came to me and demanded some answers by Friday. Well, I had already dedicated my computing resources to reducing data on the fly for a group of scientists; I was knee deep in short, small, important problems. This military person wanted me to solve his problem by the end of the day on Friday. I said, “No, I’ll give it to you Monday. I can work on it over the weekend. I’m not going to do it now.” He goes down to my boss, Schelkunoff, and Schelkunoff says, “You must run this for him; he’s got to have it by Friday.” I tell him, “Why do I?”; he says, “You have to.” I said, “Fine, Sergei, but you’re sitting in your office Friday afternoon catching the late bus home to watch as this fellow walks out that door.” I gave the military person the answers late Friday afternoon. I then went to Schelkunoff’s office and sat down; as the man goes out I say, “You see Schelkunoff, this fellow has nothing under his arm; but I gave him the answers.” On Monday morning Schelkunoff called him up and said, “Did you come in to work over the weekend?” I could hear, as it were, a pause as the fellow ran through his mind of what was going to happen; but he knew he would have had to sign in, and he’d better not say he had when he hadn’t, so he said he hadn’t. Ever after that Schelkunoff said, “You set your deadlines; you can change them.”

One lesson was sufficient to educate my boss as to why I didn’t want to do big jobs that displaced exploratory research and why I was justified in not doing crash jobs which absorb all the research computing facilities. I wanted instead to use the facilities to compute a large number of small problems. Again, in the early days, I was limited in computing capacity and it was clear, in my area, that a “mathematician had no use for machines.” But I needed more machine capacity. Every time I had to tell some scientist in some other area, “No I can’t; I haven’t the machine capacity,” he complained. I said “Go tell your Vice President that Hamming needs more computing capacity.” After a while I could see what was happening up there at the top; many people said to my Vice President, “Your man needs more computing capacity.” I got it!

I also did a second thing. When I loaned what little programming power we had to help in the early days of computing, I said, “We are not getting the recognition for our programmers that they deserve. When you publish a paper you will thank that programmer or you aren’t getting any more help from me. That programmer is going to be thanked by name; she’s worked hard.” I waited a couple of years. I then went through a year of BSTJ articles and counted what fraction thanked some programmer. I took it into the boss and said, “That’s the central role computing is playing in Bell Labs; if the BSTJ is important, that’s how important computing is.” He had to give in. You can educate your bosses. It’s a hard job. In this talk I’m only viewing from the bottom up; I’m not viewing from the top down. But I am telling you how you can get what you want in spite of top management. You have to sell your ideas there also.

Well I now come down to the topic, “Is the effort to be a great scientist worth it?” To answer this, you must ask people. When you get beyond their modesty, most people will say, “Yes, doing really first-class work, and knowing it, is as good as wine, women and song put together,” or if it’s a woman she says, “It is as good as wine, men and song put together.” And if you look at the bosses, they tend to come back or ask for reports, trying to participate in those moments of discovery. They’re always in the way. So evidently those who have done it, want to do it again. But it is a limited survey. I have never dared to go out and ask those who didn’t do great work how they felt about the matter. It’s a biased sample, but I still think it is worth the struggle. I think it is very definitely worth the struggle to try and do first-class work because the truth is, the value is in the struggle more than it is in the result. The struggle to make something of yourself seems to be worthwhile in itself. The success and fame are sort of dividends, in my opinion.

I’ve told you how to do it. It is so easy, so why do so many people, with all their talents, fail? For example, my opinion, to this day, is that there are in the mathematics department at Bell Labs quite a few people far more able and far better endowed than I, but they didn’t produce as much. Some of them did produce more than I did; Shannon produced more than I did, and some others produced a lot, but I was highly productive against a lot of other fellows who were better equipped. Why is it so? What happened to them? Why do so many of the people who have great promise, fail?

Well, one of the reasons is drive and commitment. The people who do great work with less ability but who are committed to it, get more done that those who have great skill and dabble in it, who work during the day and go home and do other things and come back and work the next day. They don’t have the deep commitment that is apparently necessary for really first-class work. They turn out lots of good work, but we were talking, remember, about first-class work. There is a difference. Good people, very talented people, almost always turn out good work. We’re talking about the outstanding work, the type of work that gets the Nobel Prize and gets recognition.

The second thing is, I think, the problem of personality defects. Now I’ll cite a fellow whom I met out in Irvine. He had been the head of a computing center and he was temporarily on assignment as a special assistant to the president of the university. It was obvious he had a job with a great future. He took me into his office one time and showed me his method of getting letters done and how he took care of his correspondence. He pointed out how inefficient the secretary was. He kept all his letters stacked around there; he knew where everything was. And he would, on his word processor, get the letter out. He was bragging how marvelous it was and how he could get so much more work done without the secretary’s interference. Well, behind his back, I talked to the secretary. The secretary said, “Of course I can’t help him; I don’t get his mail. He won’t give me the stuff to log in; I don’t know where he puts it on the floor. Of course I can’t help him.” So I went to him and said, “Look, if you adopt the present method and do what you can do single-handedly, you can go just that far and no farther than you can do single-handedly. If you will learn to work with the system, you can go as far as the system will support you.” And, he never went any further. He had his personality defect of wanting total control and was not willing to recognize that you need the support of the system.

You find this happening again and again; good scientists will fight the system rather than learn to work with the system and take advantage of all the system has to offer. It has a lot, if you learn how to use it. It takes patience, but you can learn how to use the system pretty well, and you can learn how to get around it. After all, if you want a decision No', you just go to your boss and get aNo’ easy. If you want to do something, don’t ask, do it. Present him with an accomplished fact. Don’t give him a chance to tell you No'. But if you want aNo’, it’s easy to get a `No’.

Another personality defect is ego assertion and I’ll speak in this case of my own experience. I came from Los Alamos and in the early days I was using a machine in New York at 590 Madison Avenue where we merely rented time. I was still dressing in western clothes, big slash pockets, a bolo and all those things. I vaguely noticed that I was not getting as good service as other people. So I set out to measure. You came in and you waited for your turn; I felt I was not getting a fair deal. I said to myself, “Why? No Vice President at IBM said, `Give Hamming a bad time’. It is the secretaries at the bottom who are doing this. When a slot appears, they’ll rush to find someone to slip in, but they go out and find somebody else. Now, why? I haven’t mistreated them.” Answer, I wasn’t dressing the way they felt somebody in that situation should. It came down to just that – I wasn’t dressing properly. I had to make the decision – was I going to assert my ego and dress the way I wanted to and have it steadily drain my effort from my professional life, or was I going to appear to conform better? I decided I would make an effort to appear to conform properly. The moment I did, I got much better service. And now, as an old colorful character, I get better service than other people.

You should dress according to the expectations of the audience spoken to. If I am going to give an address at the MIT computer center, I dress with a bolo and an old corduroy jacket or something else. I know enough not to let my clothes, my appearance, my manners get in the way of what I care about. An enormous number of scientists feel they must assert their ego and do their thing their way. They have got to be able to do this, that, or the other thing, and they pay a steady price.

John Tukey almost always dressed very casually. He would go into an important office and it would take a long time before the other fellow realized that this is a first-class man and he had better listen. For a long time John has had to overcome this kind of hostility. It’s wasted effort! I didn’t say you should conform; I said “The appearance of conforming gets you a long way.” If you chose to assert your ego in any number of ways, “I am going to do it my way,” you pay a small steady price throughout the whole of your professional career. And this, over a whole lifetime, adds up to an enormous amount of needless trouble.

By taking the trouble to tell jokes to the secretaries and being a little friendly, I got superb secretarial help. For instance, one time for some idiot reason all the reproducing services at Murray Hill were tied up. Don’t ask me how, but they were. I wanted something done. My secretary called up somebody at Holmdel, hopped the company car, made the hour-long trip down and got it reproduced, and then came back. It was a payoff for the times I had made an effort to cheer her up, tell her jokes and be friendly; it was that little extra work that later paid off for me. By realizing you have to use the system and studying how to get the system to do your work, you learn how to adapt the system to your desires. Or you can fight it steadily, as a small undeclared war, for the whole of your life.

And I think John Tukey paid a terrible price needlessly. He was a genius anyhow, but I think it would have been far better, and far simpler, had he been willing to conform a little bit instead of ego asserting. He is going to dress the way he wants all of the time. It applies not only to dress but to a thousand other things; people will continue to fight the system. Not that you shouldn’t occasionally!

When they moved the library from the middle of Murray Hill to the far end, a friend of mine put in a request for a bicycle. Well, the organization was not dumb. They waited awhile and sent back a map of the grounds saying, “Will you please indicate on this map what paths you are going to take so we can get an insurance policy covering you.” A few more weeks went by. They then asked, “Where are you going to store the bicycle and how will it be locked so we can do so and so.” He finally realized that of course he was going to be red-taped to death so he gave in. He rose to be the President of Bell Laboratories.

Barney Oliver was a good man. He wrote a letter one time to the IEEE. At that time the official shelf space at Bell Labs was so much and the height of the IEEE Proceedings at that time was larger; and since you couldn’t change the size of the official shelf space he wrote this letter to the IEEE Publication person saying, “Since so many IEEE members were at Bell Labs and since the official space was so high the journal size should be changed.” He sent it for his boss’s signature. Back came a carbon with his signature, but he still doesn’t know whether the original was sent or not. I am not saying you shouldn’t make gestures of reform. I am saying that my study of able people is that they don’t get themselves committed to that kind of warfare. They play it a little bit and drop it and get on with their work.

Many a second-rate fellow gets caught up in some little twitting of the system, and carries it through to warfare. He expends his energy in a foolish project. Now you are going to tell me that somebody has to change the system. I agree; somebody’s has to. Which do you want to be? The person who changes the system or the person who does first-class science? Which person is it that you want to be? Be clear, when you fight the system and struggle with it, what you are doing, how far to go out of amusement, and how much to waste your effort fighting the system. My advice is to let somebody else do it and you get on with becoming a first-class scientist. Very few of you have the ability to both reform the system and become a first-class scientist.

On the other hand, we can’t always give in. There are times when a certain amount of rebellion is sensible. I have observed almost all scientists enjoy a certain amount of twitting the system for the sheer love of it. What it comes down to basically is that you cannot be original in one area without having originality in others. Originality is being different. You can’t be an original scientist without having some other original characteristics. But many a scientist has let his quirks in other places make him pay a far higher price than is necessary for the ego satisfaction he or she gets. I’m not against all ego assertion; I’m against some.

Another fault is anger. Often a scientist becomes angry, and this is no way to handle things. Amusement, yes, anger, no. Anger is misdirected. You should follow and cooperate rather than struggle against the system all the time.

Another thing you should look for is the positive side of things instead of the negative. I have already given you several examples, and there are many, many more; how, given the situation, by changing the way I looked at it, I converted what was apparently a defect to an asset. I’ll give you another example. I am an egotistical person; there is no doubt about it. I knew that most people who took a sabbatical to write a book, didn’t finish it on time. So before I left, I told all my friends that when I come back, that book was going to be done! Yes, I would have it done – I’d have been ashamed to come back without it! I used my ego to make myself behave the way I wanted to. I bragged about something so I’d have to perform. I found out many times, like a cornered rat in a real trap, I was surprisingly capable. I have found that it paid to say, “Oh yes, I’ll get the answer for you Tuesday,” not having any idea how to do it. By Sunday night I was really hard thinking on how I was going to deliver by Tuesday. I often put my pride on the line and sometimes I failed, but as I said, like a cornered rat I’m surprised how often I did a good job. I think you need to learn to use yourself. I think you need to know how to convert a situation from one view to another which would increase the chance of success.

Now self-delusion in humans is very, very common. There are enumerable ways of you changing a thing and kidding yourself and making it look some other way. When you ask, “Why didn’t you do such and such,” the person has a thousand alibis. If you look at the history of science, usually these days there are 10 people right there ready, and we pay off for the person who is there first. The other nine fellows say, “Well, I had the idea but I didn’t do it and so on and so on.” There are so many alibis. Why weren’t you first? Why didn’t you do it right? Don’t try an alibi. Don’t try and kid yourself. You can tell other people all the alibis you want. I don’t mind. But to yourself try to be honest.

If you really want to be a first-class scientist you need to know yourself, your weaknesses, your strengths, and your bad faults, like my egotism. How can you convert a fault to an asset? How can you convert a situation where you haven’t got enough manpower to move into a direction when that’s exactly what you need to do? I say again that I have seen, as I studied the history, the successful scientist changed the viewpoint and what was a defect became an asset.

In summary, I claim that some of the reasons why so many people who have greatness within their grasp don’t succeed are: they don’t work on important problems, they don’t become emotionally involved, they don’t try and change what is difficult to some other situation which is easily done but is still important, and they keep giving themselves alibis why they don’t. They keep saying that it is a matter of luck. I’ve told you how easy it is; furthermore I’ve told you how to reform. Therefore, go forth and become great scientists!

Questions and Answers

A. G. Chynoweth: Well that was 50 minutes of concentrated wisdom and observations accumulated over a fantastic career; I lost track of all the observations that were striking home. Some of them are very very timely. One was the plea for more computer capacity; I was hearing nothing but that this morning from several people, over and over again. So that was right on the mark today even though here we are 20 – 30 years after when you were making similar remarks, Dick. I can think of all sorts of lessons that all of us can draw from your talk. And for one, as I walk around the halls in the future I hope I won’t see as many closed doors in Bellcore. That was one observation I thought was very intriguing.

Thank you very, very much indeed Dick; that was a wonderful recollection. I’ll now open it up for questions. I’m sure there are many people who would like to take up on some of the points that Dick was making.

Hamming: First let me respond to Alan Chynoweth about computing. I had computing in research and for 10 years I kept telling my management, “Get that !&@#% machine out of research. We are being forced to run problems all the time. We can’t do research because were too busy operating and running the computing machines.” Finally the message got through. They were going to move computing out of research to someplace else. I was persona non grata to say the least and I was surprised that people didn’t kick my shins because everybody was having their toy taken away from them. I went in to Ed David’s office and said, “Look Ed, you’ve got to give your researchers a machine. If you give them a great big machine, we’ll be back in the same trouble we were before, so busy keeping it going we can’t think. Give them the smallest machine you can because they are very able people. They will learn how to do things on a small machine instead of mass computing.” As far as I’m concerned, that’s how UNIX arose. We gave them a moderately small machine and they decided to make it do great things. They had to come up with a system to do it on. It is called UNIX!

A. G. Chynoweth: I just have to pick up on that one. In our present environment, Dick, while we wrestle with some of the red tape attributed to, or required by, the regulators, there is one quote that one exasperated AVP came up with and I’ve used it over and over again. He growled that, “UNIX was never a deliverable!”

Question: What about personal stress? Does that seem to make a difference?

Hamming: Yes, it does. If you don’t get emotionally involved, it doesn’t. I had incipient ulcers most of the years that I was at Bell Labs. I have since gone off to the Naval Postgraduate School and laid back somewhat, and now my health is much better. But if you want to be a great scientist you’re going to have to put up with stress. You can lead a nice life; you can be a nice guy or you can be a great scientist. But nice guys end last, is what Leo Durocher said. If you want to lead a nice happy life with a lot of recreation and everything else, you’ll lead a nice life.

Question: The remarks about having courage, no one could argue with; but those of us who have gray hairs or who are well established don’t have to worry too much. But what I sense among the young people these days is a real concern over the risk taking in a highly competitive environment. Do you have any words of wisdom on this?

Hamming: I’ll quote Ed David more. Ed David was concerned about the general loss of nerve in our society. It does seem to me that we’ve gone through various periods. Coming out of the war, coming out of Los Alamos where we built the bomb, coming out of building the radars and so on, there came into the mathematics department, and the research area, a group of people with a lot of guts. They’ve just seen things done; they’ve just won a war which was fantastic. We had reasons for having courage and therefore we did a great deal. I can’t arrange that situation to do it again. I cannot blame the present generation for not having it, but I agree with what you say; I just cannot attach blame to it. It doesn’t seem to me they have the desire for greatness; they lack the courage to do it. But we had, because we were in a favorable circumstance to have it; we just came through a tremendously successful war. In the war we were looking very, very bad for a long while; it was a very desperate struggle as you well know. And our success, I think, gave us courage and self confidence; that’s why you see, beginning in the late forties through the fifties, a tremendous productivity at the labs which was stimulated from the earlier times. Because many of us were earlier forced to learn other things – we were forced to learn the things we didn’t want to learn, we were forced to have an open door – and then we could exploit those things we learned. It is true, and I can’t do anything about it; I cannot blame the present generation either. It’s just a fact.

Question: Is there something management could or should do?

Hamming: Management can do very little. If you want to talk about managing research, that’s a totally different talk. I’d take another hour doing that. This talk is about how the individual gets very successful research done in spite of anything the management does or in spite of any other opposition. And how do you do it? Just as I observe people doing it. It’s just that simple and that hard!

Question: Is brainstorming a daily process?

Hamming: Once that was a very popular thing, but it seems not to have paid off. For myself I find it desirable to talk to other people; but a session of brainstorming is seldom worthwhile. I do go in to strictly talk to somebody and say, “Look, I think there has to be something here. Here’s what I think I see …” and then begin talking back and forth. But you want to pick capable people. To use another analogy, you know the idea called the critical mass.' If you have enough stuff you have critical mass. There is also the idea I used to callsound absorbers’. When you get too many sound absorbers, you give out an idea and they merely say, “Yes, yes, yes.” What you want to do is get that critical mass in action; “Yes, that reminds me of so and so,” or, “Have you thought about that or this?” When you talk to other people, you want to get rid of those sound absorbers who are nice people but merely say, “Oh yes,” and to find those who will stimulate you right back.

For example, you couldn’t talk to John Pierce without being stimulated very quickly. There were a group of other people I used to talk with. For example there was Ed Gilbert; I used to go down to his office regularly and ask him questions and listen and come back stimulated. I picked my people carefully with whom I did or whom I didn’t brainstorm because the sound absorbers are a curse. They are just nice guys; they fill the whole space and they contribute nothing except they absorb ideas and the new ideas just die away instead of echoing on. Yes, I find it necessary to talk to people. I think people with closed doors fail to do this so they fail to get their ideas sharpened, such as “Did you ever notice something over here?” I never knew anything about it – I can go over and look. Somebody points the way. On my visit here, I have already found several books that I must read when I get home. I talk to people and ask questions when I think they can answer me and give me clues that I do not know about. I go out and look!

Question: What kind of tradeoffs did you make in allocating your time for reading and writing and actually doing research?

Hamming: I believed, in my early days, that you should spend at least as much time in the polish and presentation as you did in the original research. Now at least 50% of the time must go for the presentation. It’s a big, big number.

Question: How much effort should go into library work?

Hamming: It depends upon the field. I will say this about it. There was a fellow at Bell Labs, a very, very, smart guy. He was always in the library; he read everything. If you wanted references, you went to him and he gave you all kinds of references. But in the middle of forming these theories, I formed a proposition: there would be no effect named after him in the long run. He is now retired from Bell Labs and is an Adjunct Professor. He was very valuable; I’m not questioning that. He wrote some very good Physical Review articles; but there’s no effect named after him because he read too much. If you read all the time what other people have done you will think the way they thought. If you want to think new thoughts that are different, then do what a lot of creative people do – get the problem reasonably clear and then refuse to look at any answers until you’ve thought the problem through carefully how you would do it, how you could slightly change the problem to be the correct one. So yes, you need to keep up. You need to keep up more to find out what the problems are than to read to find the solutions. The reading is necessary to know what is going on and what is possible. But reading to get the solutions does not seem to be the way to do great research. So I’ll give you two answers. You read; but it is not the amount, it is the way you read that counts.

Question: How do you get your name attached to things?

Hamming: By doing great work. I’ll tell you the hamming window one. I had given Tukey a hard time, quite a few times, and I got a phone call from him from Princeton to me at Murray Hill. I knew that he was writing up power spectra and he asked me if I would mind if he called a certain window a “Hamming window.” And I said to him, “Come on, John; you know perfectly well I did only a small part of the work but you also did a lot.” He said, “Yes, Hamming, but you contributed a lot of small things; you’re entitled to some credit.” So he called it the hamming window. Now, let me go on. I had twitted John frequently about true greatness. I said true greatness is when your name is like ampere, watt, and fourier – when it’s spelled with a lower case letter. That’s how the hamming window came about.

Question: Dick, would you care to comment on the relative effectiveness between giving talks, writing papers, and writing books?

Hamming: In the short-haul, papers are very important if you want to stimulate someone tomorrow. If you want to get recognition long-haul, it seems to me writing books is more contribution because most of us need orientation. In this day of practically infinite knowledge, we need orientation to find our way. Let me tell you what infinite knowledge is. Since from the time of Newton to now, we have come close to doubling knowledge every 17 years, more or less. And we cope with that, essentially, by specialization. In the next 340 years at that rate, there will be 20 doublings, i.e. a million, and there will be a million fields of specialty for every one field now. It isn’t going to happen. The present growth of knowledge will choke itself off until we get different tools. I believe that books which try to digest, coordinate, get rid of the duplication, get rid of the less fruitful methods and present the underlying ideas clearly of what we know now, will be the things the future generations will value. Public talks are necessary; private talks are necessary; written papers are necessary. But I am inclined to believe that, in the long-haul, books which leave out what’s not essential are more important than books which tell you everything because you don’t want to know everything. I don’t want to know that much about penguins is the usual reply. You just want to know the essence.

Question: You mentioned the problem of the Nobel Prize and the subsequent notoriety of what was done to some of the careers. Isn’t that kind of a much more broad problem of fame? What can one do?

Hamming: Some things you could do are the following. Somewhere around every seven years make a significant, if not complete, shift in your field. Thus, I shifted from numerical analysis, to hardware, to software, and so on, periodically, because you tend to use up your ideas. When you go to a new field, you have to start over as a baby. You are no longer the big mukity muk and you can start back there and you can start planting those acorns which will become the giant oaks. Shannon, I believe, ruined himself. In fact when he left Bell Labs, I said, “That’s the end of Shannon’s scientific career.” I received a lot of flak from my friends who said that Shannon was just as smart as ever. I said, “Yes, he’ll be just as smart, but that’s the end of his scientific career,” and I truly believe it was.

You have to change. You get tired after a while; you use up your originality in one field. You need to get something nearby. I’m not saying that you shift from music to theoretical physics to English literature; I mean within your field you should shift areas so that you don’t go stale. You couldn’t get away with forcing a change every seven years, but if you could, I would require a condition for doing research, being that you will change your field of research every seven years with a reasonable definition of what it means, or at the end of 10 years, management has the right to compel you to change. I would insist on a change because I’m serious. What happens to the old fellows is that they get a technique going; they keep on using it. They were marching in that direction which was right then, but the world changes. There’s the new direction; but the old fellows are still marching in their former direction.

You need to get into a new field to get new viewpoints, and before you use up all the old ones. You can do something about this, but it takes effort and energy. It takes courage to say, “Yes, I will give up my great reputation.” For example, when error correcting codes were well launched, having these theories, I said, “Hamming, you are going to quit reading papers in the field; you are going to ignore it completely; you are going to try and do something else other than coast on that.” I deliberately refused to go on in that field. I wouldn’t even read papers to try to force myself to have a chance to do something else. I managed myself, which is what I’m preaching in this whole talk. Knowing many of my own faults, I manage myself. I have a lot of faults, so I’ve got a lot of problems, i.e. a lot of possibilities of management.

Question: Would you compare research and management?

Hamming: If you want to be a great researcher, you won’t make it being president of the company. If you want to be president of the company, that’s another thing. I’m not against being president of the company. I just don’t want to be. I think Ian Ross does a good job as President of Bell Labs. I’m not against it; but you have to be clear on what you want. Furthermore, when you’re young, you may have picked wanting to be a great scientist, but as you live longer, you may change your mind. For instance, I went to my boss, Bode, one day and said, “Why did you ever become department head? Why didn’t you just be a good scientist?” He said, “Hamming, I had a vision of what mathematics should be in Bell Laboratories. And I saw if that vision was going to be realized, I had to make it happen; I had to be department head.” When your vision of what you want to do is what you can do single-handedly, then you should pursue it. The day your vision, what you think needs to be done, is bigger than what you can do single-handedly, then you have to move toward management. And the bigger the vision is, the farther in management you have to go. If you have a vision of what the whole laboratory should be, or the whole Bell System, you have to get there to make it happen. You can’t make it happen from the bottom very easily. It depends upon what goals and what desires you have. And as they change in life, you have to be prepared to change. I chose to avoid management because I preferred to do what I could do single-handedly. But that’s the choice that I made, and it is biased. Each person is entitled to their choice. Keep an open mind. But when you do choose a path, for heaven’s sake be aware of what you have done and the choice you have made. Don’t try to do both sides.

Question: How important is one’s own expectation or how important is it to be in a group or surrounded by people who expect great work from you?

Hamming: At Bell Labs everyone expected good work from me – it was a big help. Everybody expects you to do a good job, so you do, if you’ve got pride. I think it’s very valuable to have first-class people around. I sought out the best people. The moment that physics table lost the best people, I left. The moment I saw that the same was true of the chemistry table, I left. I tried to go with people who had great ability so I could learn from them and who would expect great results out of me. By deliberately managing myself, I think I did much better than laissez faire.

Question: You, at the outset of your talk, minimized or played down luck; but you seemed also to gloss over the circumstances that got you to Los Alamos, that got you to Chicago, that got you to Bell Laboratories.

Hamming: There was some luck. On the other hand I don’t know the alternate branches. Until you can say that the other branches would not have been equally or more successful, I can’t say. Is it luck the particular thing you do? For example, when I met Feynman at Los Alamos, I knew he was going to get a Nobel Prize. I didn’t know what for. But I knew darn well he was going to do great work. No matter what directions came up in the future, this man would do great work. And sure enough, he did do great work. It isn’t that you only do a little great work at this circumstance and that was luck, there are many opportunities sooner or later. There are a whole pail full of opportunities, of which, if you’re in this situation, you seize one and you’re great over there instead of over here. There is an element of luck, yes and no. Luck favors a prepared mind; luck favors a prepared person. It is not guaranteed; I don’t guarantee success as being absolutely certain. I’d say luck changes the odds, but there is some definite control on the part of the individual.

Go forth, then, and do great work!

At the end of the nineteen-eighties, Larry David was a standup comic in trouble. He was middle-aged, single, living in a building with subsidized housing for artists on the West Side of Manhattan, and just scraping by. He had been doing standup, with mixed success, for more than a decade; his chances for breaking through were long past. He had written for and acted on a short-lived ABC variety show called “Fridays.” He had been on the writing staff of “Saturday Night Live” in the 1984-85 season, though only one of his sketches aired. He had played bit parts in a few movies. He had written a screenplay—a dark comedy, never produced—called, appropriately, “Prognosis Negative.”

David had a reputation as a comic’s comic—“which means I sucked,” he likes to say. His material was uncompromisingly to his own taste, filled with wild tirades about apparent trivialities. In one routine, he went on at length about the use of the familiar “you” in foreign languages (“Caesar used the tu form with Brutus even after Brutus stabbed him, which I think is going too far”). He imagined himself as a professional masturbator so talented that people stopped him on the street to ask for advice (“You must practice!”). He wondered how answering machines might have changed the Old West. (For one thing, you could get out of joining a dangerous posse by screening your calls.)

David’s onstage manner was almost willfully uningratiating. He was intense and bespectacled, and often wore an old Army jacket. He had started going bald at thirty, and by his early forties “the hair was a combination of Bozo and Einstein,” the comedian Richard Lewis says. David and Lewis, who were born three days apart, originally met at Camp All America, in Cornwall-on-Hudson, New York, when they were thirteen. They disliked each other immediately. A dozen years later, they met again, at the bar of the Improv, a comedy club in Manhattan, and became close friends. “Talk about walking to the beat of your own drum,” Lewis says of David. “I mean, this guy was born in a snare drum.”

Club audiences were puzzled by David, or, worse, indifferent to him. The managers who occasionally took him on invariably recommended that he make his material more accessible. He found new managers. “I was not for everyone,” Larry David said, laughing, when I met him last October. “I was for very few.” He was sitting at his big desk in the Santa Monica offices of Larry David Productions, the company behind “Curb Your Enthusiasm,” the half-hour cinéma-vérité situation comedy on HBO.

Contrary to the cliché that comedians are stingy about giving out what they most ardently seek, Larry David is a surprisingly easy laugh. He has several laughs, ranging from a brief, wry exhalation that reveals what Shelley Berman (who plays David’s father on “Curb”) calls his “wondrously perfect teeth,” with their pronounced canines, to an uninhibited guffaw. Perhaps David’s most striking laugh, however, is a snort that is exactly like the one Jason Alexander made famous when he played “Seinfeld” ’s George Costanza—a character who, David claims, is largely based on himself.

The similarities between prototype and character are not instantly apparent. David has a lanky, wiry build and an athletic, slightly bowlegged walk. His cranium is long and sleek, surrounded by a fringe of curly whitish hair that is neatly trimmed, except for rampant sideburns. The afternoon we met, he wore what was evidently a customary outfit, in a style that I came to think of as comedy-tycoon casual: a navy-blue shirt jacket, a medium-blue zipper-neck shirt, khakis, white socks, and dingy beige Pumas.

On “Curb Your Enthusiasm,” David’s character is a semi-retired sitcom mogul who ambles through his inordinately comfortable life, routinely managing to annoy or infuriate everyone around him. This season, some of those people will include the blind, the physically handicapped, and the mentally challenged, making the show even edgier than before.

David, who, in 1988, co-created “Seinfeld,” is said to have earned more than two hundred million dollars from that show’s syndication revenues. His comedy style has remained argumentative, abrasive, and occasionally alienating, and some people claim that he has outgrown the circumstances that might have justified such a stance. Writing in The New Republic last year, Lee Siegel said, “David’s anger . . . is merely the anger of frustrated entitlement. [He] has perfected Seinfeld’s superior, uninviting stare into a cold, cruel sneer. The reason that so many people like it is that they want it to like them.” And amid the rapturous postings about “Curb” on HBO’s Web site (“the best show to hit tv in a long time”; “Larry David is the funniest, most brilliant, and most talented man on television, or possibly in entertainment”) lie voices of dissent: “Please retire this tedious program . . . a bunch of screaming jews apparently ad-libbing it is not funny.”

“It has to do with Brooklyn,” David said of his humor. “It has to do—I think—with growing up in an apartment, with my aunt and my cousins right next door to me, with the door open, with neighbors walking in and out, with people yelling at each other all the time.”

Born in 1947, the younger of two sons of a clothing salesman and a housewife, David had “a wonderful childhood,” he has said, adding, “Which is tough, because it’s hard to adjust to a miserable adulthood.” He hated the sixties. “Drugs scared me,” he said, in his hoarse tenor, with its mildly staccato rhythm. “And I couldn’t even fake my way into the sex. God knows I tried. The women were living in the sixties for everybody else, but for me they were not even in the fifties—I’d say the forties. The clothing just totally offended me. All I saw was a lot of conformity.”

After graduating from the University of Maryland in 1970, with a degree in history, he had no idea what his next step might be. “My standard response when people would ask me ‘What are you gonna do when you get out?’ was ‘Ah, somethin’ll turn up.’ ”

He moved back home to Brooklyn and got a job with a bra wholesaler in Manhattan. “The bras were seconds, actually—they were defective bras,” he said. “And that didn’t last very long. So it was this pattern of getting a job, then going on unemployment for a while. I had a job as a paralegal. I drove a cab. Until I started doing standup, there were some very bleak days. I was a private chauffeur, driving my limousine, wearing the uniform. I’m twenty-five years old. This is what I’m doing for a living. And”—he laughed, not quite happily—“wearing a uniform, outside, waiting for her while she’s shopping on Third Avenue. Seeing a guy from college walk down the street, stop in his tracks, stare at me agog in this uniform, not knowing what to do or say, you know.” His voice trailed off for a moment. “That was pretty embarrassing,” he said.

In a “Seinfeld” episode that Larry David wrote, the unemployed George Costanza is forced to move back in with his parents and endure his mother’s suggestion that he consider a career as a mailman. The scenario was apparently drawn from life. “That would’ve been my mother’s dream for me,” David said. “Take a civil-service test, work in the post office.”

Eventually, David found his way to an acting class in Manhattan. Acting made him ill at ease, but during one exercise—in which each student was supposed to pick a monologue from a play and reinterpret it—he got a surprise. “As I started putting it in my own words, everybody in the class laughed,” David said. “And I thought, Hey—that’s for me. That’s what I want. I want a laugh.”

It turned out to be harder than he had imagined. “I think that for the most part, when I started doing comedy, it had become very commercialized,” David said. “And that’s what the clubs were like. It’s not that I wasn’t suited to standup comedy. It’s that I wasn’t suited to do the kind of comedy that these people were coming to hear—mainstream comedy. Television-suitable material that had a lot of jokes that people could relate to.”

Nevertheless, David said, “I managed to put together an act that I could do, and enjoy, and kill with, on a Saturday night. But it still was difficult going on. Because I was taking my life in my hands, I felt. Every time I went up, I thought I was putting my life on the line.”

David had been known to walk off the stage if an audience wasn’t completely focussed on him. “He just needed undivided attention,” Richard Lewis says. “Even if someone would whisper, ‘I’ll have another Daiquiri’—literally—he’d storm off. I mean, it was ludicrous.”

“A night club is a place where drinks and food are served,” Jerry Seinfeld says. “A comedian is not automatically the audience’s focal point. You have to fight for their attention, and it’s not easy to get. Larry had the material, but he never had what you would call the temperament for standup.”

One night at Catch a Rising Star, a comedy club on Manhattan’s Upper East Side, David stepped onto the stage, scanned the room from side to side, said, “Never mind,” and walked off.

Despite the bravado, he had no plan. “I was hoping that somehow I could get some kind of cult following, and get by with that,” he said. “And you know what? That would have been fine with me. I just wanted laughs—that’s really what I was after. I wanted to make a living, but I really was not interested in money at all. I was interested in being a great comedian. That was really what I wanted to be.”

Larry David met Jerry Seinfeld around 1976. David had been doing standup for two years; Seinfeld was just starting out. “Our brains had a comedic connection,” Seinfeld says. “Larry was a guy open to discussing virtually any human dilemma, as long as it was something that not a lot of other people were interested in. I was exactly the same way. We weren’t interested in what was on the front page of the newspaper.” They became comedy friends, working on standup material together while walking through Central Park or sitting in a coffee shop, one helping the other if he was stuck with a bit.

Seven years younger than David, Seinfeld was boyish and charismatic, and by the late eighties he was touring steadily and making frequent appearances on the “Tonight Show” and “Late Night with David Letterman.” He reportedly earned up to twenty-five thousand dollars a weekend at comedy clubs. As the unflappable master of observational standup, Seinfeld had created a persona that was almost completely impersonal yet thoroughly engaging. Larry David pushed audiences away; Jerry Seinfeld seduced them.

In the fall of 1988, Seinfeld received the ultimate acknowledgment for a comic: NBC called, wanting to develop a show with him. “It didn’t seem like any fun to do it by myself,” he says. “So I told Larry about it.”

One night in late November, Seinfeld and David were going to share a cab back to the West Side from Catch a Rising Star but decided to stop and pick up some groceries first. “It was a Korean deli, and we were waiting to pay, and we started making fun of the products they kept by the register,” Seinfeld says. “You know, those fig bars in cellophane, without a label, that look like somebody made them in their basement?”

David turned to Seinfeld and said, “This is what the show should be—this is the kind of dialogue that we should do on the show.”

“The stuff that we would talk about was never on TV,” Seinfeld says. “The essence of the show, originally, was my desire to transplant the tone and subjects of my conversations with Larry to television. At first, the idea was to have two comedians walking around in New York, making fun of things, and in between you’d have standup bits.”

David and Seinfeld pitched the rough concept to NBC. The meeting, which was eventually immortalized in the “Seinfeld” episode that has Jerry and George pitching “a show about nothing” to NBC, was notably tense. Not only were David and Seinfeld pitching fig-bar conversations; they wanted to do a one-camera, documentary-style show. The NBC executives were not impressed; they told David and Seinfeld that they wanted a straight, three-camera sitcom.

The executives were particularly unimpressed with Larry David. He remembers Seinfeld’s looking askance at him while he protested the network’s aesthetic. “I said, ‘This is not the show.’ People looked at me like I was a little nuts—a lot of ‘Who is this guy?’ kind of looks.”

Still, the NBC executives saw something. “I guess they figured it was worth a pilot,” David said. “Well, they liked him enough that they figured it was worth a pilot. I think they would’ve gotten rid of me in a split second if they could’ve. They would have gotten rid of me without even thinking about it.”

As “Seinfeld” ’s show-runner—the head writer and the person in charge of every detail of the series and the scripts—Larry David kept clashing with the forces of conventionality. “At the beginning, Jerry’s managers were always very concerned that Jerry come off well,” Larry Charles, the former supervising producer of “Seinfeld” and now an executive producer of “Curb Your Enthusiasm,” told me. We were sitting in Charles’s book-lined office, next to Larry David’s. “I drew a caricature of him on a board on a wall, with the caption ‘Must always smell like a rose.’ ”

Charles—a tall man with a gray mane and a ZZ Top-style beard, who often wears wraparound shades—met Larry David while working on “Fridays.” “We’re both from kind of middle-earth Brooklyn—you know, Brighton Beach, Coney Island, lower middle class, under the train tracks,” Charles said. “We both understand that sort of ‘Lord of the Flies’ sensibility that requires you to be very aware as you grow up. It’s a very savage environment, in a lot of ways, a very cruel and sadistic environment. We spoke the same language—we were like brothers from different mothers.”

David wanted to bring Charles onto the staff of “Seinfeld” as a writer and producer, but he met with resistance. “The production company wouldn’t hire me, because I had no sitcom experience, and Larry was kind of new,” Charles said. “And so they flooded him with sitcom people he couldn’t stand, and he chafed under that.” When the show was picked up for a season and David was able to hire him, Charles told me, “the show sort of started to move in the direction that it was supposed to move in.” He smiled. “This idea of, like, getting very dark in a sitcom. You know, we had jackets made up—‘No Hugging, No Learning.’ This idea was anathema at that point. The idea that you would have an unhappy ending—that people would pay for their sins, or that there was no redemption sometimes—I mean, this really shook the foundation of the sitcom genre.”

The show’s pivotal moment came in the third season, in 1991. Charles remembers walking with David from the “Seinfeld” offices in Studio City up to Fryman Canyon to try to break a story: the library-cop episode, in which Jerry is investigated for keeping a book out for twenty years. “We had a couple of strands, and I don’t know if it was the oxygen from the walking, but we were very exhilarated,” Charles said. “We went, ‘What if the book that was overdue was in the homeless guy’s car? And the homeless guy was the gym teacher that had done the wedgie? And what if, when they return the book, Kramer has a relationship with the librarian?’

“Suddenly it’s like—why not? It’s like, boom boom boom, an epiphany—quantum theory of sitcom! It was, like, nobody’s doing this! Usually, there’s the A story, the B story—no, let’s have five stories! And all the characters’ stories intersect in some sort of weirdly organic way, and you just see what happens. It was like—oh my God. It was like finding the cure for cancer.”

In a far corner of Larry David’s office hangs a framed poster for a movie called “Sour Grapes.” David spent much of the time after he left “Seinfeld,” following the seventh season, writing and then directing the film, which was about the disharmony that arises between two cousins when one wins a slot-machine jackpot with two quarters borrowed from the other. The picture came out in 1998 and didn’t do well critically (“slightly threadbare,” according to the Times) or at the box office.

That spring, David returned to “Seinfeld” just long enough to write the show’s final episode, but he still wasn’t ready to retire. When I asked why not, he shifted into a declamatory Yiddish-inflected voice from his standup act. “You grow up, you need something to do!” he cried, clapping his hands. “You need”—he clapped his hands again—“a place to go.” He became himself again. “A place to go—that’s what my mother always instilled in me. You need a place to go. And you’re worthless unless you have a place to go. So I needed a place to go.”

He found himself thinking about doing standup again. Jeff Garlin, an acquaintance from the comedy clubs, had recently directed HBO specials for the comedians Jon Stewart and Denis Leary. He offered to do the same for David. “Jeff said, ‘You haven’t done it in ten years—just film it,’ ” David said. Then they got an idea: film it as a mockumentary about the making of an HBO special, with standup and fictional scenes interspersed. The show was to be built around a character entirely new to television: Larry David, the co-creator of the most successful sitcom in history. He would need a wife (David had married Laurie Lennard, a former talent coördinator for the Letterman show, in 1993) and a manager (he chose Garlin to stand in for his actual manager, Gavin Polone). He decided that all the scenes would have to be improvised. “There’s no way”—he rapped his desk vehemently—“that you can get that sort of documentary feel, that cinéma-vérité thing, unless you’re improvising. And I’d always liked improvising—whenever I’ve done it in the past, I felt I had a knack for it. So that was it.”

David called another friend, a documentary filmmaker named Robert Weide, an assiduous student of comedy history who had made documentaries about the Marx Brothers, Lenny Bruce, and Mort Sahl. Fifteen years earlier, Weide had read “Prognosis Negative” and loved it, and had become a devoted fan of David’s standup. He agreed to direct.

Shooting started in early 1999, after Cheryl Hines, a young member of the Groundlings, a Los Angeles improvisational group, was cast as David’s wife. As the show developed, the improv scenes kept expanding. Soon, it had become a show about the buildup to David’s big standup show—a buildup that led to an inevitable letdown. David’s standup routines—performed in the barking, hyperactive style familiar to those who had seen his act and those who had heard his voice-overs for the George Steinbrenner character on “Seinfeld”—became the filler.

“Larry David: Curb Your Enthusiasm” aired in October, 1999. The title was an ironic, almost superstitious reference to David’s low profile. But it was also a billboard for the vastly less famous “Seinfeld” co-creator.

The show wasn’t quite like anything that had been on TV before. The real-life details (there were deadpan talking-head interviews with Seinfeld, Lewis, Jason Alexander, and Rick Newman, the founder of Catch a Rising Star), the handheld camera (an acknowledged presence in several scenes), and the improvised dialogue made the show much closer to the bone than “Seinfeld.” “Seinfeld” was scherzo, its fun stemming from the constantly shifting play among its troupe of four. David’s new form was simpler and starker. There was a basic triangle: Larry; Jeff, his manager, who helps get him into trouble (usually in the form of telling lies and keeping secrets, Larry being spectacularly bad at the latter); and Cheryl, his wife, who calls him to account.

The special got mainly positive reviews. Tom Shales wrote in the Washington Post that it was “a peek into the life and mind of a brilliant creative talent who is also clearly a huge pain in the neck and has no TV presence whatsoever.” Belinda Acosta, a TV critic for the Austin Chronicle, had fewer reservations. She wrote, “My only gripe with the show was that I was left wondering: Could this be the launch of a comedy series on HBO? One can dream.”

Larry David had begun the special with no idea of going any further, but as it came together he began to think differently: “We realized as we were doing it that this thing seemed like it could be—a show! The scenes came out very well, better than I had expected. I didn’t cringe when I saw myself—I mean, sometimes I did, but it wasn’t a big cringe-fest for me. And it was fun. The scenes themselves were fun to do. I found myself laughing.”

Chris Albrecht, the chairman and C.E.O. of HBO, had the same idea when he watched the special. He asked for thirteen episodes. David suggested ten, and said he’d do it for a year. The series is now in its fourth season.

Like many comedians, Larry David carries a pocket notebook for writing down ideas. “You’re in a parking garage, and Larry’s wallet is empty—he forgot to ask his assistant to go to the cash machine,” Weide, who directs several episodes a year, says. “So he says, ‘Shit, I have no money for the valet—could you give me a few bucks?’ So you find yourself giving money to Larry David, who has a few bucks. And then out comes the little notebook.”

“What would I have done if he hadn’t been there?” David said. “That could have been funny.”

The notebook is a ratty brown thing that looks as if it might have cost forty-nine cents at a stationery store. Its pages are covered with David’s illegible scrawl. “Somebody commits suicide after arguing with wife over a ‘Seinfeld’ episode,” reads one entry. “[A comedy notable] asks me to go out to dinner,” another begins. “Me and Weide meet him and [the notable’s wife].”

David relishes everyday ambiguities, like the one that arose over the question of who would pay for the meal with the notable and his wife. “You know, if I don’t pick up that check, this guy’s never gonna talk to me again,” he said. “And I’m not picking up the check, ’cause he invited me out to dinner!

“Every relationship is just so tenuous and precarious,” he went on. “One tiny miscommunication or mistake and it could be all over. I’m talking about siblings! A Thanksgiving thing that somehow goes wrong—bringing the wrong dish—all of a sudden, sisters aren’t talking after forty-five years!”

He leafed through the notebook. “Most of the ideas stink,” he said. “But you’d be surprised. See, a lot of these I’ll use, not as a big story but like a little piece of filler. And then all of a sudden it somehow leads into something.”

When the time comes to begin writing the new season, David scans his notebook for possibilities. “He’ll go through the notebook and find three or four stories and extrapolate them to worst-case,” Weide says. “He starts to weave them together. Sometimes you can brainstorm ideas with him—you can even pitch B stories to him. He’s used stories from Larry Charles and me. Cheryl got a story in there. And then he just sits down and sweats it out.”

Every outline runs seven or eight pages and comprises fifteen or so scenes; each is tight and layered, comically concise and full of shootable detail. The outline for Show 8, Season 3, “Krazee Eyez Killa,” begins at “a racially mixed party”:

cheryl is talking to wanda and her boyfriend, krazee eyez killa. They have some interaction with an older black couple, who are Wanda’s parents. During all of this, we keep hearing a popping noise that sounds like a cap-gun going off. We then pan to larry and find him stomping on packing bubbles. Cheryl approaches Larry and tells him to cease and desist. . . . Larry starts chatting with Krazee Eyez Killa. Larry asks where he lives, and . . . Krazee Eyez Killa abruptly changes the subject and asks Larry if he likes to eat pussy.

In the episode itself, the chat between Larry and Krazee Eyez Killa (a rap star, played by Chris Williams) becomes a freewheeling improvised exchange in which Krazee Eyez Killa reads one of his raps and asks Larry for a critique. Larry nods judiciously. “I like it—I got one tiny little comment,” he says. “I would lose the ‘motherfucker’ at the end—’cause you already said ‘fuck’ once. . . . I would change the ‘motherfucker’ to ‘bitch.’ ”

Krazee Eyez Killa beams, and gives Larry a hug. “You my dog,” he says, warmly. “You my nigger. . . . I like you! Check it out—you like eatin’ pussy?”

Larry shrugs. “I like it, I like it,” he says. “But I’m a little too lazy. . . . It’s a whole to-do, you know. It hurts my neck.”

“Aw, man—you got to eat the pussy!” Krazee Eyez Killa says. “You know what the best pussy is to eat? Asian pussy.”

Larry blinks. “Krazee Eyez Killa—you’re getting married,” he says. “Wanda’s gonna find out.”

“Wanda ain’t gonna find out shit,” Krazee Eyez Killa says. “You my nigger, right?”

“Yes,” Larry says, utterly straight-faced. “I’m your nigger.”

“So how she gonna find out?” Krazee Eyez Killa asks. “She ain’t gonna find out—ain’t she.”

“Not from me,” Larry says. “Absolutely not.”

Within a few minutes, he has told Cheryl everything, and she is on her way to tell Wanda (who is played by the sublime Wanda Sykes).

Actors on the show are given only the barest details of their characters or the scenes they’re going to play. There are no rehearsals. Cheryl Hines says, “If we’re about to shoot a scene and I’m asking questions, Larry says, ‘Just save it for the camera—you’ll figure it out.’ ” She laughs. “Poor cameramen—they don’t know what we’re gonna do.”

David, who spent years writing dialogue for “Seinfeld,” doesn’t miss it. “It’s very liberating, actually—I don’t have to hear my voice in every character,” he said. “For example, Krazee Eyez Killa. Could I have written those words in a million years better than that guy said them? No fucking way! I wouldn’t have had the balls to do it! But he comes in and does the character—what could be better than that? I can’t write better than that.

“I’m not gonna lie,” David went on. “There are times when I’m driving home after a day’s shooting, thinking to myself, That scene would’ve been so much better if I had written it out. But that’s the exception. Most of the time I’m thinking, I’m glad that scene was improvised.”

Larry Charles told me, “This is Larry’s thing, and it was true on ‘Seinfeld’ also. If he wants the actors to say something, he’ll tell them to say it. He also will tell them how to say it. Larry hears it a certain way in his head and tries to communicate that to the actor, really giving them the line reading, in a sense.

“Sometimes he’ll have that line in the outline, sometimes he won’t,” Charles continued. “Sometimes he’ll have it in the outline and throw it away when we’re shooting it—he doesn’t need it anymore. Or throw it away in the editing, ultimately.”

“Eventually, I’ll get what I want,” David said.

A couple of days later, I accompanied David to an allergy-shot appointment in downtown Santa Monica. “I hope you get to see a traffic altercation,” he said. “There was a real good one on my way to work today.”

We went in David’s car, a hybrid-powered Toyota Prius. (David drove a Lexus in real life and on the show until three years ago; his wife, an environmentalist, may have influenced his choice.) I asked him what kind of television he liked to watch when he was growing up.

“Well, my favorite show of all time was Bilko,” he said, referring to the classic sitcom “The Phil Silvers Show,” which starred Silvers as the scheming Sergeant Ernest G. Bilko, in charge of the motor pool at the fictional Fort Baxter, Kansas. “I just thought that was head and shoulders above any other show I had seen. You know, in analyzing it now, you could see that Bilko was a manipulative character”—he smiled, giving me a pointed look—“who did a lot of kind of unlikable, despicable things. But, because he was so funny doing it, it all just worked.”

David’s style and process are remarkably similar to those of Bilko’s creator, Nat Hiken, a towering figure of nineteen-fifties situation comedy, who died in 1968. As David Everitt points out in his biography of Hiken, “King of the Half Hour,” the nineteen-fifties was an era of simply plotted sitcoms, like “I Love Lucy.” Hiken’s achievement was to create plots based on multiple interweaving strands that resolved perfectly (and hilariously). Like David, Hiken often gave an actor a line or a sight gag just before he went on camera. And in Phil Silvers he had one of the best ad-libbers in the business.

David may also have absorbed Hiken’s aversion to jokiness. For the first two seasons of “Curb,” Cheryl Hines says, David refused to show her an outline. He was worried that the actors would try to prepare funny bits and work them into the scenes. “One of the things about our show is, the more you try not to be funny, the better,” she says.

David has a sardonic, slightly depressive presence onscreen, and is quite natural playing his worst self. Some of his finest moments are when he gets into arguments—arguments that he always loses—with children. In one episode, he refers to Ted Danson as an asshole—spelling the word out—in front of Danson’s young daughter. (Carefully chosen celebrity friends of David’s regularly appear on the show, adding to its voyeuristic interest.) Moments later, Danson’s daughter—who, it turns out, knows how to spell—accidentally knocks out David’s front teeth with a piñata bat.

David constantly speaks his mind (or fudges the truth) at inopportune moments, violating subtle boundaries, and then delivers forced apologies. There are boundaries on the show, however. One afternoon, during editing, I saw him frown sharply during a scene when Cheryl Hines called him an ass. “It was way too harsh,” he told me later. “It wasn’t funny, and it was demeaning.”

He tries to avoid overusing the word “fuck,” but not out of prudishness. “Susie”—Susie Essman, who plays David’s manager’s wife—“uses it all the time. That’s her character,” David said. “That’s the way she talks. But I don’t want to be a guy who’s depending on using four-letter words to get my laughs. Sometimes it comes out of my mouth when we’re shooting. And I’ll look at it afterward and go, ‘Find another take where I don’t say it.’ ”

Thanks to the freedom of cable TV, the show can be casually raunchy, but it also has a certain sunniness, signalled by the tuba-and-mandolin opening theme song (Luciano Michelini’s “Frolic”) and the musical transitions, many of which have the same Italian-circus flavor. The sweetness, of course, has an edge: David is fond of cutting in an especially upbeat air, Franco Micalizzi’s “Amusement”—the staff call it “Everything’s Fine”—immediately after his character has perpetrated some disaster.

One afternoon in late October, the “Curb” cast and crew—twenty people, seven trucks of equipment, and various cars containing the principals—descended on an airport hotel not far from LAX, for a reshoot of a scene between Larry David and Richard Lewis.

The scene was originally shot in an expository fashion, with the two men standing and talking on the street. David felt that it just wasn’t funny enough. This time, he decided to set it at the urinals in a men’s room and add a former N.B.A. superstar.

Driving down to the reshoot, David was ebullient. “ ‘I’ll be seeing youuuu,’ ” he crooned, then added, brightly, “It’s gonna be fun seeing Richard.” Lewis, who has been a periodic presence on the show since the first episode, is a perfect foil to the otherwise less demonstrative David: they get each other going. Their scenes crackle with a comic intensity that frequently causes David to burst into laughter in the middle of a take. “We could start screaming and yelling over the slightest thing,” Lewis says. “I always forget the mike is on.”

Arriving at the hotel, David walked into a conference room that had been turned into a command center. A temporary assemblage of three rolling carts holding monitors and audio and video controls was lined up along one wall. Facing the carts were three director’s chairs: one for the script supervisor, one for Larry Charles (who was directing the episode), and one for the sound supervisor. Larry David’s chair stood off to the side.

While the cameras rolled, Charles leaned close to the monitors, anxiously hoping for serendipity. During the takes, the men’s room resembled the stateroom scene in “A Night at the Opera”: David, Lewis, and the ex-basketball player at the urinals; lighting men, soundmen, and two cameramen with shoulder-mounted Sony Digital Betacams packed in close behind them. The actors improvised, changing their invented dialogue, and their delivery, each time. Once, when David was standing at the wrong angle while the cameras ran, Charles reached out and tapped one of the monitors in frustration, as if to move David with his fingertips.

Between takes, Lewis, in a dark suit, stalked the conference room, shoulders slumped forward, doing shtick, chatting with the wardrobe woman, the basketball player, the producers. David was a more remote figure, pacing, practicing his golf swing, leaning on the railing and looking down at the lobby. He whistled “I Can’t Get Started.” He was affable when approached, but seldom approached anybody else. It struck me that he was nervous. He whistled “Blue Danube.” “I smell a shit song coming on,” Lewis said. “I smell an Andy Williams summer special.”

Five hours later, the cast and crew dispersed unceremoniously. Larry David later told me that the new scene had come out beautifully. After editing, it ran one minute and thirty-five seconds.

Because several—sometimes many—different takes are shot for every scene, there are thousands upon thousands of feet of videotape, every inch of which must be reviewed in the editing room. (For economic reasons, the show is shot on Digital Betacam and converted by computer into a format that looks like film.) Editing an episode of “Curb” is an excruciatingly subtle process, in which decisions about narrative are made frame by frame—each frame representing one-thirtieth of a second of actual airtime. “If it were up to me, I’d spend two to three days editing an episode, then move on,” Robert Weide says. “But Larry’s a deconstructionist—he has to look at every frame.” It usually takes David two full weeks of rigorous cutting to finish an episode.

“I’m Vishnu the preserver, you’re Shiva the destroyer,” Larry Charles once told him.

As the star and primary producer of “Curb Your Enthusiasm,” Larry David is the only person on the staff who is intricately involved in all three phases of production: preproduction (writing, casting, and location scouting), shooting, and editing.

“At the end of each season of ‘Seinfeld,’ he used to pray that the show would get cancelled,” Charles told me. “Now he knows the show’s not gonna get cancelled—he’ll have to make the decision to stop it at some point.” David told me that, in the first season, “I thought I’d be working seven months, maybe eight months at the most. And, of course, now it’s turned into twelve months. As soon as we’re done editing, I’m back into writing the next season.”

One day this fall, David sat in a darkened room in his Santa Monica offices, editing an episode for the upcoming season. He slouched on a sofa, his legs stretched out in front of him, and his hands clasped behind his neck. His long face was, as it often is, an impenetrable deadpan.

In front of David, at an Avid video-editing machine, sat a young man named Jon Corn, who wore a distressed baseball cap and a few days’ worth of stubble. Corn had one hand on a computer keyboard and the other on a mouse. Before him were three screens: two computer terminals and a video monitor. With a few clicks, Corn could summon any of the episode’s dozens of takes from the hard drives on which they were stored, and display them singly or in split screen.

At David’s left on the couch was Weide, a thin, bearded, professorial man with a perpetual air of being about to make a joke. In a chair to the right sat Larry Charles, who had directed the episode. Charles was wearing a tall plaid knit cap, a hooded jacket in a different plaid, blue pajama pants, and blue-and-yellow Converse high-tops. With his right hand, he fiddled with a string of worry beads. The scene, which depicted David being egged by a carful of teen-agers, was funny, but something was slightly off. Corn kept clicking the mouse, playing and replaying alternative sequences that ran only a few seconds each.

“What do you think?” Charles asked David. “Do we cut too quickly to, like, get the impact?”

“I kinda like seeing them driving away, enjoying themselves,” David said.

“Want to just add a couple seconds to the closeup?” Weide asked.

“That was about three o’clock in the morning, when we did that shot,” David mused. “That was the fourth egg I took.”

“What do you think about not cutting away?” Charles asked. “What if you just took out the shot of the guys driving away and just stayed on Larry in that shot? Heard the screeching and the laughing as they drove away, and the music—but stayed on the shot? It just lets it land a little bit.”

Corn clicked the mouse once more. The scene played on all three monitors, this time staying on David’s face after the egg hit. The image froze. His face was on all three screens, covered with egg yolk and bits of shell.

“Not bad,” David said. “Well, I don’t know! I don’t know. I mean, it’s kind of fun to see them drive away. On the other hand, it’s fun to see that egg.”

Summary: when doing Bayesian inference, sometimes the prior knowledge is vague enough for tractability concerns to come into play, so we use a prior distribution on parameters that is compatible with our likelihood function but also leads to a tractable posterior distribution after applying Bayes rule. Conjugate priors let you do this. Formally, if you have a family of likelihood functions L = { p(g|f) | f in F }, then a family of priors P = { p(f|θ) | θ in Θ } is a conjugate family for L if p(g|f) in L and p(f|θ) in P implies p(f|g) in P. The point is that in the Bayesian “posterior ~ likelihood * prior” relationship, you generally have some model form specified for the likelihood function, so you have some wiggle room in choosing the prior. If you choose a prior from the conjugate family for the likelihood function, the math becomes much nicer. Some of the most used examples: the Gaussian family is self-conjugate, and the Dirichlet distribution is conjugate to the multinomial distribution. More detailed examples are worked out in the slides.

We formulate necessary and sufficient conditions for an arbitrary discrete probability distribution to factor according to an undirected graphical model, or a log-linear model, or other more general exponential models. For decomposable graphical models these conditions are equivalent to a set of conditional independence statements similar to the Hammersley–Clifford theorem; however, we show that for nondecomposable graphical models they are not. We also show that nondecomposable models can have nonrational maximum likelihood estimates. These results are used to give several novel characterizations of decomposable graphical models.

I’d like to read this sometime. [Digression: the Hammersley-Clifford theorem still confuses me -- I have had trouble figuring out why one needs to assume that the probability distribution is positive even after seeing counterexamples showing that the theorem breaks otherwise. Fine, it breaks, but I still want an intuitive reason why. Apparently the assumption bothered Hammersley and Clifford too -- to the point of delaying publication -- so at least it's not just me being stupid.]

I became curious about the “toric algebra” business in the title, and this language apparently comes from a field called algebraic statistics; here’s a description from a short course on it:

Algebraic statistics advocates the use of algebraic geometry, commutative algebra, and geometric combinatorics as tools for making statistical inferences. The starting point for this connection is the observation that most statistical models for discrete random variables are, in fact, algebraic varieties. While some of the varieties that appear are classical varieties (like Segre varieties and toric varieties), most are new, and there are many challenging open problems about the algebraic structure of these varieties. These lectures will provide an introduction to algebraic statistics, emphasizing both the interesting algebraic questions that arise and the statistical consequences of the algebraic analysis.

This is neat, because I like algebra, and I’ve recently become interested in probability and statistics for machine learning, but as with math mashups, there is always the question of what the point is. But this sounds both interesting and applicable; the paper specifically discussing its application to graphical models sounds like a good way to get started after getting some background from the short course.

[Side note: Apparently, there are people who actually read this blog, though it was basically intended as a dumping ground for things I read. This is why there is almost no context when things are posted and insufficient explanation of technical material. In the last few months, I've actually read too much rather than too little to keep posting material here, though I would like to start again. Hopefully, one or two of the six or seven people who have looked here may be pleased about this. :) ]

THE NON-DENIAL OF THE NON-SELF
Aug 31st 2006

IN THE 1940s a philosopher called Carl Hempel showed that by manipulating the logical statement “all ravens are black”, you could derive the equivalent “all non-black objects are non-ravens”. Such topsy-turvy transformations might seem reason enough to keep philosophers locked up safely on university campuses, where they cannot do too much damage. However, a number of computer scientists, led by Fernando Esponda of Yale University, are taking Hempel’s notion as the germ of an eminently practical scheme. They are applying such negative representations to the problem of protecting sensitive data. The idea is to create a negative database. Instead of containing the information of interest, such a database would contain everything except that information.

The concept of a negative database took shape a couple of years ago, while Dr Esponda was working at the University of New Mexico with Paul Helman, another computer scientist, and Stephanie Forrest, an expert on modelling the human immune system. The important qualification concerns that word “everything”. In practice, that means everything in a particular set of things.

What interested Dr Esponda was how the immune system represents information. Here, “everything” is the set of possible biological molecules, notably proteins. The immune system is interesting, because it protects its owner from pathogens without needing to know what a pathogen will look like. Instead, it relies on a negative database to tell it what to destroy. It learns early on which biological molecules are “self”, in the sense that they are routine parts of the body it is protecting. Whenever it meets one that is “not self” and thus likely to be part of a pathogen, it destroys it. In Hempel’s terms, this can be expressed as “all non-good agents [pathogens] are non-self”.

The analogy with a computer database is not perfect. The set of possible biomolecules is not infinite, but it is certainly huge, and probably indeterminable. The immune system does not care about this, because it has to recognise only what is not in its own database. Make one adjustment, though, and you have something that might work for computers. That adjustment is to define “everything” as a finite set, all of whose members can be known–for instance, all phrases containing a fixed maximum number of characters.

A database of names, addresses and Social Security numbers (a common form of identification in America) might require only 200 characters to contain all possible combinations. That would limit the total number of character combinations. A positive database containing all the data in question would be a small subset of those combinations. The negative counterpart of this database would be much larger and contain all possible names and addresses that were not in the positive database plus a lot of gibberish. But it would not be infinite. By looking at the negative database, it would be possible to deduce what was in the positive database it complemented.

That would not guarantee security against a search for the presence or absence of a particular name and address. Indeed, the whole point is that such searches should be possible. But it would prevent fishing expeditions by making it impossible, for example, to look for the Social Security numbers of all the people living on one street.

Dr Esponda sees great potential for using negative databases when there is a need to look at the intersection of many sets of data owned by different parties. Two or more banks, for example, might wish to work out which transactions they have in common without revealing the whole contents of their databases. Using negative databases to do this would, according to Dr Esponda, provide a robust back-up to traditional cryptography, which relies on codes that can be broken.

An interesting extension of the idea might be to use negative surveys to collect sensitive information privately. Dr Esponda gives the example of a negative survey in which respondents are asked to tick the box of one sexually transmitted disease they do not have. He reckons that this would be sufficient to estimate the population frequency of each disease, without having to ask people whether they actually suffer from such diseases–which is intrusive and also invites lying. As he puts it: “In Hindu philosophy, to find out who you are, you ask what are you not. Then you are left with what you are.”

This paper is quite long, and not all of it is that interesting, so the entire article is not pasted below. Part of the introduction nicely summarizes the thesis:

One should not think of analogy-making as a special variety of reasoning (as in the dull and uninspiring phrase “analogical reasoning and problem-solving,” a long-standing cliché in the cognitive-science world), for that is to do analogy a terrible disservice. After all, reasoning and problem-solving have (at least I dearly hope!) been at long last recognized as lying far indeed from the core of human thought. If analogy were merely a special variety of something that in itself lies way out on the peripheries, then it would be but an itty-bitty blip in the broad blue sky of cognition. To me, however, analogy is anything but a bitty blip — rather, it’s the very blue that fills the whole sky of cognition — analogy is everything, or very nearly so, in my view. [...]

The thrust of my chapter is to persuade readers of this unorthodox viewpoint, or failing that, at least to give them a strong whiff of it. In that sense, then, my article shares with Richard Dawkins’s eye-opening book The Selfish Gene (Dawkins 1976) the quality of trying to make a scientific contribution mostly by suggesting to readers a shift of viewpoint — a new take on familiar phenomena. For Dawkins, the shift was to turn causality on its head, so that the old quip “a chicken is an egg’s way of making another egg” might be taken not as a joke but quite seriously. In my case, the shift is to suggest that every concept we have is essentially nothing but a tightly packaged bundle of analogies, and to suggest that all we do when we think is to move fluidly from concept to concept — in other words, to leap from one analogy-bundle to another — and to suggest, lastly, that such concept-to-concept leaps are themselves made via analogical connection, to boot.

This essay reminds me of two things: the Barry Mazur essay on category theory that I posted a few days ago, and Jeff Hawkins’ On Intelligence. (Warning: Hofstadter uses the word “category” a lot in his essay, but this has nothing to do with category theory.)

Analogies and Functors

There is an obvious analogy to be made between functors (or any morphisms) and analogies. A functor translates the objects and relationships from one mathematical theory to another; the canonical example is that of the fundamental group functor in algebraic topology, which lets us turn problems of topology into those of group theory. What is this if not a rigorous kind of analogy?

The Hofstadter paper is interesting in that it proposes an extension of this relationship. In particular, Mazur talks about replacing a mathematical object with its network of relationships, while Hofstadter talks about replacing concepts with their “bundle of analogies,” which is more or less the same idea. Recall what Mazur says:

The lights are dimmed on mathematical objects and beamed rather on the corresponding functors; that is, on the networks of relationships entailed by the objects. The functor has center stage, the object that it represents appears almost as an afterthought.

What Hofstadter proposes in his paper is to similarly dim the lights on “concepts” and beam them instead on their networks of analogies; analogies should be center stage when discussing cognition in the same way that functors are frequently center stage when discussing mathematics.

Is there anything deeper to this connection? I don’t know. At least if it’s a superficiality, it’s a neat one.

Analogies and Hierarchical Temporal Memories

It would be interesting to go back to On Intelligence and see how much of this fits with Jeff Hawkins’ theory of how the brain works. At first glance, many of the passages from the essay seem to agree with Hawkins’ ideas. I’m not going to compare them or review them in detail here, but just quote some relevant passages from both.

In this passage, Hofstadter discusses something that he calls “chunking,” which has obvious similarities to the hierarchy in the perceptual system that Hawkins describes.

We begin with a couple of simple queries about familiar phenomena: “Why do babies not remember events that happen to them?” and “Why does each new year seem to pass faster than the one before?”

I wouldn’t swear that I have the final answer to either one of these queries, but I do have a hunch, and I will here speculate on the basis of that hunch. And thus: the answer to both is basically the same, I would argue, and it has to do with the relentless, lifelong process of chunking — taking “small” concepts and putting them together into bigger and bigger ones, thus recursively building up a giant repertoire of concepts in the mind.

How, then, might chunking provide the clue to these riddles? Well, babies’ concepts are simply too small. They have no way of framing entire events whatsoever in terms of their novice concepts. It is as if babies were looking at life through a randomly drifting keyhole, and at each moment could make out only the most local aspects of scenes before them. It would be hopeless to try to figure out how a whole room is organized, for instance, given just a keyhole view, even a randomly drifting keyhole view.

Or, to trot out another analogy, life is like a chess game, and babies are like beginners looking at a complex scene on a board, not having the faintest idea how to organize it into higher-level structures. As has been well known for decades, experienced chess players chunk the setup of pieces on the board nearly instantaneously into small dynamic groupings defined by their strategic meanings, and thanks to this automatic, intuitive chunking, they can make good moves nearly instantaneously and also can remember complex chess situations for very long times. Much the same holds for bridge players, who effortlessly remember every bid and every play in a game, and months later can still recite entire games at the drop of a hat.

Here, Hofstadter discusses the disconnect between sensory input and high-level perception. This is consistent with what Hawkins says would happen in an HTM at the higher levels of the hierarchy.

In fact, I should stress that the upper echelons of high-level perception totally transcend the normal flavor of the word “perception,” for at the highest levels, input modality plays essentially no role. Let me explain. Suppose I read a newspaper article about the violent expulsion of one group of people by another group from some geographical region, and the phrase “ethnic cleansing,” nowhere present in the article, pops into my head. What has happened here is a quintessential example of high-level perception — but what was the input medium? Someone might say it was vision, since I used my eyes to read the newspaper. But really, was I perceiving ethnic cleansing visually? Hardly. Indeed, I might have heard the newspaper article read aloud to me and had the same exact thought pop to mind. Would that mean that I had aurally perceived ethnic cleansing? Or else I might be blind and have read the article in Braille — in other words, with my fingertips, not my eyes or ears. Would that mean that I had tactilely perceived ethnic cleansing? The suggestion is absurd.

The sensory input modality of a complex story is totally irrelevant; all that matters is how it jointly activates a host of interrelated concepts, in such a way that further concepts (e.g., “ethnic cleansing”) are automatically accessed and brought up to center stage. [...]

The triggering of prior mental categories by some kind of input — whether sensory or more abstract — is, I insist, an act of analogy-making. Why is this? Because whenever a set of incoming stimuli activates one or more mental categories, some amount of slippage must occur (no instance of a category ever being precisely identical to a prior instance). Categories are quintessentially fluid entities; they adapt to a set of incoming stimuli and try to align themselves with it. The process of inexact matching between prior categories and new things being perceived (whether those “things” are physical objects or bite-size events or grand sagas) is analogy-making par excellence. How could anyone deny this? After all, it is the mental mapping onto each other of two entities — one old and sound asleep in the recesses of long-term memory, the other new and gaily dancing on the mind’s center stage — that in fact differ from each other in a myriad of ways.

Below, Hofstadter makes some comments on the common core underlying various things out in the world, which gels with Hawkins’ emphasis on “discovering causes.”

I now make an observation that, though banal and obvious, needs to be made explicitly nonetheless — namely, things “out there” (objects, situations, whatever) that are labeled by the same lexical item have something, some core, in common; also, whatever it is that those things “out there” share is shared with the abstract mental structure that lurks behind the label used for them. Getting to the core of things is, after all, what categories are for. In fact, I would go somewhat further and claim that getting to the core of things is what thinking itself is for-thus once again placing high-level perception front and center in the definition of cognition.

For comparison, consider the following paragraph from Numenta’s whitepaper on how HTMs work. The connection to the passage directly above should be fairly clear.

The HTM receives the spatio-temporal pattern coming from the senses. At first, the HTM has no knowledge of the causes in the world, but through a learning process that will be described below, it “discovers” what the causes are. The end goal of this process is that the HTM develops internal representations of the causes in the world. In a brain, nerve cells learn to represent causes in the world, such as a cell that becomes active whenever you see a face. In an HTM, causes are represented by numbers in a vector. At any moment in time, given current and past input, an HTM will assign a likelihood that individual causes are currently being sensed. The HTM’s output is manifest as a set of probabilities for each of the learned causes. This moment-to- moment distribution of possible causes is called a “belief”. If an HTM knows about ten causes in the world, it will have ten variables representing those causes. The value of those variables – its belief – is what the HTM believes is happening in its world at that instant.

While none of this is necessarily that unexpected, and I may be making a mountain out of a molehill, it’s still interesting that so much of what they say appears to overlap. It seems that Hofstadter agrees with Hawkins about what the core activity of the brain is, and in particular, he wants to call that activity analogy-making. (Perhaps Hawkins mentioned analogies in his book too, and I’ve simply forgotten.) In any case, I need to think about it more; I’m still not sure whether this is something or nothing.

On the evening of June 20th, several hundred physicists, including a Nobel laureate, assembled in an auditorium at the Friendship Hotel in Beijing for a lecture by the Chinese mathematician Shing-Tung Yau. In the late nineteen-seventies, when Yau was in his twenties, he had made a series of breakthroughs that helped launch the string-theory revolution in physics and earned him, in addition to a Fields Medal—the most coveted award in mathematics—a reputation in both disciplines as a thinker of unrivalled technical power.

Yau had since become a professor of mathematics at Harvard and the director of mathematics institutes in Beijing and Hong Kong, dividing his time between the United States and China. His lecture at the Friendship Hotel was part of an international conference on string theory, which he had organized with the support of the Chinese government, in part to promote the country’s recent advances in theoretical physics. (More than six thousand students attended the keynote address, which was delivered by Yau’s close friend Stephen Hawking, in the Great Hall of the People.) The subject of Yau’s talk was something that few in his audience knew much about: the Poincaré conjecture, a century-old conundrum about the characteristics of three-dimensional spheres, which, because it has important implications for mathematics and cosmology and because it has eluded all attempts at solution, is regarded by mathematicians as a holy grail.

Yau, a stocky man of fifty-seven, stood at a lectern in shirtsleeves and black-rimmed glasses and, with his hands in his pockets, described how two of his students, Xi-Ping Zhu and Huai-Dong Cao, had completed a proof of the Poincaré conjecture a few weeks earlier. “I’m very positive about Zhu and Cao’s work,” Yau said. “Chinese mathematicians should have every reason to be proud of such a big success in completely solving the puzzle.” He said that Zhu and Cao were indebted to his longtime American collaborator Richard Hamilton, who deserved most of the credit for solving the Poincaré. He also mentioned Grigory Perelman, a Russian mathematician who, he acknowledged, had made an important contribution. Nevertheless, Yau said, “in Perelman’s work, spectacular as it is, many key ideas of the proofs are sketched or outlined, and complete details are often missing.” He added, “We would like to get Perelman to make comments. But Perelman resides in St. Petersburg and refuses to communicate with other people.”

For ninety minutes, Yau discussed some of the technical details of his students’ proof. When he was finished, no one asked any questions. That night, however, a Brazilian physicist posted a report of the lecture on his blog. “Looks like China soon will take the lead also in mathematics,” he wrote.

Grigory Perelman is indeed reclusive. He left his job as a researcher at the Steklov Institute of Mathematics, in St. Petersburg, last December; he has few friends; and he lives with his mother in an apartment on the outskirts of the city. Although he had never granted an interview before, he was cordial and frank when we visited him, in late June, shortly after Yau’s conference in Beijing, taking us on a long walking tour of the city. “I’m looking for some friends, and they don’t have to be mathematicians,” he said. The week before the conference, Perelman had spent hours discussing the Poincaré conjecture with Sir John M. Ball, the fifty-eight-year-old president of the International Mathematical Union, the discipline’s influential professional association. The meeting, which took place at a conference center in a stately mansion overlooking the Neva River, was highly unusual. At the end of May, a committee of nine prominent mathematicians had voted to award Perelman a Fields Medal for his work on the Poincaré, and Ball had gone to St. Petersburg to persuade him to accept the prize in a public ceremony at the I.M.U.’s quadrennial congress, in Madrid, on August 22nd.

The Fields Medal, like the Nobel Prize, grew, in part, out of a desire to elevate science above national animosities. German mathematicians were excluded from the first I.M.U. congress, in 1924, and, though the ban was lifted before the next one, the trauma it caused led, in 1936, to the establishment of the Fields, a prize intended to be “as purely international and impersonal as possible.”

However, the Fields Medal, which is awarded every four years, to between two and four mathematicians, is supposed not only to reward past achievements but also to stimulate future research; for this reason, it is given only to mathematicians aged forty and younger. In recent decades, as the number of professional mathematicians has grown, the Fields Medal has become increasingly prestigious. Only forty-four medals have been awarded in nearly seventy years—including three for work closely related to the Poincaré conjecture—and no mathematician has ever refused the prize. Nevertheless, Perelman told Ball that he had no intention of accepting it. “I refuse,” he said simply.

Over a period of eight months, beginning in November, 2002, Perelman posted a proof of the Poincaré on the Internet in three installments. Like a sonnet or an aria, a mathematical proof has a distinct form and set of conventions. It begins with axioms, or accepted truths, and employs a series of logical statements to arrive at a conclusion. If the logic is deemed to be watertight, then the result is a theorem. Unlike proof in law or science, which is based on evidence and therefore subject to qualification and revision, a proof of a theorem is definitive. Judgments about the accuracy of a proof are mediated by peer-reviewed journals; to insure fairness, reviewers are supposed to be carefully chosen by journal editors, and the identity of a scholar whose pa-per is under consideration is kept secret. Publication implies that a proof is complete, correct, and original.

By these standards, Perelman’s proof was unorthodox. It was astonishingly brief for such an ambitious piece of work; logic sequences that could have been elaborated over many pages were often severely compressed. Moreover, the proof made no direct mention of the Poincaré and included many elegant results that were irrelevant to the central argument. But, four years later, at least two teams of experts had vetted the proof and had found no significant gaps or errors in it. A consensus was emerging in the math community: Perelman had solved the Poincaré. Even so, the proof’s complexity—and Perelman’s use of shorthand in making some of his most important claims—made it vulnerable to challenge. Few mathematicians had the expertise necessary to evaluate and defend it.

After giving a series of lectures on the proof in the United States in 2003, Perelman returned to St. Petersburg. Since then, although he had continued to answer queries about it by e-mail, he had had minimal contact with colleagues and, for reasons no one understood, had not tried to publish it. Still, there was little doubt that Perelman, who turned forty on June 13th, deserved a Fields Medal. As Ball planned the I.M.U.’s 2006 congress, he began to conceive of it as a historic event. More than three thousand mathematicians would be attending, and King Juan Carlos of Spain had agreed to preside over the awards ceremony. The I.M.U.’s newsletter predicted that the congress would be remembered as “the occasion when this conjecture became a theorem.” Ball, determined to make sure that Perelman would be there, decided to go to St. Petersburg.

Ball wanted to keep his visit a secret—the names of Fields Medal recipients are announced officially at the awards ceremony—and the conference center where he met with Perelman was deserted. For ten hours over two days, he tried to persuade Perelman to agree to accept the prize. Perelman, a slender, balding man with a curly beard, bushy eyebrows, and blue-green eyes, listened politely. He had not spoken English for three years, but he fluently parried Ball’s entreaties, at one point taking Ball on a long walk—one of Perelman’s favorite activities. As he summed up the conversation two weeks later: “He proposed to me three alternatives: accept and come; accept and don’t come, and we will send you the medal later; third, I don’t accept the prize. From the very beginning, I told him I have chosen the third one.” The Fields Medal held no interest for him, Perelman explained. “It was completely irrelevant for me,” he said. “Everybody understood that if the proof is correct then no other recognition is needed.”

Proofs of the Poincaré have been announced nearly every year since the conjecture was formulated, by Henri Poincaré, more than a hundred years ago. Poincaré was a cousin of Raymond Poincaré, the President of France during the First World War, and one of the most creative mathematicians of the nineteenth century. Slight, myopic, and notoriously absent-minded, he conceived his famous problem in 1904, eight years before he died, and tucked it as an offhand question into the end of a sixty-five-page paper.

Poincaré didn’t make much progress on proving the conjecture. “Cette question nous entraînerait trop loin” (“This question would take us too far”), he wrote. He was a founder of topology, also known as “rubber-sheet geometry,” for its focus on the intrinsic properties of spaces. From a topologist’s perspective, there is no difference between a bagel and a coffee cup with a handle. Each has a single hole and can be manipulated to resemble the other without being torn or cut. Poincaré used the term “manifold” to describe such an abstract topological space. The simplest possible two-dimensional manifold is the surface of a soccer ball, which, to a topologist, is a sphere—even when it is stomped on, stretched, or crumpled. The proof that an object is a so-called two-sphere, since it can take on any number of shapes, is that it is “simply connected,” meaning that no holes puncture it. Unlike a soccer ball, a bagel is not a true sphere. If you tie a slipknot around a soccer ball, you can easily pull the slipknot closed by sliding it along the surface of the ball. But if you tie a slipknot around a bagel through the hole in its middle you cannot pull the slipknot closed without tearing the bagel.

Two-dimensional manifolds were well understood by the mid-nineteenth century. But it remained unclear whether what was true for two dimensions was also true for three. Poincaré proposed that all closed, simply connected, three-dimensional manifolds—those which lack holes and are of finite extent—were spheres. The conjecture was potentially important for scientists studying the largest known three-dimensional manifold: the universe. Proving it mathematically, however, was far from easy. Most attempts were merely embarrassing, but some led to important mathematical discoveries, including proofs of Dehn’s Lemma, the Sphere Theorem, and the Loop Theorem, which are now fundamental concepts in topology.

By the nineteen-sixties, topology had become one of the most productive areas of mathematics, and young topologists were launching regular attacks on the Poincaré. To the astonishment of most mathematicians, it turned out that manifolds of the fourth, fifth, and higher dimensions were more tractable than those of the third dimension. By 1982, Poincaré’s conjecture had been proved in all dimensions except the third. In 2000, the Clay Mathematics Institute, a private foundation that promotes mathematical research, named the Poincaré one of the seven most important outstanding problems in mathematics and offered a million dollars to anyone who could prove it.

“My whole life as a mathematician has been dominated by the Poincaré conjecture,” John Morgan, the head of the mathematics department at Columbia University, said. “I never thought I’d see a solution. I thought nobody could touch it.”

Grigory Perelman did not plan to become a mathematician. “There was never a decision point,” he said when we met. We were outside the apartment building where he lives, in Kupchino, a neighborhood of drab high-rises. Perelman’s father, who was an electrical engineer, encouraged his interest in math. “He gave me logical and other math problems to think about,” Perelman said. “He got a lot of books for me to read. He taught me how to play chess. He was proud of me.” Among the books his father gave him was a copy of “Physics for Entertainment,” which had been a best-seller in the Soviet Union in the nineteen-thirties. In the foreword, the book’s author describes the contents as “conundrums, brain-teasers, entertaining anecdotes, and unexpected comparisons,” adding, “I have quoted extensively from Jules Verne, H. G. Wells, Mark Twain and other writers, because, besides providing entertainment, the fantastic experiments these writers describe may well serve as instructive illustrations at physics classes.” The book’s topics included how to jump from a moving car, and why, “according to the law of buoyancy, we would never drown in the Dead Sea.”

The notion that Russian society considered worthwhile what Perelman did for pleasure came as a surprise. By the time he was fourteen, he was the star performer of a local math club. In 1982, the year that Shing-Tung Yau won a Fields Medal, Perelman earned a perfect score and the gold medal at the International Mathematical Olympiad, in Budapest. He was friendly with his teammates but not close—“I had no close friends,” he said. He was one of two or three Jews in his grade, and he had a passion for opera, which also set him apart from his peers. His mother, a math teacher at a technical college, played the violin and began taking him to the opera when he was six. By the time Perelman was fifteen, he was spending his pocket money on records. He was thrilled to own a recording of a famous 1946 performance of “La Traviata,” featuring Licia Albanese as Violetta. “Her voice was very good,” he said.

At Leningrad University, which Perelman entered in 1982, at the age of sixteen, he took advanced classes in geometry and solved a problem posed by Yuri Burago, a mathematician at the Steklov Institute, who later became his Ph.D. adviser. “There are a lot of students of high ability who speak before thinking,” Burago said. “Grisha was different. He thought deeply. His answers were always correct. He always checked very, very carefully.” Burago added, “He was not fast. Speed means nothing. Math doesn’t depend on speed. It is about deep.”

At the Steklov in the early nineties, Perelman became an expert on the geometry of Riemannian and Alexandrov spaces—extensions of traditional Euclidean geometry—and began to publish articles in the leading Russian and American mathematics journals. In 1992, Perelman was invited to spend a semester each at New York University and Stony Brook University. By the time he left for the United States, that fall, the Russian economy had collapsed. Dan Stroock, a mathematician at M.I.T., recalls smuggling wads of dollars into the country to deliver to a retired mathematician at the Steklov, who, like many of his colleagues, had become destitute.

Perelman was pleased to be in the United States, the capital of the international mathematics community. He wore the same brown corduroy jacket every day and told friends at N.Y.U. that he lived on a diet of bread, cheese, and milk. He liked to walk to Brooklyn, where he had relatives and could buy traditional Russian brown bread. Some of his colleagues were taken aback by his fingernails, which were several inches long. “If they grow, why wouldn’t I let them grow?” he would say when someone asked why he didn’t cut them. Once a week, he and a young Chinese mathematician named Gang Tian drove to Princeton, to attend a seminar at the Institute for Advanced Study.

For several decades, the institute and nearby Princeton University had been centers of topological research. In the late seventies, William Thurston, a Princeton mathematician who liked to test out his ideas using scissors and construction paper, proposed a taxonomy for classifying manifolds of three dimensions. He argued that, while the manifolds could be made to take on many different shapes, they nonetheless had a “preferred” geometry, just as a piece of silk draped over a dressmaker’s mannequin takes on the mannequin’s form.

Thurston proposed that every three-dimensional manifold could be broken down into one or more of eight types of component, including a spherical type. Thurston’s theory—which became known as the geometrization conjecture—describes all possible three-dimensional manifolds and is thus a powerful generalization of the Poincaré. If it was confirmed, then Poincaré’s conjecture would be, too. Proving Thurston and Poincaré “definitely swings open doors,” Barry Mazur, a mathematician at Harvard, said. The implications of the conjectures for other disciplines may not be apparent for years, but for mathematicians the problems are fundamental. “This is a kind of twentieth-century Pythagorean theorem,” Mazur added. “It changes the landscape.”

In 1982, Thurston won a Fields Medal for his contributions to topology. That year, Richard Hamilton, a mathematician at Cornell, published a paper on an equation called the Ricci flow, which he suspected could be relevant for solving Thurston’s conjecture and thus the Poincaré. Like a heat equation, which describes how heat distributes itself evenly through a substance—flowing from hotter to cooler parts of a metal sheet, for example—to create a more uniform temperature, the Ricci flow, by smoothing out irregularities, gives manifolds a more uniform geometry.

Hamilton, the son of a Cincinnati doctor, defied the math profession’s nerdy stereotype. Brash and irreverent, he rode horses, windsurfed, and had a succession of girlfriends. He treated math as merely one of life’s pleasures. At forty-nine, he was considered a brilliant lecturer, but he had published relatively little beyond a series of seminal articles on the Ricci flow, and he had few graduate students. Perelman had read Hamilton’s papers and went to hear him give a talk at the Institute for Advanced Study. Afterward, Perelman shyly spoke to him.

“I really wanted to ask him something,” Perelman recalled. “He was smiling, and he was quite patient. He actually told me a couple of things that he published a few years later. He did not hesitate to tell me. Hamilton’s openness and generosity—it really attracted me. I can’t say that most mathematicians act like that.

“I was working on different things, though occasionally I would think about the Ricci flow,” Perelman added. “You didn’t have to be a great mathematician to see that this would be useful for geometrization. I felt I didn’t know very much. I kept asking questions.”

Shing-Tung Yau was also asking Hamilton questions about the Ricci flow. Yau and Hamilton had met in the seventies, and had become close, despite considerable differences in temperament and background. A mathematician at the University of California at San Diego who knows both men called them “the mathematical loves of each other’s lives.”

Yau’s family moved to Hong Kong from mainland China in 1949, when he was five months old, along with hundreds of thousands of other refugees fleeing Mao’s armies. The previous year, his father, a relief worker for the United Nations, had lost most of the family’s savings in a series of failed ventures. In Hong Kong, to support his wife and eight children, he tutored college students in classical Chinese literature and philosophy.

When Yau was fourteen, his father died of kidney cancer, leaving his mother dependent on handouts from Christian missionaries and whatever small sums she earned from selling handicrafts. Until then, Yau had been an indifferent student. But he began to devote himself to schoolwork, tutoring other students in math to make money. “Part of the thing that drives Yau is that he sees his own life as being his father’s revenge,” said Dan Stroock, the M.I.T. mathematician, who has known Yau for twenty years. “Yau’s father was like the Talmudist whose children are starving.”

Yau studied math at the Chinese University of Hong Kong, where he attracted the attention of Shiing-Shen Chern, the preëminent Chinese mathematician, who helped him win a scholarship to the University of California at Berkeley. Chern was the author of a famous theorem combining topology and geometry. He spent most of his career in the United States, at Berkeley. He made frequent visits to Hong Kong, Taiwan, and, later, China, where he was a revered symbol of Chinese intellectual achievement, to promote the study of math and science.

In 1969, Yau started graduate school at Berkeley, enrolling in seven graduate courses each term and auditing several others. He sent half of his scholarship money back to his mother in China and impressed his professors with his tenacity. He was obliged to share credit for his first major result when he learned that two other mathematicians were working on the same problem. In 1976, he proved a twenty-year-old conjecture pertaining to a type of manifold that is now crucial to string theory. A French mathematician had formulated a proof of the problem, which is known as Calabi’s conjecture, but Yau’s, because it was more general, was more powerful. (Physicists now refer to Calabi-Yau manifolds.) “He was not so much thinking up some original way of looking at a subject but solving extremely hard technical problems that at the time only he could solve, by sheer intellect and force of will,” Phillip Griffiths, a geometer and a former director of the Institute for Advanced Study, said.

In 1980, when Yau was thirty, he became one of the youngest mathematicians ever to be appointed to the permanent faculty of the Institute for Advanced Study, and he began to attract talented students. He won a Fields Medal two years later, the first Chinese ever to do so. By this time, Chern was seventy years old and on the verge of retirement. According to a relative of Chern’s, “Yau decided that he was going to be the next famous Chinese mathematician and that it was time for Chern to step down.”

Harvard had been trying to recruit Yau, and when, in 1983, it was about to make him a second offer Phillip Griffiths told the dean of faculty a version of a story from “The Romance of the Three Kingdoms,” a Chinese classic. In the third century A.D., a Chinese warlord dreamed of creating an empire, but the most brilliant general in China was working for a rival. Three times, the warlord went to his enemy’s kingdom to seek out the general. Impressed, the general agreed to join him, and together they succeeded in founding a dynasty. Taking the hint, the dean flew to Philadelphia, where Yau lived at the time, to make him an offer. Even so, Yau turned down the job. Finally, in 1987, he agreed to go to Harvard.

Yau’s entrepreneurial drive extended to collaborations with colleagues and students, and, in addition to conducting his own research, he began organizing seminars. He frequently allied himself with brilliantly inventive mathematicians, including Richard Schoen and William Meeks. But Yau was especially impressed by Hamilton, as much for his swagger as for his imagination. “I can have fun with Hamilton,” Yau told us during the string-theory conference in Beijing. “I can go swimming with him. I go out with him and his girlfriends and all that.” Yau was convinced that Hamilton could use the Ricci-flow equation to solve the Poincaré and Thurston conjectures, and he urged him to focus on the problems. “Meeting Yau changed his mathematical life,” a friend of both mathematicians said of Hamilton. “This was the first time he had been on to something extremely big. Talking to Yau gave him courage and direction.”

Yau believed that if he could help solve the Poincaré it would be a victory not just for him but also for China. In the mid-nineties, Yau and several other Chinese scholars began meeting with President Jiang Zemin to discuss how to rebuild the country’s scientific institutions, which had been largely destroyed during the Cultural Revolution. Chinese universities were in dire condition. According to Steve Smale, who won a Fields for proving the Poincaré in higher dimensions, and who, after retiring from Berkeley, taught in Hong Kong, Peking University had “halls filled with the smell of urine, one common room, one office for all the assistant professors,” and paid its faculty wretchedly low salaries. Yau persuaded a Hong Kong real-estate mogul to help finance a mathematics institute at the Chinese Academy of Sciences, in Beijing, and to endow a Fields-style medal for Chinese mathematicians under the age of forty-five. On his trips to China, Yau touted Hamilton and their joint work on the Ricci flow and the Poincaré as a model for young Chinese mathematicians. As he put it in Beijing, “They always say that the whole country should learn from Mao or some big heroes. So I made a joke to them, but I was half serious. I said the whole country should learn from Hamilton.”

Grigory Perelman was learning from Hamilton already. In 1993, he began a two-year fellowship at Berkeley. While he was there, Hamilton gave several talks on campus, and in one he mentioned that he was working on the Poincaré. Hamilton’s Ricci-flow strategy was extremely technical and tricky to execute. After one of his talks at Berkeley, he told Perelman about his biggest obstacle. As a space is smoothed under the Ricci flow, some regions deform into what mathematicians refer to as “singularities.” Some regions, called “necks,” become attenuated areas of infinite density. More troubling to Hamilton was a kind of singularity he called the “cigar.” If cigars formed, Hamilton worried, it might be impossible to achieve uniform geometry. Perelman realized that a paper he had written on Alexandrov spaces might help Hamilton prove Thurston’s conjecture—and the Poincaré—once Hamilton solved the cigar problem. “At some point, I asked Hamilton if he knew a certain collapsing result that I had proved but not published—which turned out to be very useful,” Perelman said. “Later, I realized that he didn’t understand what I was talking about.” Dan Stroock, of M.I.T., said, “Perelman may have learned stuff from Yau and Hamilton, but, at the time, they were not learning from him.”

By the end of his first year at Berkeley, Perelman had written several strikingly original papers. He was asked to give a lecture at the 1994 I.M.U. congress, in Zurich, and invited to apply for jobs at Stanford, Princeton, the Institute for Advanced Study, and the University of Tel Aviv. Like Yau, Perelman was a formidable problem solver. Instead of spending years constructing an intricate theoretical framework, or defining new areas of research, he focussed on obtaining particular results. According to Mikhail Gromov, a renowned Russian geometer who has collaborated with Perelman, he had been trying to overcome a technical difficulty relating to Alexandrov spaces and had apparently been stumped. “He couldn’t do it,” Gromov said. “It was hopeless.”

Perelman told us that he liked to work on several problems at once. At Berkeley, however, he found himself returning again and again to Hamilton’s Ricci-flow equation and the problem that Hamilton thought he could solve with it. Some of Perelman’s friends noticed that he was becoming more and more ascetic. Visitors from St. Petersburg who stayed in his apartment were struck by how sparsely furnished it was. Others worried that he seemed to want to reduce life to a set of rigid axioms. When a member of a hiring committee at Stanford asked him for a C.V. to include with requests for letters of recommendation, Perelman balked. “If they know my work, they don’t need my C.V.,” he said. “If they need my C.V., they don’t know my work.”

Ultimately, he received several job offers. But he declined them all, and in the summer of 1995 returned to St. Petersburg, to his old job at the Steklov Institute, where he was paid less than a hundred dollars a month. (He told a friend that he had saved enough money in the United States to live on for the rest of his life.) His father had moved to Israel two years earlier, and his younger sister was planning to join him there after she finished college. His mother, however, had decided to remain in St. Petersburg, and Perelman moved in with her. “I realize that in Russia I work better,” he told colleagues at the Steklov.

At twenty-nine, Perelman was firmly established as a mathematician and yet largely unburdened by professional responsibilities. He was free to pursue whatever problems he wanted to, and he knew that his work, should he choose to publish it, would be shown serious consideration. Yakov Eliashberg, a mathematician at Stanford who knew Perelman at Berkeley, thinks that Perelman returned to Russia in order to work on the Poincaré. “Why not?” Perelman said when we asked whether Eliashberg’s hunch was correct.

The Internet made it possible for Perelman to work alone while continuing to tap a common pool of knowledge. Perelman searched Hamilton’s papers for clues to his thinking and gave several seminars on his work. “He didn’t need any help,” Gromov said. “He likes to be alone. He reminds me of Newton—this obsession with an idea, working by yourself, the disregard for other people’s opinion. Newton was more obnoxious. Perelman is nicer, but very obsessed.”

In 1995, Hamilton published a paper in which he discussed a few of his ideas for completing a proof of the Poincaré. Reading the paper, Perelman realized that Hamilton had made no progress on overcoming his obstacles—the necks and the cigars. “I hadn’t seen any evidence of progress after early 1992,” Perelman told us. “Maybe he got stuck even earlier.” However, Perelman thought he saw a way around the impasse. In 1996, he wrote Hamilton a long letter outlining his notion, in the hope of collaborating. “He did not answer,” Perelman said. “So I decided to work alone.”

Yau had no idea that Hamilton’s work on the Poincaré had stalled. He was increasingly anxious about his own standing in the mathematics profession, particularly in China, where, he worried, a younger scholar could try to supplant him as Chern’s heir. More than a decade had passed since Yau had proved his last major result, though he continued to publish prolifically. “Yau wants to be the king of geometry,” Michael Anderson, a geometer at Stony Brook, said. “He believes that everything should issue from him, that he should have oversight. He doesn’t like people encroaching on his territory.” Determined to retain control over his field, Yau pushed his students to tackle big problems. At Harvard, he ran a notoriously tough seminar on differential geometry, which met for three hours at a time three times a week. Each student was assigned a recently published proof and asked to reconstruct it, fixing any errors and filling in gaps. Yau believed that a mathematician has an obligation to be explicit, and impressed on his students the importance of step-by-step rigor.

There are two ways to get credit for an original contribution in mathematics. The first is to produce an original proof. The second is to identify a significant gap in someone else’s proof and supply the missing chunk. However, only true mathematical gaps—missing or mistaken arguments—can be the basis for a claim of originality. Filling in gaps in exposition—shortcuts and abbreviations used to make a proof more efficient—does not count. When, in 1993, Andrew Wiles revealed that a gap had been found in his proof of Fermat’s last theorem, the problem became fair game for anyone, until, the following year, Wiles fixed the error. Most mathematicians would agree that, by contrast, if a proof’s implicit steps can be made explicit by an expert, then the gap is merely one of exposition, and the proof should be considered complete and correct.

Occasionally, the difference between a mathematical gap and a gap in exposition can be hard to discern. On at least one occasion, Yau and his students have seemed to confuse the two, making claims of originality that other mathematicians believe are unwarranted. In 1996, a young geometer at Berkeley named Alexander Givental had proved a mathematical conjecture about mirror symmetry, a concept that is fundamental to string theory. Though other mathematicians found Givental’s proof hard to follow, they were optimistic that he had solved the problem. As one geometer put it, “Nobody at the time said it was incomplete and incorrect.”

In the fall of 1997, Kefeng Liu, a former student of Yau’s who taught at Stanford, gave a talk at Harvard on mirror symmetry. According to two geometers in the audience, Liu proceeded to present a proof strikingly similar to Givental’s, describing it as a paper that he had co-authored with Yau and another student of Yau’s. “Liu mentioned Givental but only as one of a long list of people who had contributed to the field,” one of the geometers said. (Liu maintains that his proof was significantly different from Givental’s.)

Around the same time, Givental received an e-mail signed by Yau and his collaborators, explaining that they had found his arguments impossible to follow and his notation baffling, and had come up with a proof of their own. They praised Givental for his “brilliant idea” and wrote, “In the final version of our paper your important contribution will be acknowledged.”

A few weeks later, the paper, “Mirror Principle I,” appeared in the Asian Journal of Mathematics, which is co-edited by Yau. In it, Yau and his coauthors describe their result as “the first complete proof” of the mirror conjecture. They mention Givental’s work only in passing. “Unfortunately,” they write, his proof, “which has been read by many prominent experts, is incomplete.” However, they did not identify a specific mathematical gap.

Givental was taken aback. “I wanted to know what their objection was,” he told us. “Not to expose them or defend myself.” In March, 1998, he published a paper that included a three-page footnote in which he pointed out a number of similarities between Yau’s proof and his own. Several months later, a young mathematician at the University of Chicago who was asked by senior colleagues to investigate the dispute concluded that Givental’s proof was complete. Yau says that he had been working on the proof for years with his students and that they achieved their result independently of Givental. “We had our own ideas, and we wrote them up,” he says.

Around this time, Yau had his first serious conflict with Chern and the Chinese mathematical establishment. For years, Chern had been hoping to bring the I.M.U.’s congress to Beijing. According to several mathematicians who were active in the I.M.U. at the time, Yau made an eleventh-hour effort to have the congress take place in Hong Kong instead. But he failed to persuade a sufficient number of colleagues to go along with his proposal, and the I.M.U. ultimately decided to hold the 2002 congress in Beijing. (Yau denies that he tried to bring the congress to Hong Kong.) Among the delegates the I.M.U. appointed to a group that would be choosing speakers for the congress was Yau’s most successful student, Gang Tian, who had been at N.Y.U. with Perelman and was now a professor at M.I.T. The host committee in Beijing also asked Tian to give a plenary address.

Yau was caught by surprise. In March, 2000, he had published a survey of recent research in his field studded with glowing references to Tian and to their joint projects. He retaliated by organizing his first conference on string theory, which opened in Beijing a few days before the math congress began, in late August, 2002. He persuaded Stephen Hawking and several Nobel laureates to attend, and for days the Chinese newspapers were full of pictures of famous scientists. Yau even managed to arrange for his group to have an audience with Jiang Zemin. A mathematician who helped organize the math congress recalls that along the highway between Beijing and the airport there were “billboards with pictures of Stephen Hawking plastered everywhere.”

That summer, Yau wasn’t thinking much about the Poincaré. He had confidence in Hamilton, despite his slow pace. “Hamilton is a very good friend,” Yau told us in Beijing. “He is more than a friend. He is a hero. He is so original. We were working to finish our proof. Hamilton worked on it for twenty-five years. You work, you get tired. He probably got a little tired—and you want to take a rest.”

Then, on November 12, 2002, Yau received an e-mail message from a Russian mathematician whose name didn’t immediately register. “May I bring to your attention my paper,” the e-mail said.

On November 11th, Perelman had posted a thirty-nine-page paper entitled “The Entropy Formula for the Ricci Flow and Its Geometric Applications,” on arXiv.org, a Web site used by mathematicians to post preprints—articles awaiting publication in refereed journals. He then e-mailed an abstract of his paper to a dozen mathematicians in the United States—including Hamilton, Tian, and Yau—none of whom had heard from him for years. In the abstract, he explained that he had written “a sketch of an eclectic proof” of the geometrization conjecture.

Perelman had not mentioned the proof or shown it to anyone. “I didn’t have any friends with whom I could discuss this,” he said in St. Petersburg. “I didn’t want to discuss my work with someone I didn’t trust.” Andrew Wiles had also kept the fact that he was working on Fermat’s last theorem a secret, but he had had a colleague vet the proof before making it public. Perelman, by casually posting a proof on the Internet of one of the most famous problems in mathematics, was not just flouting academic convention but taking a considerable risk. If the proof was flawed, he would be publicly humiliated, and there would be no way to prevent another mathematician from fixing any errors and claiming victory. But Perelman said he was not particularly concerned. “My reasoning was: if I made an error and someone used my work to construct a correct proof I would be pleased,” he said. “I never set out to be the sole solver of the Poincaré.”

Gang Tian was in his office at M.I.T. when he received Perelman’s e-mail. He and Perelman had been friendly in 1992, when they were both at N.Y.U. and had attended the same weekly math seminar in Princeton. “I immediately realized its importance,” Tian said of Perelman’s paper. Tian began to read the paper and discuss it with colleagues, who were equally enthusiastic.

On November 19th, Vitali Kapovitch, a geometer, sent Perelman an e-mail:

Hi Grisha, Sorry to bother you but a lot of people are asking me about your preprint “The entropy formula for the Ricci . . .” Do I understand it correctly that while you cannot yet do all the steps in the Hamilton program you can do enough so that using some collapsing results you can prove geometrization? Vitali.

Perelman’s response, the next day, was terse: “That’s correct. Grisha.”

In fact, what Perelman had posted on the Internet was only the first installment of his proof. But it was sufficient for mathematicians to see that he had figured out how to solve the Poincaré. Barry Mazur, the Harvard mathematician, uses the image of a dented fender to describe Perelman’s achievement: “Suppose your car has a dented fender and you call a mechanic to ask how to smooth it out. The mechanic would have a hard time telling you what to do over the phone. You would have to bring the car into the garage for him to examine. Then he could tell you where to give it a few knocks. What Hamilton introduced and Perelman completed is a procedure that is independent of the particularities of the blemish. If you apply the Ricci flow to a 3-D space, it will begin to undent it and smooth it out. The mechanic would not need to even see the car—just apply the equation.” Perelman proved that the “cigars” that had troubled Hamilton could not actually occur, and he showed that the “neck” problem could be solved by performing an intricate sequence of mathematical surgeries: cutting out singularities and patching up the raw edges. “Now we have a procedure to smooth things and, at crucial points, control the breaks,” Mazur said.

Tian wrote to Perelman, asking him to lecture on his paper at M.I.T. Colleagues at Princeton and Stony Brook extended similar invitations. Perelman accepted them all and was booked for a month of lectures beginning in April, 2003. “Why not?” he told us with a shrug. Speaking of mathematicians generally, Fedor Nazarov, a mathematician at Michigan State University, said, “After you’ve solved a problem, you have a great urge to talk about it.”

Hamilton and Yau were stunned by Perelman’s announcement. “We felt that nobody else would be able to discover the solution,” Yau told us in Beijing. “But then, in 2002, Perelman said that he published something. He basically did a shortcut without doing all the detailed estimates that we did.” Moreover, Yau complained, Perelman’s proof “was written in such a messy way that we didn’t understand.”

Perelman’s April lecture tour was treated by mathematicians and by the press as a major event. Among the audience at his talk at Princeton were John Ball, Andrew Wiles, John Forbes Nash, Jr., who had proved the Riemannian embedding theorem, and John Conway, the inventor of the cellular automaton game Life. To the astonishment of many in the audience, Perelman said nothing about the Poincaré. “Here is a guy who proved a world-famous theorem and didn’t even mention it,” Frank Quinn, a mathematician at Virginia Tech, said. “He stated some key points and special properties, and then answered questions. He was establishing credibility. If he had beaten his chest and said, ‘I solved it,’ he would have got a huge amount of resistance.” He added, “People were expecting a strange sight. Perelman was much more normal than they expected.”

To Perelman’s disappointment, Hamilton did not attend that lecture or the next ones, at Stony Brook. “I’m a disciple of Hamilton’s, though I haven’t received his authorization,” Perelman told us. But John Morgan, at Columbia, where Hamilton now taught, was in the audience at Stony Brook, and after a lecture he invited Perelman to speak at Columbia. Perelman, hoping to see Hamilton, agreed. The lecture took place on a Saturday morning. Hamilton showed up late and asked no questions during either the long discussion session that followed the talk or the lunch after that. “I had the impression he had read only the first part of my paper,” Perelman said.

In the April 18, 2003, issue of Science, Yau was featured in an article about Perelman’s proof: “Many experts, although not all, seem convinced that Perelman has stubbed out the cigars and tamed the narrow necks. But they are less confident that he can control the number of surgeries. That could prove a fatal flaw, Yau warns, noting that many other attempted proofs of the Poincaré conjecture have stumbled over similar missing steps.” Proofs should be treated with skepticism until mathematicians have had a chance to review them thoroughly, Yau told us. Until then, he said, “it’s not math—it’s religion.”

By mid-July, Perelman had posted the final two installments of his proof on the Internet, and mathematicians had begun the work of formal explication, painstakingly retracing his steps. In the United States, at least two teams of experts had assigned themselves this task: Gang Tian (Yau’s rival) and John Morgan; and a pair of researchers at the University of Michigan. Both projects were supported by the Clay Institute, which planned to publish Tian and Morgan’s work as a book. The book, in addition to providing other mathematicians with a guide to Perelman’s logic, would allow him to be considered for the Clay Institute’s million-dollar prize for solving the Poincaré. (To be eligible, a proof must be published in a peer-reviewed venue and withstand two years of scrutiny by the mathematical community.)

On September 10, 2004, more than a year after Perelman returned to St. Petersburg, he received a long e-mail from Tian, who said that he had just attended a two-week workshop at Princeton devoted to Perelman’s proof. “I think that we have understood the whole paper,” Tian wrote. “It is all right.”

Perelman did not write back. As he explained to us, “I didn’t worry too much myself. This was a famous problem. Some people needed time to get accustomed to the fact that this is no longer a conjecture. I personally decided for myself that it was right for me to stay away from verification and not to participate in all these meetings. It is important for me that I don’t influence this process.”

In July of that year, the National Science Foundation had given nearly a million dollars in grants to Yau, Hamilton, and several students of Yau’s to study and apply Perelman’s “breakthrough.” An entire branch of mathematics had grown up around efforts to solve the Poincaré, and now that branch appeared at risk of becoming obsolete. Michael Freedman, who won a Fields for proving the Poincaré conjecture for the fourth dimension, told the Times that Perelman’s proof was a “small sorrow for this particular branch of topology.” Yuri Burago said, “It kills the field. After this is done, many mathematicians will move to other branches of mathematics.”

Five months later, Chern died, and Yau’s efforts to insure that he-—not Tian—was recognized as his successor turned vicious. “It’s all about their primacy in China and their leadership among the expatriate Chinese,” Joseph Kohn, a former chairman of the Prince-ton mathematics department, said. “Yau’s not jealous of Tian’s mathematics, but he’s jealous of his power back in China.”

Though Yau had not spent more than a few months at a time on mainland China since he was an infant, he was convinced that his status as the only Chinese Fields Medal winner should make him Chern’s successor. In a speech he gave at Zhejiang University, in Hangzhou, during the summer of 2004, Yau reminded his listeners of his Chinese roots. “When I stepped out from the airplane, I touched the soil of Beijing and felt great joy to be in my mother country,” he said. “I am proud to say that when I was awarded the Fields Medal in mathematics, I held no passport of any country and should certainly be considered Chinese.”

The following summer, Yau returned to China and, in a series of interviews with Chinese reporters, attacked Tian and the mathematicians at Peking University. In an article published in a Beijing science newspaper, which ran under the headline “SHING-TUNG YAU IS SLAMMING ACADEMIC CORRUPTION IN CHINA,” Yau called Tian “a complete mess.” He accused him of holding multiple professorships and of collecting a hundred and twenty-five thousand dollars for a few months’ work at a Chinese university, while students were living on a hundred dollars a month. He also charged Tian with shoddy scholarship and plagiarism, and with intimidating his graduate students into letting him add his name to their papers. “Since I promoted him all the way to his academic fame today, I should also take responsibility for his improper behavior,” Yau was quoted as saying to a reporter, explaining why he felt obliged to speak out.

In another interview, Yau described how the Fields committee had passed Tian over in 1988 and how he had lobbied on Tian’s behalf with various prize committees, including one at the National Science Foundation, which awarded Tian five hundred thousand dollars in 1994.

Tian was appalled by Yau’s attacks, but he felt that, as Yau’s former student, there was little he could do about them. “His accusations were baseless,” Tian told us. But, he added, “I have deep roots in Chinese culture. A teacher is a teacher. There is respect. It is very hard for me to think of anything to do.”

While Yau was in China, he visited Xi-Ping Zhu, a protégé of his who was now chairman of the mathematics department at Sun Yat-sen University. In the spring of 2003, after Perelman completed his lecture tour in the United States, Yau had recruited Zhu and another student, Huai-Dong Cao, a professor at Lehigh University, to undertake an explication of Perelman’s proof. Zhu and Cao had studied the Ricci flow under Yau, who considered Zhu, in particular, to be a mathematician of exceptional promise. “We have to figure out whether Perelman’s paper holds together,” Yau told them. Yau arranged for Zhu to spend the 2005-06 academic year at Harvard, where he gave a seminar on Perelman’s proof and continued to work on his paper with Cao.

On April 13th of this year, the thirty-one mathematicians on the editorial board of the Asian Journal of Mathematics received a brief e-mail from Yau and the journal’s co-editor informing them that they had three days to comment on a paper by Xi-Ping Zhu and Huai-Dong Cao titled “The Hamilton-Perelman Theory of Ricci Flow: The Poincaré and Geometrization Conjectures,” which Yau planned to publish in the journal. The e-mail did not include a copy of the paper, reports from referees, or an abstract. At least one board member asked to see the paper but was told that it was not available. On April 16th, Cao received a message from Yau telling him that the paper had been accepted by the A.J.M., and an abstract was posted on the journal’s Web site.

A month later, Yau had lunch in Cambridge with Jim Carlson, the president of the Clay Institute. He told Carlson that he wanted to trade a copy of Zhu and Cao’s paper for a copy of Tian and Morgan’s book manuscript. Yau told us he was worried that Tian would try to steal from Zhu and Cao’s work, and he wanted to give each party simultaneous access to what the other had written. “I had a lunch with Carlson to request to exchange both manuscripts to make sure that nobody can copy the other,” Yau said. Carlson demurred, explaining that the Clay Institute had not yet received Tian and Morgan’s complete manuscript.

By the end of the following week, the title of Zhu and Cao’s paper on the A.J.M.’s Web site had changed, to “A Complete Proof of the Poincaré and Geometrization Conjectures: Application of the Hamilton-Perelman Theory of the Ricci Flow.” The abstract had also been revised. A new sentence explained, “This proof should be considered as the crowning achievement of the Hamilton-Perelman theory of Ricci flow.”

Zhu and Cao’s paper was more than three hundred pages long and filled the A.J.M.’s entire June issue. The bulk of the paper is devoted to reconstructing many of Hamilton’s Ricci-flow results—including results that Perelman had made use of in his proof—and much of Perelman’s proof of the Poincaré. In their introduction, Zhu and Cao credit Perelman with having “brought in fresh new ideas to figure out important steps to overcome the main obstacles that remained in the program of Hamilton.” However, they write, they were obliged to “substitute several key arguments of Perelman by new approaches based on our study, because we were unable to comprehend these original arguments of Perelman which are essential to the completion of the geometrization program.” Mathematicians familiar with Perelman’s proof disputed the idea that Zhu and Cao had contributed significant new approaches to the Poincaré. “Perelman already did it and what he did was complete and correct,” John Morgan said. “I don’t see that they did anything different.”

By early June, Yau had begun to promote the proof publicly. On June 3rd, at his mathematics institute in Beijing, he held a press conference. The acting director of the mathematics institute, attempting to explain the relative contributions of the different mathematicians who had worked on the Poincaré, said, “Hamilton contributed over fifty per cent; the Russian, Perelman, about twenty-five per cent; and the Chinese, Yau, Zhu, and Cao et al., about thirty per cent.” (Evidently, simple addition can sometimes trip up even a mathematician.) Yau added, “Given the significance of the Poincaré, that Chinese mathematicians played a thirty-per-cent role is by no means easy. It is a very important contribution.”

On June 12th, the week before Yau’s conference on string theory opened in Beijing, the South China Morning Post reported, “Mainland mathematicians who helped crack a ‘millennium math problem’ will present the methodology and findings to physicist Stephen Hawking. . . . Yau Shing-Tung, who organized Professor Hawking’s visit and is also Professor Cao’s teacher, said yesterday he would present the findings to Professor Hawking because he believed the knowledge would help his research into the formation of black holes.”

On the morning of his lecture in Beijing, Yau told us, “We want our contribution understood. And this is also a strategy to encourage Zhu, who is in China and who has done really spectacular work. I mean, important work with a century-long problem, which will probably have another few century-long implications. If you can attach your name in any way, it is a contribution.”

E. T. Bell, the author of “Men of Mathematics,” a witty history of the discipline published in 1937, once lamented “the squabbles over priority which disfigure scientific history.” But in the days before e-mail, blogs, and Web sites, a certain decorum usually prevailed. In 1881, Poincaré, who was then at the University of Caen, had an altercation with a German mathematician in Leipzig named Felix Klein. Poincaré had published several papers in which he labelled certain functions “Fuchsian,” after another mathematician. Klein wrote to Poincaré, pointing out that he and others had done significant work on these functions, too. An exchange of polite letters between Leipzig and Caen ensued. Poincaré’s last word on the subject was a quote from Goethe’s “Faust”: “Name ist Schall und Rauch.” Loosely translated, that corresponds to Shakespeare’s “What’s in a name?”

This, essentially, is what Yau’s friends are asking themselves. “I find myself getting annoyed with Yau that he seems to feel the need for more kudos,” Dan Stroock, of M.I.T., said. “This is a guy who did magnificent things, for which he was magnificently rewarded. He won every prize to be won. I find it a little mean of him to seem to be trying to get a share of this as well.” Stroock pointed out that, twenty-five years ago, Yau was in a situation very similar to the one Perelman is in today. His most famous result, on Calabi-Yau manifolds, was hugely important for theoretical physics. “Calabi outlined a program,” Stroock said. “In a real sense, Yau was Calabi’s Perelman. Now he’s on the other side. He’s had no compunction at all in taking the lion’s share of credit for Calabi-Yau. And now he seems to be resenting Perelman getting credit for completing Hamilton’s program. I don’t know if the analogy has ever occurred to him.”

Mathematics, more than many other fields, depends on collaboration. Most problems require the insights of several mathematicians in order to be solved, and the profession has evolved a standard for crediting individual contributions that is as stringent as the rules governing math itself. As Perelman put it, “If everyone is honest, it is natural to share ideas.” Many mathematicians view Yau’s conduct over the Poincaré as a violation of this basic ethic, and worry about the damage it has caused the profession. “Politics, power, and control have no legitimate role in our community, and they threaten the integrity of our field,” Phillip Griffiths said.

Perelman likes to attend opera performances at the Mariinsky Theatre, in St. Petersburg. Sitting high up in the back of the house, he can’t make out the singers’ expressions or see the details of their costumes. But he cares only about the sound of their voices, and he says that the acoustics are better where he sits than anywhere else in the theatre. Perelman views the mathematics community—and much of the larger world—from a similar remove.

Before we arrived in St. Petersburg, on June 23rd, we had sent several messages to his e-mail address at the Steklov Institute, hoping to arrange a meeting, but he had not replied. We took a taxi to his apartment building and, reluctant to intrude on his privacy, left a book—a collection of John Nash’s papers—in his mailbox, along with a card saying that we would be sitting on a bench in a nearby playground the following afternoon. The next day, after Perelman failed to appear, we left a box of pearl tea and a note describing some of the questions we hoped to discuss with him. We repeated this ritual a third time. Finally, believing that Perelman was out of town, we pressed the buzzer for his apartment, hoping at least to speak with his mother. A woman answered and let us inside. Perelman met us in the dimly lit hallway of the apartment. It turned out that he had not checked his Steklov e-mail address for months, and had not looked in his mailbox all week. He had no idea who we were.

We arranged to meet at ten the following morning on Nevsky Prospekt. From there, Perelman, dressed in a sports coat and loafers, took us on a four-hour walking tour of the city, commenting on every building and vista. After that, we all went to a vocal competition at the St. Petersburg Conservatory, which lasted for five hours. Perelman repeatedly said that he had retired from the mathematics community and no longer considered himself a professional mathematician. He mentioned a dispute that he had had years earlier with a collaborator over how to credit the author of a particular proof, and said that he was dismayed by the discipline’s lax ethics. “It is not people who break ethical standards who are regarded as aliens,” he said. “It is people like me who are isolated.” We asked him whether he had read Cao and Zhu’s paper. “It is not clear to me what new contribution did they make,” he said. “Apparently, Zhu did not quite understand the argument and reworked it.” As for Yau, Perelman said, “I can’t say I’m outraged. Other people do worse. Of course, there are many mathematicians who are more or less honest. But almost all of them are conformists. They are more or less honest, but they tolerate those who are not honest.”

The prospect of being awarded a Fields Medal had forced him to make a complete break with his profession. “As long as I was not conspicuous, I had a choice,” Perelman explained. “Either to make some ugly thing”—a fuss about the math community’s lack of integrity—“or, if I didn’t do this kind of thing, to be treated as a pet. Now, when I become a very conspicuous person, I cannot stay a pet and say nothing. That is why I had to quit.” We asked Perelman whether, by refusing the Fields and withdrawing from his profession, he was eliminating any possibility of influencing the discipline. “I am not a politician!” he replied, angrily. Perelman would not say whether his objection to awards extended to the Clay Institute’s million-dollar prize. “I’m not going to decide whether to accept the prize until it is offered,” he said.

Mikhail Gromov, the Russian geometer, said that he understood Perelman’s logic: “To do great work, you have to have a pure mind. You can think only about the mathematics. Everything else is human weakness. Accepting prizes is showing weakness.” Others might view Perelman’s refusal to accept a Fields as arrogant, Gromov said, but his principles are admirable. “The ideal scientist does science and cares about nothing else,” he said. “He wants to live this ideal. Now, I don’t think he really lives on this ideal plane. But he wants to.”

In particular, the whole discussion of replacing an object with its network of relationships (and the following discussion of object and representation) is interesting. It’s also probably worth following up on the bit about a Wittgenstenian interpretation of Yoneda’s Lemma. This paragraph from near the end sums up some of this well:

It sometimes happens that the introduction of a term in a mathematical discussion is the signal that an important shift of viewpoint is taking place, or is about to take place. An emphasis on “representability” of functors in a branch of mathematics suggests an ever so slight, but ever so important, shift. The lights are dimmed on mathematical objects and beamed rather on the corresponding functors; that is, on the networks of relationships entailed by the objects. The functor has center stage, the object that it represents appears almost as an afterthought. The lights are dimmed on on equality of mathematical objects as well, and focussed, rather, on canonical isomorphisms, and equivalence.

The awkwardness of equality

One can’t do mathematics for more than ten minutes without grappling, in some way or other, with the slippery notion of equality. Slippery, because the way in which objects are presented to us hardly ever, perhaps never, immediately tells us — without further commentary — when two of them are to be considered equal. We even see this, for example, if we try to define real numbers as decimals, and then have to mention aliases like 20 = 19.999…, a fact not unknown to the merchants who price their items $19.99.

The heart and soul of much mathematics consists of the fact that the “same” object can be presented to us in different ways. Even if we are faced with the simple-seeming task of “giving” a large number, there is no way of doing this without also, at the same time, “giving” a hefty amount of extra structure that comes as a result of the way we pin down — or the way we present — our large number. If we write our number as 1729 we are, sotto voce, offering a preferred way of “computing it” (add one thousand to seven hundreds to two tens to nine). If we present it as 1 + 12^3 we are recommending another mode of computation, and if we pin it down – as Ramanujuan did – as the first number expressible as a sum of two cubes in two different ways, we are being less specific about how to compute our number, but have underscored a characterizing property of it within a subtle diophantine arena.

The issue of “presentation” sometimes comes up as a small pedagogical hurdle – no more than a pebble in the road, perhaps, but it is there – when one teaches young people the idea of congruence mod N. How should we think of 1, 2, 3, … mod 691? Are these ciphers just members of a new number system that happens to have similar notation as some of our integers? Are we to think of them as equivalence classes of integers, where the equivalence relation is congruence mod 691? Or are we happy to deal with them as the good old integers, but subjected to that equivalence relation? The eventual answer, of course, is: all three ways — having the flexibility to adjust our viewpoint to the needs of the moment is the key. But that may be too stiff a dose of flexibility to impose on our students all at once.

To define the mathematical objects we intend to study, we often — perhaps always — first make it understood, more often implicitly than explicitly, how we intend these objects to be presented to us, thereby delineating a kind of super- object; that is, a species of mathematical objects garnished with a repertoire of modes of presentation. Only once this is done do we try to erase the scaffolding of the presentation, to say when two of these super-objects—possibly presented to us in wildly different ways— are to be considered equal. In this oblique way, the objects that we truly want enter the scene only defined as equivalence classes of explicitly presented objects. That is, as specifically presented objects with the specific presentation ignored, in the spirit of “ham and eggs, but hold the ham.”

This issue has been with us, of course, forever: the general question of abstraction, as separating what we want from what we are presented with. It is neatly packaged in the Greek verb aphairein, as interpreted by Aristotle1 in the later books of the Metaphysics to mean simply separation: if it is whiteness we want to think about, we must somehow separate it from white horse, white house, white hose, and all the other white things that it invariably must come along with, in order for us to experience it at all.

[...]

One of the templates of modern mathematics, category theory, offers its own formulation of equivalence as opposed to equality; the spirit of category theory allows us to be content to determine a mathematical object, as one says in the language of that theory, up to canonical isomorphism. The categorical viewpoint is, however, more than merely “content” with the inevitability that any particular mathematical object tends to come to us along with the contingent scaffolding of the specific way in which it is presented to us, but has this inevitability built in to its very vocabulary, and in an elegant way, makes profound use of this. It will allow itself the further flexibility of viewing any mathematical object “as” a representation of the theory in which the object is contained to the proto-theory of modern mathematics, namely, to set theory.

Defining Natural Numbers

[...]

For only in terms of [the structure intrinsic to the whole gamut of natural numbers] (packaged, perhaps, as a version of Peano’s axioms) do we have a criterion to determine when your understanding of “natural numbers,” and mine, admit “faithful translations” one to another. A consequence of such an approach — which is the standard modus operandi of mathematics ever since Hilbert — is that any single mathematical object, say the number 5, is understood primarily in terms of the structural relationship it bears to the other natural numbers. Mathematical objects are determined by – and understood by — the network of relationships they enjoy with all the other objects of their species.

Objects versus Structure

Mathematics thrives on going to extremes whenever it can. Since the “compromise” we sketched above has “mathematical objects determined by the network of relationships they enjoy with all the other objects of their species,” perhaps we can go to extremes within this compromise, by taking the following further step. subjugate the role of the mathematical object to the role of its network of relationships — or, a further extreme — simply replace the mathematical object by this network. This may seem like an impossible balancing act. But one of the elegant – and surprising – accomplishments of category theory is that it performs this act, and does it with ease.

[...]

Equality versus Isomorphism

The major concept that replaces equality in the context of categories is isomorphism. An isomorphism f : A → B between two objects A, B of the category C is a morphism in the category C that can be “undone,” in the sense that there is another morphism g : B → A playing the role of the inverse of f ; that is, the composition gf : A → A is the identity morphism [on A] and the composition fg : B → B is the identity morphism [on B] . The essential lesson taught by the categorical viewpoint is that it is usually either quixotic, or irrelevant, to ask if a certain object X in a category C is equal to an object Y . The query that is usually pertinent is to ask for a specific isomorphism from X to Y.

Note the insistence, though, on a specific isomorphism; although it may be useful to be merely assured of the existence of isomorphisms between X and Y , we are often in a much better position if we can pinpoint a specific isomorphism f : X → Y characterized by an explicitly formulated property, or list of properties. In some contexts, of course, we simply have to make do without being able to pin- point a specific isomorphism. If, for example, I manage to construct an algebraic closure of the finite field [with two elements], and am told that someone halfway around the world has also constructed such an algebraic closure, I know that there exists an isomorphism between the two algebraic closures but – without any further knowledge – I have no way of pinpointing a specific isomorphism. In contrast, desipte my ignorance of the manner in which my colleague at the opposite end of the world went about constructing her algebraic closure, I can, with utter confidence, put my finger on a specific isomorphism between the group of automorphisms of my algebraic closure and the group of automorphisms of the other algebraic closure. The fact that the algebraic closures are not yoked together by a specified isomorphism is the source of some theoretical complications at times, while the fact that their automorphism groups are seen to be isomorphic via a cleanly specified isomorphism is the source of great theoretical clarity, and some profound number theory.

A uniquely specified isomorphism from some object X to an object Y characterized by a list of explicitly formulated properties – this list being sometimes, the truth be told, only implicitly understood – is usually dubbed a “canonical isomorphism.” The “canonicality” here depends, of course, on the list. It is this brand of equivalence, then, that in category theory replaces equality: we wish to determine objects, as people say, “up to canonical isomorphism.”

Representing one theory in another

If categories package entire mathematical theories, it is natural to imagine that we might find the shadow of one mathematical theory (as packaged by a category C) in another mathematical theory (as packaged by a category D). We might do this by establishing a “mapping” of the entire category C to the category D. Such a “mapping” should, of course, send basic features (i.e., objects, morphisms) of C to corresponding features of the category D, and moreover, it must relate the composition law of morphisms in C to the corresponding law for morphisms of D; we call such a “mapping” a functor from C to D.

[...]

In this way, we have a vocabulary for establishing bridges between whole disciplines of mathematics; we have a way of representing grand aspects of, say, topology in algebra (or conversely) by establishing functors from the category of topological spaces to the category of groups (or conversely): construct the pertinent functors from the one category to the other! The easiest thing to do, at least in mathematics, is to forget, and the forgetting process offers us some elementary functors, such as the functor from topological spaces to sets that passes from a topological space to its underlying set, thereby forgetting its topology. The more profound bridges between fields of mathematics are achieved by more interesting constructions. But there is a ubiquitous type of functor, as easy to construct as one can imagine, and yet extraordinarily revealing. Given any object X in any category C we will construct an important functor from C to the category of sets upon which C was built. This functor alone will be enough to “reconstruct” X , but – as you might guess – only “up to canonical isomorphism.”

[...]

The object as a functor from the theory-in-which-it-lives to set theory

Given an object X of a category C , we can define a mapping (a functor that we will denote FX) that encodes the essence of the object X. The functor FX will, in fact, determine X up to canonical isomorphism. This functor FX maps the category C to the category S of sets (the same category of sets on which C is “modelled,” as we’ve described above). Here is how it is defined. The functor FX assigns to any object Y of C the set of morphisms from X to Y ; that is, FX(Y) := Mor(X, Y).

[...]

The fundamental, but miraculously easy to establish, fact is that the object X is entirely retrievable (however, only up to canonical isomorphism, of course) from knowledge of this functor FX. This fact, a consequence of a result known as Yoneda’s Lemma, can be expressed this way:

Theorem: Let X, X’ be objects in a category C . Suppose we are given an isomorphism of their associated functors n : FX ~= FX’. Then there is a unique isomorphism of the objects themselves, h : X ~= X that gives rise – as in the process described above – to this isomorphism of functors.

The beauty of this result is that it has the following decidedly structuralist, or Wittgensteinian language-game, interpretation:

an object X of a category C is determined (always, only up to canonical isomorphism, the recurrent theme of this article!) by the network of relationships that the object X has with all the other objects in C.

Yoneda’s lemma, in its fuller expression, tells us that the set of morphisms (of the category C) from an object X to an object Y is naturally in one-one correspondence with the set of morphisms of the functor FY to the functor FX. In brief, we have (or rather, Yoneda has) reconstructed the category C, objects and morphisms alike, purely in terms of functors to sets; i.e., in terms of networks of relationships that deal with the entire category at once.

Representable Functors

The following definition (especially as it pervades the mathematical work of Alexander Grothendieck) marked the beginning of a significantly new viewpoint in our subject.

A functor F : C → S, from a category C to the category of sets S on which it is modelled, is said to be represented by an object X of C if an isomorphism of functors F ~= FX is given. The functor F is said, simply, to be representable if it can be represented by some object X.

If you consult the theorem quoted at the end of the last section you see that Yoneda’s lemma, then, guarantees that if a functor F is representable, then F determines the object X that represents it up to unique isomorphism.

One of the noteworthy lessons coming from subjects such as algebraic geometry is that often, when it is important for a theory to make a construction of a particular object that performs an important function, we have a ready description of the functor F that it would represent, if it exists. Often, indeed, the basic utility of the object X that represents this functor F comes exactly from that: that X represents the functor! Although a specific construction of X may tell us more about the particularities of X, there is no guarantee that all the added information a construction provides – or any of it – furthers our insight beyond guaranteeing representability of F.

Some of the important turning points in the history of mathematics can be thought of as moments when we achieve a fuller understanding of what it means for one “thing” to represent another “thing.” The issue of representation is already implicit in the act of counting, as when we say that these two mathematical units “represent” those two cows. Leibniz dreamed of a scheme for a universal language that would reduce ideas “to a kind of alphabet of human thought” and the ciphers in his universal language would be manipulable representations of ideas.

Kant reserved the term representation (Vorstellung) for quite a different role. Here is the astonishing way in which this concept makes its first appearance in the Critique of Pure Reason:

There are only two possible ways in which synthetic representations and their objects … can meet one another. Either the object (Gegenstand) alone must make the representation possible, or the representation alone must make the object possible.

It is this either-or, this dance between object and representation, that animates lots of what follows in Kant’s Critique of Pure Reason. With meanings quite remote from Kant’s, the same two terms, object and representation, each provide grounding for the other, in our present discussion.

Nowadays, whole subjects of mathematics are seen as represented in other subjects, the “represented” subject thereby becoming a powerful tool for the study of the “representing” subject, and vice versa.

It sometimes happens that the introduction of a term in a mathematical discussion is the signal that an important shift of viewpoint is taking place, or is about to take place. An emphasis on “representability” of functors in a branch of mathematics suggests an ever so slight, but ever so important, shift. The lights are dimmed on mathematical objects and beamed rather on the corresponding functors; that is, on the networks of relationships entailed by the objects. The functor has center stage, the object that it represents appears almost as an afterthought. The lights are dimmed on on equality of mathematical objects as well, and focussed, rather, on canonical isomorphisms, and equivalence.

[...]

Object and equality

A stark alternative – the viewpoint of categories – is precisely to dim the lights where standard mathematical foundations shines them brightest. Instead of focussing on the question of modes of justification, and instead of making any explicit choice of set theory, the genius of categories is to provide a vocabulary that keeps these issues at bay. It is a vocabulary that can say nothing whatsoever about proofs, and that works with any – even the barest – choice of a set theoretic language, and that captures the essential template nature of the mathematical concepts it studies, showing these concepts to be – indeed – separable from modes of justification, and from the substrate of ever-problematic set theory. Separable but not forever separated, effecting the kind of aphairesis that Aristotle might have wanted, for, as we have said, you must bring your own set theory, and your own mode of proof, to this party. With the other lights low, the mathematical concepts shine out in this new beam, as pinned down by the web of relations they have with all the other objects of their species. What has receded are set theoretic language and logical apparatus. What is now fully incorporated, center stage under bright lights, is the curious class of objects of the category, a template for the various manners in which a mathematical object of interest might be presented to us. The basic touchstone is that, in appropriate deference to the manifold ways an object can be presented to us, objects need only be given up to unique isomorphism, this being an enlightened view of what it means for one thing to be equal to some other thing.

Bayesian Networks have been successfully applied in many areas such as pharmaceutical, decision making by doctors, air control, marketing. Structural learning of Bayesian Networks is usually a desirable but costly operation. In some domains it is possible to collect expert knowledge to manually create a structure for a Bayes Net. However, social networks, warehousing data, or supermarket purchasing records may contain hundreds of thousands of attributes. Providing expert Bayes Net structure in such cases is cumbersome if not impossible, even if as in the case with many of those domains the events are choices of very small subsets of the large pool of available entities. The complexity of existing algorithms for structural search prevents Bayes Net learning on datasets of that size.

This work introduces an algorithm for tractable structural learning in Bayes Nets by exploring structures on the local level. The algorithm exploits the computational efficiency of Frequent Sets for gathering statistics that are most likely to be useful for structure search given the assumption of sparse data. I will show the relevance of this work to modeling Social Networks. Finally, I will present an empirical evaluation of our algorithm applied to several massive datasets.

Note: If the time is left and there is a sufficient interest in the audience, I will in addition present a new generative model for evolution of social networks that I have developed in collaboration with Alice Zheng.

Graphical models are a marriage between probability theory and graph theory. They provide a natural tool for dealing with two problems that occur throughout applied mathematics and engineering – uncertainty and complexity – and in particular they are playing an increasingly important role in the design and analysis of machine learning algorithms. Fundamental to the idea of a graphical model is the notion of modularity – a complex system is built by combining simpler parts. Probability theory provides the glue whereby the parts are combined, ensuring that the system as a whole is consistent, and providing ways to interface models to data. The graph theoretic side of graphical models provides both an intuitively appealing interface by which humans can model highly-interacting sets of variables as well as a data structure that lends itself naturally to the design of efficient general-purpose algorithms.

Many of the classical multivariate probabalistic systems studied in fields such as statistics, systems engineering, information theory, pattern recognition and statistical mechanics are special cases of the general graphical model formalism – examples include mixture models, factor analysis, hidden Markov models, Kalman filters and Ising models. The graphical model framework provides a way to view all of these systems as instances of a common underlying formalism. This view has many advantages – in particular, specialized techniques that have been developed in one field can be transferred between research communities and exploited more widely. Moreover, the graphical model formalism provides a natural framework for the design of new systems.