Solitude vs Serendipity

June 05, 2024

One tension I notice in grad school is that between solitude and serendipity.

More precisely, it’s the tension between prolonged periods of distraction free and intense focus, and the exposure to chance ideas via serendipitous encounters. The latter can be more structured (talks and seminars in the case of academics) or less structured (lunchtime chats with friends and colleagues). Both solitude and serendipity are required to do good work, but they trade off against one another. Time spent pursuing one is necessarily time spent not doing the other.

Dig into the backstory of any great writer, composer, or researcher and you’ll find they spent many hours alone obsessing over their work (doing “deep work” as Cal Newport calls it). From Derek Parfit and Jerry Seinfeld to Carl Jung and Virginia Woolf, all prioritized finding hours of interrupted time to focus on their work. “The mind is sharper and keener in seclusion and uninterrupted solitude,” wrote Nikolai Tesla, “be alone, that is when ideas are born.” “Without great solitude, no serious work is possible,” said Picasso. Vladimir Nabokov organized his day around two multi-hour writing blocks. Emily Dickinson rarely left her family’s homestead throughout the 1950s.

Virginia Woolf's writing lodge. Image Credit.

This kind of intense solitude is required to do great work. But if you talk to most successful academics (and other successful knowledge workers such as writers, engineers, lawyers, architects, etc.), you’ll find that their best work often came as a result of a chance event. Perhaps they went to lunch with a colleague and realized they were working on a similar problem from different angles. Or they were at a seminar and the speaker said something which sparked an idea. Maybe they were just having a casual conversation with a friend.

Many institutions have recognized the importance of serendipitous encounters and bake this insight into their design. The machine learning department at CMU was designed with the intention of everyone working with their doors open, to promote conversations between different researchers. This decision was in turn based on the famous open door policy of Bell Labs—the research lab which created the transistor, laser, charged-coupled device and photovoltaic cell, information theory, UNIX and C, among other things—where all senior researchers were encouraged to work with open doors to foster collaboration.

MIT’s building 20 (a.k.a the magical incubator) was mourned when demolished because it housed researchers from across disciplines—linguists, architects, physicists, computer scientists, and biologists. This, it’s claimed, was responsible for inter-disciplinary innovations such as loudspeakers, instruments to measure the cosmic-microwave background, LIGO, and Chomskian grammars.

MIT's building 20. Image credit

Where does this leave us? Solitude is required to fully flesh out an idea and to deeply understand a topic. But serendipity is helpful for generating new ideas. The problem is that serendipitous encounters, by definition, can’t be planned. Not all interactions will leave you with new ideas (in fact, most won’t). Most talks will be unenlightening, most lunches unproductive. By contrast, if you spend two hours thinking hard about a problem, you will often make progress (even if that progress is better understanding what it is you don’t understand). So the tradeoff between solitude and serendipity is one between a sure thing and a gamble.

Newport also notices this tension. He advises you to split your time between the two activities: “Separate your pursuit of serendipitous encounters from your efforts to think deeply and build on these inspirations. You should try to optimize each effort separately, as opposed to mixing them together into a sludge that impedes both goals.” (pg. 76 of Deep Work.) This is good advice: you shouldn’t simultaneously try to write a paper and listen to a talk, you should choose one or the other.

But can we say any more than this? Unfortunately, I’m not sure. But I think this tradeoff is worth thinking a lot about. In academia, each day—each hour—you have the choice of whether to sit quietly by yourself and work on an interesting problem, attend some talk, or knock on your neighbor’s door and talk to them about a problem. And how you spend your time matters.

I don’t think there’s a master formula dictating how to spend every minute of every day in order to maximize productivity, just as there’s no equation for creativity. But just as you can help foster creativity (e.g., incubation, hypnagogia, free-flow writing) can you foster interactions that will result in more innovation? Can you design your day so that the interactions you do have are more likely to lead to new ideas? Here are some concrete questions I think are worth considering. These are focused specifically on academia but they have analogues in other jobs.

How much time should you spend working on your own and how much going to talks, or going to coffee with your labmates?
When should you do your deep work? Morning, evening?
Which talks should you attend? How related to your area should they be? How well should you know the material beforehand?

My guess is the answer to these questions changes depending on where you are in your career. When you’re a grad student you most likely want to spend more time doing deep work, because you need to get familiar with a field and become competent with its tools. Exposure to new ideas is only useful if you have the knowledge to take advantage of the insights.

Beyond considerations of productivity only, how much you prioritize solitary work also affects the kinds of things you work on. On one hand, working on your own can shelter you from trends and fads. This can lead to paradigm-shifting insights that require dismissing the status quo. If Einstein had been in academia, working on the popular problems of the day, would he have had insights necessary to develop special and general relativity? Maybe, but maybe not.

On the other hand, it’s possible to work on problems that truly don’t matter. Richard Hamming, in his famous talk You and Your Research, noted that researchers at Bell labs who talked more with others often ended up working on more consequential problems:

I noticed the following facts about people who work with the door open or the door closed. I notice that if you have the door to your office closed, you get more work done today and tomorrow, and you are more productive than most. But 10 years later somehow you don’t quite know what problems are worth working on; all the hard work you do is sort of tangential in importance. He who works with the door open gets all kinds of interruptions, but he also occasionally gets clues as to what the world is and what might be important.

In other words, when obtaining ideas from others via conversation or talks, you are more likely to work on the kind of things they find interesting. You’ll pick up ideas that are in the air. They’ll criticize your ideas more effectively and this leads to faster progress and new insights. But you’ll probably be less original. Solitary work, on the other hand, lets you take your own ideas more seriously without having to justify them to other people. This is important for fostering creativity.

So what’s the right tradeoff?

Back to all writing