« Purpose and Pragmatism | Main | The Affect Heuristic »

November 26, 2007

Comments

Hm, maybe the insiders should deliberately bring in a few outsiders to supplement their decision-making. Also, maybe journals could self-consciously publish articles 20 percent of the time that they would normally have rejected as too weird. They could then over time see how well the weird stuff did compared to the normal stuff in terms of citations. Perhaps even if one journal did it (maybe one does something like this?) it would have some data that could be used to argue for or against weird articles that normally wouldn't be published.

Are you sure that it's the referees who are doing the rejecting, as opposed to the editors? Almost all of the papers I get sent to referee are closely related to work I've done, some of them definitely obscure enough that the work shouldn't have been done in the first place.

I only wish I were sent good papers on neglected topics!

Mike, each journal privately wants to become as prestigious as possible which means publishing the currently prestigious/fashionable articles.

Ben, both referees and editors are responsible.

Robin, I bet that's largely true, but obviously new stuff has to come in that isn't fashionable, that then becomes fashionable, or what is fashionable would be constant perpetually. My impression is Becker was weird, and now is fashionable. How'd it happen?

Robin, I think the desire of each journal to become prestigious implies a desire to publish articles which its editors predict will be regarded in the near future as prestigious. Surely, if an editor saw an article which would turn his field upside down, he would rush to publish it as a "scoop". The editor could be risk-averse, of course, but that's different from having only the desire to follow the current trends.

Academics may clump because it is productive (competition between loosely grouped scholars working on similar problems may generate better results than the monopoly production of the disconnected iconoclast) or socially rewarding (more enjoyable to the scholar to work in a field with colleagues and conversations rather than working in a narrow sub-specialty that no one else finds worthwhile).

Re: "How can we decide between these two views?"

A prior issue, useful for addressing this one, is why it might be valuable to decide between these two views. If the "outsider" is a grant-making foundation, the issue matters because the grants panel can be composed of either insiders or outsiders. If clumping is something to be fought (or more neutrally, if clumping leaves profitable lines of research untapped), then use an outsider panel.

If the "outsider" is, say, an academic dean trying to decide whether to support a department's proposal to add one more public finance scholar to the five already on staff or push for an alternative candidate in some under served field (economics of romance?), the considerations may be different.

According to Paglia, absorption in technical areas is a male escape from women. Romance is hardly going to be introduced back in them as a subject.

Not that you can't find men who are interested (Mallarme) but it's hard to see how to make it technical.

Mike and Alex, yes journals want to publish what is about to become fashionable, but that still leaves out a lot.

Ron, economists can make any subject technical, I promise you.

Mike, I have in mind the distinction between private and social value. We can presume each actor is doing what is privately best in clumping - the question is whether and how that deviates from what is socially best. Arguments that clumping is productive are arguments for social value in the practice.

Robin, clumping is a major problem of academia and needs to be fought. But in understanding why it's hard to fight, I think there's a key point you overlooked: academics need to study not just interesting topics, but topics they can say something nontrivial about.

I think of it this way: academics have built "result mines" in various places around the intellectual landscape, of which maybe a thousand are active at any given time. If you slave away in one of these mines, there's a good chance you'll emerge with a gem -- or maybe even find a previously-unexplored shaft with hundreds of gems.

You could also choose to avoid the result mines entirely, and just dig with your bare hands in a random patch of grass. Who knows? You might discover a whole new vein of result ore, and go on to start your own mine. Or you might just end up with a two-foot-deep hole in the dirt.

Actually, my analogy fails in one respect: even the people who discover new mines usually do so by digging in the existing ones. This is because the intellectual landscape has a completely different geometry from the physical one: you can start digging in one mine and emerge in a completely different mine on the other side of the world. In this strange landscape, one of the only successful heuristics known is to follow the gems. (Which doesn't mean we shouldn't be on the lookout for other heuristics.)

Scott, yes, whether a topic is currently productive is one of those "details" that insiders may know which could justify their clumping.

But it's not a detail!

Are most academics capable of producing high quality research in unknown areas? Is not the clumping resulting from a few brilliant and original ideas by one or two individuals, followed by legions of followers refining and correcting them?

If this is so, it may be worth encouraging a new type of academic position: one geared specifically to someone who would produce research in new unexplored domains every year. Not too many positions of this type, just a few, but with a specific novelty mandate. This would presumably attract those who were good at exploring away from the clumps; then monitoring of references and such would establish just how useful such "thinking outside the clumps" really is.

Stuart,

The academic position you're talking about is called "statistician." We can work on whatever we want!

"Are most academics capable of producing high quality research in unknown areas? Is not the clumping resulting from a few brilliant and original ideas by one or two individuals, followed by legions of followers refining and correcting them?"

I don't think the problem is just one of capability, but also of inertia. The originators of brilliant ideas often get to train a lot of graduate students at top departments. When those students decide what to write their next paper on, their local incentives push them to build on their existing expertise (less time to the next publishable paper) rather than train for a whole new area, which leads to path-dependence and a career very heavily influenced by one's initial mentor(s).

To overcome this effect, one might provide very strong (and costly) incentives for some high-status academics to work in a novel field, and then fellowships and the like to increase their share of graduate students in the next academic generation.

Robin's question is interesting. I don't see it as special to academia--I'm not quite sure how to measure trendiness or clumpiness, but by any measure I assume it would be higher in art, literature, journalism, business, and lots of other fields. Back in the 50s and 60s, there were lots of westerns on TV, now not so much. And check out the Museum of Modern Art if you want to see trendiness. Or look at cars over the years.

However, academia is what I know best, and also there is some sense that academia should be held to higher standards, so Robin's original question seems worth addressing. I have three main thoughts here:

1. The scientific landscape is fractal. When an area is studied in depth, it commonly spawns related research. From this perspective, I think clumpiness is inevitable. I think it's misleading to think of research topics as being situated on a smooth Euclidean-type space.

2. I think there's a feedback-and-overcorrection mechanism. Overstudied fields often seem to be things that were recently the hot new thing. For example, for the past 15 years or so, there have been a zillion statistics Ph.D. theses in genetics. Much of this comes from funding, I'm sure, but I think a lot comes because people (students and faculty alike) think of genetics as exciting and new, a lot more exciting than seemingly-boring topics such as sample surveys (which isn't actually boring at all!).

3. Three words: Division of labor. Economists study auctions, psychologists study romance. Gains can be made by people working outside their fields (psychologists studying auctions, economists studying romance), but, by and large, it makes sense for people to work on their topics of expertise.

But starting a new journal is a small barrier to entry for academics breaking open a new frontier. The Journal of Uninteresting Economics, anyone?

Mobile, there are significant barriers to starting a journal with a reputation such that capable academics will expend time on writing articles for it. If you want tenure at a top university you will want to publish in the American Economic Review, not the JUE.

http://news.bbc.co.uk/2/hi/science/nature/7113098.stm
The above is an extraordinarily illustrative anecdote, not a theoretical contribution. Try it. Most of us can't work in leaps and bounds. We work work in small increments, chipping away at existing seams ( thanks Scott). What's so hard about that ?

Well, it is not quite fully serious, but there is the old Journal of Irreproducible Results, although maybe it does not exist anymore. Have not seen it around for awhile.

As a journal editor, I would say that Robin is right that both editors and referees are involved in the clumping phenomenon. Needless to say the issue is whether or not the clumping is excessive or not. There are good reasons to expect clumping in an "optimal" research environment, although I think editors need to be careful about too much pounding on a dead, or badly beaten up, horse. People have mentioned funding, but that often reflects social and political priorities. Thus, there is a lot of funding for papers on the climate change in many disciplines because this is now a very front and center topic of concern by many people around the world. Auctions have made lots of money for governments, thus they fund research on it, especially as it is also known that many people try to manipulate those markets.

There is also the case in the harder sciences, and I think to some degree in economics as well, that at certain times it is easier to discover new ideas and move forward because of certain developments of technical advances. Thus part of the push on auctions has arisen from improvements in game theory. The spread of interacting heterogeneous agents models has made it easier to model and understand certain kinds of financial market dynamics (which is also a hot topic for obvious reasons), and so forth.

Needlees to say, the wise editor is, or should be, keeping his/her eye out for that odd duck paper that really is innovative, but not on a topic that is currently hot. This is a fine art.

I can think of one "field" where there's a deliberate attempt to exert pressure against clumping, and new ideas seem to be genuinely valued - in practice, not just in claimed principle.

That field is science fiction, where a successful trope is forcibly retired by the journal (excuse me, magazine) editors. For example, I've heard that a substantial fraction of all stories in the unsolicited-manuscript slushpile are about a man and a woman stranded on a planet, ending with the surprise twist, "His name was Adam. Hers was Eve." It is a lot harder for an amateur author to come up with a novel SF idea than amateurs imagine.

It's a fascinating illustration of a counterintuitive point - that in SF, outsiders are not capable of doing anything different; only experienced insiders are capable of departing from the norm. This also appears to be largely the case in Artificial Intelligence.

Perhaps it's not the journal editors who are to blame, but the practitioners? SF editors are always screaming for new ideas, but most of the time they have to settle for old ideas and decent characterization. And - this is amazing, now that I think about it - I can't ever recall hearing a professional SF author complaining about editors' abilities to recognize or publish new ideas. Genuine novelty is just really hard. And surely novelty is easier for an SF author than for a scientist - scientists have to be right, and they have to do experiments to prove it...

Of course, if there's concrete signs of a significant problem with science journal editors, than an analogy to SF publishing is not enough evidence to disprove that. I'm just saying that fixing the editors might not fix the problem.

Eliezer, I feel pretty confident that I could write an SF novel that was quite different from any other out there, but I also don't estimate a very high chance editors would like it enough to publish.

Eliezer, I feel pretty confident that I could write an SF novel that was quite different from any other out there, but I also don't estimate a very high chance editors would like it enough to publish.

That's actually an interesting statement. My responses would be:

1) How much SF have you read?

2) Different from any other novel out there?

3) Do you think the editors would reject because of novelty, or because you may have good ideas but would need to work on characterization/plot/description for a few years to get good at that too?

I could write an SF novel different from any other novel I've read, but only if I had more writing talent.

Eliezer, most of the fiction I have read in my long life has been science fiction. I do think it would take time to learn to do characters and plots better, but I also think much of that learning would be about what sort of characters and plots readers want to see. On those dimensions, SF editors are not looking for diversity. "High" literature editors are more interested in diversity along those dimensions, but surely not unlimited diversity.

Eliezer, I've read a lot of science fiction, though at this point it is now becoming known as the 'older' stuff. Believe me, science fiction is comfort literature. Science fiction readers, as far as I can tell, are a very conservative lot, and far from actively discouraging clumping, the editors seem to actively promote it: in the 70's it was all O'Neill colonies; in the 80's, cyberpunk (in fact, this rapidly became a parody of itself due to clumping); in the 90's, a lot of nanotech and singularity stories. [1]

I would make a guess and say that, statistically at any rate, a certain amount of clumping is mathematically unavoidable. How much of the clumping is nonstatistical is anyone's guess, depending on the field.

[1] A new trope? Ha! Just how old do you think the newest trope is? A lot older than five years, probably older than fifteen, as far as I can tell. My reading of the genre has dropped off dramatically in the 21st century, I admit, but I'm sure that if there was anything new out there I'd have heard of it.

What would 'novel' SciFi look like? Would the movie 'Primer' qualify? No offense to Robin, but I doubt a newcomer could write a truly novel, truly good novel in any genre.

Science fiction novels are different from each other in many ways but one way is in the speculations that they make about the future. When people talk about writing "new" or "novel" science fiction novels, one thing they might mean by "novel" is "novel speculation". If this is what people are arguing about, then the argument comes down to this: one side argues that novel speculation is easy but is rejected by publishers, and the other side argues that publishers encourage novel speculation but it's hard to do.

The main problem here is that there are two alternative proposed causes for the lack of novel speculation in science fiction: one side says that it's the publishers who reject novel speculation (which they do presumably because of market pressure), and the other side says it's the difficulty of coming up with novel speculations.

It may be possible to resolve this argument by looking at an area where one of the two proposed causes is absent. There is such an area. Futurologists speculate about the future and are not at the mercy of market pressure. We can ask, then, just how abundant are the speculations of futurologists.

My impression is that they are not abundant. This suggests that the problem in science fiction is not with the publishers with their dependence on the market, but with the intrinsic difficulty of coming up with novel speculations.

GrayArea: Would the movie 'Primer' qualify?

Only as a movie. By the standards of literary SF, absolutely nothing you see on a screen is a new idea.

ScentOfViolets, you and I must have been reading different science fiction. Consider Neverness - was it the first story to have godlike AIs, faster-than-light travel? No. But I can't think of any other book like it. Was Iain Banks (Consider Phlebas, Player of Games) the first author to depict a future civilization focused on having fun? No, but I can't think of any predecessor like him, though others came after.

Is Singularity Sky original? No. Was A Fire Upon the Deep original? Yes. I could discurse at length as to why this is so (Vinge cares about his ideas, Stross is trying to show off how cool and detached he is) but the presence of a lot of non-original work doesn't mean that a science-fiction editor wouldn't be bursting with joy to get the manuscript for the next Neverness. It means that most of the novels, they take what they can get.

Robin, I'm still puzzled as to why, if you know SF, you think that a novel with your original idea and otherwise acceptable literary quality wouldn't be published. Do we just have different pictures of the field, or is your idea too terrible for mortal minds to comprehend, or what?

Eliezer, my concern is doubts about my ability to produce "otherwise acceptable literary quality." I was trying to note that a lot of preferences for conformity are tied up in that.

Eliezer: It doesn't look to me like it's hard to come up with novel sf ideas. I never read more than 50 or 100 sf stories, but I always found it very easy to come up with sf ideas that were novel in the opinions of people who devoted their whole lives to reading the stuff. I also used to ask such people, years ago, for any light that sf could shine on my thinking about technology, for instance, the mature development of immersive mobile VR cellphone etiquette, even asking them to draw on teleportation stories. Nothing. Not even any insightful stories on the immediate social effects of aging reversal. Countless interesting alternative histories unexplored. The list goes on, but I generalized and, due to a defect in human cognition, or possibly just the need for compression, lost most of the data producing the generalization.

Eliezer, your original assertion was about the appearance of new tropes, or 'new' ideas. None of the works that you mentioned have those. My challenge: name once such genuinely 'new' trope or 'new' idea that has appeared in the last five years (or ten!) in science fiction. One of the reasons I've stopped reading it as anything other than a bedtime pleasantry is precisely because there are none that I have seen. A fairly plausible argument has been made recently by one well-known writer that the genre is, if not dead, actively ossified in the ways lots of other genres have become ossified, though still practiced - swing and big band for example (and I would nominate rock & roll as well.)

I don't read as much science fiction as I used to, so the undeniably new novel that comes to mind is Permutation City from 1994. Though A Deepness in the Sky also comes to mind as containing at least one new idea: the programmer-at-arms who has to write code in the middle of space warfare. It doesn't have the sheer shock value as Permutation City, though, or even A Fire Upon the Deep.

I don't know about this business of new tropes. The best defy imitation - which, come to think of it, is a fundamental difference between science and science fiction. But it's probably a good bet that by the time anything becomes a trope, it was invented at least a decade ago and probably longer.

Rejecta Mathematica is an interesting experiment that may help with 'clumping:' math.rejecta.org.

the mature development of immersive mobile VR cellphone etiquette

Not sure what you have in mind, but a couple of days ago there was a scene in Ghost in the Shell in which we were allowed to see behind the scenes of a videoconference - one of the participants was clipping his toenails in his pajamas, but what the other participants saw was him perfectly dressed, facing forward, etc. It looked like the real him but it was a puppet.

I have read lots of academic and professional journals, and attended academic & professional conferences related to my field. Academic journals and conferences definitely clump around things some prestigious writer or journal raised which tend to be interesting items to ponder and nosh over for a little while. Professional journals tend to focus on what readers need to know-- the latest legislative promulgation, commercial development or interesting case studies. These persist in importance until the rules or commercial trends change. As long as the rule, structure or case remains relevant, the articles will be called up and utilized. I have found most academic articles have a short half-life and are quickly forgotten when the next comely hot-topic arrives on the scene. Academic writing is also formulaic and written with little humor or style or talent in comparison to professionals writing on a lark or for resume stuffing.

It would be interesting to read a professional journal on literature, with articles breaking down reading patterns from Amazon statistics.

I think it's hard to tell what's novel and what isn't.

One other issue around clumping is finding anyone able to distinguish interesting results from crap. An awful lot of dealing with conference submissions in crypto conferences is filtering out the crap, and it's easy to see how not-very-well-explained-but-innovative results can get discarded because they're hard to tell from nonsense. I've seen that happen with important results from people who didn't know the language of the field, and been involved in reviewing clever (albeit not earth shaking) results where some of the reviewers just didn't get the point of the paper.

The thing is, nobody wants to publish something that is then widely regarded as crap, because that's embarrassing. So there's an incentive to avoid stuff that's too far outside what you're used to, as well as a good reason to avoid it.

Robin,

The editors want to publish material which will soon become fashionable because the ideas are so revolutionary that their publication will create the new fashion. I don't refer to editors trying to ride an upcoming wave. I mean editors seeking to create the wave. Also, just because editors might have this desire, it still might be swamped by cowardice, herding or myopia.

Perhaps what works is exploited as much as possible by as many researchers and editors as possible, and when the gains diminish sufficiently, new territory is sought, and then outsiders are perhaps turned to and become centers for new clumps. And over time, many clumps fuse into a larger super-clump.

So in order to explore new areas more rapidly, one would have to more rapidly exhaust promising fields of their insights. Would the clumps unify more rapidly if there were many more researchers?

Why is such an overwhelming majority of comments directed at editors? After all, at least in biology, there are thousands of journals to choose from, many directed at the most obscure niches you can imagine.

The elephant in the room is funding. At least in biology, most basic research is funded by the government, which tremendously influences research priorities. Want to get funded? Study cancer, AIDS, Alzheimer's, bio-terrorism, or drug abuse. Diabetes and metabolic syndrome have become big in recent years.

On the bright side, scientists are smarter that bureaucrats, and will do their best to study what interests them by repackaging it as superficially relating to the purpose of a grant: "sure, we'll study diabetes, using these mutant mice that happen to also have signs of delayed aging..."

Carl Shulman: retraining incentives do exist. For example, I've been told that it's easier to get some NIH training grants if you are segueing from one sub-field to another rather than just continuing to study as a post-doc what you studied as a grad student.

The comments to this entry are closed.

Less Wrong (sister site)

May 2009

Sun Mon Tue Wed Thu Fri Sat
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31