How does one find the best ideas at present?
I think an important heuristic is to consider a person’s error correction mechanism. The vast majority of people try to create the impression that they are never wrong. That makes them untrustworthy by my heuristic.
One of the reasons that Scott Alexander is (was? will SlateStarCodex come back?) such a useful source of ideas is that he is very diligent, and even systematic, about error correction.
It is is claimed that Charles Darwin diligently wrote down in his notebooks every surprising and disconcerting fact that opposed what he was expecting to see. He noted in writing everything that opposed his theoretical models. He was convinced that if he didn’t write such things down he would promptly forget them.
Apparently this is in Darwin’s autobiography.
Slate Star Codex is back–kind of–by the way.
But the announcement that it is going away seems to be pinned at the top of the blog.
It’s been back for a week or two, methinks. The indexing seems to have gone away, and the links to other blogs. Some parts of it seem to have been retained.
An exercise for our fabulous commenters would be to be look for their favorite article there from before and see if they can still find it with a web search. But I digress.
What tips do you have for demonstrating a trustworthy error-correction mechanism? It is actually somewhat hard to avoid giving the impression that you don’t think you are never wrong when you speak freely and in a confident way.
There are internally-motivated mechanisms, and externally enforced mechanisms.
The trouble with judging individuals as ‘trustworthy’ based on personal habits and character is that most people are not consistent across different areas. They may be perfectly reliable in some areas – often their field of expertise – and you would be justified in trusting your life to their advice in those fields. But the same people can sincerely and passionately hold to all kinds of crazy stuff you know to be erroneous in other areas.
So the existence of systems of external enforcement that penalize verifiably wrong claims, and the willingness of the individual to put their ‘skin in the game’ and submit themselves and their confident predictions to such mechanisms of accountability and public scrutiny is a good sign that someone is trying harder to be rigorous and epistemically disciplined in their claims. Of course those systems themselves have to be reliable and robustly resistant to various kinds of compromise and bias in order for this to work.
Accountability requires negative personal consequences for being wrong, and one question is how negative. You want it to be somehow proportional in seriousness to the stakes – how much will be lost from being wrong. You want it to be big enough to hurt, but not so ruinous that it discourages anybody from taking any risks at all.
I think the sweet spot is a system that will weed people out, or encourage them to weed themselves out, after a few iterations of a track record of being more wrong than right. If you go to the casino with $1,000 and the minimum play is $100, then if you are a bad gambler, it won’t take long for you to run out of money and thus have to leave the game. People will justifiably believe that most long-term players who are coming out ahead over the long term are smart and that their decisions are strategically sound.
When there are no effective systems of accountability, that is a recipe for the production of a lot of bunk. One of the more distressing things about the recent collapse in the health of our society’s intellectual culture and the breakdown in coherence and rigor in discourse is precisely because the system of accountability has broken down entirely.
If there is a tiny silver lining to that huge dark cloud, it might be that the urge and pressure to censor and silence will diminish, because even public contradictions and refutations are neutralized and ineffective since they can’t get any attention or media-amplification, and thus don’t have to be worried about. Orwell didn’t imagine that past a certain point of degeneration, the memory hole would be unnecessary. Oh, there’s a book that proves that we *haven’t* always been at war with Eastasia? Who cares, probably written by that rascal Goldstein, and anyway, no one cares to read it.
Some recent examples are Nancy MacLean’s “Democracy in Chains”, Nikole Hannah-Jones’ “The 1619 Project”, or, or Jill Lepore’s New Yorker Article, “The Invention of the Police”. All of these immediately met with devastating criticism and explanation of multiple factual errors, for example, check out Peter Moskos on Lepore. Did it matter in terms of causing the kind of personal or reputational damage or weed-out effect described above? No, perhaps even the contrary. So, the system for accountability in those fields is completely broken, and one can’t trust that long-term players are likely reliable sources of good ideas.
Nice comment.
This deserves a follow-up post. Perhaps Prof. Kling can recycle this comment into the lead for a post sometime in the future.
Free associating on Handle’s comment, I was reminded of Nassim Nicholas Taleb’s assertion (probably true) that “some fields don’t have experts.” You can find it in _The black swan_ using the index. Taleb is really at his best when covering such matters.
Handle, your point is a different one. How to phrase it succinctly.
“In some fields, bullshit runs rampant and is never flagged as such and then removed from the corpus of reasonably plausible research findings.”
This is, quite likely, because of what Handle has noted.
“Accountability requires negative personal consequences for being wrong…”
Somehow the 1619 Project is not good history, but the author and the NYTimes gets enough positive feedback from doing it that they don’t stop what they are doing and say “Sorry, never mind, I got carried away.”
Here are examples where bad theory bites the dust.
A couple examples in case I can make sense today.
1. There are many problems with economic theory, for example, but at some point Marx stopped being taken seriously when examining cost, price, value, market prices and similar issues. Marginal cost = price turns out to be useful in commodity markets over the long run. Marxist labor theory of value not so much. Where do you find Marx now? In Sociology and History, but not in price theory or business economics.
2. Most biologists aren’t Lysenkoists. They don’t have a big representation in agricultural research labs these days. Whatever is going on at the big research labs, Lysenko doesn’t have a lot of defenders.
= – = – = – =
In contrast, we see bad research that seems to get traction based on “wishful thinking.” Perhaps it is the demand for ideas that make people feel comfortable with my beliefs.
Presumably this is easier to get away with in environments where organizations don’t compete on the basis of how the theories perform.
Thanks for listening.
P.S.: Sometime more than a year ago, Prof. Kling proposed that the most popular books to be assigned to undergraduates are left leaning. For example, “The New Jim Crow” is not the best thing you could assign, but it’s highly popular anyway.
Handle, your point is a different one. How to phrase it succinctly.
CLARIFICATION:
This first quote is Charlie Abbott’s gloss on Handle: “In some fields, bullshit runs rampant and is never flagged as such and then removed from the corpus of reasonably plausible research findings.”
The following quote is from Handle: “Accountability requires negative personal consequences for being wrong…”
Intellectual humility – it seems such a long ago concept. Where have all the Carl Sagans gone? Let’s hope the pendulum still swings both ways and that this to shall pass.
And from that hour did I with earnest thought
Heap knowledge from forbidden mines of lore;
Yet nothing that my tyrants knew or taught
I cared to learn, but from that secret store
-Percy Bysshe Shelley
FWIW, not much assuredly, poetry classics might be a good source of new ideas for some. Something about rhythm and rhyme that sets the noggin noodling. The classics engage with profound questions but require engagement from the reader. New ideas of all kinds fly out of that dynamic, and with luck, some may be good. This year I’ve returned to the Germans (Goethe, Schiller, Klopstock (try chewing on https://m.poemhunter.com/poem/hermann-and-thusnelda/ ) et al, but perhaps most rewarding have been repeated trips to the well of Olavo Bilac
(just for naughty fun check out Delirium ).
The best classic fiction of course is another deep well. The best book that I have read so far this year, a Bildungsroman entitled Green Henry by Gottfried Keller, puts a fresh perspective on so many ideas from the changing perspective of a man through life’s changes that, even if it doesn’t generate new ideas might polish older ones.
Attending nonprofit functions in the DC area, AEI, Cato, Heritage, lunch briefings on the Hill, etc) for over 30 years certainly provided access to many good, new ideas. Something about the live atmosphere of events like a Conversation with Tyler seems to let good new ideas shine more than they would otherwise just reading the transcript. The live experience takes more time and engagement which might make it more rewarding and you never know, you might even see Dr Kling in attendance.
Been to about fifty events at the National Press Club, and I would have to say more, and better ideas were encountered traveling. I’ve a deep interest in agriculture so visiting farms and staying on sheep stations in New Zealand was very rewarding in terms of ideas. Seeing what you know in a different context spins off a lot of ideas. A few months in Australia should be enough to reset anyone’s world view. But perhaps the time that the light really went on for me was sitting one Saturday night in a medium size rural town in the interior of Brazil. The stray dogs laying in the street, the happy sertanejos riding their horses down the street drinking beer, playing checkers on the sidewalk with a really smart guy who had probably never made more than $50 in a day in his life…that type of experience can generate a lot of great ideas about priorities, letting go, and life in general.
Along the error correction mechanism theme, there is this from a talk by Richard Hamming in 1986 on being a scientist. He warns to find the right people who will bounce rather than absorb ideas.
“Question: Is brainstorming a daily process?
“Hamming: Once that was a very popular thing, but it seems not to have paid off. For myself I find it desirable to talk to other people; but a session of brainstorming is seldom worthwhile. I do go in to strictly talk to somebody and say, “Look, I think there has to be something here. Here’s what I think I see …” and then begin talking back and forth. But you want to pick capable people. To use another analogy, you know the idea called the `critical mass.’ If you have enough stuff you have critical mass. There is also the idea I used to call `sound absorbers’. When you get too many sound absorbers, you give out an idea and they merely say, “Yes, yes, yes.” What you want to do is get that critical mass in action; “Yes, that reminds me of so and so,” or, “Have you thought about that or this?” When you talk to other people, you want to get rid of those sound absorbers who are nice people but merely say, “Oh yes,” and to find those who will stimulate you right back.”
http://www.paulgraham.com/hamming.html
There is a related problem with finding new good ideas when you are surrounded by sounding boards for the socially desirable mainstream consensus who are like more that parrots repeating conventional wisdom which prevents any ability to question assumptions or move beyond orthodox priors and who view such proposed deviations with suspicion.
This can be a huge problem in certain fields, indeed, when one considers the full depth of the issue and how profoundly difficult it is to address, it is perhaps *the* problem we face.
This problem makes it incredibly difficult for a layman observer to discover when the mainstream consensus is wrong. Without oneself becoming an expert in a field, which is not always possible when such expertise requires high levels of ability and years of familiarization, one will have a hard time following chains of logic and evidence to make a reasoned conclusion and an independent judgment one way or the other.
One has no choice but to rely on experts and authorities, but if the path to becoming an expert authority filters out anyone who doesn’t give full support to the consensus or mainstream view, then one has to start taking risks with outsiders with contrary views, and most of these are unreliable cranks, kooks, crackpots, etc.
What’s worse is that smart people need positive incentives to pursue new good ideas in these fields, to encourage them to allocate their time and take risks when they always have other ways to make a comfortable living. If the award for advocating for contrarian ideas is becoming a pariah, few well-adjusted people are going to bother.
When institutions responsible for knowledge generation and custodianship become compromised in this way, there is really nothing anyone can do but abandon them altogether and start over with new institutions less vulnerable to that particular kind of social corruption. But the trouble is, how can you convince anyone an institution needs replacement? You are backed against the same wall. No one can make this argument without engaging on the merits on why some institution is producing nothing but exactly the kind of error that cannot be mentioned without becoming a pariah.
For centuries the attempt to deal with this problem was to set up systems like tenure or guaranteed lifetime employment. But everything humans do and every system we set up is always fighting an evolutionary arms race with the corrupting pressures of the pursuit of power and status, and it was only a matter of time until such tactics would no longer serve their original purpose.