Supposedly clever statistical analysis

Russ Roberts writes,

It would be tempting to say that this is just a working paper. Perhaps it will get no traction. But I doubt it. The Becker-Friedman Institute will spread it around — I only knew about the study because the Institute sent me an email. The media will be eager to repeat the finding because people have strong feelings about Uber and Lyft: “U of Chicago Study Finds Ridesharing Kills 1000 People Each Year.” Taxicab owners and their supporters will cite it.

The fact is that economists are almost always doing observational studies, not experiments. At the very least, economists should make more use of the Hill Criteria.

  • Strength (effect size): A small association does not mean that there is not a causal effect, though the larger the association, the more likely that it is causal.
  • Consistency (reproducibility): Consistent findings observed by different persons in different places with different samples strengthens the likelihood of an effect.
  • Specificity: Causation is likely if there is a very specific population at a specific site and disease with no other likely explanation. The more specific an association between a factor and an effect is, the bigger the probability of a causal relationship.
  • Temporality: The effect has to occur after the cause (and if there is an expected delay between the cause and expected effect, then the effect must occur after that delay).
  • Biological [or economic] gradient: Greater exposure should generally lead to greater incidence of the effect. However, in some cases, the mere presence of the factor can trigger the effect. In other cases, an inverse proportion is observed: greater exposure leads to lower incidence.
  • Plausibility: A plausible mechanism between cause and effect is helpful (but Hill noted that knowledge of the mechanism is limited by current knowledge).
  • Coherence: Coherence between epidemiological and laboratory findings increases the likelihood of an effect. However, Hill noted that “… lack of such [laboratory] evidence cannot nullify the epidemiological effect on associations”.
  • Experiment: “Occasionally it is possible to appeal to experimental evidence”.
  • Analogy: The effect of similar factors may be considered.
  • Some authors consider also, the Reversibility: If the cause is deleted then the effect should disappear as well

Many of Russ’ criticisms of the paper can be mapped back to some of these criteria.

7 thoughts on “Supposedly clever statistical analysis

  1. Interestingly, while Russ’ reminds us of increased smartphone-usage among drivers in his call for skepticism, he does not mention increased usage (and distraction) among the second party in the statistic: the pedestrians. This should heighten the call for a more granular empirical analysis.

    In any event, what should an economist prefer: broad base data sets like these researchers used, and access to broad base data that may provide controls; or narrower data sets (in time, location, etc.) that may would have tracked incidents by driver identity (ride share, cab, personal trip), cause of accident, etc?

    Incidentally, if one does not like blaming Uber/Lyft for the increases in traffic deaths, others “may” say it is all the Fed’s fault…

    ://www.manhattan-institute.org/html/lyft-uber-traffic-worse-finance-interest-rates

  2. Why should 1,000 death in traffic matter? Seems like the cost of doing business and not all that weird or notable given the scale of these operations.

  3. As a cyclist, I can state the Uber and Lyft drivers are definitely less experienced pulling to the side of the road once they see their fare. I say this as someone ideologically opposed to taxi cartels and indifferent to Uber and Lyft. One can also see a stark difference between their driving with a fare and going to a fare. That is something Uber and Lyft should be able to address with their app. Overall taxi drivers are the best drivers on the road, and public bus drivers are the worst (they get a paid vacation if they kill someone – I swear that goes through mind when they see a cyclist)

  4. Like MG – distracted pedestrians are more likely a cause for ped deaths than ridesharing — see the many YouTube meme of distracted walkers into pools, walls, non-open doors, etc. These are usually funny; but I’m sure a lot of the ped deaths are distracted pedestrians. Including drunk, distracted, Uber and Taxi ordering clients who get run over while doing something stupid when waiting for their ride. With lower cost Uber, perhaps there are more “drunk nights”, too.

    More data would help answer the MB issue of bad Uber drivers pulling over. Not sure whether cyclist deaths are included in ped deaths, and suspect they are not.

    Excellent critique by Russ of the study.

    Arnold mad a good addition, but I wish he added a bit more info about the Hill Criteria: “establishing epidemiologic evidence of a causal relationship between a presumed cause and an observed effect.” (Wiki) These are good criteria.

  5. I’d like to see an opportunity-cost-type comparison. People who use ride-sharing services would presumably have used other modes of transportation to get where they were going if the former were unavailable, and these other modes carry their own risks. It is hardly fair to compare mortality from ride-sharing with mortality from teleportation.

  6. All exceptions are counted. This meets the ‘exceptions that prove the rule”’
    Like proof by contradiction, there is no other way to explain the effect unless you use these three exceptions, for example.

  7. Prof. Andrew,

    Thank you for sharing this.

    Immediate the essay that jumped into my mind is from the blog Andrew Gelman is part of. It may be apropos, and uses Randy Newman lyrics for the outline.

    https://statmodeling.stat.columbia.edu/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/

    Gelman is extremely witty and writes well–it’s not my field so I’m not competent to evaluate his work.

    The fourth paragraph of the essay below is worth keeping in mind whenever we hear of new and exciting results:

    https://statmodeling.stat.columbia.edu/2016/03/03/more-on-replication-crisis/

Comments are closed.