An Evidence-Based Approach to Law and Science

Please note that this post is from 2009. Evaluate with care and in light of later events.

John Pfaff has been writing a series of articles for PrawfsBlawg over the last month or so, focusing on “Empirical Legal Scholarship” (ELS). ELS brings empirical social science research, including especially statistical studies, into the realm of the law. (Law & Economics would be another, related attempt to bring math and the law together.)

One of the problems he points out in his introductory piece is the lack of formal training that many legal academics have in statistics (as contrasted with, say, theoretical economists). Certainly most of the lawyers, law students, and law professors I know (though far from all of them) seem to come from a humanities background (with the exception of those working in the patent field, which tends to attract those from a more technical or scientific background). While this often leads to the ability to deal with a wide range of issues in an effective manner, it does make it challenging for some of us (I include myself in this, since despite my background in software development, academically I come from the humanities) to grapple with statistical data in an effective and sophisticated manner.

In the courtroom, this is supposed to be solved by the adversarial process, which requires each side to present experts capable of explaining their analyses to a (potentially non-mathematically trained) judge and a law jury. Daubert and Frye increasingly put the initial screening burden on judges, and this has increased the benefit (and, I think the need) for judges to grasp sophisticated analyses presented by experts. I have seen this struggle in the employment discrimination context (where courts have grappled with how to deal with data that may or may not demonstrate systemic discrimination) and, perhaps most notably, in the torts context, especially when dealing with pharmaceuticals (how should epidemiological data be treated, for example?)

Professor Pfaff writes:

Empirical evidence has long posed a problem to the adversarial, common law system. As Tal Golan points out in Laws of Men and Laws of Nature, courts have struggled for at least three centuries with how to use complex scientific evidence in the courtroom, and the problem is only going to get worse in the years to come. Lay judges and lay jurors have never had the epistemic competence to understand technical scientific and empirical evidence, and thanks to the technological revolution of the past three decades the volume and sophistication of such evidence is only going to grow.

(Incidentally, Tal Golan is the primary professor I’ll be working with when I begin in the History of Science PhD program at UCSD in the fall.)

Outside the courtroom, in the world of law journals, it can be hard for student editors untrained in statistics and data analysis, to differentiate good empirical studies in legal articles from bad ones. Adding in peer review might arguably help, but only if the “peers” involved themselves have a background in such empirical research. Otherwise, law professors may be no better than young law students in dealing with statistics—and, given that many younger students are more comfortable with technology (having grown up with Excel, for example), may even be worse at it.

Professor Pfaff believes that “[s]ome sort of reform is inevitable, and I think a shift towards courtroom-EBP is the way to go.” This means focusing on an “evidence-based” policy (similar to evidence-based medicine), as he describes in The Path Ahead: Evidence Based Empirical Work. The point is to apply the scientific method and to critically evaluate the quality of evidence, data, and studies.

In addition to looking to evidence and its quality, this path ahead means doing overview (and synthesis) studies to confirm a view of the big picture, instead of extrapolating from individual studies to larger answers without checking alternatives across various studies and approaches.

I recommend reading the series of articles (and the comments) and PrawfsBlawg.