At the forefront of data and the law: an effort to grade briefs.
Just as sabermetrics and Moneyball changed how sports teams evaluate prospective players’ skills — moving the analysis from simplistic and subjective measures to sophisticated and objective measures — litigation data is finally detailed enough, and legal technology robust enough, to provide an objective measure of lawyer and law firm skills.
For good or ill, this effort does not instill confidence. It's a program into which you input briefs, and it outputs a score. The idea is that briefs can be graded based on typos / missed citations (fair). But past that, the whole idea seems iffy, and there's a big hole where there should be a "here's how we do it" section. The article talks about looking at "argumentation." (First, isn't that just 'argument'? And if you use the wrong word in evaluating writing, maybe that casts a little doubt on your other conclusions?) How does it evaluate argumentation? Well . . . I'm sure that's proprietary, so we don't know. We're told it can determine whether a case is being cited for the correct proposition, but that seems unlikely: assuming the software is accurate, I don't know of a good brief-writer who can't argue that "case x extends to bound y and no further." What would this program do with such an argument? I suspect the failure to tell us is revealing.
What's suggested here is an automated program that can grade writing, and grade it well. The authors don't tell you, in any meaningful sense, how it works. If I had a program to grade writing well, I would sell it to a publisher. Or it stands to reason that if it can grade writing, it can write. Sell that! When the program starts spitting out better briefs than the lawyers, I'll start losing sleep. Until then, I suspect what's happening is this: the same folks paying piles of money might want to pay piles of money to grade the law firms it's paying, and hey, if there's money to be made, somebody will find a way.
Coming next: Harry Morse LLC d/b/a Harry Big Data Regression Analysis Cryptolawcoin Morse.