It's The End Of Citation As We Know It & I Feel Fine
from the the-ai-can-free-us dept
Legal scholarship sucks. It’s interminably long. It’s relentlessly boring. And it’s confusingly esoteric. But the worst thing about legal scholarship is the footnotes. Every sentence gets one1. Banal statement of historical fact? Footnote. Recitation of hornbook law? Footnote. General observation about scholarly consensus? Footnote. Original observation? Footnote as well, I guess.
It’s a mess. In theory, legal scholarship should be free as a bird. After all, it’s one of the only academic disciplines to have avoided peer review. But in practice, it’s every bit as formalistic as any other academic discipline, just in a slightly different way. You can check out of Hotel Academia, but you can’t leave.
Most academic disciplines use peer review to evaluate the quality of articles submitted for publication. In a nutshell, anonymous scholars working in the same area read the article and decide whether it’s good enough to publish. Sounds great, except for the fact that the people reviewing an article have a slew of perverse incentives. After all, what if the article makes arguments you dislike? Even worse, what if it criticizes you? And if you are going to recommend publication, why not insist on citations to your own work? After all, it’s obviously relevant and important.
But the problems with peer review run even deeper. For better or worse, it does a pretty good job of ensuring that articles don’t jump the shark and conform to the conventional wisdom of the discipline. Of course, conformity can be a virtue. But it can also help camouflage flaws. Peer review is good at catching outliers, but not so good at catching liars. As documented by websites like Retraction Watch, plenty of scholars have sailed through the peer review process by just fabricating data to support appealing conclusions. Diederik Stapel, eat your heart out!
Anyway, legal scholarship is an outlier, because there’s no peer review. Of course, it still has gatekeepers. But unusually, the people deciding which articles to publish are students, not professors. Why? Historical accident. Law was a profession long before it became an academic discipline, and law schools are a relatively recent invention. Law students invented the law review in the late 19th century, and legal scholars just ran with it.
Asking law students to evaluate the quality of legal scholarship and decide what to publish isn’t ideal. They don’t know anything about legal scholarship. They don’t even know all that much about the law yet. But they aren’t stupid! After all, they’re in law school. So they rely on heuristics to help them decide what to publish. One important heuristic is prestige. The more impressive the author’s credentials, the more promising the article. Or at least, chasing prestige is always a safe choice, a lesson well-observed by many practicing lawyers as well.
Another key heuristic is footnotes. Indeed, footnotes are almost the raison d’etre of legal scholarship. An article with no footnotes is a non-starter. An article with only a few footnotes is suspect. But an article with a whole slew of footnotes is enticing, especially if they’re already properly Bluebooked. After all, much of the labor of the law review editor is checking footnotes, correcting footnotes, adding footnotes, and adding to footnotes. So many footnotes!
Most law review articles have hundreds of footnotes. Indeed, the footnotes often overwhelm the text. It’s not uncommon for law review articles to have entire pages that consist of nothing but a footnote.
It’s a struggle. Footnotes can be immensely helpful. They bolster the author’s credibility by signaling expertise and point readers to useful sources of additional information. What’s more, they implicitly endorse the scholarship they cite and elevate the profile of its author. Every citation matters, every citation is good. But how to know what to cite? And even more vexing, how to know when a citation is missing? So much scholarship gets published, it’s impossible to read it all, let alone remember what you’ve read. It’s easy to miss or forget something relevant and important. Legal scholars tend to cite anything that comes to mind and hope for the best.
There’s gotta be a better way. Thankfully, in 2020, Rob Anderson and Trent Wenzel created ScholarSift, a computer program that uses machine learning to analyze legal scholarship and identify the most relevant articles. Anderson is a law professor at Pepperdine University Caruso School of Law and Wenzel is a software developer. They teamed up to produce a platform intended to make legal scholarship more efficient. Essentially, ScholarSift tells authors which articles they should be citing, and tells editors whether an article is novel.
It works really well. As far as I can tell, ScholarSift is kind of like Turnitin in reverse. It compares the text of a law review article to a huge database of law review articles and tells you which ones are similar. Unsurprisingly, it turns out that machine learning is really good at identifying relevant scholarship. And ScholarSift seems to do a better job at identifying relevant scholarship than pricey legacy platforms like Westlaw and Lexis.
One of the many cool things about ScholarSift is its potential to make legal scholarship more equitable. In legal scholarship, as everywhere, fame begets fame. All too often, fame means the usual suspects get all the attention, and it’s a struggle for marginalized scholars to get the attention they deserve. Unlike other kinds of machine learning programs, which seem almost designed to reinforce unfortunate prejudices, ScholarSift seems to do the opposite, highlighting authors who might otherwise be overlooked. That’s important and valuable. I think Anderson and Wenzel are on to something, and I agree that ScholarSift could improve citation practices in legal scholarship.
But I also wonder whether the implications of ScholarSift are even more radical than they imagine? The primary point of footnotes is to identify relevant sources that readers will find helpful. That’s great. And yet, it can also be overwhelming. Often, people would rather just read the article, and ignore the sources, which can become distracting, even overwhelming. Anderson and Wenzel argue that ScholarSift can tell authors which articles to cite. I wonder if it couldn’t also make citations pointless. After all, readers can use ScholarSift, just as well as authors.
Maybe ScholarSift could free legal scholarship from the burden of oppressive footnotes? Why bother including a litany of relevant sources when a computer program can generate it automatically? Maybe legal scholarship could adopt a new norm in which authors only cite works a computer wouldn’t flag as relevant? Apparently, it’s still possible. I recently published an essay titled “Deodand.” I’m told that ScholarSift generated no suggestions about what it should cite. But I still thought of some. The citation is dead; long live the citation.
Brian L. Frye is Spears-Gilbert Professor of Law at the University of Kentucky College of Law
1. Orin S. Kerr, A Theory of Law, 16 Green Bag 2d 111 (2012). (“It is a common practice among law review editors to demand that authors support every claim with a citation. These demands can cause major headaches for legal scholars. Some claims are so obvious or obscure that they have not been made before. Other claims are made up or false, making them more difficult to support using references to the existing literature.”).↩
Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community.
Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis.
While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.
–The Techdirt Team
Filed Under: ai, citations, legal scholarship
Reader Comments
Subscribe: RSS
View by: Time | Thread
I'm going to have to disagree about the citations. As a physicist, the citations in an article are important for looking up the original source for what is cited. That allows one to state a result without rederiving it along with all of the assumptions that went into the original derivation. Once something becomes well enough known, it doesn't get cited. For example, an article on general relativity is not going to cite Einstein's original 1916 paper. I don't know about citations in the legal field however.
[ link to this | view in chronology ]
Re:
i think the idea is that this is about the legal field specifically, given the unique weirdness of legal scholarship. it is completely not science.
[ link to this | view in chronology ]
Re:
I agree with you and as the author said "The citation is dead; long live the citation."
I believe this kind of approach could be adapted to all other areas as well easing the burden of peer reviews and helping identify plagiarism. At the very least flag weird articles for further scrutiny. At the very least it's worth a try. Scientific production is not only about proving a hypothesis or something right. It's also about either proving the opposite and actually coming out with nothing (ie: your method wasn't appropriate can be a conclusion).
Could be interesting.
[ link to this | view in chronology ]
“Now We are all Sons of Bitches.”
-Abraham Lincoln
[ link to this | view in chronology ]
Good cite there. Off to the presses with ye.
[ link to this | view in chronology ]
Footnotes can be accurate pointers to other works. They can at the same time be reference that (in the most charitable light) don't mean what the author thinks they do.
More ambitious frauds can create entirely specious references. If the first dozen or two citations are accurate, the chance of catching the fraud goes down dramatically.
And, of course, the occasional footnote referring to an entirely fictional authority happens ...
[ link to this | view in chronology ]
I hope you are paying REM for the use of their song in your title. They will starve if you don't.
[ link to this | view in chronology ]
Re:
LOL. Pity the poor landlord!
[ link to this | view in chronology ]
Ending "Oppressive footnotes"
Don't they realize that citations drive traffic to other books?
[ link to this | view in chronology ]
Legal scholarship's citation and student review is not the probl
I'd like to shut down the lazy attack on legal scholarships' supposedly twinned failures: (1) substantial citation and (2) student instead of peer review.
First, legal citation is not the problem. Poor writing is the problem. If you do much in any social sciences field—history, political science, social studies, and others—then you will run into good and poor writing. Law is no different.
When I studied and wrote legal scholarship in England I would often hear my European colleagues mock American legal scholarship for its citation propensity. I would hear jokes about reading a page with one sentence on it and the rest covered in footnotes. However, they miss the point.
Legal citations are important. They are intended to point readers to other sources that either support or contrast positions, provide either additional arguments or primary source material, and are key to any legal researcher being able to delve deep into an issue.
More often than not the break in legal research on a topic can result from finding a work that only tangentially touches on the subject you are looking for, but includes a reference to the argument you need with citations to works that more fully expound upon it.
Citations, therefore, are not necessarily for the work immediate. They are intended for that immediate work, but also for others and the body of legal scholarship as a whole.
Having had to do research in legal scholarship and other areas of scholarship, I can say that legal scholarship—especially American legal scholarship—has a more richly-woven, and well connected, tapestry of knowledge that truly benefits any serious legal scholar.
In contrast, history and political science are more often found wanting, lacking, redundant, and pedantic in the quality and utility of their citations.
This does not mean, however, that all legal scholarship is written well, or that all citations in said scholarship are done well. As with any area, there are the good and great works, and the poor and terrible ones.
However, any lawyer or legal scholar should have the skills to be able to quickly skim the legal citations in a work in order to parse out the ones they need based on the signals used in each cite and (where they exist) the parentheticals added to the end of each explaining that cites' purpose.
This may take some training, but if you have not received it after three years of law school then perhaps legal scholarship is not up your alley. (And, really, it usually comes in the form of one 1-3 hour class in your first semester that is not solely focused on this one aspect of legal scholarship.)
Legal citations are more clear, more useful, and more beneficial to their scholarly body than any other social science field I've dealt with.
(NOTE—Most legal scholarship uses footnotes, not inline citations. Most legal filings will use inline citations, although that is also changing. This makes even the most riddled-with-citations work more readable, any the reader can easily read the prose while ignoring the footnotes if they so desire.)
Second, as to the issue of student-reviewed versus peer-reviewed scholarship—I can tell you now that top law students gunning for good grades and accolades spend far more time on the review, selection, revision, editing, and polishing of legal scholarship than peer-reviewed journals in other areas.
I've been through the student reviewed process on both ends. I have numerous friends and colleagues who go through the peer-reviewed process in their fields. (Including the legal fields in Europe and elsewhere.)
Frankly, the overall quality of top-notch journals is the same. To claim that a third-year law student that is in the top of their class is not competent to sort through legal scholarship and choose well is utter and absolute hogwash perpetuated by legal academic dandies who have never worked a day in their lives as lawyers and instead yearn for some reason to be treated as social sciences in other fields. (This is especially ridiculous because the folks they want to be like often get paid by their respective institutions far less, except for the fools who somehow do think law should be treated as a science and they treated like physicists or chemists or biologists. And, well, those folks are amazingly obtuse.)
Having had to clean up mess written by some fairly well-regarded legal academics during my law journal days, and seeing some of the trash published by peer-reviewed journals in other fields, I can safely say that student-reviewed journals are a strength of the legal academia rather than a weakness.
That is not to say all peer-reviewed works should shift to student-reviewed works. Hardly.
Law schools in the US train lawyers—professionals who have professional doctorates. The structure of the academic process is fundamentally different than the one faced by, say, a history or political science PhD. One simple difference is the lack of a dissertation. In replace of that, most schools have at least one law journal that only accepts its best students—either from a rigorously judged writing contest, or based on grades, or some combination thereof.
In short, law journals are staffed often by the top 5%, 10%, 15%, 20% of students who are top-quality legal writers in their own right. These students run the journals, and while burdened with difficult coursework, are not burdened by a large and complex dissertation process.
This works in law. This will not work in history, political science, or other social science areas.
However, neither is the student-reviewed process wrong. Nor is the peer-reviewed process wrong.
Both have their strengths and weaknesses.
Finally, as to the topic of this article—once again, if a trained lawyer can't sort through citations in case law or articles when skimming a work, then I'd truly worry about hiring them for your case. This is a key and core part of the process of learning to be a lawyer in law school—how to research, how to argue, how to synthesize disparate works into your own arguments. These skills should also be reflected in the professors who so-often write legal academia.
I'm not sure the value here.
This is a tool, not a revelation nor a revolution.
Heck, it sounds like this is just a variation on Shephardizing: https://www.lawinsider.com/dictionary/shephardizing
In the end, though, this is an algorithm meant to save time. And we all know, with algorithms, garbage in equals garbage out. Plus, assumptions in the design can lead to noise, or errors, in the output.
"ScholarSift tells authors which articles they should be citing, and tells editors whether an article is novel."
Really, now? Well, I'm glad someone can tell me who to cite, and whether an article is novel. Because, you know, I didn't go get a law degree, or hell, an undergraduate degree, for that purpose.
I'll note, finally, that the author is a law professor. It really does make me sad that this law professor seems to hate his field so much, or think so little of the students he teaches, that he believes his colleagues and those students can't use their own mind and skills to determine whether an article is novel, or how and when to cite a work.
I guess, in the end, we can just let the AIs write our articles: https://www.theguardian.com/commentisfree/2020/sep/08/robot-wrote-this-article-gpt-3
All hail laziness, all hail our robot overlords.
Good grief, Charlie Brown. Good grief, indeed.
(Also, in the spirit of this kvetch, get off my dang lawn you kids!)
[ link to this | view in chronology ]
Re: Legal scholarship's citation and student review is not the p
Epic comment. How could I have guessed you were a legal scholar?
[ link to this | view in chronology ]
Re: Re: Legal scholarship's citation and student review is not t
Like all good kvetches, rants, or other polemical screeds, they grow in power and length in proportion to the odiousness of the work seeking to be avoided through the inherent procrastination represented by the polemical work.
In my case, drafting and reviewing extensive discovery requests in a document intensive case in federal litigation.
In all seriousness, having experienced both sides of the legal scholarship coin—student review and legal scholar—and watching the process in other areas of academia, American legal scholarship is doing just fine.
I can tell you quite truthfully that I would not trade the general quality of legal scholarship, or the publication process, for what other social scientists go through in their fields.
A unified set of journals, streamlined application processes, reviewers who actually care about their reviewing work, reviewers who actually engage with the submitted material meaningfully instead of advancing their own positions or recommending self-serving citations—my friends in academia publishing in other social scientists look across the fence and see our grass as much greener.
As to citations—my European colleagues can joke all they want about the thickness of our citations.
American law journals are useful to practitioners here. Truly. I use them regularly in my practice.
European lawyers do not say the same. In fact, they laugh at their legal academy as out of touch, and presume as a matter of first principles that nothing in a law journal will help them in their day-to-day practice.
I don't think this is a state for the legal academy to envy, or seek to emulate.
Even in the academic realm, however, I can say having to write legal academic papers and works both here and across the pond that it is far easier to delve deeply into issues using American legal academic works as opposed to their European counterparts.
Moreover, when I applied American citation approaches grounded in true Bluebook principals to my European works, reviewers who joked at the pub previously about American citation habits responded glowingly to the thoroughness of discussion they helped provide in minimal space, not even understanding how ironic their appreciation for the approach was despite their prior jokes.
Finally, while I think paper impact scores are important and relevant, and can be good tools to give high-level analysis of research impact and utility, in the end the value of good legal scholarship may not always be apparent until years later.
To this day I regularly pull and find law journal articles from decades ago that provide insight into the development of an arcane legal area.
In legal scholarship in the US I find that the good works stand on the shoulders of titans past, and the ability to see those titans through citations and find other directions those works might lead is invaluable.
If you cannot tell, this is a particular grinding axe for myself.
There are, actually, peer-review journals in American legal scholarship. They are high quality journals, generally, and difficult to get an article placed in them. The value of works in those journals tends to be higher because of the obvious network effects, but because they are good quality.
But these are few in part because of the structure of post-graduate legal study in the U.S. (How many research assistants do the average law professor have versus professors in other social sciences? I can tell you it is lower for law.)
At the same time, they truly are labors of love and dedication for those law professors who contribute.
In the end, I'd take the quality of the American legal academy against any international law academy, and indeed most social science academies in the US and internationally as well.
It does have its issues. Sure. But as both a sometimes-scholar and a currently all-time practitioner, the value and quality of the US law scholarship is clear, measurably, and above that of other areas.
The only thing missing is a renaissance on more practice-focused common law and statutory law surveys coupled with analysis. These roles have been offloaded to BigLaw white papers and the Thomson-Reuters and Wolters-Kluwers of the world, locked in proprietary databases, or sent off to the realm of special interest organizations.
Which is a shame. Because the more theoretical and philosophical works benefit from the more grounded works, just as the more-grounded works benefit from the more philosophical. So to the law-and-economic, or other law-and-field works.
Now, if you'll excuse me, I have complaints to draft designed to overcome Spokeo Article III objections from Federalist-inspired, US Chambers of Commerce-backed judges preach law and economics but ignore the value of internalizing externalities despite their shrines to Coase.
[ link to this | view in chronology ]
Re: Re: Re: Legal scholarship's citation and student review is n
LOL, I feel your pain. The more I tried to be a real lawyer, the more I longed for academia. Look at the results! I hope it's clear that my essay was tongue in cheek. I agree with your assessment of the usefulness of legal scholarship. But I think it could be even better! We avoid many of the weaknesses of other scholarly disciplines, but add some of our own. Every day, in every way, legal scholarship could be getting better & better.
[ link to this | view in chronology ]