Keyword search is back in the news again after the recent order in United States v. New Mexico State University (No. 1:16-cv-00911-JAP-LF, 2017 WL 4386358 (D.N.M. Sept. 29, 2017) and with it, the predictable spate of articles and posts demeaning keyword search and using as their basis the reference to J. Facciola and his “where angels fear to tread” quote.
It’s like hurricanes returning every year to the Gulf of Mexico: we know they’re going to be strong, we just don’t know when they will appear or exactly how strong they will be. As that great legal analyst James W. Buffet once observed, “aint no reasonin with hurricane season.”
Or misplaced legal analysis.
So first off, I’d recommend you read Ralph Loseys article on this subject, Judge Goes Where Angels Fear To Tread: Tells the Parties What Keyword Searches to Use at https://e-discoveryteam.com/ . Ralph is, as always, erudite, well researched and exceedingly thorough. But as much as I respect him both personally and professionally I have to disagree with a part of his analysis. I’ll set forth my disagreement below but first let’s take a look at that the “angels fear to tread quote” itself.
The original was written by Alexander Pope in his 1711 poem An Essay on Criticism and refers to inexperienced or rash people attempting things that more experienced people avoid. At the time Pope wrote the phrase, the word fool meant an inexperienced person and not one with inferior intelligence. Key point here.
The quote has been used many times, eg
- Edmund Burke in his work Reflections on the Revolution in France in 1790.
- Abraham Lincolns speech in the Lincoln Douglas debates at Peoria, Illinois on October 16, 1854
- “Fools Rush In (Where Angels Fear to Tread)”, a 1940 song written by Johnny Mercerand Rube Bloom, sung by Frank Sinatra
- Bob Dylan’s song “Jokerman“
Sometimes, however, it has been misinterpreted, as in:
- Wise men say, only fools rush in, from Elvis Presley’s “Can’t Help Falling in Love“, released in 1961.
- “Angels rush in where fools fear to tread”, Cary Grant as the angel in the 1948 film The Bishop’s Wife. (my personal favorite)
And I believe the interpretation of J. Facciola’s words as a complete proscription against keyword search is one of those misinterpretations. (I’ve written on this many times and my 2016 blog post on the subject is appended at the end of this post) In the O’Keefe case, he was being asked to overturn search terms that had been used by a party after consulting with an expert. The objecting party offered no expert of their own and simply asked the Court to set aside the existing terms and substitute his own.
The Judge, it seems to me, was hesitant to undertake that task. To that specific point, he wrote that “[w]hether search terms or ‘keywords’ will yield the information sought is a complicated question involving the interplay, at least, of the sciences of computer technology, statistics and linguistics. … Given this complexity, for lawyers and judges to dare opine that a certain search term or terms would be more likely to produce information than the terms that were used is truly to go where angels fear to tread.” (my emphasis added).
The Judge wasn’t saying keyword search was deficient per se. He was saying that substituting a list of words drawn up by a non-technical person (himself) would be necessarily inferior to a list done by technically adroit people. And that rather than ask him to create a new list, the objecting party should bring their own technical expert to the table.
And I think we can safely assume that J. Facciola understood completely the meaning of Popes phrase. Some of you may recall his keynote lunch address at the Eleventh Annual Georgetown Advanced eDiscovery Institute in Nov of 2014 where he carried on an imagined conversation on the state of litigation and advocacy. for an hour … with Socrates. His understanding of the classics is clearly well established.
So this is where I disagree with Ralph. In his columns, he states..
“Why did she make multiple detailed, technical decisions on legal search, including specific keywords to be used, without the benefit of expert testimony? Was that foolish as several judges have suggested, or was she just doing her job by making the decisions that the parties asked her to make?”
“Going it alone with legal search in a complex case is a fool’s errand.”
But the dictionary definition of a “fools errand” is “a task or activity that has no hope of success.” And as noted above that is not what either Pope or J. Facciola meant to say. Rather they meant that only an educated or trained person could succeed.
Two reports always under reported in this debate are instructive. The first is the mostly ignored EDI/Oracle study which found that experienced SME (subject matter expert) attorneys out performed computer search engines. (see the article at http://www.americandiscovery.com/resources/industry-insights/2015/03/edi-oracle-study-electronic-document-review-inferior-to-human-review/ ) .The second is the test run by LawGeex which initially reported that their AI product “…. achieved an average 94% accuracy rate, whereas the “highly-experienced, US trained” human lawyers achieved just 85% accuracy on average.” (see the article at https://www.legalfutures.co.uk/latest-news/ai-beats-average-legal-mind-not-best-performing-lawyers )
But a close look at that report reveals that the two best-performing lawyers each achieved 95% accuracy across the five NDAs, which was 1% BETTER than the computer. In one specific document, the AI spotted 100% of issues, while the best lawyers found 97%. Not much of a difference.
So although the computers were, of course, always faster than humans (humans took an average of 92 minutes to complete a review of the five proffered exemplar documents while the software took just 26 seconds) did the overall difference in average accuracy of only 9% justify the costs of a computer search? Without pricing information, a factor always left out of discussions about computer searches, it’s impossible to say.
Where I totally agree with Ralph is where he states, in his penultimate paragraph,
I urge the judges and litigants faced with a situation like Judge Fashing had to deal with in New Mexico State University, to consider the three choices set out by Judge Francis in Greater New York Taxi Association:
- Cooperation with the other side and their technical consultants to attempt to agree on an appropriate set of search criteria.
- Motions supported by expert testimony and facts regarding the search.
- Appointment of a neutral consultant who will design a search strategy.
Indeed, bring in an expert. A human expert. Computers are a good tool but as Ralph notes in his conclusion, Comment 2 to the ABA Model Rules of Professional Conduct states that “Competent representation can also be provided through the association of a lawyer of established competence in the field in question.”
REPORTS OF THE DEATH OF KEYWORD SEARCHING ARE GREATLY EXAGGERATED
Published: April 4, 2016
Author: Tom O’Connor, Senior eDiscovery Consultant, Advanced Discovery
This week has seen several websites recirculating a law.com article, first released in January just before LegalTech New York, entitled And the Judges Say: It’s Time to Adopt New Legal Technologies. It’s an excellent piece that was an introduction to a great LegalTech session with Judges Andrew J. Peck (SDNY), James C. Francis (also S.D.N.Y.), Elizabeth D. Laporte (N.D. Cal.), and Pamela Meade Sargent (W.D. Va.), in which the judges discussed what they are currently seeing in their courts regarding big data, analytics, e-discovery and other technologies.
However, part of the article is a quote from an interview with Judge Peck in which he says, “I think there’s just too much data to try and do it the old fashioned way. That’s whether you’re talking really old fashioned with eyes-on-everything for review, or the still-old fashioned in my view use of keywords.”
With all due deference to Judge Peck, who I respect and admire both professionally and personally, I’ve disagreed with that position in the past and I still disagree with it. Here is my reasoning.
Much of the lack of confidence in keyword searches is laid at the feet (or pen) of Judge John Facciola, in the case of United States v. O’Keefe, 537 F. Supp. 2d 14 (D.D.C. 2008), with his famous quote about going where angels dare to tread. But that’s not exactly what Judge Facciola opined in that case. He actually dismissed a defendant’s objection to the adequacy of keywords used by the prosecution and ruled that a party challenging the efficacy of an opposing party’s search terms must do so through expert testimony.
His hesitancy was about the Court itself undertaking the complexity of search in the identification and production of electronically stored information. To that specific point, he stated that “[w]hether search terms or ‘keywords’ will yield the information sought is a complicated question involving the interplay, at least, of the sciences of computer technology, statistics and linguistics. … Given this complexity, for lawyers and judges to dare opine that a certain search term or terms would be more likely to produce information than the terms that were used is truly to go where angels fear to tread.”
To be fair, in an earlier decision, Disability Rights Council of Greater Washington v. Washington Metro Transit Authority, 242 F.R.D. 139 (D.D.C. 2007), Judge Facciola did, in fact, state that concept searching is more likely to produce comprehensive results and is more efficient than keyword searches. However there, as in O’Keefe, he questioned the litigants’ ability to demonstrate to him that the results were defensible.
I’ve noted numerous times in the past six months that despite the strong scientific evidence in support of TAR, numerous polls from organizations such as eDJournal and Kroll, as well as client surveys from firms such as Gibson Dunn and Norton Rose, show that the majority of people are NOT using TAR. And if they are not, it seems clear they must be using keyword searches.
This position was supported publicly last year at the Today General Counsel conference in New York by Gene Eames, Pfizer Inc.’s director of Search and Analytics in their Legal Division. Gene made it clear that he is strongly in favor of keyword searches IF the keywords and the results can be tested and validated. His point was that you use keywords to propagate a seed set for the eventual computer search, so why not use it on all the documents as a first pass.
Several people noted that pointing any search tool at all the data may be costly and inefficient. Co-chair David Kessler of Norton Rose Fulbright stated, “If I’m playing hide-and-seek with my kids and it’s my turn to seek, I’m not looking in the breadbox. They won’t be there because they don’t fit there.” That comment prompted Gene to recall a discussion he had with a federal judge about the best way to proceed in a search, where he said if he returned from a meeting at the courthouse to find he had lost his keys he wouldn’t begin a search in Penn Station; he’d start in the lobby of the courthouse.
Everyone in the conversation did agree that the best practice is to bring some common sense to your search process. Technology is great, but it’s not an “Easy Button”, and the best technology for your project depends on a number of variables, including budget, time constraints and search needs. As Maura put it, “TAR is a process, not a product.”
The point about keywords was brought home again several weeks ago at the ASU Arkfeld EDiscovery Conference in Tempe, when I spoke on a panel about keyword searches to an SRO crowd. As I said in my recap of that session, “…keyword search is far from dead and is probably still the most common search method used by eDiscovery staff. TAR may be up and coming, but keyword search is still king.”
