Why new AI tools for legal research fail to deliver the goods

A host of companies in Canada and the US are touting expensive new products that promise to bring the magic of generative AI to law. The result, they say, is a revolution in research – instant memos with cases directly on point. They don’t come cheap, but they more than make up for it in time saved.

A closer look at these apps reveals that they don’t quite live up to their promise. But the reason why isn’t obvious at first.

Few lawyers currently understand how AI is being used here and why it can’t – and possibly won’t ever – do what we’re hoping it will do: make it easier to find law buried somewhere in a database.

New products like Lexis+ AI and Westlaw Edge boast impressive capabilities. They can be prompted with a fact pattern and produce a short memo listing cases and statutes directly on point.

Techno-optimists will encourage us to believe that the days of running Boolean searches are over. The power of AI has finally been harnessed to make all Canadian law accessible to plain-language queries.

But running several prompts through these tools quickly proves this optimism to be premature. Much of the time, results would include cases containing some of the things I was asking about but not in the specific combinations or on the precise question I prompted the tool.

The failure has to do with how these apps really work. None of these AI research tools are reading the cases. As you can tell from the hit-and-miss quality of the results, they are still relying on Boolean searches.

In order to find the cases we’re looking for – and do so consistently – the AI would have to read all of the cases in the Canadian legal database every time we run a query.

So why don’t they do that?

The reason has to do with a language model’s “context window,” or the amount of data a model can handle when prompting it. When it appeared in 2022, ChatGPT had a window of roughly 2,000 words. The various frontier models can now hold up to 50-100 books in their context window, enabling them to do all sorts of things with text, along with audio and video.

To bring about a real revolution in legal research, we would need to place the entire legal database in the context window. As big as they are now, the windows are not likely to be big enough to fit all of CanLII any time soon.

But of course, case retrieval isn’t the only thing these apps can do. They can also summarize cases or documents you upload. Do these features not make the new products worthwhile?

It is certainly convenient to pull up a case and click a button to summarize it. Many legal researchers are currently doing this for free by copying the link of a case we come across on CanLII, dropping it into GPT4o or Claude and asking for a 200-word summary – but that involves two or three steps.

I expect that CanLII will offer this feature soon (it’s begun to offer summaries of caselaw in some provinces). When Apple Intelligence comes to Canada in December, those of us with a recent Mac or iPhone will have instant summaries of any webpage right in the web browser, Safari. Microsoft’s Edge browser (with Copilot) has offered this function for a while. And all the frontier models can do things with documents, links, or files we upload.

Be wary of products promising the sky when it comes to AI research. But there is good reason to be optimistic. Better research tools – many of them free – are on the horizon.