I wouldn’t have it do that. I’m thinking more along the lines of an AI that drafts legal documents.
It would find a reference to a case, something an ai is good at.
Then it would send out a query to the database to check if that case exists, something you would need to build some tooling for.
If it retrieves anything it would then check if the source and the document agree
if not, it would run a loop to re-write that section.
I’m not like an agent engineer or anything like that, and I’m sure there’s more efficient ways of having an LLM draft legal documents. But so long as you can get it to reference primary sources there should be a lot less cleanup work to do.
You need to format a database query from the file. Then you need to compare what was said in the file to the actual text. Both are things an AI is good at.
You dont need to - you just have to make sure you set up one or two to find holes. Make it 50 points per hole, and 1000 bonus if all is proved wrong. Make two compete and let them know
I don’t think LLMs are great at figuring out if something is written by an LLM, or factual.
I wouldn’t have it do that. I’m thinking more along the lines of an AI that drafts legal documents.
I’m not like an agent engineer or anything like that, and I’m sure there’s more efficient ways of having an LLM draft legal documents. But so long as you can get it to reference primary sources there should be a lot less cleanup work to do.
A database query would be a better too for finding factual information than an LLM.
You need to format a database query from the file. Then you need to compare what was said in the file to the actual text. Both are things an AI is good at.
You dont need to - you just have to make sure you set up one or two to find holes. Make it 50 points per hole, and 1000 bonus if all is proved wrong. Make two compete and let them know
And then you can have more that check those results for faults!! With only 500 agents working full time you have quite a great setup
“Who watches the watchers?”, that problem is far older than both me, you and every LLM.