The Supreme Court (SC) on Thursday cautioned against the unchecked use of Artificial Intelligence (AI) in legal proceedings after finding that the National Company Law Tribunal (NCLT) had relied on fake and non-existent case laws generated by AI while delivering the verdict on a case.

Setting aside its order in the Essel Infraprojects insolvency case, the top court emphasised that while AI can assist the legal system, judicial decision making must remain firmly under human control at every stage. A Bench of the Apex Court, comprising Justices PS Narasimha and Alok Aradhe noted that the tribunal had relied on fabricated and ‘hallucinated’ judicial precedents generated by means of AI tools, calling the development a ‘serious threat’ to the integrity of the justice delivery system.

Court likens AI-generated fake precedents to toxic gas

In a strongly-worded remark by the bench, it likened the use of fabricated AI-generated legal material to the release of the Methyl Isocyanate (MIC) gas, terming it an invisible yet potentially devastating danger.

The bench remarked, “This byproduct, that is AI, the production of fake, non-existing and hallucinated material, and its utilisation as precedents in law, is like the release of methyl isocyanate within the province of law and justice. Invisible, insidious and catastrophic, by the time anyone notices, it not only contaminates but takes away the very lifeline of judicial determination.”

The court further said that reliance on such non-existent precedents strikes at the very foundation of adjudication and undermines the confidence of the general public in the judicial process.

AI can assist, but cannot replace human reasoning

The top court thereafter acknowledged the transformative capabilities of AI, noting that the technology has increasingly demonstrated the ability to match, and in certain situations, even surpass human effort in routine as well as intellectual tasks.

However, the bench underscored that AI should serve merely as an aid to adjudication and not become a substitute for judicial reasoning. The judgment stated, “What is significant for our decision making is to resolve to adopt AI technology in aid of adjudication while asserting total and absolute control over adjudication to the human in the loop at every stage.”

Furthermore, it warned that professionals could gradually become overly dependent on AI if its use remains unregulated, ultimately resulting in technology eclipsing independent legal reasoning and decision-making.

Bar Council directed to examine use of AI in legal profession

The case stemmed from a petition filed by suspended director Pooja Ramesh Singh challenging the initiation of the corporate insolvency resolution process against Essel Infraprojects. The NCLT’s Mumbai Bench had admitted the insolvency plea filed by Jammu and Kashmir Bank Limited over an alleged default of Rs 87.43 crore, and the National Company Law Appellate Tribunal (NCLAT) later upheld the decision.

However, the SC discovered that the tribunal had relied on non-existent AI-generated precedents while deciding the matter. Consequently, it junked the orders of both the NCLT and the NCLAT, directing the tribunals to reconsider the case solely on the basis of the facts and applicable law.

Furthermore, the Apex Court also directed the Bar Council of India (BCI) to constitute a committee to undertake a comprehensive examination of the implications of AI in legal practice and recommend an appropriate regulatory framework.