Can Artificial Intelligence Predict Judicial Decisions? A Systematic Review of International Research
Can Artificial Intelligence Predict Judicial Decisions? A Systematic Review of International Research
DOI:
https://doi.org/10.14515/monitoring.2024.5.2580Keywords:
artificial intelligence, prediction of judicial decisions, machine learning, deep learning, legal document classification, algorithmic accuracyAbstract
Advancements in artificial intelligence technologies and the emergence of open databases containing judicial decisions have led to rapid improvements in algorithms capable of classifying legal documents and forecasting decisions made by judges. This article examines a body of international research dedicated to how accurately AI can predict judges’ decisions and whether it could potentially replace human judges in the future. The answer to this question is formed by analyzing two key aspects: the capability and accuracy of predicting judicial decisions and the various constraints associated with using AI.
Analysis of international experience shows that the accuracy of predictions has increased in recent years; however, the quality of the models depends greatly on the specificity of the tasks and the available data. Most studies analyze decisions from higher courts worldwide, significantly reducing their practical potential for dealing with mass categories of cases. Moreover, concerns have arisen regarding the use of models that operate on a “black box” principle, as their decisions are difficult to interpret. Despite the rapid development of AI technologies, the complete replacement of judges is unlikely because of the range of methodological limitations, including insufficient quality and volume of data, issues with interpretability, challenges in understanding legal and cultural context, and limitations in transferring models to other legal systems. However, AI technologies can be used to reduce the costs associated with case material handling.
Acknowledgments. The research was funded by the Russian Science Foundation grant No. 23-78-10073 “Development and approbation of the algorithm for automated analysis of the court decisions texts for socio-legal studies (based on cases of violent crimes)” (see more: https://rscf.ru/project/23-78-10073/).
References
Захарцев С. И., Сальников В. П. Судья-робот в уголовном процессе: хорошо или плохо? // Юридическая наука; история и современность. 2018. № 7. С. 176—180.
Zakharcev S. I., Salnikov V. P. (2018) The Robot Judge in Criminal Proceedings: Good or Bad? Legal Science: History and Modernity. No. 7. P. 176—180. (In Russ.)
Коваленко К. Е., Печатнова Ю. В., Стаценко Д. А., Коваленко Н. Е. Судья-робот как преодоление противоречий судебного усмотрения (юридический аспект) // Юридический вестник Дагестанского Государственного Университета. 2020. Т. 36. № 4. С. 169—173.
Kovalenko K. E., Pechatnova Yu. V., Statsenko D. A., Kovalenko N. E. (2020) The Robot Judge as a Resolution of Judicial Discretion Contradictions (Legal Aspect). Legal Bulletin of Dagestan State University. Vol. 36. No. 4. P. 169—173. (In Russ.)
Кравчук Н. В. Искусственный интеллект как судья: перспективы и опасения (Обзор) // Социальные и гуманитарные науки. Отечественная и зарубежная литература. Серия 4: Государство и право. 2021. № 1. С. 115—122.
Kravchuk N. V. (2021) Artificial Intelligence as a Judge: Prospects and Concerns (Review). Social and Human Sciences. Domestic and Foreign Literature. Series 4: State and Law. No. 1. P. 115—122. (In Russ.)
Степанов О. А., Басангов Д. А. О перспективах влияния искусственного интеллекта на судопроизводство // Вестник Томского Государственного Университета. Философия. Социология. Политология. 2022. № 475. С. 229—237.
Stepanov O. A., Basangov D. A. (2022) On the Prospects of Artificial Intelligence Impact on Judiciary. Bulletin of Tomsk State University. Philosophy, Sociology, Political Science. No. 475. P. 229—237. (In Russ.)
Alali M., Syed S., Alsayed M., Patel S., Bodala H. (2021) JUSTICE: A Benchmark Dataset for Supreme Court’s Judgment Prediction. arXiv. Art. 2112.03414. https://doi.org/10.48550/arXiv.2112.03414.
Alarie B., Niblett A., Yoon A. (2017) Using Machine Learning to Predict Outcomes in Tax Law. SSRN. https://doi.org/10.2139/ssrn.2855977.
Albonetti C. A. (1991) An Integration of Theories to Explain Judicial Discretion. Social Problems. Vol. 38. No. 2. P. 247―266. https://doi.org/10.2307/800532.
Alcántara Francia O. A., Nunez-del-Prado M., Alatrista-Salas H. (2022) Survey of Text Mining Techniques Applied to Judicial Decisions Prediction. Applied Sciences. Vol. 12. No. 20. Art. 20. https://doi.org/10.3390/app122010200.
Aletras N., Tsarapatsanis D., Preoţiuc-Pietro D., Lampos V. (2016) Predicting Judicial Decisions of the European Court of Human Rights: A Natural Language Processing Perspective. PeerJ Computer Science. Vol. 2. No. 10. Art. e93. https://doi.org/10.7717/peerj-cs.93.
Alghazzawi D., Bamasag O., Albeshri A., Sana I., Ullah H., Asghar M. Z. (2022) Efficient Prediction of Court Judgments Using an LSTM+CNN Neural Network Model with an Optimal Feature Set. Mathematics. Vol. 10. No. 5. Art. 5. https://doi.org/10.3390/math10050683.
Ashley K. D., Brüninghaus S. (2009) Automatically Classifying Case Texts and Predicting Outcomes. Artificial Intelligence and Law. Vol. 17. No. 2. P. 125―165. https://doi.org/10.1007/s10506-009-9077-9.
Bagherian-Marandi N., Ravanshadnia M., Akbarzadeh-T M.-R. (2021) Two-Layered Fuzzy Logic-Based Model for Predicting Court Decisions in Construction Contract Disputes. Artificial Intelligence and Law. Vol. 29. No. 4. P. 453―484. https://doi.org/10.1007/s10506-021-09281-9.
Barysė D., Sarel R. (2023) Algorithms in the Court: Does It Matter Which Part of the Judicial Decision-Making is Automated? Artificial Intelligence and Law. Vol. 32. P. 117―146. https://doi.org/10.1007/s10506-022-09343-6.
Beriain I. D. M. (2018) Does the Use of Risk Assessments in Sentences Respect the Right to Due Process? A Critical Analysis of the Wisconsin v. Loomis Ruling. Law, Probability & Risk. Vol. 17. No. 1. P. 45―53. https://doi.org/10.1093/lpr/mgy001.
Brennan, T., Dieterich W. (2018) Correctional Offender Management Profiles for Alternative Sanctions (COMPAS). In: Singh J. P., Kroner D. G., Wormith J. S., Desmarais S. L., Hamilton Z. (eds.) Handbook of Recidivism Risk/Needs Assessment Tools. Hoboken: John Wiley & Sons. P. 49–75. https://doi.org/10.1002/9781119184256.ch3.
Chalkidis I., Androutsopoulos I., Aletras N. (2019) Neural Legal Judgment Prediction in English. arXiv. Art. 1906.02059. https://doi.org/10.48550/arXiv.1906.02059.
Chalkidis I., Fergadiotis M., Malakasiotis P., Aletras N., Androutsopoulos I., Androutsopoulos I. (2020) LEGAL-BERT: The Muppets Straight Out of Law School. arXiv. Art. 2010.02559. https://arxiv.org/abs/2010.02559.
Chicco D., Jurman G. (2020) The Advantages of the Matthews Correlation Coefficient (MCC) over F1 Score and Accuracy in Binary Classification Evaluation. BMC Genomics. Vol. 21. No. 1. Art. 6. https://doi.org/10.1186/s12864-019-6413-7.
Devlin J., Chang M.-W., Lee K., Toutanova K. (2019) BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. In: Burstein J., Doran C., Solorio T. (eds.) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Vol. 1. Minneapolis, Association for Computational Linguistics. P. 4171—4186. https://doi.org/10.18653/v1/N19-1423.
Doerner J. K. (2015) The Joint Effects of Gender and Race/Ethnicity on Sentencing Outcomes in Federal Courts. Women & Criminal Justice. Vol. 25. No. 5. P. 313—338. https://doi.org/10.1080/08974454.2014.989298.
Franklin C. A., Fearn N. E. (2008) Gender, Race, and Formal Court Decision-Making Outcomes: Chivalry/Paternalism, Conflict Theory or Gender Conflict? Journal of Criminal Justice. Vol. 36. No. 3. P. 279—290. https://doi.org/10.1016/j.jcrimjus.2008.04.009.
Freitas A. L., Allende‐Cid H., Santana O., Oliveira-Lage L. (2022) Predicting Brazilian Court Decisions. PeerJ Computer Science. Vol. 8. Art. e904. https://doi.org/10.7717/peerj-cs.904.
Gerber M. S. (2014) Predicting Crime Using Twitter and Kernel Density Estimation. Decision Support Systems. Vol. 61. P. 115—125. https://doi.org/10.1016/j.dss.2014.02.003.
Jacob de Menezes-Neto E., Clementino M. B. M. (2022) Using Deep Learning to Predict Outcomes of Legal Appeals Better than Human Experts: A Study with Data from Brazilian Federal Courts. PLoS ONE. Vol. 17. No. 7. Art. e0272287. https://doi.org/10.1371/journal.pone.0272287.
Dressel J., Farid H. (2018) The Accuracy, Fairness, and Limits of Predicting Recidivism. Science Advances. Vol. 4. No. 1. Art. eaao5580. https://doi.org/10.1126/sciadv.aao5580.
Cui J., Shen X., Wen S. (2023) A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges. IEEE Access. Vol. 11. P. 102050―102071. https://doi.org/10.1109/access.2023.3317083.
Katz D. M., Bommarito M. J., Blackman J. (2017) A General Approach for Predicting the Behavior of the Supreme Court of the United States. PLoS ONE. Vol. 12. No. 4. Art. e0174698. https://doi.org/10.1371/journal.pone.0174698.
Kaufman A. R., Kraft P., Sen M. (2019) Improving Supreme Court Forecasting Using Boosted Decision Trees. Political Analysis. Vol. 27. No. 3. P. 381―387. https://doi.org/10.1017/pan.2018.59.
Kaur H., Choudhury T., Singh T. P., Shamoon Mohammad. (2019) Crime Analysis using Text Mining. In: Ming Fong A.Ch., Hong G.Y., Fong B. (eds.) 2019 International Conference on Contemporary Computing and Informatics (IC3I). Singapore. P. 283―288. https://doi.org/10.1109/IC3I46837.2019.9055606.
Kort F. (1957) Predicting Supreme Court Decisions Mathematically: A Quantitative Analysis of the “Right to Counsel” Cases. American Political Science Review. Vol. 51. No. 1. P. 1―12. https://doi.org/10.2307/1951767.
Kowsrihawat K., Vateekul P., Boonkwan P. (2018) Predicting Judicial Decisions of Criminal Cases from Thai Supreme Court Using Bi-directional GRU with Attention Mechanism. In: Do Van T., Do Duc H., Nguyen G. (eds.) 5th Asian Conference on Defense Technology (ACDT). Hanoi, Vietnam. P. 50―55. https://doi.org/10.1109/acdt.2018.8592948.
Malik V., Sanjay R., Nigam S. K., Ghosh K., Guha S. K., Bhattacharya A., Bhattacharya A., Bhattacharya A., Modi A. (2021) ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation. In: Zong Ch., Xia F., Li W., Navigli R. (eds.) Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer. Stroudsburg: Association for Computational Linguistics. P. 4046-4062. https://doi.org/10.18653/v1/2021.acl-long.313.
Manresa-Yee C., Ramis S. (2022) Assessing Gender Bias in Predictive Algorithms using eXplainable AI arXiv. Art. 2203.10264. https://doi.org/10.48550/arXiv.2203.10264.
Medvedeva M., Üstun A., Xu X., Vols M., Wieling M. (2021) Automatic Judgement Forecasting for Pending Applications of the European Court of Human Rights. In: Ashley K.D., Atkinson K., Branting K., Francesconi E., Grabmair M., Walker V.R., Waltl B., Wyner A.Z. (eds.) Proceedings of the fifth workshop on automated semantic analysis of information in legal text (ASAIL 2021), São Paulo, Brazil. CEUR Workshop Proceedings. P. 1-12. https://ceur-ws.org/Vol-2888/paper2.pdf.
Medvedeva M., Vols M., Wieling M. (2020) Using Machine Learning to Predict Decisions of the European Court of Human Rights. Artificial Intelligence and Law. Vol. 28. No. 2. P. 237―266. https://doi.org/10.1007/s10506-019-09255-y.
Medvedeva M., Wieling M., Vols M. (2022) Rethinking the Field of Automatic Prediction of Court Decisions. Artificial Intelligence and Law. Vol. 31. P. 195―212. https://doi.org/10.1007/s10506-021-09306-3.
Medvedeva M., Xiao X., Wieling M., Vols M. (2020) JURI SAYS: An Automatic Judgement Prediction System for the European Court of Human Rights. Frontiers in Artificial Intelligence and Applications. Vol. 334. P. 277―280. https://doi.org/10.3233/faia200883.
O’Sullivan C., Beel J. (2019) Predicting the Outcome of Judicial Decisions made by the European Court of Human Rights. arXiv. Art. 1912.10819. https://doi.org/10.48550/arXiv.1912.10819.
Quemy A., Wrembel R. (2020) On Integrating and Classifying Legal Text Documents. In: Hartmann S., Küng J., Kotsis G., Tjoa A. M., Khalil I. (eds.) Database and Expert Systems Applications. Cham, Switzerland: Springer International Publishing. P. 385―399. https://doi.org/10.1007/978-3-030-59003-1_25.
Rudin C. (2019) Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead. Nature Machine Intelligence. Vol. 1. No. 5. Art. 5. https://doi.org/10.1038/s42256-019-0048-x.
Ruger T. W., Kim P. T., Martin A., Martin A. D., Quinn K. M. (2004) The Supreme Court Forecasting Project: Legal and Political Science Approaches to Predicting Supreme Court Decisionmaking. Columbia Law Review. Vol. 104. No. 4. P. 1150―1209. https://doi.org/10.2307/4099370.
Rusakova E. P. (2021) Integration of “Smart” Technologies in the Civil Proceedings of the People’s Republic of China. RUDN Journal of Law. Vol. 25. No. 3. P. 622―633. https://doi.org/10.22363/2313-2337-2021-25-3-622-633.
Sebők M., Kiss R., Járay I. (2023) Introducing HUNCOURT: A New Open Legal Database Covering the Decisions of the Hungarian Constitutional Court for Between 1990 and 2021. Journal of the Knowledge Economy. Vol. 15. P. 6507―6540. https://doi.org/10.1007/s13132-023-01395-6.
Sert M. F., Yıldırım E., Haşlak İ. (2022) Using Artificial Intelligence to Predict Decisions of the Turkish Constitutional Court. Social Science Computer Review. Vol. 40. No. 6. P. 1416―1435. https://doi.org/10.1177/08944393211010398.
Shaikh R. A., Sahu T. P., Anand V. (2020) Predicting Outcomes of Legal Cases based on Legal Factors using Classifiers. Procedia Computer Science. Vol. 167. P. 2393―2402. https://doi.org/10.1016/j.procs.2020.03.292.
Sharma R. D., Mittal S., Tripathi S., Acharya S. (2015) Using Modern Neural Networks to Predict the Decisions of Supreme Court of the United States with State-of-the-Art Accuracy. In: Arik, S., Huang, T., Lai, W., Liu, Q. (eds.) Neural Information Processing. ICONIP 2015. Lecture Notes in Computer Science. Vol. 9490. Cham: Springer. P. 475–483. https://doi.org/10.1007/978-3-319-26535-3_54.
Shi C., Sourdin T., Li B. (2021) The Smart Court — A New Pathway to Justice in China? International Journal of Court Administration. Vol. 12. No. 1. Art. 1. https://doi.org/10.36745/ijca.367.
Song D., Gao S., He B., Schilder F. (2022) On the Effectiveness of Pre-Trained Language Models for Legal Natural Language Processing: An Empirical Study. IEEE Access. Vol. 10. P. 75835—75858. https://doi.org/10.1109/ACCESS.2022.3190408.
Sourdin T., Cornes R. (2018) Do Judges Need to Be Human? The Implications of Technology for Responsive Judging. In: Sourdin T., Zariski A. (eds.) The Responsive Judge: International Perspectives. Singapore: Springer. P. 87—119. https://doi.org/10.1007/978-981-13-1023-2_4.
Steffensmeier D., Ulmer J., Kramer J. (1998) The Interaction of Race, Gender, and Age in Criminal Sentencing: The Punishment Cost of Being Young, Black, and Male. Criminology. Vol. 36. No. 4. P. 763—798. https://doi.org/10.1111/j.1745-9125.1998.tb01265.x.
Tahura U. S., Selvadurai N. (2023) The Use of Artificial Intelligence in Judicial Decisionmaking: The Example of China. International Journal of Law, Ethics, and Technology. Vol. 3. P. 1—20. https://doi.org/10.55574/pyeb5374.
Tan H., Zhang B., Zhang H., Li R. (2020) The Sentencing-Element-Aware Model for Explainable Term-of-Penalty Prediction. In: Zhu, X., Zhang, M., Hong, Y., He, R. (eds.) Natural Language Processing and Chinese Computing. NLPCC 2020. Lecture Notes in Computer Science. Vol. 12431. Cham: Springer. P. 16―27. https://doi.org/10.1007/978-3-030-60457-8_2.
Taylor I. (2023) Justice by Algorithm: The Limits of AI in Criminal Sentencing. Criminal Justice Ethics. Vol. 42. No. 3. P. 193―213. https://doi.org/10.1080/0731129X.2023.2275967.
Virtucio M. B. L., Aborot J. A., Abonita J. K. C., Avinante R. S., Copino R. J. B., Neverida M. P., Osiana V. O., Peramo E. C., Syjuco J. G., Tan G. B. A. (2018) Predicting Decisions of the Philippine Supreme Court Using Natural Language Processing and Machine Learning. In: Honiden S., Fujii R. (eds.) 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC). Tokyo. P. 130―135. https://doi.org/10.1109/compsac.2018.10348.
Volkov V. (2016) Legal and Extralegal Origins of Sentencing Disparities: Evidence from Russia’s Criminal Courts. Journal of Empirical Legal Studies. Vol. 13. No. 4. P. 637―665. https://doi.org/10.1111/jels.12128.
Waltl B., Bonczek G., Scepankova E., Landthaler J., Matthes F. (2017) Predicting the Outcome of Appeal Decisions in Germany’s Tax Law. Electronic Participation. In: Parycek P. et al. (eds.) Electronic Participation. ePart 2017. Lecture Notes in Computer Science. Vol. 10429. Cham: Springer. P. 89-99. https://doi.org/10.1007/978-3-319-64322-9_8.
Xiao C., Zhong H., Guo Z., Tu C., Liu Z., Sun M., Feng Y., Han X., Hu Z., Wang H., Xu J. (2018) CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction. arXiv. Art. 1807.02478. https://arxiv.org/abs/1807.02478.
Xu Z. (2022) Human Judges in the Era of Artificial Intelligence: Challenges and Opportunities. Applied Artificial Intelligence. Vol. 36. No. 1. Art. 2013652. https://doi.org/10.1080/08839514.2021.2013652.
Zhong H., Xiao C., Tu C., Tianyang Zhang, Zhiyuan L., Liu Z., Liu Z., Sun M. (2020) How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence. In: Jurafsky D., Chai J., Schluter N., Tetreault J. (eds.) Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg. Association for Computational Linguistics. P. 5218―5230. https://aclanthology.org/2020.acl-main.466/.
Zhuchkova S., Kazun A. (2023) Exploring Gender Bias in Homicide Sentencing: An Empirical Study of Russian Court Decisions Using Text Mining. Homicide Studies. https://doi.org/10.1177/10887679231217159.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Monitoring of Public Opinion: Economic and Social Changes
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
