A comprehensive evaluation of large language models in mining gene relations and pathway knowledge

扫码查看

原文链接

万方数据
维普

外文摘要：Understanding complex biological pathways,including gene-gene in-teractions and gene regulatory networks,is critical for exploring disease mechanisms and drug development.Manual literature curation of biological pathways cannot keep up with the exponential growth of new discoveries in the literature.Large-scale language models(LLMs)trained on extensive text corpora contain rich biological information,and they can be mined as a biological knowledge graph.This study assesses 21 LLMs,including both application programming interface(API)-based models and open-source models in their capacities of retrieving biological knowledge.The evalua-tion focuses on predicting gene regulatory relations(activation,inhibition,and phosphorylation)and the Kyoto Encyclopedia of Genes and Genomes(KEGG)pathway components.Results indicated a significant disparity in model performance.API-based models GPT-4 and Claude-Pro showed superior performance,with an F1 score of 0.4448 and 0.4386 for the gene regulatory relation prediction,and a Jaccard similarity index of 0.2778 and 0.2657 for the KEGG pathway prediction,respectively.Open-source models lagged behind their API-based counterparts,whereas Falcon-180b and Ilama2-7b had the highest F1 scores of 0.2787 and 0.1923 in gene regulatory relations,respectively.The KEGG pathway recognition had a Jaccard similarity index of 0.2237 for Falcon-180b and 0.2207 for Ilama2-7b.Our study suggests that LLMs are informative in gene network analysis and pathway mapping,but their effectiveness varies,necessitating careful model selection.This work also provides a case study and insight into using LLMs das knowledge graphs.Our code is publicly available at the website of GitHub(Muh-aza).

外文关键词：

biomedical text mininggene-gene interactionKEGG pathwaylarge language model

作者：

Muhammad Azam、Yibo Chen、Micheal Olaolu Arowolo、Haowang Liu、Mihail Popescu、Dong Xu

展开 >

作者单位：

Department of Electrical Engineering and Computer Science,University of Missouri,Columbia,Missouri,USA

Bond Life Sciences Center,University of Missouri,Columbia,Missouri,USA

Institute for Data Science and Informatics,University of Missouri,Columbia,Missouri,USA

Department of Biomedical Informatics,Biostatistics and Medical Epidemiology,University of Missouri,Columbia,Missouri,USA

展开 >

出版年：

2024

DOI：

10.1002/qub2.57

定量生物学(英文版)

ISSN：

年,卷(期)：2024.12(4)