With the increase of system complexity,the scale of log grows larger,making it impractical to analyze them manually.Some researchers have proposed deep learning methods combined with log anomaly detection.However,these methods face several challenges,existing log anomaly detection methods based on deep learning often have issues such as high training cost.Additionally,they rely heavily on high-quality training data and need to be retrained regularly.Recently,Large Language Models have shown promising results in various domains such as machine translation,language under-standing and so on.In our work,we combine Large Language Models with log anomaly detection.By leveraging the rich pre-training knowledge of Large Language Models,we propose an efficient log anomaly detection method in few-shot scenarios without fine-tuning.The method employs hierarchical clustering to extract a small,diverse,and representative collection of normal log messages as a candidate set,which can reflect a wide range of normal log patterns.Additionally,we propose explanation-based prompt learning,which is used to explain each normal log in the candidate set,this method can enhance the model's understanding of normal log patterns.According to the characteristics of log datasets,a specific prompt template for different log datasets is constructed by using the chain of thought strategy.Therefore,the specific prompt template proposed in this paper can also effectively detect log anomalies in zero-shot scenarios.Compared with the existing log anomaly detection methods,the method only requires a very small amount of training data and can achieve high accuracy,which greatly reduces the cost of model training.When the log is updated on a large scale,there is no need to retrain the model.To evaluate the performance of the method,we use two public datasets to verify the effectiveness of the model.The F1 scores of the proposed method on BGL and Spirit datasets reach 81.54%and 96.55%respectively,and the recall scores on two datasets reach 95.00%and 97.77%respectively.The proposed method has high recall scores and F1 scores on two datasets.The results demonstrate that the proposed method is able to effectively achieve log anomaly detection with only a very small amount of training data.
关键词
异常检测/深度学习/大语言模型/ChatGPT
Key words
anomaly detection/deep learning/large language model/ChatGPT
维普
万方数据