Punctuation Restoration Method Based on MEGA Network and Hierarchical Prediction
Punctuation restoration,also known as punctuation prediction,refers to the task of adding appropriate punctuation marks to a text without punctuation to enhance its readability.This is a classic Natural Language Processing(NLP)task.In recent years,with the development of pretraining models and deepening research on punctuation restoration,the performance of punctuation restoration tasks has continuously improved.However,Transformer-based pretraining models have limitations in extracting local information from long-sequence inputs,which hinders the prediction of the final punctuation marks.In addition,previous studies have treated punctuation labels as symbols to be predicted by overlooking the contextual attributes of different punctuation marks and their relationships.To address these issues,this study introduces a Moving average Equipped Gated Attention(MEGA)network as an auxiliary module to enhance the ability of the model to extract local information.Moreover,a hierarchical prediction module is constructed to fully utilize the contextual attributes of different punctuation marks and the relationships between them for the final classification.Experiments are conducted using various transformer-based pretraining models on datasets in different languages.The experimental results on the English punctuation dataset IWSLT demonstrate that applying the MEGA and hierarchical prediction modules to most pretraining models leads to performance gains.Notably,DeBERTaV3 xlarge achieved an F1 score of 85.5％on the REF test set of the IWSLT,which is a 1.2 percentage points improvement compared to the baseline.The proposed model achieved the highest accuracy for the Chinese punctuation dataset.
punctuation restorationNatural Language Processing(NLP)pretrained modelTransformer structurehierarchical prediction
万方数据
维普
