Most of the current problem generation models are based on the Transformer structure,but as the text length increases,the KV caching mechanism of the Transformer leads to a linear increase in GPU occupancy,a decrease in throughput,and an increase in inference cost.To solve this problem,RetNet model was used to construct RetNet-Bert problem generation model.The model uses the multi-scale holding mechanism instead of the multi-head attention mechanism,and has the dual form of parallel and cyclic,which improves the inference efficiency.Experiments prove that RetNet-Bert performs better on long sequence modeling,while achieving training parallelism,low-cost deployment and efficient inference,and achieves a high level of feasibility and effectiveness on the building municipal information generation problem.
problem generation modelRetNet modellong sequence modelingconstruction and municipal information
维普
万方数据
