Real-time Processing Technologies for High-throughput Infrastructure Detection Data
To meet the real-time processing requirements of detection data in 5G based vehicle ground wireless transmission systems,a high-throughput real-time detection data processing method was proposed,which integrates the distributed stream processing platform Apache Kafka(Kafka)and the distributed processing engine Apache Flink(Flink).A real-time data warehouse was built based on Kafka to achieve orderly and fast reading and storage of real-time detection data.Using Flink as a real-time computing engine to achieve real-time processing and analysis of detection data under high-throughput data streams.To shorten the time for real-time detection data association with ledger data,a ledger association method based on spatiotemporal index was proposed.Through the processing of real-time detection data collected from comprehensive inspection trains,it has been proven that the proposed method can ensure stable and efficient data processing under high-throughput data streams,and the average processing time of real-time detection data is within 1 min,which can meet the remote real-time monitoring needs of ground staff for the status of comprehensive inspection train detection equipment.
high speed railwayreal-time data processing architectureexperimental studyinfrastructure testing dataKafkaFlink
NETL
NSTL
万方数据
