Research on distributed processing framework for big data analysis of real-time water environment monitoring
The rapid accumulation and growth of water quality monitoring data brings new opportunities and challenges to water quality research.In view of the problems of poor timeliness,unintuitive data display,and inefficient data processing faced by water quality monitoring and analysis at present,this paper constructs WaterSpark,a distributed processing framework for real-time water environment monitoring and big data analysis,based on big data technology and data visualization technology,using the improved Canadian Council of Ministers of the Environment water quality index(CCME-WQI)and the Spark machine learning library(MLlib),Applying the water quality monitoring data of nine plateau lakes in Yunnan province,the results show that WaterSpark has excellent performance in real-time water quality transmission,cleaning and archiving,and efficient computing and analysis.It can enable large-scale water quality data to be captured and analyzed timely and accurately,and the distributed data sets and clusters can cope with the growing water quality data to ensure performance scalability,and to support more water quality indicators and water quality monitoring on a larger scale.
water quality monitoringdistributed processing frameworkwater quality indexnine plateau lakes in Yunnan province
万方数据
维普
