Design of Provincial Satellite Direct-Receiving Data Sharing Service Based on Data Lake
Meteorological satellites play an increasingly important role in meteorological observation,disaster prevention and mitigation,and weather forecasting.Direct reception,satellite broadcasting,and website download are common methods for obtaining satellite data.A number of high-speed data ground satellite direct reception stations have been built in some regions of each province across China to receive and process various satellite data products such as FY-3 and FY-4 in real-time.Currently,the satellite direct-receiving data are stored on different devices,which poses significant inconvenience for data sharing applications.Convenient and efficient data access with hierarchical security management is urgently needed.Although satellite data product files can be read and accessed through the Tianqing unified data service interface,the files need to be downloaded time after time and stored onto a local disk directory for further processing.This traditional service mode has disadvantages of low time efficiency and short time sequences,which can not meet the demand of big data applications such as artificial intelligence for long time series product files.Therefore,there is an urgent need to build a unified big data file sharing system.A data lake can store any type of massive data and has good scalability.Various data files centralisedly stored can be directly accessed through the data lake engine for analysis and mining.Consequently,a data lake can reduce the repeated storage and migration of data files and significantly improve the support capability in big data applications.So,in this paper,we propose a convenient and efficient access and security hierarchical management scheme for satellite multi-source data using a data lake.We introduce data lake technology to build a provincial meteorological data lake system based on the meteorological big data cloud platform with a set of unified national standards.A substantial volume of raw data and processed products from FY-3 and FY-4 satellite ground receiving stations are being aggregated into the data lake,enabling a unified data sharing service.The technical challenges such as massive multi-source heterogeneous data storage,data management,and data sharing service are being partially resolved to a certain extent.Through the unified authority control technology,we achieve the hierarchical permission management of data access,ensuring the stability and security in data applications.Furthermore,the unified file directory service technology facilitates fast and convenient retrieval of long-time series files,providing significant support for the service applications and satellite data mining in the era of big data.
data lakesatellite direct reception datameteorological big data cloud platformsharing service
NETL
NSTL
万方数据
