Research on Dynamic Web Page Crawling Method Based on SVM
This paper proposes a dynamic web page recognition method based on Support Vector Machine(SVM),and combines it with the Scrapy open source web crawler framework to build a web crawler for dynamic web pages,achieving efficient recognition and content capture of dynamic web pages.This paper uses httpbin.org as the test website,uses the SVM model to classify static and dynamic web pages,and then uses the Scrapy framework to dynamically adjust the crawling strategy to verify the feasibility and effectiveness of this method.
Support Vector Machine(SVM)dynamic web page recognitionScrapy frameworkweb crawler
NETL
NSTL
万方数据
