news-please - A Generic News Crawler and Extractor
news-please - A Generic News Crawler and Extractor
复制标题
news-please - 通用新闻爬虫和提取器
DOI：
--
复制
发表时间：
2017
期刊：
影响因子：
--
通讯作者：
Bela Gipp
中科院分区：
文献类型：
--
作者：
Felix Hamborg;Norman Meuschke;Corinna Breitinger;Bela Gipp
近年来，在线发布和阅读的新闻数量大大增加，这使得新闻数据成为许多研究学科的有趣资源，例如社会科学和语言学。但是，由于缺乏爬网和提取此类数据的通用工具，大规模的新闻数据集合很麻烦。我们提出新闻愉快，这是一个通用，多语言，开源轨道和新闻的提取器，可为各种新闻网站开箱即用。我们的系统允许爬行任意新闻网站并在这些网站上提取新闻文章的主要元素，即标题，铅段，主要内容，出版日期，作者和主要图像。与现有工具相比，新闻愉快的特征功能仅需要根URL的完整网站提取。
The amount of news published and read online has increased tremendously in recent years, making news data an interesting resource for many research disciplines, such as the social sciences and linguistics. However, large scale collection of news data is cumbersome due to a lack of generic tools for crawling and extracting such data. We present news-please, a generic, multi-language, open-source crawler and extractor for news that works out-of-the-box for a large variety of news websites. Our system allows crawling arbitrary news websites and extracting the major elements of news articles on those websites, i.e., title, lead paragraph, main content, publication date, author, and main image. Compared to existing tools, news-please features full website extraction requiring only the root URL.