design-documents

  网页信息采集子系统： 网页采集：动态查找及实时分析新增网页，读取网页回帖信息。 网页过滤：，对获取的网页，通过网页清洗模块清除网页中的广告、导航信息、 图片、版权说明等噪声数据，萃取出相关网页的标题、正文、链接地址、采集时 间、回帖、发帖人数等数据。 网页信息预处理子系统： 网页审查脏字：主要功能包括中文分词、词性标注、命名实体识别、新词识别， 建立数据库说明每类词库，建立敏感词词库。 网页舆情监测：监测是否出现了一定影响的舆论，即回帖的回帖或支持、顶等 达到一个预先设定的阈值； 舆情分析子系统： 汇总网站舆情：汇总整个网站的舆情监控信息，自动生成报告。(Web information collection subsystem: the pages collection: dynamic lookup and real-time analysis of the new website, read pages Replies. Web filtering: to clear advertising pages, navigation information, images, copyright noise data access pages via web cleaning module, extract the relevant page title, text, link address, acquisition time, posts, posting number and other data. Web information preprocessing subsystem: Web review dirty: The main features include the Chinese word segmentation, POS tagging, named entity recognition, new word identification, to establish a database description of each type of thesaurus, and the establishment of a sensitive word thesaurus. Page opinion monitoring: monitoring whether there is a certain influence public opinion, that a reply to the post or support other top reaches a preset threshold public opinion analysis subsystems: summary website public opinion, public opinion monitoring information: summary of the entire site, automatically generate re)

design documents.pdf,771713,2013-01-09