Article Preview
Top1. Introduction
The web is a collection of documents connected by hyperlinks or accessed through search forms. The collection of documents accessed by the hyperlinks is termed as the surface web. Search engines can easily crawl the surface web. The part of the web which is hidden behind the search forms is termed as the deep web or hidden web. These forms are the entry points to deep web. Crawling the deep web is a challenging task because of its non-accessibility. Such documents can only be accessed by formulating the relevant queries, filling, and submitting in the search forms. Search form attributes can have various labels, text boxes, dropdown lists, radio buttons, etc. Thus, understanding the search forms and their attributes is essential. Text box comes under the infinite domain wherein values need to be predicted rather than available in the search form. Whereas dropdown lists, radio buttons, etc., are the finite domains wherein their values are given in the search forms and do not require any prediction. Most of the deep web information is freely available without any subscription. Moreover, it has more structured and quality content than surface web content. Therefore, significance of the deep web is more than the surface web.
Unranked and ranked deep web are two types of deep web. In unranked deep web, the user enters a query in the search form, and then all the matched results are returned. While in the ranked deep web, a particular query is filled in the search form, then only k matched results are returned rather than all the results. This query is known as an overflow query that returns more than k matched results, and only k matched results are visible to the user. Crawling the ranked deep web is more challenging than unranked deep web because of ranking and return limit k. This combination of ranking and return limit is known as ranking bias. High ranked documents are retrieved more frequently than low ranked documents as limited results are available in the ranked deep web. Most existing techniques assumed that all documents were returned for a given query. Such techniques are not suitable for the ranked deep web. The ranked deep web has a ranking bias which prevents the exhaustive crawling of deep web database and thus more challenging.
There are four challenging steps to crawl the deep web. First is identifying the search forms from billions of surface web pages (Moraes, Heuser, Moreira, & Barbosa, 2013). Second is the semantic identification of search form attributes (Furche et al., 2012). Third is the formulation and optimization of a query to retrieve the deep web content, which is termed as query selection problem (Y. Wang, Lu, Chen, & Li, 2017) (Xu, Yoon, & Tourassi, 2014). The last fourth step is the crawling path process, which requires further crawling to explore the relevant documents (Liakos & Ntoulas, 2012). With the advancement of crawling methods, such challenges can be easily implemented as compared to the third challenge of deep web. The proper query set selection helps in retrieving the deep web content with less redundancy. Query selection for the finite domain is easier. But, the infinite domain requires prediction, which is a very challenging task.
Wang et al. proposed the document frequency based crawling technique (Y. Wang et al., 2017). In this technique, the selection of queries was done from a query set whose document frequency is less than or nearly equal to return limit k. This document frequency technique proved to give significant coverage of the database. There was no discussion about how to select the small queries if there exist more overflow queries in the sample documents. However, the use of multiple keywords was suggested to form small queries in their future work. The proposed query optimization technique has extended the existing work of Wang’s method as described in section 3.2 and 3.3.