Скачать или смотреть Search Engine with Relative Weight for Query Words - Web Search

We developed a search engine and web crawler. In addition of entering a standard query where each word's importance is 1/N, this applications allows the user to increase/decrease the relative importance of particular words in the query. The query input is comparable to a tag cloud, where the most prominent term is considered the most important term for the user. The user can diminish or enlarge specific words in order to add more weight/importance to a word.

The web crawler is a batch crawler that runs on a localhost installation, mimicking a LAMP stack. The crawler is programmed using the CakePHP framework and we used the Bootstrap 3 frontend framework. The crawler crawls using a indegree ordering policy, but half of the frontier URL list is based on random documents to prevent too greedy crawling. The indexer uses a Porter stemmer to stem words. While it has been shown that Porter stemming can improve document retrieval, we have added stemming to decrease the amount of different terms in the index, to improve performance. The inverted index stores the following metrics: term frequency (TF), log frequency weight, document frequency (DF), collection frequency (CF) and inverse document frequency (IDF). This allows the user to retrieve a ranked list of results, based on TF-IDF (term frequency--inverse document frequency).

Please note that our index is too small to see a significant difference and the utility of the 'relative search'. We are convinced of the usefulness in a real world search engine.