forked from anishmahesh/Web-Search-Engines-PageRank
-
Notifications
You must be signed in to change notification settings - Fork 0
/
README
147 lines (105 loc) · 6 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
######################################
#
# Web Search Engines
# CSCIGA - 2580
#
# Homework 3
# Group 8
#
# 21, November, 2016
# @Authors: avm358, sm7029,nk2238
#####################################
------------------------------------------------------------
Commands to run the search engine
------------------------------------------------------------
From the root directory
1.Compile Code :
javac -cp "lib/jsoup-1.10.1.jar:src" src/edu/nyu/cs/cs2580/*.java
2.Generate PageRank ad Numviews
java -cp "lib/jsoup-1.10.1.jar:src" edu.nyu.cs.cs2580.SearchEngine --mode=mining --options=conf/engine.conf
3.Index Corpus
java -cp "lib/jsoup-1.10.1.jar:src" edu.nyu.cs.cs2580.SearchEngine --mode=index --options=conf/engine.conf
4.Run Server
java -cp "lib/jsoup-1.10.1.jar:src" -Xmx512m edu.nyu.cs.cs2580.SearchEngine --mode=serve --options=conf/engine.conf --port=25808
------------------------------------------------------------
Running Pseudo Relavance
------------------------------------------------------------
1. Run Command :
curl "http://localhost:25808/prf?query=$q&ranker=favorite&numdocs=10&numterms=5"
***NOTE:
i. Run above command at time of serving to get spearman generated for any query.
ii. The result is displayed to the client.
iii. Default value for numdocs = 10 and numterms = 10.
------------------------------------------------------------
Running Spearman
------------------------------------------------------------
1. Run Command :
java -cp src edu.nyu.cs.cs2580.Spearman <PATH-TO-PAGERANKS> <PATH-TO-NUMVIEWS>
For our code :
<PATH-TO-PAGERANKS> = data/index/PageRank.tsv
<PATH-TO-NUMVIEWS> = data/index/numViews.tsv
***NOTE:
i. Needs to be run from root directory.
ii. If giving the path, give the exact path of both files with file name.
iii. Run the above after serving if you don't want to give path to page rank and num view,
load method from page rank and num views classes are used for getting data.
------------------------------------------------------------
Running Bhattacharya
------------------------------------------------------------
***NOTE : We are assuming the set V is set of all frequently used terms in document from both queries, hence the result would be same for irrespective of the order of the queries.
1. If given a prf.tsv file, execute the following command:
java -cp src edu.nyu.cs.cs2580.Bhattacharyya prf.tsv qsim.tsv
***NOTE:
It is assumed the above command will be run from the root directory.
i. It should have a "query:File-Name" structure.
ii. Query shouldn't have space but %20 for space.
iii. The file should be at the same directory level as Root Folder.
iv. qsim file is at the same directory level as Root Folder.
2. If given a folder location for finding all prf files, run the following command :
java -cp src edu.nyu.cs.cs2580.Bhattacharyya PrfPath qsim.tsv
***NOTE:
It is assumed the above command will be run from the root directory.
i. There should be a list of queries file called queries.tsv, stored at Data.
ii. Query shouldn't have space but %20 for space.
iii. qsim file is at the same level as Root Directory.
3.Running Through Script
Script:
#!/bin/bash
rm -f prf*.tsv
i=0
while read q ; do
i=$((i + 1));
prfout=prf-$i.tsv;
curl "http://localhost:25808/prf?query=$q&ranker=favorite&numdocs=10&numterms=5" > $prfout;
echo $q:$prfout >> prf.tsv
done < queries.tsv
java -cp src edu.nyu.cs.cs2580.Bhattacharyya prf.tsv qsim.tsv
------------------------------------------------------------
Indexing
------------------------------------------------------------
****NOTE ::: As per the instructions in the question, We have "only" used IndexerInvertedCompressed for this Homework. We assume the graders will run the code with just this indexer.
--------------------------------------------------------------
Implementation Description
--------------------------------------------------------------
1. Num Views
- For NumViews, we compute the num views for each document in our corpus during the mining mode.
- We store the data in data/index/numViews.tsv
- The data is stored in the format: <document name> <numViews>.
- During indexing, this data is loaded and every document has its numViews stored in its object.
- This intrinsic property is then used during ranking with Ranker Comprehensive.
***NOTE : In case you’re providing you own NumViews file for spearman , please ensure it is tab delimited.
2. Page Rank
- We have chosen a value of λ = 0.9 because since our corpus is not a representation of internet which can have many dangling links for a random surfer. We know that the all documents having outlines to each other will be present locally in our corpus. Hence , a higher lambda value would more accurately represent the page ranks.(Having a higher component of transition matrix in ranks).
- For Page Rank we have used the formula G^n*V , where G is the Google matrix
where,
G = λ(Transition Matrix) + (1 - λ) * (Dangling Link Matrix)
n = no of iterations
V = (No of Docs)x1, Matrix with each value = 1
- To save n^3 computations instead of calculating G^n first we first calculate G*V to generate
a 1*(No Of Docs) Matrix . We then multiply this Matrix with G matrix each time for further iterations.
- We have also optimised the matrix multiplication by storing matrix in terms of adjacency list (as Vectors). By this way, we will only store the value of docs that has outlines from a particular docs and not all the documents. As the matrix could be sparse, this will optimise memory space used.
- We store the data in data/index/PageRank.tsv
- The data is stored in the format: <document name> <PageRank Score>.
- During indexing, this data is loaded and every document has its PageRank stored in its object.
- This intrinsic property is then used during ranking with Ranker Comprehensive.
***NOTE : In case you’re providing you own PageRanks file for spearman , please ensure it is tab delimited.