Let's look at another example of using distances and similarities. This time, we'll look at a hypothetical search engine index and see how we can use k-nearest-neighbor search to identify the documents that are most similar to a specified query.¶
In [1]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
In [2]:
cd D:\Documents\Class\CSC478\Data
In our data set, we have 15 documents. We assume that the documents have already been preprocessed, converted into word vectors (bags of words), and inserted into an index. After preprocessing and removing "stop words" we are left with 10 index terms (used as dimensions for the document vectors).¶
In [3]:
DF = pd.read_csv('term-doc-mat.csv', header=None)
DF
Out[3]:
Let's remove the column containing the terms¶
In [4]:
# TD = genfromtxt('term-doc-mat.csv',delimiter=',',usecols=(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15),dtype=int)
TD = DF.ix[:,1:]
TD
Out[4]:
In [5]:
# terms = genfromtxt('term-doc-mat.csv',delimiter=',',usecols=(0),dtype=str)
terms = DF.ix[:,0]
terms
Out[5]:
Transposing the TD matrix.¶
In [6]:
DT = TD.T
Now we have a document-term matrix:¶
In [7]:
DT
Out[7]:
In [8]:
DT.shape
Out[8]:
In [9]:
len(DT.ix[1])
Out[9]:
In [10]:
numTerms=DT.shape[1]
# could also say numTerms = len(DT.ix[1])
NDocs = DT.shape[0]
In [11]:
print numTerms
print NDocs
Next, let's compute term frequencies to get an idea of their distributions across the corpus.¶
In [12]:
termFreqs = TD.sum(axis=1)
print termFreqs
In [16]:
dictTF = {}
for i in range(numTerms):
dictTF[terms[i]] = termFreqs[i]
print sorted(dictTF.items())
sortedTF = sorted(dictTF.values(), reverse=True)
print sortedTF
In [14]:
plt.plot(sortedTF)
plt.show()
We convert the dataframe into a Numpy array which will be used as input for our search function.¶
In [17]:
DT = np.array(DT)
DT
Out[17]:
The search function takes a query object (in this case a vector of words), and searches for the K most similar (least distant) items in the data (our index of documents). The "measure" parameter allows us to use either the Euclidean distance or the inverse of Cosine similarity as our ditance metric. The function returns the indices of the K most similar neighbors and a list of their distances to the query object.¶
In [18]:
def knn_search(x, D, K, measure):
""" find K nearest neighbours of data point x among D """
if measure == 0:
# euclidean distances from the other points
dists = np.sqrt(((D - x)**2).sum(axis=1))
elif measure == 1:
D_norm = np.array([np.linalg.norm(D[i]) for i in range(len(D))])
x_norm = np.linalg.norm(x)
sims = np.dot(D,x)/(D_norm * x_norm)
dists = 1 - sims
idx = np.argsort(dists) # sorting
# return the indexes of K nearest neighbors
return idx[:K], sorted(dists)[:K]
Let's now try this on a new query object¶
In [19]:
x = np.array([3, 22, 0, 17, 9, 6, 1, 12, 0, 22])
x
Out[19]:
In [20]:
# Finding the k=5 nearest neighbors using inverse of Cosine similarity as a distance metric
neigh_idx, distances = knn_search(x, DT, 5, 1)
In [21]:
neigh_idx
Out[21]:
In [22]:
distances
Out[22]:
In [23]:
DT[neigh_idx]
Out[23]:
In [28]:
# Finding the k=5 nearest neighbors using Euclidean distance metric
neigh_idx, distances = knn_search(x, DT, 5, 0)
In [29]:
neigh_idx
Out[29]:
In [30]:
distances
Out[30]:
In [31]:
DT[neigh_idx]
Out[31]: