Experimental Setup and Evaluation Metrics

Next: Experimental Results Up: Experimental Evaluation Previous: Experimental Evaluation

3.1 Experimental Setup and Evaluation Metrics

For our experiments, we used the access logs from the Web site of the Association for Consumer Research (ACR) Newsletter (www.acr-news.org). After data preprocessing, the data set contains a total of 18342 transactions and 122 URLs. For our analysis, we eliminate both pageviews which appear in less than 0.5% or more than 80% of transactions and short transactions which contain less than 6 pageviews. This data set was divided into a training set and an evaluation set. After these preprocessing steps, the total number of remaining pageview URLs was 40. Approximately 70% of transactions were randomly selected as training set and the remaining transactions were used for evaluation.

Our evaluation methodology is as follows. Each transaction t in the evaluation set is divided into two parts. The first n pageviews in t are used for generating recommendations, whereas, the remaining portion of t is used to evaluate the generated recommendations. The value n reflects the maximum allowable window size for the experiments (in our case 4). Given a window size $w \leq n$ , we select a subset of the first n pageviews as the surrogate for a user's active session window. The active session window is the portion of the user's clickstream used by the recommendation engine in order to produce a recommendation set. We call this portion of the transaction t the active session with respect to t, denoted by as_t. The recommendation engine takes as_t and a recommendation threshold $\tau$ as inputs and produce a set of pageviews as recommendations. We denote this recommendation set by $R(as_t,\tau)$ . Note that $R(as_t,\tau)$ contains all pageviews whose recommendation score is at least $\tau$ (in particular, if $\tau = 0$ , then $R(as_t,\tau) = P$ , where P is the set of all pageviews).

The set of pageviews $R(as_t,\tau)$ can now be compared with the remaining |t|-n, pageviews in t. We denote this portion of t by eval_t. Our comparison of these sets is based on 2 different metrics, namely, precision and coverage. The precision of $R(as_t,\tau)$ is defined as:

$\begin{displaymath}precision(R(as_t,\tau)) = \frac{{\vert R(as_t,\tau) \cap eval_t\vert}}{{\vert R(as_t,\tau)\vert}}, \end{displaymath}$

and the coverage of $R(as_t,\tau)$ is defined as:

$\begin{displaymath}coverage(R(as_t,\tau)) = \frac{{\vert R(as_t,\tau) \cap eval_t\vert}}{{\vert eval_t\vert}}. \end{displaymath}$

Precision measures the degree to which the recommendation engine produces accurate recommendations (i.e., the proportion of relevant recommendations to the total number of recommendations). Coverage measures the ability of the recommendation engine to produce all of the pageviews that are likely to be visited by the user (i.e., the proportion of relevant recommendations to all pageviews that should be recommended).

Finally, for a given recommendation threshold $\tau$ , the mean over all transactions in the evaluation set was computed as the overall evaluation score for each measure. We ran each set of experiments for thresholds ranging from 0.1 to 1.0. The results of these experiments are presented below.

Next: Experimental Results Up: Experimental Evaluation Previous: Experimental Evaluation

Bamshad Mobasher (mobasher@cs.depaul.edu)
2001-07-29