Tutorial Overview

The easiest way to understand how to use the toolkit is through the examples. There are 3 (working) examples in the repository that all use the Microsoft Learning to Rank Web30k dataset in the examples module. They are presented below in order of simplest to most complex.

This was one of the first projects where I have attempted to dive deeper into the core TensorFlow API rather than rely solely on the high level primitives of Keras. So I hve tried to provide as much detail as I can while walking through the code, both to help me and in hopes of providing a resource for others who are not already experts. However, these tutorials still assume a degree of familiarity with TensorFlow and Keras and will be difficult to follow without having gone through some of their tutorials. This is especially true in regards to implementing custom Models.

MSLR Dataset

The Microsoft Learning to Rank dataset is a standard corpus used to compare ranking algorithms. It has approximately 30,000 queries and 136 features. The maximum number of documents in a query is 1,251 and on average each query has 120 documents. More information about the dataset can be found on the microsoft website.

Multi Layer Perceptron (MLP)

The first example demonstrates an implementation of Bruch et al. [Bruch2019], which uses a simple feed forward network with a differentiable approximation of the normalized discounted cumulative gain quality measure for the loss function. If you would prefer to just read through the code rather than follow the tutorial it is available here.

Groupwise Scoring Function (GSF)

The second example is an implementation of Ai et al. [Ai2019], which uses a more complex network whose input considers multiple documents in order to learn comparative features between documents. If you would prefer to just read through the code rather than follow the tutorial it is available here.

Groupwise Attention Scoring Function (ATTN)

The third example uses the self attention mechanism of Vaswani et al. [Vaswani2017] to try to improve upon the Groupwise Scoring Function model. If you would prefer to just read through the code rather than follow the tutorial it is available here.

Tutorial Outline

• Data Preparation
  • Overview
  • Data Conversion
  • Command Line Options
• Example 1: Multi-Layer Perceptron
  • Architecture
  • Overview
  • Load The Data
  • Preprocessing
  • Setup The Model
  • Train The Model
  • Test The Model
  • Running The Script
• Example 2: Groupwise Scoring Function
  • Architecture
  • Overview
  • Load The Data
  • Preprocessing
  • Sample Documents
  • Setup The Model
  • Train The Model
  • Test The Model
  • Running The Script
• Example 3: Groupwise Attention Scoring Function
  • Architecture
  • Overview
  • Load The Data
  • Preprocessing
  • Sample Documents
  • Set Up The Model
  • Train The Model
  • Test The Model
  • Running The Script

References

Ai2019

Qingyao Ai, Xuanhui Wang, Sebastian Bruch, Nadav Golbandi, Michael Bendersky, and Marc Najork. 2019. Learning Groupwise Multivariate Scoring Functions Using Deep Neural Networks. In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, Santa Clara, CA, October.

Bruch2019

Sebastian Bruch, Masrour Zoghi, Mike Bendersky, and Marc Najork. 2019. Revisiting Approximate Metric Optimization in the Age of Deep Neural Networks. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’19), pages 1241–1244.

Vaswani2017

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems 30, pages 5998–6008, Long Beach, CA, December.