TensorFlow Ranking

TensorFlow Ranking is a library for Learning-to-Rank (LTR) techniques on the
TensorFlow platform. It contains the following components:

• Commonly used loss functions including pointwise, pairwise, and listwise
  losses.
• Commonly used ranking metrics like
  Mean Reciprocal Rank (MRR)
  and
  Normalized Discounted Cumulative Gain (NDCG).
• Multi-item (also known as groupwise) scoring functions.
• LambdaLoss implementation for
  direct ranking metric optimization.
• Unbiased Learning-to-Rank
  from biased feedback data.

We envision that this library will provide a convenient open platform for
hosting and advancing state-of-the-art ranking models based on deep learning
techniques, and thus facilitate both academic research and industrial
applications.

Tutorial Slides

TF-Ranking was presented at premier conferences in Information Retrieval,
SIGIR 2019 and
ICTIR 2019! The slides are available
here.

Demos

We provide a demo, with no installation required, to get started on using
TF-Ranking. This demo runs on a
colaboratory notebook, an
interactive Python environment. Using sparse features and embeddings in
TF-Ranking
.
This demo demonstrates how to:

*   Use sparse/embedding features
*   Process data in TFRecord format
*   Tensorboard integration in colab notebook, for Estimator API

Also see Running Scripts for executable scripts.

Linux Installation

Stable Builds

To install the latest version from
PyPI, run the following:

# Installing with the `--upgrade` flag ensures you'll get the latest version.
pip install --user --upgrade tensorflow_ranking

To force a Python 3-specific install, replace pip with pip3 in the above
commands. For additional installation help, guidance installing prerequisites,
and (optionally) setting up virtual environments, see the
TensorFlow installation guide.

Note: Since TensorFlow is now included as a dependency of the TensorFlow Ranking
package (in setup.py). If you wish to use different versions of TensorFlow
(e.g., tensorflow-gpu), you may need to uninstall the existing verison and
then install your desired version:

$ pip uninstall tensorflow
$ pip install tensorflow-gpu

Installing from Source

1. To build TensorFlow Ranking locally, you will need to install:

   • Bazel, an open
     source build tool.

     $ sudo apt-get update && sudo apt-get install bazel
     

   • Pip, a Python package manager.

     $ sudo apt-get install python-pip
     

   • VirtualEnv, a tool
     to create isolated Python environments.

     $ pip install --user virtualenv
     

2. Clone the TensorFlow Ranking repository.

   $ git clone https://github.com/tensorflow/ranking.git
   

3. Build TensorFlow Ranking wheel file and store them in /tmp/ranking_pip
   folder.

   $ cd ranking  # The folder which was cloned in Step 2.
   $ bazel build //tensorflow_ranking/tools/pip_package:build_pip_package
   $ bazel-bin/tensorflow_ranking/tools/pip_package/build_pip_package /tmp/ranking_pip
   

4. Install the wheel package using pip. Test in virtualenv, to avoid clash with
   any system dependencies.

   $ ~/.local/bin/virtualenv -p python3 /tmp/tfr
   $ source /tmp/tfr/bin/activate
   (tfr) $ pip install /tmp/ranking_pip/tensorflow_ranking*.whl
   

   In some cases, you may want to install a specific version of tensorflow,
   e.g., tensorflow-gpu or tensorflow==2.0.0. To do so you can either

   (tfr) $ pip uninstall tensorflow
   (tfr) $ pip install tensorflow==2.0.0
   

   or

   (tfr) $ pip uninstall tensorflow
   (tfr) $ pip install tensorflow-gpu
   

5. Run all TensorFlow Ranking tests.

   (tfr) $ bazel test //tensorflow_ranking/...
   

6. Invoke TensorFlow Ranking package in python (within virtualenv).

   (tfr) $ python -c "import tensorflow_ranking"
   

Running Scripts

For ease of experimentation, we also provide
a TFRecord example
and
a LIBSVM example
in the form of executable scripts. This is particularly useful for
hyperparameter tuning, where the hyperparameters are supplied as flags to the
script.

TFRecord Example

1. Set up the data and directory.

   MODEL_DIR=/tmp/tf_record_model && \
   TRAIN=tensorflow_ranking/examples/data/train_elwc.tfrecord && \
   EVAL=tensorflow_ranking/examples/data/eval_elwc.tfrecord && \
   VOCAB=tensorflow_ranking/examples/data/vocab.txt
   

2. Build and run.

   rm -rf $MODEL_DIR && \
   bazel build -c opt \
   tensorflow_ranking/examples/tf_ranking_tfrecord_py_binary && \
   ./bazel-bin/tensorflow_ranking/examples/tf_ranking_tfrecord_py_binary \
   --train_path=$TRAIN \
   --eval_path=$EVAL \
   --vocab_path=$VOCAB \
   --model_dir=$MODEL_DIR \
   --data_format=example_list_with_context
   

LIBSVM Example

1. Set up the data and directory.

   OUTPUT_DIR=/tmp/libsvm && \
   TRAIN=tensorflow_ranking/examples/data/train.txt && \
   VALI=tensorflow_ranking/examples/data/vali.txt && \
   TEST=tensorflow_ranking/examples/data/test.txt
   

2. Build and run.

   rm -rf $OUTPUT_DIR && \
   bazel build -c opt \
   tensorflow_ranking/examples/tf_ranking_libsvm_py_binary && \
   ./bazel-bin/tensorflow_ranking/examples/tf_ranking_libsvm_py_binary \
   --train_path=$TRAIN \
   --vali_path=$VALI \
   --test_path=$TEST \
   --output_dir=$OUTPUT_DIR \
   --num_features=136 \
   --num_train_steps=100
   

TensorBoard

The training results such as loss and metrics can be visualized using
Tensorboard.

1. (Optional) If you are working on remote server, set up port forwarding with
   this command.

   $ ssh <remote-server> -L 8888:127.0.0.1:8888
   

2. Install Tensorboard and invoke it with the following commands.

   (tfr) $ pip install tensorboard
   (tfr) $ tensorboard --logdir $OUTPUT_DIR
   

Jupyter Notebook

An example jupyter notebook is available in
third_party/tensorflow_ranking/examples/handling_sparse_features.ipynb.

1. To run this notebook, first follow the steps in installation to set up
   virtualenv environment with tensorflow_ranking package installed.

2. Install jupyter within virtualenv.

   (tfr) $ pip install jupyter
   

3. Start a jupyter notebook instance on remote server.

   (tfr) $ jupyter notebook third_party/tensorflow_ranking/examples/handling_sparse_features.ipynb \
           --NotebookApp.allow_origin='https://colab.research.google.com' \
           --port=8888
   

4. (Optional) If you are working on remote server, set up port forwarding with
   this command.

   $ ssh <remote-server> -L 8888:127.0.0.1:8888
   

5. Running the notebook.

   • Start jupyter notebook on your local machine at
     http://localhost:8888/ and browse to the
     ipython notebook.

   • An alternative is to use colaboratory notebook via
     colab.research.google.com and open
     the notebook in the browser. Choose local runtime and link to port 8888.

References

• Rama Kumar Pasumarthi, Sebastian Bruch, Xuanhui Wang, Cheng Li, Michael
  Bendersky, Marc Najork, Jan Pfeifer, Nadav Golbandi, Rohan Anil, Stephan
  Wolf. TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank.
  KDD 2019.

• Qingyao Ai, Xuanhui Wang, Sebastian Bruch, Nadav Golbandi, Michael
  Bendersky, Marc Najork. Learning Groupwise Scoring Functions Using Deep
  Neural Networks. ICTIR 2019

• Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. Learning
  to Rank with Selection Bias in Personal Search.
  SIGIR 2016.

• Xuanhui Wang, Cheng Li, Nadav Golbandi, Mike Bendersky, Marc Najork. The
  LambdaLoss Framework for Ranking Metric Optimization.
  CIKM 2018.

Citation

If you use TensorFlow Ranking in your research and would like to cite it, we
suggest you use the following citation:

@inproceedings{TensorflowRankingKDD2019,
   author = {Rama Kumar Pasumarthi and Sebastian Bruch and Xuanhui Wang and Cheng Li and Michael Bendersky and Marc Najork and Jan Pfeifer and Nadav Golbandi and Rohan Anil and Stephan Wolf},
   title = {TF-Ranking: Scalable TensorFlow Library for Learning-to-Rank},
   booktitle = {Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining},
   year = {2019},
   pages = {2970--2978},
   location = {Anchorage, AK}
}