Paper Ranking Methodology

Review Data Collection:

I start by gathering all available reviews for each paper from the OpenReview platform for the specified venue. This includes numerical scores (like overall rating, confidence, soundness, etc.) and any textual comments provided by the reviewers. Each review is associated with a specific paper and reviewer.

Normalization with Z-scores:

Reviewers often have different grading tendencies. Some might consistently give high scores, while others are more critical. To make the scores comparable, I use a technique called z-score normalization. A z-score tells you how far away a particular score is from the average score, measured in units of standard deviation. The standard deviation represents the typical spread or variability in the scores.

• Calculation: For each review aspect (e.g., "rating"), I calculate the z-score using this formula: z = (reviewer's score - average score for that aspect) / standard deviation of scores for that aspect
• Example: A z-score of +1 means the reviewer's score is one standard deviation above the average. A z-score of -1 means it's one standard deviation below the average. A z-score of 0 means the score is the average.
• Benefit: This puts all scores on a common scale, so a "high" score from a generally lenient reviewer is comparable to a "high" score from a strict reviewer.

Aggregation of Normalized Scores:

After calculating z-scores for each aspect of each review, I combine the scores for each paper. For each aspect (like "rating" or "confidence"), I average the z-scores across all reviewers who reviewed that paper. This gives me a single, aggregated z-score for each aspect, for each paper.

Beta Distribution Modeling:

To represent the overall quality of a paper, and to acknowledge the inherent uncertainty in the review process, I model the aggregated, normalized scores using a Beta distribution.

• Why Beta? The Beta distribution is a probability distribution defined on the interval [0, 1]. It's often used to model probabilities or proportions, making it a good fit for representing the "quality" of a paper, which can be thought of as the probability that the paper is "good."
• Parameters (alpha and beta): The Beta distribution has two parameters, often called alpha (α) and beta (β). These parameters control the shape of the distribution.
• Transformation: Before using the Beta Distribution, I transform z-scores to be between 0.01 and 0.99. This step is necessary to fit them.
• My Approach: For each aspect of a paper, I calculate the transformed z-score. I then create a Beta distribution where α = 1 + transformed_z` and `β = 1 + (1 - transformed_z). The mean of this Beta distribution is then calculated as α / (α + β).
• Interpretation: The mean of the Beta distribution represents my best estimate of the paper's quality for that aspect, taking into account both the average reviewer scores and the level of agreement (or disagreement) among reviewers. Higher mean values indicate higher estimated quality.

Final Score Calculation:

To arrive at a single final score for each paper, I multiply together the means of the Beta distributions calculated for each aspect (e.g., rating, soundness, contribution). This combines all the individual aspect scores into one overall quality metric. The product is used to ensure that a paper scores highly in all aspects to get a high final score. A low score in even one aspect will significantly reduce the final score.