Measuring IIA Violations in Similarity Choices with Bayesian Models

Similarity choice data arise when humans make choices among alternatives based on their similarity to a target, \emph{e.g.}, in information retrieval and embedding learning settings. Classical metric-based models of similarity choice typically assume independence of irrelevant alternatives (IIA), a property that enables simpler formulations. While IIA violations have been documented in many discrete choice settings, the similarity choice setting has received comparatively little attention, largely because the target-dependent nature of the choice complicates IIA testing. We propose two statistical methods to test for IIA: a classical goodness-of-fit test and a Bayesian counterpart based on the framework of Posterior Predictive Checks (PPC). The Bayesian approach, which constitutes our main technical contribution, quantifies the degree of IIA violation beyond mere statistical significance. We curate two datasets: one with choice sets designed to elicit IIA violations, and another with randomly generated choice sets drawn from the same item universe. Our tests reveal significant IIA violations on both datasets, and notably, we find a comparable degree of violation between them. In addition, we introduce a new PPC-based test for population homogeneity. The results indicate that the population is homogeneous, suggesting that the observed IIA violations are driven by context effects, specifically interactions within the choice sets. These findings highlight the need for new similarity choice models that explicitly account for such context effects.

Publisher Page | ArXiv PDF