Dimensionality and Measurement Precision in HLE’s Multiple-Choice Subset

Mayank Sharma
Stanford University
masharma@stanford.edu &Savira Nadela
Stanford University
savira@stanford.edu &Tyler Matteson
Stanford University
tylerjm@stanford.edu

Abstract

Humanity’s Last Exam (HLE) is widely used to evaluate frontier language models. HLE organizes its questions into eight subject-domain categories, whose subscores are often interpreted as evidence of distinct capabilities. However, no study has assessed whether these labels correspond to empirically separable latent constructs, nor whether the benchmark effectively differentiates between models of similar ability. We evaluate 29 LLMs on the text-only multiple-choice subset of HLE ( $J=428$ items) and apply psychometric methods to assess both the dimensionality of the benchmark and the distribution of its measurement precision. Fitting a two-parameter logistic IRT model, we find convergent evidence that HLE measures a single general reasoning factor: McDonald’s $\omega_{h}=0.998$ , domain labels explain only 3.5% of item response variance, within- and between-domain residual correlations are nearly identical (Cohen’s $d=0.016$ ), and domain-specific ability estimates are near-redundant with the total score ( $r\geq 0.81$ ). A separate analysis of the test information function reveals that measurement precision concentrates at moderate ability levels and drops sharply above $\theta=0$ , where frontier models sit. These findings suggest that HLE’s domain subscores do not warrant distinct capability interpretations and that the benchmark’s ability to discriminate among the strongest models is limited.

1 Introduction

Humanity’s Last Exam (HLE) has quickly emerged as a prominent benchmark for evaluating advanced language models, appearing in capability assessments and policy discussions shortly after its release (Phan et al., 2026). Designed to resist simple retrieval and pattern matching through expert-level questions spanning mathematics, the natural sciences, and the humanities, HLE represents a plausible candidate for evaluating higher-order reasoning. However, widespread adoption has outpaced systematic evaluation of its measurement properties. Most benchmark studies, including HLE, report aggregate accuracy and domain-specific subscores without testing whether the reported domains correspond to empirically distinct latent factors. These subscores are often interpreted as evidence that one model outperforms another in domains such as mathematics or chemistry, an interpretation that implicitly assumes benchmark categories recover as separable latent capabilities rather than as facets of a single general reasoning factor (Jiang et al., 2026). Whether HLE’s domain labels reflect psychometrically distinct abilities remains largely untested, and the implications are concrete: if the eight domains do not correspond to distinct latent constructs, then per-domain model rankings reported in leaderboards are not warranted by the instrument, and developers and policy audiences should treat HLE as a measure of general reasoning rather than a profile of domain-specific skill.

A second concern is measurement precision. Even if HLE is unidimensional, its ability to differentiate between models depends on where along the ability continuum its items concentrate information. If most items are calibrated for average-performing models, score differences between the strongest frontier models may reflect measurement noise rather than genuine capability gaps, a problem that will intensify as models continue to improve.

This paper addresses both concerns directly. We evaluate 29 LLMs on the text-only MCQ subset of HLE ( $J=428$ items) and apply psychometric methods to investigate two questions: (a) Does HLE’s eight-domain structure reflect distinct latent constructs, or do the domains collapse into a general reasoning factor?, addressed through McDonald’s $\omega_{h}$ , PCA of item response profiles, residual correlation analysis, and domain-level ability comparisons; and (b) Where along the ability continuum does HLE concentrate measurement precision, and which domains contribute most to discrimination among frontier models?, addressed through the test information function decomposed by subject domain.

1.1 Related Work

Benchmarks are the primary instrument by which progress in large language models is evaluated, but their useful lifespan is often short. Popular benchmarks such as MMLU have shown signs of saturation within only a few years of release (Phan et al., 2026). This has motivated the development of HLE, a benchmark consisting of 2,500 expert-level questions spanning mathematics, humanities, and the natural sciences, crowdsourced from nearly 1,000 subject-matter experts across 50 countries and filtered to ensure that contemporary language models could not reliably answer them at submission time (Phan et al., 2026). HLE and related studies (e.g., MMLU (Hendrycks et al., 2021), GPQA (Rein et al., 2023)) typically report aggregate accuracy as the primary evaluation metric, but aggregate scores alone obscure whether items meaningfully discriminate between models and whether domain subscores reflect distinct latent constructs.

These measurement properties matter because benchmark rankings increasingly inform high-stakes decisions ranging from model deployment to AI governance (Liang et al., 2023). When domain subscores are interpreted as evidence of distinct capabilities, the validity of those comparisons is often assumed rather than empirically demonstrated, and decisions based on poorly validated measurements risk misrepresenting both relative model capability and the nature of progress in language model development (Bommasani et al., 2022).

Psychometric methods offer a framework for evaluating these properties directly. Applying IRT across 29 NLP datasets, Vania et al. (Vania et al., 2021) showed that several widely used benchmarks contain items that fail to discriminate effectively between models, raising questions about whether aggregate scores reflect meaningful capability differences. TinyBenchmarks (Polo et al., 2024) uses IRT to enable efficient evaluation on subsets of MMLU, and Chatbot Arena (Chiang et al., 2024) applies Bradley-Terry models to pairwise comparisons. These studies demonstrate the feasibility of treating benchmarks as measurement instruments, but they focus on ranking efficiency and item difficulty rather than on latent dimensionality: the question of whether domain-specific capability claims are structurally warranted. More broadly, benchmark leaderboards frequently report domain-specific performance as evidence of separable capabilities without testing whether such categories recover as empirically distinct latent factors (Jiang et al., 2026). To our knowledge, no psychometric dimensionality analysis has yet been applied to HLE. This study addresses that gap.

2 Methods

2.1 HLE Subset

The full benchmark contains two question formats: exact-match short-answer ( $\approx$ 76%) and multiple-choice ( $\approx$ 24%). Approximately 14% of all questions require image comprehension. Subject coverage skews toward quantitative domains: mathematics (41%), biology/medicine (11%), computer science/AI (10%), physics (9%), humanities/social science (9%), other (9%), chemistry (7%), and engineering (4%). We restrict to the text-only multiple-choice subset, which is the only subset amenable to automated binary scoring without multimodal infrastructure or LLM-judge extraction. This subset comprises $J=513$ items, with individual items containing between 2 and 21 answer choices. The full dataset is publicly available via HuggingFace ¹¹1https://huggingface.co/datasets/cais/hle (cais/hle, split="test") and is filtered to this subset by selecting answer_type == "multipleChoice" and excluding image-dependent items. These items (see examples in Appendix A.2) demonstrate that they cannot be answered by retrieval or pattern matching and require domain expertise, making them valuable for an IRT analysis.

2.2 Model Selection

For generating responses on the subset, we used 37 contemporary language models across five model families: OpenAI ( $n=12$ ): GPT-4.1 series (standard, mini, nano), GPT-5 series (mini, nano, 5.4-mini, 5.4-nano), GPT-4o series (2024-11-20, mini), and reasoning models (o3-mini, o4-mini, o1-mini); Anthropic ( $n=6$ ): Claude Opus (4.7, 4.6, 4.5), Claude Sonnet (4.6, 4.5), and Claude Haiku 4.5; Google ( $n=5$ ): Gemini 3.5 Flash, Gemini 3.1 Flash Lite, Gemini 2.5 series (Pro, Flash, Flash Lite); DeepSeek ( $n=4$ ): DeepSeek-V3, DeepSeek-V4 (Flash, Pro), and DeepSeek-R1 (reasoning); Open-weight models ( $n=10$ ): Gemma (2-9B, 3-27B), Phi-4, Qwen2.5-7B, QwQ-32B (reasoning), Sky-T1-32B (reasoning), Mistral-7B, OLMo-2-7B, Falcon3-10B, and Llama-3.1-8B. Models represented both reasoning-specialized architectures and standard instruction-tuned models.

2.3 Response Collection

For each model-item pair, we prompted models using a standardized format (see prompt in Appendix A.1). To ensure reproducibility, we set temperature=0.0 for all models supporting this parameter. Reasoning-specialized models (OpenAI o-series, Claude Opus 4.7) used their default extended reasoning configurations without temperature control. Maximum response lengths were set to 8,192 tokens, with model-specific adjustments for known constraints (e.g., 4,096 tokens for smaller models). Proprietary models (OpenAI, Anthropic, Google, DeepSeek) were queried via their respective API endpoints with asynchronous request handling and automatic retry logic for rate-limit management. Open-weight models were deployed using vLLM (Kwon et al., 2023), a high-throughput inference engine optimized for large language models, on cloud GPU infrastructure (Modal Labs) using Nvidia A100-40GB, Nvidia A100-80GB, and Nvidia H100 GPUs. Data collection spanned approximately 20 hours, with the majority of latency attributable to reasoning-specialized models.

2.4 Response Parsing and Scoring

Model responses were parsed using rule-based extraction with a hierarchical strategy: (1) regex pattern matching for explicit answer declarations (“Answer:”, “Final Answer”), supporting markdown formatting and parenthetical notation ((A)); (2) flexible pattern matching for natural language phrasings (“the correct answer is A,” “choice is B”); and (3) fallback extraction of the last isolated letter (A-Z) when no explicit answer marker was present. Parsed responses (91.8%) were scored as correct (1) or incorrect (0) by exact-match comparison to the ground-truth answer. Unparsed responses (8.2%) were coded as missing data and predominantly resulted from empty model outputs (DeepSeek variants), safety-filtered refusals (Gemini 2.5 Pro), or inference failures (Phi-4, Gemma-9B).

2.5 Response Matrix Construction

We constructed a binary response matrix $\mathbf{X}\in\{0,1,\text{NA}\}^{N\times J}$ where $N$ represents models and $J=513$ items. From the initial 37 models evaluated, one model (o1-mini) was excluded due to zero coverage (0% valid responses). Coverage among the remaining 36 models ranged from 50.9% to 100%, with median 99.7% and mean 91.8%. Because missingness was not at random (attributable to content safety filtering, inference failures, and rate limiting rather than item difficulty), we retained only the 29 models with excellent coverage ( $\geq$ 95%). Among the 513 items evaluated with these 29 models, 428 items (83.4%) exhibited non-zero variance across models; the remaining 85 items (16.6%) showed zero variance, with all models responding incorrectly, indicating extreme difficulty. We removed zero-variance items to obtain a final analytic matrix of $\mathbf{X}\in\{0,1\}^{29\times 428}$ , comprising 12,412 model-item observations, with remaining sparse missing values (0.79% of observations; $n=117$ ) coded as incorrect responses, as their negligible proportion introduces minimal bias to estimates.

2.6 Analytic Overview

First, we ask whether HLE’s eight subject-domain labels correspond to empirically distinct latent constructs or collapse to a single general factor. Second, we ask how well HLE measures along the underlying factor(s): where measurement precision concentrates along the ability continuum, and which domains contribute most. We address both questions through a unified analysis, first estimating a two-parameter logistic (2PL) IRT model to obtain item-level discrimination and difficulty parameters, which serve as shared inputs to both stages, the dimensionality analysis and the measurement precision analysis.

2.6.1 2PL model estimation

We modeled item-level measurement properties using a two-parameter logistic (2PL) IRT model (Baker and Kim, 2004) fit to $\mathbf{X}$ . The probability that model $i$ correctly answers item $j$ was modeled as:

P(X_{ij}=1\mid\theta_{i},a_{j},b_{j})=\frac{1}{1+\exp[-a_{j}(\theta_{i}-b_{j})]},

(1)

where $\theta_{i}$ represents latent model ability, $a_{j}$ is the item discrimination parameter, and $b_{j}$ is the item difficulty parameter. Higher discrimination values indicate items that better differentiate between stronger and weaker models, whereas higher difficulty values indicate items requiring greater latent ability for a 50% probability of correct response. The model was estimated using marginal maximum likelihood implemented in torch_measure (Truong and others, 2026), with optimization run for up to 2,000 epochs using a learning rate of 0.05. For each item, we extracted estimated discrimination ( $\hat{a}_{j}$ ) and difficulty ( $\hat{b}_{j}$ ).

2.6.2 Dimensionality analysis

We examined whether the benchmark’s 8 subject-domain labels reflected empirically distinct latent constructs. Using data from $N=29$ models and $J=428$ items, we found that the inter-item tetrachoric correlation matrix was rank-deficient and non-positive-definite, preventing the use of confirmatory factor analysis because the resulting fit indices would be invalid (Flora and Curran, 2004). We therefore substituted three $N$ -robust alternatives that together address the same question without requiring inversion of the correlation matrix.

McDonald’s $\omega_{h}$ .

Our primary evidence for unidimensionality is McDonald’s hierarchical omega (McDonald, 1999), computed analytically from the 2PL discrimination parameters. Under the normal-ogive parameterization, item $j$ ’s loading on the general factor is $\lambda_{j}=a_{j}/\sqrt{1+a_{j}^{2}}$ , and $\omega_{h}$ is defined as:

\omega_{h}=\frac{\left(\sum_{j}\lambda_{j}\right)^{2}}{\left(\sum_{j}\lambda_{j}\right)^{2}+\sum_{j}(1-\lambda_{j}^{2})}

(2)

$\omega_{h}$ quantifies the proportion of item variance attributable to the general factor, ranging from 0 (no general factor) to 1 (perfectly unidimensional). Unlike CFA-based indices, this estimator requires no matrix inversion and is stable at small $N$ . We report $\omega_{h}$ overall and separately for each of the eight subject domains, with 95% bootstrap CIs constructed by resampling models with replacement ( $B=200$ ) and refitting the 2PL at each iteration.

PCA on item response profiles.

As a model-free complement, we transposed the response matrix to $\mathbf{X}^{\top}\in\{0,1\}^{J\times N}$ , treating each item as a point in $N$ -dimensional model space, and applied standard PCA. If domain labels capture real structure, items from the same domain should cluster together in this space. We quantified this using domain $R^{2}$ , the proportion of variance in the first three principal components explained by domain membership, where $R^{2}\approx 0$ would indicate that items from the same domain respond no more similarly across models than items from different domains.

Residual item correlations.

To further assess whether domain membership explains structure beyond the general factor, we computed Pearson correlations between item response vectors, subtracted the general-factor-implied correlation matrix $\mathbf{R}_{g}=\boldsymbol{\lambda}\boldsymbol{\lambda}^{\top}$ , and compared within-domain to between-domain residual correlations. If domains capture distinct latent structure, within-domain residuals should be higher than between-domain residuals.

Domain-level $\hat{\theta}$ correlations.

To assess whether domain subscores carry any incremental information beyond the total score, we fit separate 2PL models within each domain and computed the Pearson and Spearman correlations between domain-specific ability estimates $\hat{\theta}_{\text{domain}}$ and overall ability estimates $\hat{\theta}$ . If domain subscores reflect distinct latent dimensions, models should exhibit differential ability profiles across domains; if a single underlying dimension dominates, $r\approx 1$ across all domains would indicate that the total score is sufficient and domain subscores carry little information.

2.6.3 Measurement precision

Taking the 2PL estimates and dimensionality findings together, we characterize where measurement precision concentrates along the ability continuum using the test information function (Baker and Kim, 2004),

I(\theta)=\sum_{j=1}^{J}a_{j}^{2}P_{j}(\theta)\left[1-P_{j}(\theta)\right],

(3)

which quantifies measurement precision at different ability levels, with higher values corresponding to lower conditional SEM. We decomposed the TIF by subject domain to evaluate which domains contributed most strongly to discrimination.

3 Results and Discussion

3.1 2PL Model Estimation

Across the full item pool, estimated difficulty parameters ranged from $-2.06$ to $5.67$ , with a median of $b=0.41$ , indicating that the benchmark was moderately challenging for the evaluated models. The positively skewed distribution further suggests a substantial concentration of advanced items capable of differentiating performance across a wide range of contemporary LLMs (Liang et al., 2023). Discrimination parameters ranged from $0.00$ to the imposed upper cap of $5.00$ , with a median of $a=1.49$ . Many items demonstrated moderate-to-high discrimination values, indicating that the benchmark effectively differentiated between stronger and weaker models (see Figure 1). Mean empirical accuracy across items was relatively low ( $M=0.17$ ), further supporting the conclusion that the benchmark was challenging for most models. See Appendix A.3 for supplemental results.

Refer to caption — Figure 1: Distributions of item difficulty ( $b$ ) and item discrimination ( $a$ ).

Substantial variation emerged across disciplinary categories (see Figure 2). Engineering items exhibited the highest median difficulty ( $b=1.65$ ), followed by Physics ( $b=1.47$ ), whereas Computer Science/AI and Humanities/Social Science showed the lowest median difficulty values (both approximately $b=0.10$ ). In contrast, discrimination patterns differed from difficulty trends. Computer Science/AI demonstrated the highest median discrimination ( $a=2.63$ ), while Engineering showed the lowest ( $a=0.77$ ), indicating that highly difficult domains did not necessarily provide the strongest differentiation between models. Category-level empirical accuracies aligned broadly with difficulty estimates, with Humanities/Social Science producing the highest mean accuracy ( $M=0.19$ ) and Engineering the lowest ( $M=0.13$ ). Figure 3 displays estimated latent ability $\hat{\theta}$ for all 29 models with 95% confidence intervals alongside raw accuracy. Claude Opus 4.7 achieved the highest estimated ability, followed by Gemini 3.5 Flash and Claude Opus 4.6. GPT-4o-2024-11-20 exhibited an unusually low ability estimate ( $\hat{\theta}\approx-1.8$ ) with wide confidence intervals, suggesting unstable parameter estimation likely due to atypically low or inconsistent response patterns. IRT-based ability estimates were strongly correlated with raw accuracy rankings ( $\rho=0.857$ , $p<10^{-8}$ ), confirming that the 2PL model recovers a latent dimension consistent with aggregate performance while additionally characterizing item-level discrimination and measurement precision unavailable from raw scores alone. However, category-level differences in item parameters do not necessarily imply that HLE measures multiple distinct latent abilities: domains could still reflect a single underlying reasoning dimension if models that perform well in one domain tend to perform well across all others. The following analysis examines this directly.