The MADRS Pipeline: Supporting Depression Assessment in Clinical Trials

^*Mila Fodor ^*Katalin Ócsai ^*+Francesco Periti Rien Sonck
Alex Boudreau
^*These authors contributed equally. ⁺Corresponding author
Clario, part of Thermo Fisher Scientific
name.surname@clario.com

Abstract

Depression is a major mental disorder for which diagnosis relies primarily on clinical assessments. Automated methods to support its detection via the psychiatric MADRS scale are getting more and more attention. While existing solutions primarily focus on detecting the disorder from different text sources (e.g., online text, social media), there is still limited support for clinical trials, where clinical assessments are conducted through structured interviews based on standard guidelines such as SIGMA. In this work, we develop a LLM pipeline specifically designed to support clinicians in supporting the assessment of depression in patients enrolled in clinical trials. Our pipeline converts audio interviews into transcripts, maps them into the ten MADRS symptom items, estimates their severity, and identify problematic clinical ratings associated with them. Evaluation on real clinical interviews shows a strong overall correlation of 0.867 with expert ratings, providing interpretable support for future assessments in clinical trials.

^*Mila Fodor and ^*Katalin Ócsai and ^*+Francesco Periti and Rien Sonck and Alex Boudreau ^*These authors contributed equally. ⁺Corresponding author Clario, part of Thermo Fisher Scientific name.surname@clario.com

1 Introduction

Depression is a major mental health disorder and a significant risk factor for suicidal behavior Weinberger et al. (2018). In the past few years, notable scientific progress has been made in understanding its molecular and neurobiological underpinnings Bu et al. (2025); Chourpiliadis et al. (2024); Fries et al. (2023). Despite these advances, definitive biological markers and validated diagnostic tests are still lacking Lakhan et al. (2010). As a result, clinical trials for depression continues to rely on structured interviews and longitudinal assessments where clinicians are required to interpret behavioral symptoms Stuart et al. (2014).

The quality of these assessments is particularly critical for trial outcomes, as it can make the difference between a failed study and one in which the target drug separates from placebo Kobak et al. (2005). To improve consistency and reliability across sites and clinicians, symptom severity is quantified using validated rating scales, such as the Montgomery–Åsberg Depression Rating Scale (MADRS) Montgomery and Åsberg (1979), and interviews are conducted under standardized administration guidelines, such as the Structured Interview Guide for the MADRS (SIGMA) Williams and Kobak (2008). Nevertheless, the rating remains dependent on clinician expertise and judgment, often introducing variability that can affect trial outcomes Lipsitz et al. (2004). This challenge makes clinical trial interviews a particularly attractive setting for automated depression analysis.

Recent years have seen growing interest in automated methods for depression detection, with the aim of providing confidential and rapid triage for patients while supporting clinicians in their assessments Fisher et al. (2026); Zhang et al. (2026); Milintsevich et al. (2023). Central to these efforts is the analysis of language Viduani et al. (2026), as language is a well‑established indicator of depression Trifu et al. (2024); Segrin (1990) and clinical interviews naturally unfold through speech. Accordingly, existing research has focused on increasingly sophisticated models that infer depression from text Liu et al. (2025). However, much of this research has been conducted on online or non-clinical text, with comparatively little attention devoted to standardized clinical interviews used in depression trials Hengle et al. (2024); Squires et al. (2023); Ji et al. (2022); Fröhlich et al. (2018). Furthermore, researchers have primarily concentrated on binary classification model Zhang et al. (2025); Bader et al. (2024), offering limited interpretability into individual symptoms that characterize depressive disorders.

Our original contributions.

In this work, we address these gaps by developing a large language model (LLM) pipeline for supporting the assessment of depression in patients enrolled in clinical trials. Inspired by the structure of MADRS, we introduce the MADRS Pipeline, which orchestrates different LLMs to evaluates depressive symptoms directly from clinician–patient interview recordings. The pipeline first transcribes interview audio into text and segments the transcript according to the symptom items of MADRS. Then, for each item, it estimates symptom severity to support clinical rating. Finally, it compares these estimations against ratings provided by human clinicians and identifies instances where the human ratings may not fully comply with MADRS.

We conduct extensive experiments across different LLMs to evaluate the performance of the complete pipeline. We validate it on real-world clinical data, comprising approximately 16,000 expert-rated MADRS instances. The results demonstrate strong agreement with expert assessments, achieving Spearman correlation of 0.867 on total MADRS scores.

2 Background and Related Work

Given the interdisciplinary nature of this work, we first summarize the clinical foundations underlying depression assessment and then review computational approaches to depression detection in NLP.

Clinical foundations

MADRS is one of the most widely used clinician-administered instruments for assessing depression severity in both clinical practice and drug‑development trials Montgomery and Åsberg (1979). The scale consists of ten symptom items,¹¹1Apparent Sadness, Reported Sadness, Inner Tension, Reduced Sleep, Reduced Appetite, Concentration Difficulties, Lassitude, Inability to Feel, Pessimistic Thoughts, and Suicidal Thoughts each rated on a severity scale from 0 to 6.

Although well‑validated and sensitive to symptomatic change over time, the original MADRS offers no guidance on how clinicians should elicit information, leaving interview quality dependent on individual interviewing style and expertise. To reduce assessment variability across raters and study sites, structured administration protocols such as SIGMA were developed Williams and Kobak (2008). SIGMA standardizes interview administration by providing scripted questions and recommended follow-up probes for each symptom item. These guidelines ensure that clinicians systematically collect the information required for reliable symptom assessment and consistent scoring.

Refer to caption — Figure 1: Overview of the MADRS Pipeline. The MADRS Pipeline orchestrates four components (1) interview transcription, (2) transcript segmentation, (3) MADRS assessment, and (4) quality assessment.

Computational approaches

Research on depression detection spans multiple modalities, including audio‑based analyses Sardari et al. (2022), video‑derived behavioral markers Mahayossanunt et al. (2023), and multimodal systems Shangguan et al. (2022). Despite this advancements, text remains the most widely used data source due to data privacy concerns (§ 6), and the central role of language in clinical assessment.

State‑of‑the‑art approaches rely on deep learning Squires et al. (2023). Earlier work primarily uses CNNs and LSTMs to capture sentiment and semantic signals of depression Amanat et al. (2022). More recent studies leverage LLMs for their unprecedented, natural-language capabilities Rinaldi et al. (2020); Liu et al. (2025); Viduani et al. (2026).

Prompt engineering is sometimes used to process interview transcripts as monolithic units Raganato et al. (2024); however, this approach requires long prompts and suffers from context overload, a problem referred to as “lost-in-the-middle” Liu et al. (2024). Other approaches instead relies on multi-step prompting Kebe et al. (2026); Liu et al. (2025) to decompose interviews into fine-grained units—e.g., symptom-specific text segments Rinaldi et al. (2020), thematic blocks Zhao et al. (2025), and clinician–patient dialogue turns Lee et al. (2026)— and then analyze each unit independently.

Our work builds upon these recent efforts. Similar to Vail et al. (2026), we employ a LLM orchestration that leverages a mixture-of-models to solve an end-to-end clinical assessment: unlike preliminary experiments relying on manually prepared transcripts Raganato et al. (2024), MADRS Pipeline directly processes audio recordings, performs speaker-aware transcription, and identifies symptom-specific evidence. In contrast to Vail et al. (2026), MADRS Pipeline is specifically designed to support standard MADRS scoring.

Similar to Weber et al. (2025), MADRS Pipeline employs a shared encoder with 9 item-specific heads for MADRS assessment. However, they focus on a subset of MADRS items, using real and synthetic German data. In contrast, MADRS Pipeline is trained only on real English clinical interviews and predicts all 10 MADRS items.

Kebe et al. (2026) represents another relevant comparison. However, their end-to-end reliance on larger-scale models (e.g., 70B parameters) introduces substantial computational costs in terms of both time and resources. In contrast, by leveraging different models, MADRS Pipeline provides a practical trade-off between clinical utility, computational efficiency, and scalability.

Finally, to the best of our knowledge, MADRS Pipeline is the first computational solution specifically designed for depression clinical trials following SIGMA administration guidelines. Beyond supporting symptom severity estimation, MADRS Pipeline enables quality monitoring by assessing the reliability of clinical ratings and identifying potentially non-compliant assessments.

Component	Data split	Interviews	Utterances	Annotations	# Annotations
Interview transcription	Test	8	2,642	Transcripts	2,642
Transcript segmentation	Test	14	3,970	10 SIGMA sections	$3,970\times 10$
MADRS assessment	Train	1100	371,715	10 MADRS scores (0–6)	$1100\times 10$
	Dev	251	87,364		$251\times 10$
	Test	251	86,789		$251\times 10$
Quality assessment	Train	305	119,017	$\{$ Compliant, Non-compliant $\}$	$305\times 2$
	Test	305	117,324	$\{$ Compliant, Non-compliant $\}$	$305\times 2$

Table 1: Datasets: for each component and data split, we report the number of interviews (duration: ~40 min), the total number of utterances across interviews, the annotation format, and the total number of annotations.

3 The MADRS Pipeline

Let $a$ denote the audio recording of a clinician–patient interview conducted within a depression clinical trial according to the SIGMA guidelines. The problem addressed by the MADRS Pipeline is to support the depression assessment of the patient across the ten symptom items of the MADRS scale, denoted by $\mathcal{M}={m_{1},\ldots,m_{10}}$ .

The MADRS Pipeline is designed as a computational clinician grounded in standardized clinical interviewing procedures. It is implemented as a directed acyclic graph comprising an orchestration of four sequential components, where each stage operates on the outputs of preceding stages. The orchestration consists of: (1) interview transcription, (2) transcript segmentation, (3) MADRS assessment, and (4) quality assessment. An overview of the proposed framework is shown in Figure 1.

Interview transcription

This component coverts the audio recording $a$ into a textual transcript $t$ . The resulting transcript contains a chronological sequence of clinician and patient utterances and serves as the input to subsequent processing stages.

To generate transcripts, we follow an established pipeline used in related research Deonise et al. (2026); Desai et al. (2024); Weber et al. (2025). Specifically, we employ an open-source deep learning toolkit (i.e., pyannote.audio 4.0; Bredin et al., 2020) for voice activity detection and speaker diarization, combined with an automatic speech recognition (ASR) system (i.e., Whisper Medium; Radford et al., 2023; Desai et al., 2024) to transcribe the audio recordings.

Transcript segmentation

Clinical interviews are organized around the MADRS items $\mathcal{M}$ . This component decompose a transcript $t$ into 10 symptom-specific segments $S={s_{1},\ldots,s_{10}}$ , where each segment $s_{i}$ contains the interview utterances relevant to the assessment of the corresponding MADRS item $m_{i}$ . The resulting representation aligns the interview with the structure of the clinical instrument, enabling subsequent components to operate on localized symptom-specific evidence rather than the transcript as a whole.

To segment transcripts, we employ a LLM (i.e., GPT-4.1) guided by the corresponding SIGMA administration guidelines. For each MADRS item, the model receives the complete transcript together with the item-specific SIGMA description and it is instructed to identify the utterances relevant to the target symptom using a zero-shot prompting strategy. This procedure is repeated independently for all ten MADRS items (Appendix A.2).

MADRS assessment

This is the core component of MADRS Pipeline, responsible for estimating the severity of each MADRS symptom. For each symptom item $m_{i}$ , the component analyzes the corresponding segment $s_{i}$ and predicts a severity rating $r_{i}\in\{0,\ldots,6\}$ , where 0 indicates the absence of the symptom and 6 corresponds to the highest level of severity defined by the MADRS guidelines.

To estimate symptom severity, we employ a Transformer encoder model fine-tuned for multi-task ordinal regression Baly et al. (2019); Li and Lin (2007). The model processes each segment $s_{i}$ independently and predicts the corresponding MADRS ordinal rating. Specifically, a shared encoder first maps the input text into a latent representation, which is subsequently processed by item-specific prediction heads corresponding to the ten MADRS symptoms. Each head models the ordinal structure of MADRS ratings through a sequence of threshold decisions, enabling the prediction of scores on the discrete 0–6 scale while explicitly accounting for their ordered nature.

Quality assessment

In clinical trials, reviewer teams routinely monitor clinical raters to ensure their adherence to MADRS scoring guidelines. This component is designed to support this process by detecting interviews where the rater assessment is not consistent with expected clinical practice.

Given the MADRS scores ${\hat{r}_{1},\ldots,\hat{r}_{10}}$ assigned by the clinical rater as additional input, this component compares them with the reference scores ${r_{1},\ldots,r_{10}}$ . The reference scores can either be generated directly by the MADRS Pipeline (when a review by the reviewer team is not available), or provided/refined by the reviewer team following their assessment. The component then predicts a binary quality label, $q\in\{$ Compliant, Non-compliant $\}$ , indicating whether the rater scores are considered reliable or potentially problematic.

To estimate interview compliance, we train a Random Forest binary classifier using the absolute differences, $|r_{i}-\hat{r}_{i}|$ for $i\in\{1,\ldots,10\}$ , as input features. As additional features, we include the total and the mean of the absolute differences.

4 Experimental setup

We evaluate MADRS Pipeline on a collection of interviews annotated with MADRS scores from both clinical raters and reviewer team at the participating clinical sites. Since interview transcription and transcript segmentation constitute essential pre-processing steps for the subsequent clinical assessments, we conduct targeted validation analyses to verify the expected quality-level of the speech recognition pipeline.²²2Larger-scale evaluation of these components would require extensive review of interview audio recordings and the generation of additional ground-truth labels, an effort which is outside the scope of this work.

4.1 Dataset

Our dataset consists of 1,602 interviews administered according to SIGMA. Each interview contains approximately 40 minutes of audio. Trained raters in private clinics conducted real-time interviews and assigned 10 MADRS scores during each assessment. A reviewer team evaluated the audio recordings of these interviews and assigned an additional set of MADRS scores to assess scoring consistency. They then labeled a subset of the interviews as compliant or non-compliant based on discrepancies between the two sets of MADRS scores. Ground-truth collection and clinical training procedures were defined by the study protocol and are thus outside the scope of this work (Appendix E).

We randomly selected two subsets of interviews to assess interview transcription and transcript segmentation. For interview transcription, we manually transcribed 8 interviews (2,642 utterances) to serve as the reference standard. For transcript segmentation, we annotated 14 interviews (3,970 utterances) by assigning each utterance to one of the 10 SIGMA interview sections, each corresponding to a MADRS item.

For madrs assessment, we randomly partitioned interviews into Train, Dev, and Test sets using an 80/10/10 split³³3No patient appears in more than one set. and considered ground truth provided by the reviewer team. For quality assessment, the subset annotated with quality labels was smaller and exhibited class imbalance. We therefore conducted 10-fold cross-validation using random 50/50 Train/Test splits while preserving the class distribution. A summary of our dataset is available in Table 1.

4.2 Task setting

Interview transcription

We conduct a quality assessment by computing the similarity between the automatically generated transcripts and the manually curated references. First, we use the temporal Dice metric to align utterances across the two transcript versions. Then, we quantify lexical similarity using the average Jaccard index over the aligned utterances and semantic similarity using the average cosine similarity between sentence embeddings⁴⁴4sentence-transformers/all-MiniLM-L6-v2 of the aligned utterances (Appendix A).

Transcript segmentation

We evaluate this component as a multi-class classification task, where the objective is to assign each utterance to one of the 10 SIGMA sections. Performance is measured using one-vs-all F1-score for each SIGMA section and weighted F1-score as an overall metric.

We compare one open-weight LLM, Llama 3.3 70B, with two closed-source LLMs, GPT-4.1 and Claude Sonnet 4.5. These models were selected because their large context windows (128K, 1M, and 200K tokens, respectively) enable processing of long clinical transcripts. To minimize variability due to the non-deterministic nature of LLMs, decoding is performed with temperature set to 0. Each experiment is repeated three times, and the average result across runs is considered.

MADRS assessment

We evaluate this component as a ranking task by comparing our prediction against the scores of the reviewer team at both the item and total score level. Performance is measured using complementary metrics: Spearman correlation, Mean Absolute Error (MAE), and Accuracy@1. Spearman correlation assesses ranking quality. MAE measures the absolute error in predicted overall MADRS scores. Accuracy1 reports the proportion of predictions within a $\pm 1$ point tolerance of the expert ratings.

We compare fine-tuned encoders across three settings: (i) pretraining, (ii) architecture, and (iii) training loss. For (i), we compare general-purpose and clinically adapted encoders jointly fine-tuned across all items using an ordinal regression loss. For (ii), we compare the best-performing model from (i) with independent models fine-tuned separately for each MADRS item. For (iii), we compare the same best-performing model under ordinal regression and categorical classification loss (Appendix B). All models are trained for up to 10 epochs with early stopping based on validation item-level MAE. Specifically, we evaluate RoBERTa Zhuang et al. (2021), ModernBERT Warner et al. (2025), MentalBERT Ji et al. (2022), ClinicalBERT Huang et al. (2020), and PubMedBERT Gu et al. (2021).

Depression datasets are typically confidential and therefore not publicly available (§ 6). As a result, direct comparison with state-of-the-art results is not feasible. To provide a relevant baseline with state-of-the-art, we additionally evaluate decoder LLMs used for transcript segmentation, following the zero-shot LLAMADRS prompting strategy proposed by Kebe et al. (2026).

Quality assessment

We evaluate this component as a binary classification task by comparing our predictions against the quality labels of the reviewer team. Given the imbalanced nature of the data and the importance of negative examples in this screening setting (§ Limitations), we use the Matthews Correlation Coefficient (MCC) as metric and report True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN).

We evaluate MADRS Pipeline both when the reference scores ${r_{1},\ldots,r_{10}}$ are provided by the reviewer team, and when they are directly generated by the top-performing encoder and decoder from MADRS assessment.

When reference scores are directly generated⁵⁵5To prevent information leakage, MADRS reference scores were generated using split-specific fine-tuned models that excluded the corresponding quality assessment interviews (see Appendix C)., we account for the fact that MADRS predictions are not error-free. For each MADRS item, we thus investigate the use of a noise quantile threshold estimated from the differences between the generated scores and the scores provided by the reviewer team. We then use this threshold when comparing the rater scores with the generated scores, so that only disagreements larger than the expected model error are counted. This reduces the risk of attributing model error to the clinical rater (Appendix C).

To better contextualize task difficulty, we include simple baselines, namely Random, All-1s, and All-0s. Additionally, we report performance using a simple threshold classifier tuned on the total gap (sum of the item differences) over the Train set.

4.3 Evaluation results

Interview Transcription

Our validation suggests strong alignment at the utterance level, reaching an average temporal Dice score of 0.852 between generated and manually curated utterances. On the aligned utterances, we observe a high similarity, with average Jaccard index of 0.749 and a average embedding similarity of 0.814. This analysis suggests substantial overlap in both wording and meaning, supporting the use of generated transcripts for the subsequent components.

Transcript segmentation

The overall results across three independent evaluation runs are reported in Table 2. Our results show that closed-source models outperform the open-weight model, with GPT-4.1 achieving slightly higher performance and therefore being selected as the reference model. Figure 2 reports the performance of GPT-4.1 across the individual MADRS items. Overall, the segmentation module achieved consistently strong performance, with one-vs-all F1 scores exceeding 0.94 for all MADRS items and an overall weighted F1 above 0.98. Performance was highly stable across runs (std from 0.005 to 0.01), as evidenced by the close clustering of individual run-level scores around their corresponding means, indicating that the segmentation procedure is robust to residual variability in LLM inference. The high performance suggests that symptom-specific segmentation is a well-structured task in standardized SIGMA interviews. The predefined interview structure creates substantial lexical and semantic overlap between the symptom descriptions supplied in the prompt and the corresponding interview content (Appendix A.2). This redundancy may simplify symptom identification, aligning with recent evidence that repeated contextual information can improve the performance of non-reasoning LLMs Leviathan et al. (2025).

weigh. F1 score	Run1	Run2	Run3	Avg.
GPT-4.1	0.984	0.981	0.984	0.983
Llama 3.3 70B	0.654	0.609	0.582	0.615
Claude Sonnet 4.5	0.978	0.977	0.978	0.978

Table 2: Weighted F1 for transcript segmentation.

Model	Setting	MAE	Accuracy@1	Spearman
RoBERTa	(i-iii) - general purpose	2.956	0.895	0.867
RoBERTa - Independent fine-tuning	(ii) - general purpose	3.493	0.860	0.800
RoBERTa - Categorical multiclass loss	(iii) - general purpose	3.039	0.900	0.850
ModernBERT	(i) - general purpose	4.200	0.842	0.716
MentalBERT	(i) - clinically adapted	3.288	0.879	0.842
PubMedBERT	(i) - clinically adapted	4.161	0.836	0.756
ClinicalBERT	(i) - clinically adapted	4.322	0.832	0.754
Baselines: Kebe et al. (2026)	Prompt-engineering	MAE	Accuracy@1	Spearman
GPT-4.1	LLAMADRS	4.044	0.813	0.849
Claude Sonnet 4.5	LLAMADRS	4.080	0.814	0.848
Llama 3.3 70B	LLAMADRS	4.133	0.811	0.851

Table 3: Summary performance for MADRS assessment: MAE and Spearman are computed over total MADRS scores. For Accuracy@1, we report the average result per item, as the metric would be too aggressive at the total level. For (i), encoders are all jointly fine-tuned across all items using an ordinal regression loss.

MADRS Assessment

Table 3 summarizes the results of our evaluation considering the total MADRS scores. Overall, RoBERTa achieves the best performance, with an MAE of 2.956, an avg. Accuracy@1 of 0.895, and a Spearman correlation of 0.867. Baseline LLMADRS also achieve strong performance, although it consistently underperform the fine-tuned encoder models.

Figure 3 reports the correlation for individual items for fine-tuned RoBERTa and a decoder baselines, GPT-4.1. Specifically, we observe that performance varies across items, and that GPT-4.1 outperform RoBERTa on most items. RoBERTa achieves higher correlations than GPT-4.1 only for Apparent Sadness and Suicidal Thoughts, with the largest gap observed for the latter. We hypothesize that the sensitive nature of the item may interact with safety guardrails Dong et al. (2025), potentially leading to more conservative predictions and reduced performance. Despite weaker per-item correlations, we shown in Table 8 and 7, that RoBERTa outperforms all baselines in terms of MAE and Accuracy@1. This translates into higher performance on the total MADRS score, suggesting the aggregation of item predictions can mitigate the overall measurement noise.

Figure 4 and Table 3 summarize our evaluation over pretraining (i), architecture (ii), and loss (iii). For pretraining (i), the performance differences are relatively small; However, in line with recent observations Vishwanath et al. (2026), general-purpose RoBERTa achieves better point estimates than its clinical counterparts. Among the evaluated models, RoBERTa achieves the strongest correlation (Figure 5) and performance (Appendix D) and is therefore selected for the experiments (ii) and (iii).

Considering (ii), shared fine-tuning consistently outperforms independent fine-tuning. This suggests that a shared model benefits from common patterns across MADRS items. For example, negative responses to key symptom questions (e.g., “No” for symptom presence) are informative across multiple items, as they are consistently associated with lower scores. Similarly, recurring severity anchors such as normal versus reduced provide consistent signals that generalize across different symptoms.

Considering (iii), we observe that the categorical loss yields nearly identical performance, while the ordinal loss achieves slightly higher Spearman correlation on the total score.

Quality assessment

We report in Figure 6 the results of our evaluation while varying the quantile threshold used to discount expected prediction error when generated scores are used as reference scores. Overall, lower levels of discounting lead to higher MCC, with the best performance obtained when no discount is applied. Performance progressively decreases as the model becomes more tolerant to prediction errors.

Table 4 reports the result of our evaluation when no discount is applied. The highest performance is achieved when using MADRS scores provided by the reviewer team as reference scores, reaching an MCC of 0.704. When using MADRS scores automatically generated by MADRS Pipeline as reference scores, performance remains above the baselines but decreases due to error propagation from the automatic predictions. In particular, we obtain a MCC of 0.123 when using predictions generated by fine-tuned RoBERTa, and a MCC of 0.061 when using predictions generated by GPT-4.1. This suggests that, although reviewer scores remain the preferred option, RoBERTa scores can still provide a useful signal for identifying non-compliant ratings. The superior performance of RoBERTa is consistent with its higher Accuracy@1 on MADRS assessment.

Compared with the thresholding baseline approach, the Random Forest classifier achieves comparable performance when reviewer scores are used as references (MCC 0.704 vs. 0.713). However, the comparison differs for RoBERTa and GPT-4.1. With the more accurate RoBERTa predictions, the Random Forest consistently outperforms the baseline, suggesting that the quality label $q$ depends on structured patterns of item-level discrepancies rather than only on their overall magnitude. In contrast, when GPT-4.1 predictions are used, the threshold baseline performs better than the Random Forest. For GPT-4.1, the confidence intervals are substantially wider for both methods, reflecting the higher uncertainty introduced by the lower accuracy of the underlying MADRS predictions.

Ran. For.	MCC	TN	TP	FP	FN
reviewer	0.704 $\pm$ 0.1	24.1 $\pm$ 1.0	263.0 $\pm$ 3.9	5.9 $\pm$ 1.0	12.0 $\pm$ 3.9
RoBERTa	0.123 $\pm$ 0.0	8.2 $\pm$ 2.8	239.4 $\pm$ 12.0	21.8 $\pm$ 2.8	35.6 $\pm$ 12.0
GPT-4.1	0.049 $\pm$ 0.1	7.6 $\pm$ 3.1	223.5 $\pm$ 12.7	22.4 $\pm$ 3.1	51.5 $\pm$ 12.7
Thresh.	MCC	TN	TP	FP	FN
reviewer	0.713 $\pm$ 0.1	24.4 $\pm$ 1.3	263.5 $\pm$ 3.3	5.6 $\pm$ 1.3	11.7 $\pm$ 3.3
RoBERTa	0.098 $\pm$ 0.0	8.6 $\pm$ 6.0	120.2 $\pm$ 49.3	21.4 $\pm$ 6.0	154.8 $\pm$ 49.3
GPT-4.1	0.106 $\pm$ 0.0	9.2 $\pm$ 3.1	133.1 $\pm$ 14.9	20.8 $\pm$ 3.1	141.9 $\pm$ 14.9
Baselines	MCC	TN	TP	FP	FN
Random	0.03	–	–	–	–
All-1s	0	30	0	0	275
All-0s	0	0	275	30	0

Table 4: MCC for quality assessment. Results are reported as mean

\pm

standard dev. across 10 folds.

5 Conclusion

In this work, we introduced MADRS Pipeline, an end-to-end LLM pipeline that supports the assessment of depression in clinical trials. MADRS Pipeline is specifically designed for clinical interviews in which the interviewing clinician assesses depression using the MADRS scale and follows the SIGMA guidelines.

MADRS Pipeline orchestrates four components: interview transcription, which converts clinical interview audio into structured speaker-aware transcripts; transcript segmentation, which aligns utterances with the ten MADRS symptom items using structured SIGMA guidelines; MADRS assessment, which estimates depression severity for each MADRS item; and quality assessment, which identifies problematic clinical ratings.

We evaluated each component separately using real-world clinical trial data. First, we performed sanity checks to ensure the reliability of the generated structured representations of clinical interviews. We then evaluated the MADRS assessment and quality assessment components. Our results show strong agreement with expert reviewer ratings, reaching a Spearman correlation of 0.867 on total MADRS scores. Furthermore, we show that MADRS Pipeline can identify potentially non-compliant ratings using either generated MADRS predictions or reviewer-provided scores, although reviewer scores remain the preferred option.

This is consistent with the intended role of MADRS Pipeline as a supportive tool for clinicians rather than a replacement for clinical judgment or decision-making. Overall, MADRS Pipeline represents a step toward more consistent, scalable, and interpretable psychiatric assessment workflows by supporting clinicians in the evaluation of depression severity and rating quality.

Limitations

While the proposed MADRS Pipeline demonstrates promising results for automated MADRS assessment, the following limitations should be considered when interpreting the findings.

Modality mismatch

Human clinicians typically assess depression severity using in-person interviews conducted in real time, or audio and video recordings. In contrast, the MADRS Pipeline operates exclusively on textual transcripts for MADRS and quality assessment.

On the one hand, this design choice is motivated by practical considerations, including privacy, confidentiality, and data-sharing constraints that frequently arise in clinical research settings. On the other hand, restricting the analysis to transcripts removes access to paralinguistic and behavioral signals, such as prosody, intonation, pauses, facial expressions, and body language, which may contribute to clinical assessments.

Consequently, there is an inherent modality mismatch between human and automated evaluations. Although the proposed framework achieves substantial agreement with expert reviewers, its performance should be interpreted in light of this discrepancy. Future work could investigate multimodal approaches that integrate acoustic and visual information alongside textual evidence.

Transcription quality

The MADRS Pipeline relies on automatically generated transcripts, making it inherently sensitive to transcription errors. Although modern ASR systems generally achieve high accuracy and our evaluation indicates strong transcription quality, errors may still arise due to poor audio conditions, overlapping speech, speaker diarization failures, or model hallucinations. In clinical interviews, even subtle transcription inaccuracies can alter symptom descriptions and affect downstream assessments.

While aggregating predictions across multiple MADRS items may reduce the impact of isolated errors, transcription quality remains a fundamental dependency of the overall pipeline. Improvements in downstream modeling cannot fully compensate for missing or incorrectly transcribed information. Consequently, the reported results should be interpreted relative to the quality of the initial recordings and subsequent transcripts.

Imbalanced data in quality assessment

The dataset used for quality assessment is highly imbalanced. While this poses challenges for model training and evaluation, the observed class distribution is representative of real-world screening conditions. Clinical raters are trained to administer assessments in a standardized and consistent manner, which contributes to the relatively low prevalence of negative cases.

Constructing a reliable ground truth requires the sampling and independent validation of a large number of interviews in order to identify a relatively small number of erroneous instances, making the annotation process both costly and time-consuming. In our dataset, only 60 out of 610 interviews (approximately 407 hours of audio) are labeled as problematic according to MADRS-based scoring.

Despite this imbalance, the setting is of practical importance, as automated quality assessment has the potential to substantially support and scale human expertise in clinical trial environments. As a first work based on real-world data, our work contributes toward the development of supportive tools for clinical trials. As a future direction, we envision a human-in-the-loop framework in which the dataset can be progressively expanded through additional expert annotations, and model performance iteratively improved through feedback-driven refinement.

6 Ethical Considerations

This work involves the analysis of sensitive mental health data collected as part of clinical trials. All interview recordings and associated annotations were de-identified prior to analysis in accordance with the data governance procedures of the sponsoring studies. No personally identifiable information was used during model development or evaluation.

To further reduce privacy risks, the proposed framework operates primarily on textual transcripts rather than raw audio or video recordings. While this design choice introduces a modality mismatch with human assessments, it limits exposure to potentially identifiable acoustic and visual information, thereby facilitating privacy-preserving analysis of clinical interviews.

The proposed MADRS Pipeline is intended as a decision-support tool and not as a replacement for trained clinical raters. Automated assessments of depression severity may be affected by transcription errors, model biases, and ambiguity inherent to psychiatric evaluation. Consequently, system outputs should be reviewed by qualified professionals and should not be used as the sole basis for clinical or treatment decisions.

Due to confidentiality agreements and regulatory restrictions governing clinical trial data, neither the underlying datasets nor the trained models can be publicly released. To support transparency and reproducibility, we release model configurations, prompting templates, and synthetic examples that replicate the structure of the evaluation pipeline without exposing confidential material.

Acknowledgments

This work was supported by the NeuRev project, funded by Clario, part of Thermo Fisher Scientific. AI coding tools were used to support software development; all generated code suggestions were reviewed, adapted, validated, and integrated by the authors, who remain fully responsible for the final implementation. We thank Matthew Agard, and Rachel Alexander for their constructive discussions, brainstorming, feedback, and support throughout this project. We also thank Milos Ivankovic and Cuong Lai for developing the backend and user interface of the pipeline, which eventually led our scientific research toward a usable product; and Volodymyr Pozniak for facilitating access to the necessary infrastructure. Finally, we are grateful to Barbara Echevarria and Mark Opler for their leadership of the Clinical Science team and their efforts in curating and collecting the foundational data that supported this work.

References

A. Amanat, M. Rizwan, A. R. Javed, M. Abdelhaq, R. Alsaqour, S. Pandya, and M. Uddin (2022) Deep learning for depression detection from textual data. Electronics 11 (5). External Links: Link, ISSN 2079-9292, Document Cited by: §2.
M. Bader, M. Abdelwanis, M. Maalouf, and H. F. Jelinek (2024) Detecting Depression Severity Using Weighted Random Forest and Oxidative Stress Biomarkers. Scientific Reports 14 (1), pp. 16328. External Links: Document, Link, ISSN 2045-2322 Cited by: §1.
R. Baly, G. Karadzhov, A. Saleh, J. Glass, and P. Nakov (2019) Multi-Task Ordinal Regression for Jointly Predicting the Trustworthiness and the Leading Political Ideology of News Media. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio (Eds.), Minneapolis, Minnesota, pp. 2109–2116. External Links: Link, Document Cited by: §3.
H. Bredin, R. Yin, J. M. Coria, G. Gelly, P. Korshunov, M. Lavechin, D. Fustes, H. Titeux, W. Bouaziz, and M. Gill (2020) Pyannote.audio: neural building blocks for speaker diarization. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vol. , pp. 7124–7128. External Links: Document Cited by: §3.
T. Bu, J. Yang, J. Zhou, Y. Liu, K. Qiao, Y. Wang, J. Zhang, E. Zhao, B. K. Owura, X. Qiu, Z. Qiao, and Y. Yang (2025) LncRNA of peripheral blood mononuclear cells: hymai acts as a potential diagnostic and therapeutic biomarker for female major depressive disorder. Frontiers in Psychiatry Volume 16 - 2025. External Links: Link, Document, ISSN 1664-0640 Cited by: §1.
C. Chourpiliadis, Y. Zeng, A. Lovik, D. Wei, U. Valdimarsdóttir, H. Song, N. Hammar, and F. Fang (2024) Metabolic profile and long-term risk of depression, anxiety, and stress-related disorders. JAMA Network Open 7 (4), pp. e244525. External Links: Document Cited by: §1.
J. Cohen (1968) Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 70 (4), pp. 213–220. External Links: Document Cited by: Appendix E.
C. Deonise, T. Coconu, M. K. Z. Bajwa, C. Negru, B. Mocanu, A. Castiglione, and F. Pop (2026) Speaker-Attributed Meeting Transcription Refinement with Constrained Open-Weight Language Models. Future Generation Computer Systems 185, pp. 108648. External Links: ISSN 0167-739X, Document, Link Cited by: §3.
A. Desai, N. V. J. K. Kartik, P. Gupta, Vinayak, A. T S, M. K. Vanahalli, and R. Rajendran (2024) Advancing speaker diarization with whisper speech recognition for different learning environments. In 2024 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE), Vol. , pp. 1–8. External Links: Document Cited by: §3.
Y. Dong, R. Mu, Y. Zhang, S. Sun, T. Zhang, C. Wu, G. Jin, Y. Qi, J. Hu, J. Meng, S. Bensalem, and X. Huang (2025) Safeguarding large language models: a survey. Artificial Intelligence Review 58 (12), pp. 382. External Links: Document, Link, ISSN 1573-7462 Cited by: §4.3.
H. Fisher, N. M. Jaffe, K. Pidvirny, A. O. Tierney, M. S. Vaidean, P. Dongre, and C. A. Webb (2026) Language-based detection of depression with machine learning: systematic review and meta-analysis. npj Digital Medicine. External Links: ISSN 2398-6352, Document, Link Cited by: §1.
G. R. Fries, V. A. Saldana, J. Finnstein, and T. Rein (2023) Molecular pathways of major depressive disorder converge on the synapse. Molecular Psychiatry 28 (1), pp. 284–297. External Links: Document, Link, ISSN 1476-5578 Cited by: §1.
H. Fröhlich, R. Balling, N. Beerenwinkel, O. Kohlbacher, S. Kumar, T. Lengauer, M. H. Maathuis, Y. Moreau, S. A. Murphy, T. M. Przytycka, M. Rebhan, H. Röst, A. Schuppert, M. Schwab, R. Spang, D. Stekhoven, J. Sun, A. Weber, D. Ziemek, and B. Zupan (2018) From hype to reality: data science enabling personalized medicine. BMC Medicine 16 (1), pp. 150. External Links: Document, Link, ISSN 1741-7015 Cited by: §1.
Y. Gu, R. Tinn, H. Cheng, M. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao, and H. Poon (2021) Domain-specific language model pretraining for biomedical natural language processing. ACM Trans. Comput. Healthcare 3 (1). External Links: Link, Document Cited by: §4.2.
A. Hengle, A. Kulkarni, S. D. Patankar, M. Chandrasekaran, S. D’silva, J. S. Jacob, and R. Gupta (2024) Still not quite there! evaluating large language models for comorbid mental health diagnosis. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, pp. 16698–16721. External Links: Link, Document Cited by: §1.
T. Hu and N. Collier (2024) Quantifying the Persona Effect in LLM Simulations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 10289–10307. External Links: Link, Document Cited by: §A.2.
K. Huang, J. Altosaar, and R. Ranganath (2020) ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission. In Proceedings of the ACM Conference on Health, Inference, and Learning (CHIL) Workshops, Note: Workshop 20W13 Cited by: §4.2.
S. Ji, T. Zhang, L. Ansari, J. Fu, P. Tiwari, and E. Cambria (2022) MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, J. Odijk, and S. Piperidis (Eds.), Marseille, France, pp. 7184–7190. External Links: Link Cited by: §1, §4.2.
G. Y. Kebe, J. M. Girard, E. Liebenthal, J. Baker, F. D. la Torre, and L. Morency (2026) LLAMADRS: evaluating open-source llms on real clinical interviews–to reason or not to reason?. External Links: 2501.03624, Link Cited by: Appendix B, Table 6, Table 7, Table 8, §2, §2, §4.2, Table 3.
K. A. Kobak, A. D. Feiger, and J. D. Lipsitz (2005) Interview quality and signal detection in clinical trials. American Journal of Psychiatry 162 (3), pp. 628. External Links: Document, Link Cited by: §1.
S. E. Lakhan, K. Vieira, and E. Hamlat (2010) Biomarkers in psychiatry: drawbacks and potential for misuse. International Archives of Medicine 3 (1), pp. 1. External Links: Document, Link, ISSN 1755-7682 Cited by: §1.
J. Lee, J. Han, and C. Woo (2026) Interpretable depression assessment using a large language model. PLOS Digital Health 5 (2), pp. 1–18. External Links: Document, Link Cited by: §2.
Y. Leviathan, M. Kalman, and Y. Matias (2025) Prompt Repetition Improves Non-Reasoning LLMs. External Links: 2512.14982, Link Cited by: §4.3.
L. Li and H. Lin (2007) Ordinal regression by extended binary classification. In Advances in Neural Information Processing Systems 19, B. Schölkopf, J. C. Platt, and T. Hofmann (Eds.), pp. 865–872. External Links: Document Cited by: §3.
J. Lipsitz, K. Kobak, A. Feiger, D. Sikich, G. Moroz, and A. Engelhard (2004) The Rater Applied Performance Scale: development and reliability. Psychiatry Research 127 (1), pp. 147–155. External Links: ISSN 0165-1781, Document, Link Cited by: Appendix E, §1.
J. M. Liu, M. Gao, S. Sabour, Z. Chen, M. Huang, and T. M. C. Lee (2025) Enhanced large language models for effective screening of depression and anxiety. Communications Medicine 5 (1), pp. 457. Note: Published: 2025-11-05 External Links: ISSN 2730-664X, Document, Link Cited by: §1, §2, §2.
N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang (2024) Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12, pp. 157–173. External Links: Link, Document Cited by: §2.
Y. Mahayossanunt, N. Nupairoj, S. Hemrungrojn, and P. Vateekul (2023) Explainable depression detection based on facial expression using lstm on attentional intermediate feature fusion with label smoothing. Sensors 23 (23). External Links: Link, ISSN 1424-8220, Document Cited by: §2.
K. Milintsevich, K. Sirts, and G. Dias (2023) Towards automatic text-based estimation of depression through symptom prediction. Brain Informatics 10 (1), pp. 4. Note: Published: 2023-02-13 External Links: ISSN 2198-4026, Document, Link Cited by: §1.
S. A. Montgomery and M. Åsberg (1979) A New Depression Scale Designed to be Sensitive to Change. British Journal of Psychiatry 134 (4), pp. 382–389. External Links: Document Cited by: §1, §2.
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever (2023) Robust Speech Recognition via Large-Scale Weak Supervision. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. Cited by: §3.
A. Raganato, F. Bartoli, C. Crocamo, D. Cavaleri, G. Carrà, G. Pasi, and M. Viviani (2024) Leveraging Prompt Engineering and Large Language Models for Automating MADRS Score Computation for Depression Severity Assessment. In Ital-IA 2024: 4th National Conference on Artificial Intelligence, organized by CINI, Naples, Italy, pp. . External Links: Link Cited by: §2, §2.
A. Rinaldi, J. Fox Tree, and S. Chaturvedi (2020) Predicting Depression in Screening Interviews from Latent Categorization of Interview Prompts. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (Eds.), Online, pp. 7–18. External Links: Link, Document Cited by: §2, §2.
S. Sardari, B. Nakisa, M. N. Rastgoo, and P. Eklund (2022) Audio based depression detection using Convolutional Autoencoder. Expert Systems with Applications 189, pp. 116076. External Links: ISSN 0957-4174, Document, Link Cited by: §2.
C. Segrin (1990) A meta‐analytic review of social skill deficits in depression. Communication Monographs 57 (4), pp. 292–308. External Links: Document, Link Cited by: §1.
Z. Shangguan, Z. Liu, G. Li, Q. Chen, Z. Ding, and B. Hu (2022) Dual-stream Multiple Instance Learning for depression detection with facial expression videos. IEEE Transactions on Neural Systems and Rehabilitation Engineering 31, pp. 554–563. External Links: Document Cited by: §2.
P. E. Shrout and J. L. Fleiss (1979) Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin 86 (2), pp. 420–428. External Links: Document Cited by: Appendix E.
C. Spearman (1904) The proof and measurement of association between two things. The American Journal of Psychology 15 (1), pp. 72–101. External Links: Document Cited by: Appendix E.
M. Squires, X. Tao, S. Elangovan, R. Gururajan, X. Zhou, U. R. Acharya, and Y. Li (2023) Deep learning and machine learning in Psychiatry: a survey of current progress in Depression detection, diagnosis and treatment. Brain Informatics 10 (1), pp. 10. External Links: Document, Link, ISSN 2198-4026 Cited by: §1, §2.
A. L. Stuart, J. A. Pasco, F. N. Jacka, S. L. Brennan, M. Berk, and L. J. Williams (2014) Comparison of self-report and structured clinical interview in the identification of depression. Comprehensive Psychiatry 55 (4), pp. 866–869. External Links: ISSN 0010-440X, Document, Link Cited by: §1.
R. N. Trifu, B. Nemeș, D. C. Herta, C. Bodea-Hategan, D. A. Talaș, and H. Coman (2024) Linguistic markers for major depressive disorder: a cross-sectional study using an automated procedure. Frontiers in Psychology Volume 15 - 2024. External Links: Link, Document, ISSN 1664-1078 Cited by: §1.
A. K. Vail, M. Cicconet, K. A. Doorn, R. Maroney, and M. Aafjes (2026) ADAPTS: agentic decomposition for automated protocol-agnostic tracking of symptoms. External Links: 2605.03212, Link Cited by: §2.
A. Viduani, C. Buchweitz, A. C. Ostermann, R. M. Araújo, and C. Kieling (2026) From linguistic analyses to large language models: a scoping review of methods used to investigate language features in depression research. Psychiatry Research 360, pp. 117064. External Links: ISSN 0165-1781, Document, Link Cited by: §1, §2.
K. Vishwanath, A. Alyakin, M. Ghosh, A. Hage, S. N. Neifert, C. Orillac, N. J. Mandelberg, H. A. Khan, J. V. Lee, J. J. Yao, W. R. Small, A. Varma, D. B. Hewitt, Y. Aphinyanaphongs, D. A. Alber, and E. K. Oermann (2026) General-purpose large language models outperform specialized clinical ai tools on medical benchmarks. Nature Medicine , pp. . External Links: Document, Link, ISSN 1546-170X Cited by: §4.3.
B. Warner, A. Chaffin, B. Clavié, O. Weller, O. Hallström, S. Taghadouini, A. Gallagher, R. Biswas, F. Ladhak, T. Aarsen, G. T. Adams, J. Howard, and I. Poli (2025) Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 2526–2547. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §4.2.
S. Weber, N. Deperrois, R. Heun, L. Frühschütz, A. Monn, S. Homan, A. Häfliger, E. Seifritz, T. Kowatsch, L. Jäger, K. Schultebraucks, S. Gershov, J. Mocellin, B. Kleim, S. Olbrich, and M. consortium (2025) Using a fine-tuned large language model for symptom-based depression evaluation. npj Digital Medicine 8 (1), pp. 598. External Links: ISSN 2398-6352, Document, Link Cited by: §2, §3.
A. H. Weinberger, M. Gbedemah, A. M. Martinez, D. Nash, S. Galea, and R. D. Goodwin (2018) Trends in depression prevalence in the usa from 2005 to 2015: widening disparities in vulnerable groups. Psychological Medicine 48 (8), pp. 1308–1315. External Links: Document Cited by: §1.
J. B. W. Williams and K. A. Kobak (2008) Development and reliability of a structured interview guide forthe montgomery-Åsberg depression rating scale (sigma). British Journal of Psychiatry 192 (1), pp. 52–58. External Links: Document Cited by: §1, §2.
S. Zhang, C. Zhang, and J. Zhang (2026) Text-Based Depression Estimation Using Machine Learning With Standard Labels: Systematic Review and Meta-Analysis. Journal of Medical Internet Research 28. External Links: Document, Link Cited by: §1.
X. Zhang, C. Li, W. Chen, J. Zheng, and F. Li (2025) Optimizing depression detection in clinical doctor-patient interviews using a multi-instance learning framework. Scientific Reports 15 (1), pp. 6637. External Links: ISSN 2045-2322, Document, Link Cited by: §1.
X. Zhao, Y. Lyu, D. Wang, and B. Tang (2025) Predicting Depression in Screening Interviews from Interactive Multi-Theme Collaboration. In Findings of the Association for Computational Linguistics: ACL 2025, W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 23025–23035. External Links: Link, Document, ISBN 979-8-89176-256-5 Cited by: §2.
L. Zhuang, L. Wayne, S. Ya, and Z. Jun (2021) A robustly optimized BERT pre-training approach with post-training. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, S. Li, M. Sun, Y. Liu, H. Wu, K. Liu, W. Che, S. He, and G. Rao (Eds.), Huhhot, China, pp. 1218–1227 (eng). External Links: Link Cited by: §4.2.

Appendix A Transcription

A.1 Transcription quality

We conduct a quality assessment by computing the similarity between the automatically generated transcripts and the manually curated references. For all possible pairs of manually transcribed utterances ${\hat{u}_{i}}$ , and automatically generated transcription utterances ${{u}_{i}}$ , temporal alignment was determined using the Dice coefficient, defined as

\mathrm{Dice}(u_{i},\hat{u}_{i})=\frac{2\lvert t(u_{i})\cap t(\hat{u}_{i})\rvert}{\lvert t(u_{i})\rvert+\lvert t(\hat{u}_{i})\rvert}

where ${{t}({u}_{i})}$ denotes the temporal span of an utterance ${{u}_{i}}$ . The optimal temporal matching is defined as the pairing that maximizes the Dice coefficient.

Lexical similarity was quantified using the average Jaccard index between utterances $u_{i}$ and ${\hat{u}_{i}}$ ; where Jaccard is defined as:

\mathrm{Jac.}(u_{i},\hat{u}_{i})=\frac{\lvert words(u_{i})\cap words(\hat{u}_{i})\rvert}{\lvert words(u_{i})\cup words(\hat{u}_{i})\rvert}\ .

Similarly, semantic similarity was quantified as the average cosine similarity between sentence embeddings derived from Sentence Transformers; where Cosine is defined as:

\cos(u_{i},\hat{u}_{i})=\frac{u_{i}\cdot\hat{u}_{i}}{\|u_{i}\|\,\hat{u}_{i}\|}\ .

A.2 Transcript segmentation

We adopt a zero-shot prompt engineering approach. Specifically, we design a structured template prompt (Figure 7) consisting of four parts concatenated in a fixed order (Figure 8, 9, 10, 11, and 12). For the first part, we follow prior work showing that explicit role assignment can improve LLM performance Hu and Collier (2024). The remaining parts are designed to enforce a structured representation of inputs, constrain the output format, and provide unambiguous instructions. These parts are then paired with both the full transcript of an interview and the corresponding MADRS guideline section under examination.

In practice, given the transcript of an interview, a LLM is prompted independently for each of the 10 SIGMA sections. The LLM predicts the first and last utterance indices corresponding to each section. We then reconstruct the final annotation by assigning all utterances between the predicted start and end indices to the respective SIGMA section, resulting in the final segmentation of the transcript.

Figure 7: Prompt template used for transcript segmentation.

Figure 8: Prompt section used for defining role and task.

Figure 9: Prompt section used for defining transcript format.

Figure 10: Prompt section used for defining guidelines format.

Figure 11: Prompt section used for defining response format.

Figure 12: Prompt section used for defining intructions.

Appendix B MADRS assessment

LLAMADRS

To compare with the state of the art, we evaluate decoder LLMs using the zero-shot LLAMADRS prompt proposed by (Kebe et al., 2026). For fair comparisons, we evaluate LLAMADRS on the same Test set used for encoders. Figure 13 reports the prompt used in our experiments, as originally defined in the referenced study.

Figure 13: LLAMADRS prompt template used for MADRS assessment.

Encoder fine-tuning

All encoders are fine-tuned on SIGMA segments using a Train/Dev/Test split, with no patient appearing in more than one set. For all experiments, the input text includes both clinician and patient turns from the extracted SIGMA segment. Specifically, we use this format: clinician: [utterance] patient: [utterance]. Each SIGMA segment is paired with the MADRS reviewer score for the assessed item in the same interview. Item scores are modeled on the standard MADRS ordinal scale $y\in\{0,\ldots,6\}$ .

For setting (i), we use a shared transformer encoder with item-specific prediction heads. Given a segment $s_{i}$ , the encoder produces a representation which is passed to the prediction head corresponding to MADRS item $m_{i}$ . The model outputs six ordinal-threshold logits $z_{k}$ , where $k\in\{0,\ldots,5\}$ . Each MADRS score is represented using six binary threshold targets:

t_{i,k}=\mathbf{1}\{y_{i}>k\}\ .

The logits are converted into threshold probabilities using the sigmoid activation function:

p_{i,k}=\sigma(z_{i,k})\ .

Let $B$ denote the batch size, and let $w_{i,k}$ denote the positive-class weight applied to threshold $k$ for MADRS item $m_{i}$ , used to account for class imbalance among the ordinal threshold targets. The ordinal objective is binary cross-entropy over the six thresholds:

	$\displaystyle\mathcal{L}_{o}=-\frac{1}{B}\sum_{n=1}^{B}\frac{1}{6}\sum_{k=0}^{5}\Big[$	$\displaystyle w_{i_{n},k}t_{n,k}\log p_{n,k}\ +$
		$\displaystyle(1-t_{n,k})\log(1-p_{n,k})\Big]\ .$

where $i_{n}$ denotes the MADRS item associated with sample $n$ . At inference time, the predicted score is obtained by summing the thresholds whose predicted probability is at least 0.5:

\hat{y}_{i}=\sum_{k=0}^{5}\mathbf{1}\{p_{i,k}\geq 0.5\}.

For setting (ii), independent models are fine-tuned with the same ordinal objective separately for each MADRS item. For setting (iii), we use the architecture of (i) but replace the ordinal objective with a seven-class cross-entropy loss:

\displaystyle\mathcal{L}_{\mathrm{ce}}=-\frac{1}{B}\sum_{n=1}^{B}\alpha_{i_{n},y_{n}}\log\frac{\exp z_{n,y_{n}}}{\sum_{c=0}^{6}\exp z_{n,c}},

where $\alpha_{i_{n},y_{n}}$ is the item-specific class weight.

For RoBERTa, MentalBERT, PubMedBERT, and ClinicalBERT, we use a maximum input length of 512 tokens. Longer item segments are processed with sliding windows using a stride of 128 tokens. If an item segment yields $M_{n}$ windows, the row representation is the mean of the first-token window embeddings:

h_{n}=\frac{1}{M_{n}}\sum_{m=1}^{M_{n}}H_{\theta}(x_{n,m})_{0}.

The resulting segment-level vector is then passed to the prediction head for the corresponding MADRS item. ModernBERT is fine-tuned with a 4096-token context window, so each segment can be processed without sliding-window aggregation. For each window or full segment, we use the hidden state of the first special token as the sequence representation: [CLS] for BERT-style tokenizers and <s> for RoBERTa-style tokenizers. We apply dropout with rate 0.1 to this representation before the item-specific prediction head.

Hyperparameter	Value
Optimizer	AdamW
Learning rate	$1\times 10^{-5}$
Weight decay	0.01
Warmup ratio	0.10
Maximum epochs	10
Early stopping patience	2 validation epochs
Gradient clipping	1.0
Dropout	0.1
Precision	FP32
Random seed	13
Checkpoint selection	Lowest validation item MAE
Training batch size	2 (1 for ModernBERT)
Gradient accumulation steps	2 (4 for ModernBERT)
Evaluation batch size	4 (2 for ModernBERT)
Effective training batch size	4

Table 5: Fine-tuning hyperparameters.

Appendix C Quality assessment

Encoder-generated reference scores

Directly using the encoder models trained for MADRS scoring would introduce information leakage: the MADRS ground-truth annotations of an interview used during RoBERTa training would subsequently influence the generation of reference scores for the same interview during quality assessment.

To prevent this issue, we repeat the fine-tuning excluding the interviews from the quality assessment subset. The resulting model is then used to generate MADRS predictions for those specific interviews.

This procedure ensures that every generated MADRS score used for quality assessment is produced by a model that has not observed the corresponding interview during training. Thus, the generated reference scores represent out-of-sample predictions rather than reconstructions influenced by exposure to the target interviews.

C.1 Error tolerance estimation

When the reference scores $\{r_{1},\ldots,r_{10}\}$ are generated by MADRS Pipeline, we account for the possibility that they may themselves contain item-level prediction errors. Consequently, we do not interpret every disagreement between a rater and the generated scores as evidence of poor rater quality. Instead, we estimate an item-specific error tolerance using the Train set.

Let $\{r_{1,g},\ldots,r_{10,g}\}$ , $\{r_{1,r},\ldots,r_{10,r}\}$ , and $\{\hat{r}_{1},\ldots,\hat{r}_{10}\}$ denote the MADRS item scores generated by MADRS Pipeline, assigned by the reviewer team, and assigned by the clinical rater, respectively.

For each MADRS item $i$ and instance $j$ in the Train set, we compute the empirical generated-score error as the absolute difference between the generated score $r_{i,g}^{(j)}$ and the corresponding reviewer score $r_{i,r}^{(j)}$ :

e_{i}^{(j)}=\left|r_{i,g}^{(j)}-r_{i,r}^{(j)}\right|.

The item-specific error tolerance for MADRS item $i$ is defined as the noise threshold $\tau_{i}$ , computed as the 80th percentile of the empirical generated-score errors for item $i$ across all Train-set instances:

\tau_{i}=Q_{0.8}\left(\left\{e_{i}^{(j)}\right\}_{j=1}^{N_{\mathrm{train}}}\right),

where $N_{\mathrm{train}}$ denotes the number of instances in the Train set. This threshold represents the expected prediction variability of the generated reference score for MADRS item $i$ .

For our Random Forest classifiers, we define the excess-gap feature for item $i$ as the absolute score difference between MADRS Pipeline and the clinical rater that exceeds the expected noise level:

d_{i}=\max\left(\left|r_{i,g}-\hat{r}_{i}\right|-\tau_{i},0\right),

where $r_{i,g}$ is the MADRS score generated by MADRS Pipeline, $\hat{r}_{i}$ is the score assigned by the clinical rater, and $\tau_{i}$ is the item-specific noise threshold estimated from the Train set. Thus, score differences within the expected error range of the generated reference score are removed, while only the excess component beyond this tolerance is retained.

This procedure prevents the classifier from attributing expected uncertainty in the generated reference scores to the clinical rater. The classifier is then trained using the item-level excess-gap features ${d_{i}}_{i=1}^{10}$ together with summary statistics, including the total excess gap and the mean excess gap across MADRS items:

D_{\mathrm{sum}}=\sum_{i=1}^{10}d_{i},

D_{\mathrm{mean}}=\frac{1}{10}\sum_{i=1}^{10}d_{i}.

C.2 Random forest parameters

We use the default parameters provided by scikit-learn for the Random Forest classifier, varying only the number of estimators, which is set to 500.

Appendix D Evaluation results

For comparison, we report in Table 6, Table 7, and Table 8 the performance of MADRS assessment for each MADRS item and the final MADRS total score, using Spearman correlation, MAE, and accuracy metrics, respectively.

	Baselines: LLAMADRS Kebe et al. (2026)			Encoders
MADRS Items	GPT-4.1	Claude Sonnet 4.5	Llama 3.3 70B	RoBERTa	ModernBERT	MentalBERT	ClinicalBERT	PubMedBERT
APPARENT SADNESS	0.644	0.641	0.648	0.660	0.534	0.650	0.575	0.522
REPORTED SADNESS	0.650	0.657	0.649	0.632	0.485	0.622	0.393	0.378
INNER TENSION	0.762	0.767	0.760	0.698	0.668	0.713	0.651	0.664
REDUCED SLEEP	0.788	0.795	0.789	0.624	0.453	0.608	0.485	0.501
REDUCED APPETITE	0.832	0.828	0.827	0.752	0.599	0.700	0.671	0.684
CONCENTRATION DIFFICULTIES	0.717	0.715	0.721	0.686	0.691	0.692	0.680	0.687
LASSITUDE	0.679	0.673	0.674	0.675	0.528	0.622	0.518	0.541
INABILITY TO FEEL	0.681	0.680	0.681	0.611	0.478	0.590	0.499	0.494
PESSIMISTIC THOUGHTS	0.709	0.713	0.712	0.694	0.655	0.716	0.666	0.688
SUICIDAL THOUGHTS	0.588	0.586	0.567	0.812	0.820	0.813	0.814	0.841
TOTAL_SCORE	0.849	0.848	0.851	0.867	0.716	0.842	0.754	0.756

Table 6: Spearman correlation: performance comparison for MADRS assessment.

	Baselines: LLAMADRS Kebe et al. (2026)			Encoders
MADRS Items	GPT-4.1	Claude Sonnet 4.5	Llama 3.3 70B	RoBERTa	ModernBERT	MentalBERT	ClinicalBERT	PubMedBERT
APPARENT SADNESS	0.878	0.888	0.889	0.562	0.700	0.645	0.660	0.665
REPORTED SADNESS	0.676	0.670	0.683	0.483	0.621	0.557	0.773	0.803
INNER TENSION	0.805	0.795	0.807	0.527	0.695	0.512	0.714	0.685
REDUCED SLEEP	0.829	0.811	0.833	0.778	1.197	0.862	1.113	1.089
REDUCED APPETITE	0.748	0.759	0.756	0.709	0.901	0.793	0.862	0.857
CONCENTRATION DIFFICULTIES	0.600	0.594	0.602	0.564	0.559	0.554	0.574	0.578
LASSITUDE	0.722	0.733	0.733	0.623	0.824	0.667	0.853	0.848
INABILITY TO FEEL	0.885	0.889	0.894	0.725	0.819	0.755	0.868	0.868
PESSIMISTIC THOUGHTS	0.860	0.852	0.859	0.659	0.717	0.639	0.766	0.678
SUICIDAL THOUGHTS	0.907	0.915	0.964	0.322	0.341	0.322	0.322	0.312
TOTAL_SCORE	4.044	4.080	4.133	2.956	4.200	3.288	4.322	4.161

Table 7: Mean Absolute Error: performance comparison for MADRS assessment. Item rows report item-level MAE; the final row reports MAE over summed total MADRS scores.

	Baselines: LLAMADRS Kebe et al. (2026)			Encoders
MADRS Items	GPT-4.1	Claude Sonnet 4.5	Llama 3.3 70B	RoBERTa	ModernBERT	MentalBERT	ClinicalBERT	PubMedBERT
APPARENT SADNESS	0.794	0.790	0.789	0.906	0.872	0.887	0.892	0.892
REPORTED SADNESS	0.855	0.859	0.855	0.946	0.906	0.916	0.857	0.857
INNER TENSION	0.828	0.829	0.826	0.941	0.872	0.946	0.862	0.872
REDUCED SLEEP	0.807	0.811	0.810	0.803	0.729	0.793	0.724	0.729
REDUCED APPETITE	0.842	0.837	0.837	0.867	0.754	0.823	0.764	0.754
CONCENTRATION DIFFICULTIES	0.857	0.863	0.859	0.892	0.868	0.873	0.853	0.858
LASSITUDE	0.852	0.847	0.847	0.902	0.824	0.897	0.814	0.799
INABILITY TO FEEL	0.785	0.784	0.782	0.882	0.828	0.828	0.804	0.814
PESSIMISTIC THOUGHTS	0.745	0.750	0.748	0.873	0.834	0.883	0.810	0.839
SUICIDAL THOUGHTS	0.766	0.764	0.753	0.941	0.932	0.941	0.937	0.941
TOTAL_SCORE	0.813	0.814	0.811	0.895	0.879	0.842	0.836	0.832

Table 8: Accuracy@1: performance comparison for MADRS assessment. Item rows report item-level within-one accuracy; the final row reports Acc@1 over summed total MADRS scores.

Appendix E Human clinical assessment

All assessments were obtained through real-time, interviews conducted by trained clinicians in private clinics outside the scope of this work. Following standard practices in clinical trials, interviews were subsequently reviewed and, where necessary, adjudicated by a senior clinician based on audio recordings of the original interviews Lipsitz et al. (2004). Inter-rater agreement between interviewing and audio-reviewing clinicians is reported in Table 9, computed using Spearman’s correlation Spearman (1904), Quadratic Weighted Kappa (QWK) Cohen (1968), and Intraclass Correlation Coefficient (ICC) Shrout and Fleiss (1979). Although the ratings are not strictly independent (the audio-reviewing clinician had access to the original interview recordings), high values across all measures indicate strong concordance between the original and adjudicated scores. A subset of these recordings was re-analyzed for quality assessment, where the difference between the initial rater and reviewer scores was used to define a binary quality label.

MADRS Items	DATASET
	QWK	ICC	Spearm.
APPARENT SADNESS	0.940	0.937	0.898
REPORTED SADNESS	0.944	0.944	0.912
INNER TENSION	0.946	0.946	0.919
REDUCED SLEEP	0.950	0.944	0.929
REDUCED APPETITE	0.958	0.958	0.947
CONC. DIFFICULTIES	0.955	0.955	0.933
LASSITUDE	0.943	0.943	0.914
INABILITY TO FEEL	0.918	0.914	0.866
PESSIMISTIC THOUGHTS	0.937	0.937	0.933
SUICIDAL THOUGHTS	0.895	0.895	0.904
TOTAL_SCORE	0.980	0.980	0.963

Table 9: Inter-rater agreement between the interviewing clinician and the audio-reviewing clinician across all studies.