NepMedJP: Cross-Lingual Medical Summarization for Japanese–Nepali

The problem

Nepali speakers living in Japan receive clinical documents written in Japanese. Turning those documents into something a patient can actually act on is a cross-lingual summarization problem — but it is one where the language pair has almost no parallel data, and where a fluent-sounding summary that drops or distorts a clinical fact is worse than no summary at all. Neither the data nor the evaluation methodology for this task existed.

NepMedJPBench

I constructed the first benchmark for this task: 100 parallel source–summary pairs with clinician-authored reference summaries, built for patient-facing rather than clinician-facing output.

A clinician-grounded evaluation rubric

Rather than assuming which qualities matter, I drew candidate attributes from four literatures — medical summarization, factuality, cross-lingual evaluation, and patient communication — then validated them through a clinician survey collecting both importance ratings and criteria the clinicians volunteered themselves. Two attributes specific to the cross-lingual setting emerged and are, to my knowledge, novel:

• Clinical meaning preservation — whether the clinical content survives the language boundary intact
• Medical terminology transfer — whether Japanese medical terminology is rendered into Nepali a patient can understand without losing precision

Gold-standard human judgments

I ran a multi-rater clinician evaluation across the full attribute set, establishing the human judgments the benchmark is scored against.

What the models do

I benchmarked task-specific and frontier systems to characterize how architecture affects behavior on this task:

• mT5 (encoder–decoder), both pretrained and fine-tuned
• Qwen3-4B (decoder-only), in thinking and non-thinking configurations
• Qwen3.5-4B (multimodal)
• Frontier LLMs

Meta-evaluating the evaluation

The part I find most interesting is not which model wins — it is whether our measuring instruments work at all. I tested whether automatic metrics (ROUGE-1/2/L, sacreBLEU, chrF++, BERTScore, BARTScore, and medical entity-level precision/recall/F1) and LLM-as-a-judge approximate clinician judgment, and identified where they diverge and physician assessment remains irreplaceable.