Seed Variance and Evaluation Rigor in Memory-Augmented Cooperative MARL: A Study with CHARM
Applied Sciences (Switzerland), cilt.16, sa.15, 2026 (SCI-Expanded, Scopus)
- Yayın Türü: Makale / Tam Makale
- Cilt numarası: 16 Sayı: 15
- Basım Tarihi: 2026
- Doi Numarası: 10.3390/app16157650
- Dergi Adı: Applied Sciences (Switzerland)
- Derginin Tarandığı İndeksler: Science Citation Index Expanded (SCI-EXPANDED), Scopus, Applied Science & Technology Source, Compendex, INSPEC, Directory of Open Access Journals
- Anahtar Kelimeler: evaluation methodology, Google Research Football, hindsight experience replay, memory-augmented learning, multi-agent reinforcement learning, reproducibility, seed variance
- Hacettepe Üniversitesi Adresli: Evet
Özet
Cooperative multi-agent reinforcement learning (MARL) is hard when rewards are sparse. This is an evaluation study built around a negative result. We study CHARM (Cooperative Hindsight-Augmented Role-conditioned Memory), a four-part memory layer that wraps a standard backbone: role-indexed episodic memory (JACM/RCMP), hindsight relabeling (HCR), an auxiliary memory loss (SMAL), and cooperative replay (CER). We test it on four backbones (DQN, QMIX, MADDPG, QPLEX) against the episodic-memory baselines EMC and EMU, five to ten seeds, on Google Research Football. The mean win count barely moves on any backbone, and the small five-seed differences reverse sign at ten seeds. Because every method trains under reward shaping, the task is not fully sparse; a shaping-free control and a 30,000-episode anchor keep the result flat. A random-action policy scores within the trained range; only two of eleven configurations exceed it before correction. Inter-seed variance drops with CHARM on all four, but this is not a memory effect: plain weight decay reproduces the reduction, and removing the recall path leaves training byte-identical, so the cause is generic regularization. Evaluation is fragile: a three-seed comparison ( (Formula presented.) ) collapsed at five seeds, and no comparison survived correction. A three-tier memory added nothing: its top tier was never consumed.