Lack of rigorous reproducibility and validation are significant hurdles for scientific development across many fields. Materials science, in particular, encompasses a variety of experimental and theoretical approaches that require careful benchmarking. Leaderboard efforts have been developed previously to mitigate these issues. However, a comprehensive comparison and benchmarking on an integrated platform with multiple data modalities with perfect and defect materials data is still lacking. This work introduces JARVIS-Leaderboard, an open-source and community-driven platform that facilitates benchmarking and enhances reproducibility. The platform allows users to set up benchmarks with custom tasks and enables contributions in the form of dataset, code, and meta-data submissions. We cover the following materials design categories: Artificial Intelligence (AI), Electronic Structure (ES), Force-fields (FF), Quantum Computation (QC), and Experiments (EXP). For AI, we cover several types of input data, including atomic structures, atomistic images, spectra, and text. For ES, we consider multiple ES approaches, software packages, pseudopotentials, materials, and properties, comparing results to experiment. For FF, we compare multiple approaches for material property predictions. For QC, we benchmark Hamiltonian simulations using various quantum algorithms and circuits. Finally, for experiments, we use the inter-laboratory approach to establish benchmarks. There are 1281 contributions to 274 benchmarks using 152 methods with more than 8 million data points, and the leaderboard is continuously expanding. The JARVIS-Leaderboard is available at the website: https://pages.nist.gov/jarvis_leaderboard/

缺乏严格的再现性和验证是许多领域科学发展的重大障碍。特别是材料科学，涵盖了需要仔细基准测试的各种实验和理论方法。之前已经开发了排行榜来缓解这些问题。然而，仍然缺乏在具有多种数据模式的集成平台上对完美和缺陷材料数据进行全面的比较和对标。这项工作引入了 JARVIS-Leaderboard，这是一个开源和社区驱动的平台，可促进基准测试并增强可重复性。该平台允许用户通过自定义任务设置基准，并允许以数据集、代码和元数据提交的形式做出贡献。我们涵盖以下材料设计类别：人工智能 (AI)、电子结构 (ES)、力场 (FF)、量子计算 (QC) 和实验 (EXP)。对于人工智能，我们涵盖多种类型的输入数据，包括原子结构、原子图像、光谱和文本。对于 ES，我们考虑了多种 ES 方法、软件包、赝势、材料和属性，并将结果与​​实验进行比较。对于 FF，我们比较了多种材料属性预测方法。对于质量控制，我们使用各种量子算法和电路对哈密顿模拟进行基准测试。最后，在实验中，我们使用实验室间方法来建立基准。使用 152 种方法、超过 800 万个数据点对 274 个基准测试做出了 1281 项贡献，并且排行榜还在不断扩大。 JARVIS-Leaderboard 可在以下网站获取：https://pages.nist.gov/jarvis_leaderboard/