Summary of info provided via email on 7/9/2025

• Software releases:

  • Release of RAJA Perf can be made before official RFP goes out
  • That release should be tied to latest RAJA release
  • New released versions used for acceptance will be negotiated for SOW (run parameters, problem size, etc. are most important for this)

• Baseline results

  • Baseline numbers will be generated from the release cited in the RFP
  • We will put out preliminary results before this
  • ATS-4 will be official baseline
  • We will also generate results for H100
  • Here's an example from AMG for ATS-5: https://lanl.github.io/benchmarks/02_amg/amg.html
  • Benchmarking group has scripts to generate plots for RAJA Perf that we can use -- we will likely regenerate plots as software evolves

• Metrics/FOM

  • We want to keep RAJA Perf as a single node benchmark
    • Comm kernels may be an exception to this
  • We want to focus on varying problem size and run parameters (e.g., # thread blocks) rather than running on different numbers of cores/node
  • Setting metrics/FOM based on throughput plots may be a good way to go
  • We may need to set a single number for FOM such as geometric mean of something across a set of kernels

• Tier 1

  • We may be able to have up to 10 kernels
    • Focus on the "Apps" group and fill in gaps there based on meeting discussion Meeting:-2025-07-08

• Tier 2

  • We may be able to use the whole Suite as a compiler test with a requirement that run times of RAJA variants are within some threshold of base variants (like it was for ATS-4)