You are here

Map Challenge Assessment Guidance

We've gathered some suggestions here about how to proceed with comparisons of the map submissions.* These are not meant to be prescriptive; results from other approaches are also welcome.

FSC Analysis

FSC curves based on provided half-maps and masks have been prepared for each map challenge entry (link). In most cases the results are consistent with or very close to the submitter-reported resolution, but this initial analysis cannot be used to directly compare submissions, because of differences in masking and map sizes, and thus convolution effects. FSC is a fundamental similarity metric, but its use in standard cryoEM practice has been problematic because of the maps being compared. Many suggestions were made on how to carry out follow-up analyses:  

  • Apply a single, common mask to all entries belonging to a target (e.g. 15-20 A average of all entries, with soft-edges or low-pass filtered).
  • Employ other methods/techniques such as: 
    • post-process phase randomization 
(e.g., to investigate effects of different masking on FSC)
    • mask artifact compensation
    • determine FSC error
    • Calculate map-model FSCs

Map Density Analysis

Images of each map both by itself and aligned to a common model are provided for reference (link), but further investigation is warranted, as variations in density appearance may be due to differences in power spectra and/or filtering/sharpening schemes.  Some suggestions:

  • Both overall images and close-up views are desirable; for comparison it is best to have the exact same view

  • Both well-ordered regions and not-so well ordered regions should be investigated 

  • Views containing slices (slabs or grey-scale planes) could be useful

  • Apply a common filtering/sharpening scheme to the unfiltered (raw) map entries for a target, bringing power spectra to a “common denominator” for density comparison

  • Along this line, view density across maps attenuated at a common low resolution, and then walk the attenuation towards higher resolution

  • Density quality could be investigated by fitting defined portions of each map using modeling tools (e.g. compare rmsd's of multiple models).


*suggestion credits: Maya Holmdahl, Roberto Marabini, Sjors Scheres, Bernard Heymann, Niko Grigorieff, Pawel Penczek, Ed Egelman, Steve Ludtke, Scott Stagg, Marin van Heel

EMDataResource Validation Challenges are supported by NIH National Institute of General Medical Sciences

Please send your challenge questions, comments and feedback to

Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer