Map Challenge Assessment Guidance

We've gathered some suggestions here about how to proceed with comparisons of the map submissions.* These are not meant to be prescriptive; results from other approaches are also welcome.

FSC Analysis

FSC curves based on provided half-maps and masks have been prepared for each map challenge entry (link). In most cases the results are consistent with or very close to the submitter-reported resolution, but this initial analysis cannot be used to directly compare submissions, because of differences in masking and map sizes, and thus convolution effects. FSC is a fundamental similarity metric, but its use in standard cryoEM practice has been problematic because of the maps being compared. Many suggestions were made on how to carry out follow-up analyses:  

• Apply a single, common mask to all entries belonging to a target (e.g. 15-20 A average of all entries, with soft-edges or low-pass filtered).
• Employ other methods/techniques such as: 
• post-process phase randomization 
(e.g., to investigate effects of different masking on FSC)
• mask artifact compensation
• determine FSC error
• Calculate map-model FSCs

Map Density Analysis

Images of each map both by itself and aligned to a common model are provided for reference (link), but further investigation is warranted, as variations in density appearance may be due to differences in power spectra and/or filtering/sharpening schemes.  Some suggestions:


• Both overall images and close-up views are desirable; for comparison it is best to have the exact same view

• Both well-ordered regions and not-so well ordered regions should be investigated 

• Views containing slices (slabs or grey-scale planes) could be useful

• Apply a common filtering/sharpening scheme to the unfiltered (raw) map entries for a target, bringing power spectra to a “common denominator” for density comparison

• Along this line, view density across maps attenuated at a common low resolution, and then walk the attenuation towards higher resolution

• Density quality could be investigated by fitting defined portions of each map using modeling tools (e.g. compare rmsd's of multiple models).


------------

*suggestion credits: Maya Holmdahl, Roberto Marabini, Sjors Scheres, Bernard Heymann, Niko Grigorieff, Pawel Penczek, Ed Egelman, Steve Ludtke, Scott Stagg, Marin van Heel