2015/2016 Map Challenge

All members of the Scientific Community--at all levels of experience--are invited to participate as Challengers, and/or as Assessors.

In the Challenge Phase (July 2015-Apr 2016), 27 participants created and submitted 66 reconstructions of the challenge targets using the supplied raw image data. Challengers were encouraged to perform their own movie frame alignment, frame summation, and particle picking. Alternately, they could begin with pre-aligned, summed images and/or original author-provided particle positions. For each submission, challengers filled out a questionnaire (overview) and provided the following data:

• final unmasked map with filtering/sharpening
• final unmasked map without filtering or sharpening
• half-maps and mask used for FSC calculation
• CTF, coordinates, euler angles for each particle image used in the reconstruction

In the Blinded Assessment Phase (Nov 2016-Apr 2017), six groups have contributed reports.  Following initial review period by the map committee, the challenge data and files are now publicly available (entry authorship and software suppressed) for anyone to assess. Assessment methods could include statistical analyses, resolution estimation, or coordinate model fitting. A few suggestions gathered from software developers are summarized below. The intention is to enable comparisons of the various packages available and their options in a positive spirit.  During this period, assessors are strongly encouraged to share their plans and short result summaries on this website using the Assessment Registration Form.  Assessment results will be more fully presented and discussed via a workshop (early October 2017) as well as via manuscript submissions to a Journal special issue.

We've gathered some suggestions here about how to proceed with comparisons of the map submissions.* These are not meant to be prescriptive; results from other approaches are also welcome.

FSC Analysis

FSC curves based on provided half-maps and masks have been prepared for each map challenge entry (link). In most cases the results are consistent with or very close to the submitter-reported resolution, but this initial analysis cannot be used to directly compare submissions, because of differences in masking and map sizes, and thus convolution effects. FSC is a fundamental similarity metric, but its use in standard cryoEM practice has been problematic because of the maps being compared. Many suggestions were made on how to carry out follow-up analyses:  

• Apply a single, common mask to all entries belonging to a target (e.g. 15-20 A average of all entries, with soft-edges or low-pass filtered).
• Employ other methods/techniques such as: 
• post-process phase randomization 
(e.g., to investigate effects of different masking on FSC)
• mask artifact compensation
• determine FSC error
• Calculate map-model FSCs

Map Density Analysis

Images of each map both by itself and aligned to a common model are provided for reference (link), but further investigation is warranted, as variations in density appearance may be due to differences in power spectra and/or filtering/sharpening schemes.  Some suggestions:


• Both overall images and close-up views are desirable; for comparison it is best to have the exact same view

• Both well-ordered regions and not-so well ordered regions should be investigated 

• Views containing slices (slabs or grey-scale planes) could be useful

• Apply a common filtering/sharpening scheme to the unfiltered (raw) map entries for a target, bringing power spectra to a “common denominator” for density comparison

• Along this line, view density across maps attenuated at a common low resolution, and then walk the attenuation towards higher resolution

• Density quality could be investigated by fitting defined portions of each map using modeling tools (e.g. compare rmsd's of multiple models).


------------

*suggestion credits: Maya Holmdahl, Roberto Marabini, Sjors Scheres, Bernard Heymann, Niko Grigorieff, Pawel Penczek, Ed Egelman, Steve Ludtke, Scott Stagg, Marin van Heel