map challenge

At the request of several Map Challenge participants we enabled submission of revised maps/info for 2016 Map Challenge entries in the period between mid-October 2017 and mid-January 2018. We offered this so that updated maps could be archived and referenced, e.g. as part of a contribution to the planned journal special issue.

To support the upcoming JSB Journal Special Issue, all of the map challenge files and associated metadata are now archived, with downloadable zip files divided up by benchmark. A PDF containing a compilation of web pages related to the map challenge is also included. To reference particular map entries submitted to the challenge, manuscripts can cite doi:10.5281/zenodo.1120379 along with emcd### identifiers.  UPDATE Jan 25, 2018: All model challenge data are now also archived at https://doi.org/10.5281/zenodo.1165999.

In 2016 EMDataBank ran two community challenges in parallel to create awareness of the need for cryoEM structure validation as a routine process in research studies and publications, and to expedite development of quantitative tools for assessment

Several updates for the map challenge:

Map Challenge Assessors:

• If you have not yet completed your analysis, the deadline for submitting assessment reports is now end of April (April 30). After that time we will be releasing the masked software/submitter info.
• We are working on the planning for a two day face-to-face discussion of assessment results, and are considering several possible dates/locations either over the summer or (more likely) in September.
• Following the workshop each assessor/assessment group will have the opportunity to publish their findings in a journal special issue.

As a reminder the Blind Assessment phase of the EMDataBank Map Challenge.is now open. A 1 hour webinar held on November 17 @15:00 UTC (9:00 US-CT, 10:00 US-ET, 15:00 UK, 16:00 Eur, 23:00 China) covered these topics:

• map challenge submissions overview
• how the submitted data and files were prepared for blind assessment
• how to download/access the data
• what preliminary analyses are already available
• how to participate in the assessment phase
• Q/A period

You can view/listen to the recorded webinar here:

The Assessment Phase of the EMDataBank Map Challenge is now officially open, and all are welcome to participate.

The Challenge Phase (July 2015-Apr 2016) was a tremendous success, with 66 submitted maps spread across the 7 image data benchmark targets. We recognize the significant time commitment required and are grateful to the 27 challengers who contributed their efforts and made their results available for analysis.

Following review, the challenge data and files are now publicly available to assessors.  The authors of the entries and the software used have been suppressed to promote a blind assessment. We welcome input from the assessors as to how to most usefully evaluate these results.  For example, assessment/comparison methods could include statistical analyses, resolution estimation, or fitting of atomic models. Some suggestions are provided on the website but these are not meant to be prescriptive. 

Following two amazing challenge submission finishes in April (66 maps!) and in June (106 models!), we have been working with our respective committees to prepare and organize the data for blinded assessments and to perform preliminary analyses. This process has taken more time than originally anticipated, so we have been making adjustments to assessment phase timelines. For the map challenge, we plan to announce the beginning of the assessment phase in early November.  Watch this space for more details! 

By popular request, we are extending the map challenge submission deadline to April 15, 2016 @ 21:00 UTC (5 PM US ET).  Click here to see the new deadline time in your location.

The authors of the GroEL simulated dataset have asked us to let all map challenge participants know that 4HEL, the model from which the images were generated, is non-symmetrical, owing to unmatched crystallographic symmetry (P2₁2₁2₁).   They have just become aware of this issue following discussion with two challenge participants.