model challenge

June 15 2020: "Outcomes of the 2019 EMDataResource model challenge: validation of cryo-EM models at near-atomic resolution" is available as a preprint at BioRxiv while under consideration for publication.  Feburary 2021 update: The final version of the paper is now published in Nature Methods with an accompanying News & Views commentary.

To support the upcoming JSB Journal Special Issue, all of the map challenge files and associated metadata are now archived, with downloadable zip files divided up by benchmark. A PDF containing a compilation of web pages related to the map challenge is also included. To reference particular map entries submitted to the challenge, manuscripts can cite doi:10.5281/zenodo.1120379 along with emcd### identifiers.  UPDATE Jan 25, 2018: All model challenge data are now also archived at https://doi.org/10.5281/zenodo.1165999.

In 2016 EMDataBank ran two community challenges in parallel to create awareness of the need for cryoEM structure validation as a routine process in research studies and publications, and to expedite development of quantitative tools for assessment

Update for the Model Challenge: A face-to-face meeting was held Monday May 29, 2017 to discuss model assessments and how to move forward with completing the challenge.  The challenge has been designed to enable assessment of current approaches to interpretation of higher resolution (3-5Å) cryoEM maps with atomic models, to critically evaluate the map interpretation, model fitting, model refinement, and validation methods that are now coming into use. Initial assessments of the 106 submissions from 16 modellers have been prepared and can be viewed at the model comparison site.  Notes from the May 29 meeting are available in the provided attachment.

The model challenge assessment period is still in progress.  Interested to see how we the results are being assessed?  Please visit the model-compare pages at http://model-compare.emdatabank.org.  There you can view and compare the submitted models via more than a dozen different whole-model and residue level metrics, as summarized here.

Following two amazing challenge submission finishes in April (66 maps!) and in June (106 models!), we have been working with our respective committees to prepare and organize the data for blinded assessments and to perform preliminary analyses. This process has taken more time than originally anticipated, so we have been making adjustments to assessment phase timelines. For the map challenge, we plan to announce the beginning of the assessment phase in early November.  Watch this space for more details! 

EMDataBank Challenge Modellers, Please complete your model submissions this week--the deadline is this coming Friday at 21:00 UTC. Any questions/issues please let us know (challenges@emdatabank.org). Thanks for participating!

The model challenge guide now holds corrected symmetry center info and BIOMT matrices for the proteasome, GroEL, and β-galactosidase targets. The changes are highlighted with red text and update date.

The previously posted symmetry info had used rounded values for the voxel sizes, yielding inprecise center coordinate and matrix element values. We apologize for the errors.

The deadline for model challenge submissions is now changed to Friday June 17 (21:00 UTC).

We hope that this extension will be helpful to all of our challengers.

In addition, please be aware that indicated symmetry centers in the model challenge guide may not be exact. 
This issue was just raised by one participant—we are investigating and will send a further update about this soon.

Further note (4/28): symmetry issue has been resolved