2015/2016 Model Challenge

Goals

Establish a benchmark set of 3DEM maps in the 3.0-4.5 Å resolution range, where significant growth in the number of maps is anticipated over the next few years and where a number of technical challenges exist to map interpretation and fitting
Encourage developers of modelling software packages and biological end users to analyze these maps and present modelling results with the best practice
Evolve criteria for evaluation and validation of 3DEM map-derived models
Compare and contrast the various modelling and analysis approaches in a positive spirit

Participation Links

Model Challenge Registrants: 41

Current Count, 2019 Model Metrics Challenge Submissions: 63

Model Committee

Paul Adams (Chair), Axel Brunger, Randy Read, Torsten Schwede, Maya Topf, Gerard Kleywegt, Cathy Lawson, Wah Chiu

Model Challenge How to Participate

All members of the Scientific Community--at all levels of experience--are invited to participate as Challengers, and/or as Assessors.

Challengers will create and submit their own atomic coordinate models of one or more challenge targets using the supplied target maps.

There are four modelling categories:

Optimize the current cryoEM model for the target, improving fit to the cryo-EM map density and/or model stereochemistry
Fit known related cryoEM, crystallographic, or comparative models to the cryo-EM map, optionally followed by model optimization
Ab initio model building, without reference to any existing cryo-EM or crystallographic models related to the cryo-EM map
Any other method that seeks to interpret, or create a model based on, the cryo-EM map

For each submission, challengers will provide their final model, target map (if modified from original), and basic information about their modelling process. Researchers are expected to disclose all of the prior information used in their method. For full challenge rules and additional information see the model challenge guide.

Assessors will contribute to the challenge assessment phase. Following a short initial review period by the model committee, challenge data and files will be made publicly available (entry authorship suppressed) for anyone to assess. The intention is to enable comparisons of the various modelling methodologies available and their options in a positive spirit. Assessors will be able to share their results via a planned workshop (~Fall 2016) as well as manuscript submissions to a Journal special issue.

****All participants--challengers and assessors--are required to register****

Model Challenge Timeline

2015	DEVELOPMENT PHASE
February-June	Model Committee meets monthly to discuss possible scope of challenge
June 20/21	Face-to-face meeting to define challenge targets, goals, and parameters
July-September	Finalization of challenge formulation. Requests to map contributors to provide any missing data (e.g., half-maps, unfiltered, unsharpened, unmasked maps)
2015/2016	CHALLENGE PHASE
October 14	Pre-Challenge Announcement, Challenger and Assessor Registration Opens
January-June 17	Challenge Site Open for Model Entry Submissions
June 17 21:00 UTC	Model entry submission window closes
2016/2017	ASSESSMENT PHASE
June-November	Challenge Data review by the Model Committee
December-May	Challenge Data review opened to volunteer assessors (blind analysis--entry authorship suppressed)
May 29	Face-to-Face Meeting to discuss preliminary outcomes of assessments
mid July	Assessors provide written reports on their results
mid July	Reports on Models sent to Challengers; Full workflow info provided to all challengers and assessors
Aug 15	Feedback from Challengers on their reports
Sept 1	Deadline for Assessors to update their reports
Oct 5-8	Challenges Meeting (Maps and Models)
Post-workshop	Challenge Writeups (multiple articles) for a Journal Special Issue

Model Challenge Targets

The eight targets chosen by the model challenge committee are shown below. All are based on recently reported 3DEM structures. The target order is according to unique molecular weight, from 19 kDa (TMV) at left to 2500 kDa (ribosome) at right. For challenge rules and additional information about each target see the model challenge guide.

April 28: Corrected symmetry center info and BIOMT matrix files for the proteasome and beta-galactosidase targets, see the model challenge guide.

May 3: Corrected BIOMT matrix file containing Icosahedral transformations for Brome Mosaic Virus, see the model challenge guide.

Target Map Download: You can use this rsync script to download target maps from one of three wwPDB ftp sites. Alternately, you can download individual maps from EMDB atlas pages (click on EMDB entry link in the table below, select "download" tab).

	1. Tobacco Mosaic Virus	2. T20S Proteasome	3. GroEL	4. TRPV1 Channel	5. Brome Mosaic Virus	6. β-Galactosidase	7. γ-Secretase	8. 70S Ribosome
target
Map "A" EMDB entry Primary Citation Reported Resolution (Å)	EMD-2842 Fromm et al 3.3	EMD-5623 Li et al 3.3	EMD-6422 -- 4.1	EMD-5778 Liao et al 3.3	EMD-6000 Wang et al 3.8	EMD-5995 Bartesaghi et al 3.2	EMD-2677 Lu et al 4.5	EMD-2847 Fischer et al 2.9
Reference Model(s) Map "A"	4udv (EM) 1ei7 (Xray)	1yar (Xray) 3j9i (EM)	3cau (EM) 1ss8 (Xray) 1svt (Xray)	3j5p (EM)	3j7l (EM) 1js9 (Xray)	3j7h (EM) 1jz7 (Xray)	4upc/superceded by 5a63 (EM)	5afi (EM)
Map "B" EMDB entry Primary Citation Reported Resolution (Å)		EMD-6287 Campbell et al 2.8				EMD-2984 Bartesaghi et al 2.2	EMD-3061 Bai et al 3.4	EMD-6316 Li et al 3.6
Reference Model(s) Map "B"		1yar (Xray)				5a1a (EM) 1jz7 (Xray)	5a63 (EM)	3ja1 (EM)
Imposed Map Symmetry	Helical	Dihedral (D7)	Dihedral (D7)	Cyclic (C4)	Icosahedral (I)	Dihedral (D2)	None (C1)	None (C1)
Sample MW (MDa)	--	0.7	0.8	0.3	4.6	0.47	0.17	2.5
Unique MW (kDa)	19	50	56	80	80	120	170	2500
Map Contributors (Thank You!)	Simon Fromm, Carsten Sachse	Jean-Paul Armache, Yifan Cheng Melody Campbell, Bridget Carragher	Soung-Hun Roh, Corey Hryc, Wah Chiu	Jean-Paul Armache, Maofu Liao, Yifan Cheng	Zhao Wang, Wah Chiu	Alberto Bartesaghi, Sriram Subramaniam	Xiaochen Bai, Sjors Scheres	Niels Fischer, Holger Stark Wen Li, Zheng Liu, Joachim Frank