Map Challenge FAQ

Below are compiled questions/answers regarding the map challenge that may be of interest to all of the participants.  We'll update this FAQ as needed.

Challenge Phase

1. Is the Cs listed in the target table (2.7) correct for the Apoferritin data?  No. There was an error in the script to generate the metadata requested for the challenge. The manufacturer specified Cs value for the Polara is 2.0 mm, but this has never been measured accurately for the instrument. In practice, this is an somewhat arbitrary fitting parameter that can be input during analysis of the raw data or refined during fitting. Followup question: What Cs do the defocus values provided in the particle stack refer to?  The provided defocus values assume Cs = 2.7. posted Nov 20, 2015, thanks to Niko Grigorieff for questions, and Chris Russo for answers

2. Why are more images provided in the newly added gain-corrected subdirectory vs original non-gain-corrected subdirectory for BMV (EMPIAR-10010)?  The gain-corrected data include the whole collected dataset, including images with lower resolution; the non-gain-corrected images correspond to those selected by the deposition authors for processing. posted Nov 20, 2015, thanks to Ardan Patwardhan for question and Zhao Wang for answer.  Please also see this additional note from Ben Bammes regarding the method used for gain-correction in the original work.

3. The TRPV1 challenge dataset lists the pixel size as 1.22 A, but after downloading the movies we found that they are stored in super-resolution and the pixel size is therefore 0.61 A. Could this be clarified? The movie stacks in the section of "TRPV1 raw multi-frame micrographs" (6.4G each) are super-resolution images, and should have a pixel size of 0.61 A (or 0.6078 A). posted Dec 3, 2015, thanks to Niko Grigorieff for the question, and Maofu Liao for the answer. 

4. We cannot find particle positions for TRPV1 micrographs in EMPIAR-10005, are they missing?  Yes!, at least up until the writing of this FAQ item. Following consultation with the data contributors, particle coordinates were added and are located in a new subdirectory of the EMPIAR entry micrographs section (picked_coordinates).  Please note that these are SPIDER coordinates for all picked particles (not the final particle set).  Also, the coordinates are for 3x binned images, so values will need to be multiplied by 3 to correspond to unbinned images. The 88915 folder reflects the initial particles that were selected by Maofu Liao for further processing. The 35645 one relates to the final particles he used in his processing to obtain the high-resolution map (meaning: he initially extracted 88915 particles, then post 2d and 3d classification he proceeded to minimize the stack by subselecting the final group of particles from his “star” file -> these 35645 particles). [Previously, we erroneously pointed to a relion star file in the "final particle coordinates" row of the target table, but that file provides shift information for the picked particle stack.]  posted January 18, 2016, thanks to Jose Maria Carazo and Carlos Oscar Sorozano for the question and Yifan Cheng and Jean Paul Armache for coordinates and answers. Picked coordinates link updated March 21, 2016, thanks to John Rubenstein for reporting that the link was broken.

5. Are submissions being looked at currently by anyone who would also participate in the challenge? Or are they all 'confidential' until the deadline? We are setting up a process that copies the submissions to an ftp area with all submitter-related data removed. The content/structure of the ftp is currently under review by the map committee. We will open up ftp access to everyone during the assessment period, but submissions will remain anonymous until the end of the assessment period. posted January 28, 2016, thanks to Sjors Scheres for the question.

6. What is the correct symmetry for the GroEL simulated data?  It is C1, not C7, please see this clarification. posted Feb. 19, 2016

7. Can I fill in multiple submissions simultaneously?  This is not recommended. If you have multiple maps resulting from a single benchmark, these need to be submitted separately, and we suggest to work on one at a time. Once you have completed one submission you will get an email with your answers. You can use then use email text to cut/paste answers as appropriate into your subsequent submissions.  Also, please note that you can save drafts while you are working, and further edit your submission after you submit. posted March 29. 2016, thanks to Shabih Shakeel for the question.

Blind Assessment Phase

8. Are CTF parameters, coordinates, and Euler angles of particle images of the submitted maps available for assessment of the maps? We have not been able to find this information.  Assessment of the submitted 3D maps without additional information is the main goal at this stage. We have held-back the CTF/coordinate/euler information during the blind assessment phase because the file formats provided  (e.g., .star, .spi files) in many cases do "give away" the identity of software used for particular map submissions.  Even so, we want to encourage anyone interested to pursue analyses that require this information at a later stage when these data become available.  We appreciate that such work will likely yield insights of benefit to the community.  posted December 2, 2016, thanks to Mohammadreza Paraan/Stagg Lab for the question.

Final Assessment

9. I see that the blind assessment phase has finished ;-). Should we start the not-so blind assessment phase in which more detailed information on how the 3D map has been obtained is available?  Questions that would be of interest to answer:

1. Did the groups that picked the particles perform better than those that use the particles supplied by default ?
2. Should we -always- dose filter the movie frames?
3. Regarding movie alignment: do global methods perform consistently worse than local ones or is it the other way around (at least for some specimens)?
4. Is it better to be very selective picking particles or is it better to pick as many as possible?
5. If CTF information can be obtained, it will be very interesting to check up to which frequency ctf computed by the different participant for the same data set are equivalent.
6. Particle consensus. For those groups that picked the particles it may be interesting to analyze the intersection and difference between the selected datasets.
7. There are many question related with 2D and 3D classification.
8. Finally the more important question is what software should I use.

Answer: Yes!!! With the newly released data it should be possible to further analyze the results of the blinded assessments, and these are the kinds of questions we hope that the challenge can ultimately address.  Please have a look at the list of metadata collected along with each reconstruction, now available in the newly released spreadsheet. Particle parameter sets are also available for download (ftp://public.emdatabank.org/maps2016/particle_params). posted Jun 8, 2017, many thanks to Roberto Marabini for the question.

More questions? send email challenges@emdatabank.org