Re: [phenixbb] phenix and weak data

On Dec 12, 2012, at 10:47 AM, Randy Read wrote:

On 12 Dec 2012, at 15:36, Douglas Theobald wrote:

...

On Dec 12, 2012, at 1:46 AM, Ed Pozharski wrote:

...

On Tue, 2012-12-11 at 11:27 -0500, Douglas Theobald wrote:

...

What is the evidence, if any, that the exptl sigmas are actually negligible compared to fit beta (is it alluded to in Lunin 2002)? Is there somewhere in phenix output I can verify this myself?

Essentially, equation 4 in Lunin (2002) is the same as equation 14 in Murshudov (1997) or equation 1 in Cowtan (2005) or 12-79 in Rupp (2010). The difference is that instead of combination of sigf^2 and sigma_wc you have a single parameter, beta. One can do that assuming that sigf<

From the log file:

|-----------------------------------------------------------------------------| |R-free likelihood based estimates for figures of merit, absolute phase error,| |and distribution parameters alpha and beta (Acta Cryst. (1995). A51, 880-887)| | | | Bin Resolution No. Refl. FOM Phase Scale Alpha Beta | | # range work test error factor | | 1: 44.4859 - 3.0705 14086 154 0.93 12.12 1.00 0.98 118346.13| | 2: 3.0705 - 2.4372 13777 149 0.91 15.26 1.00 0.99 58331.77| | 3: 2.4372 - 2.1291 13644 148 0.94 11.42 1.00 0.99 23216.31|

it appears that phenix estimates alpha and beta from the R-free set rather than from the working set (I might be misreading that). Is that correct?

Yes, using the cross-validation data was a key step in getting maximum likelihood refinement to work. A long time ago (a few years before our first paper on ML refinement) I implemented a first version of the MLF target we put into CNS, but the sigmaA values were estimated from the working data. What happened was that the data would be over-fit, then the sigmaA estimates would go up (with part of the increase being a result of the overfitting), then in the next cycle the pressure to fit the data compared to the restraints would be higher, and so on. The best I could claim for this at the time was that the resulting models were at least as good as the ones from least-squares refinement, but the R-factors were higher (indicating that there was still less over-fitting). It would have been hard to sell the advantage of higher R-factors to the protein crystallography community so it was good that, when we started using cross-validated sigmaA values, the convergence radius improved and we could get significantly better models with lower R-factors. I think you'll find that all the programs use just the cross-validation data to estimate the variance parameters for the likelihood target, not just phenix.refine.

Thanks Randy. I must say I'm quite surprised by this, and coming from a likelihoodist/Bayesian POV it seems very wrong :). I realize it works in practice, and works quite well evidently, but there's a very odd marriage of statistical philosophies going on here. From a likelihood POV, the joint ML estimates really should come from the same data set (i.e. the working set --- and there's the issue that there's already been a likelihood "compromise" of sorts by excluding some of the data from estimation anyway, a practice which violates the likelihood principle). And from a frequentist cross-validation POV, you certainly should not be refining against the test set --- that violates the very rationale of a test set.

Randy

...

_______________________________________________ phenixbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/phenixbb

------ Randy J. Read Department of Haematology, University of Cambridge Cambridge Institute for Medical Research Tel: + 44 1223 336500 Wellcome Trust/MRC Building Fax: + 44 1223 336827 Hills Road E-mail: [email protected] Cambridge CB2 0XY, U.K. www-structmed.cimr.cam.ac.uk

_______________________________________________ phenixbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/phenixbb