- cctbxbb - phenix-online.org

Request for a fix in CC1/2 calculation in phenix.merging_statistics
by Keitaro Yamashita 28 Oct '13

28 Oct '13

Dear CCTBX developers, I found a big difference in CC1/2 value (say, 77.3 vs 68.3) between phenix.merging_statistics and XSCALE when more than one *anomalous* dataset was merged in XSCALE. phenix.merging_statistics seems to assume that equivalent (in case of anomalous, here I do not consider Friedel pairs as equivalent) reflections are sorted (equivalent reflections are not separeted) in the input file. Usually, this assumption should be correct. However, if multiple anomalous datasets were merged in XSCALE, this assumption is not correct. Because Friedel pairs are not necessarily separated. This is a real example (in P43212). These lines are extracted from .HKL file by XSCALE: > 1 -41 1 1.749E+04 1.657E+05 2052.0 2467.7 78.8 -113.23 1 > 41 -1 -1 4.569E+03 6.014E+03 454.1 1112.8 132.2 116.90 1 > 41 1 -1 8.721E+03 2.637E+03 473.0 1064.4 130.4 116.11 2 > 41 1 1 7.634E+03 2.417E+03 498.2 1010.8 135.3 115.65 2 > -1 -41 -1 4.847E+03 1.997E+03 1892.9 2543.0 85.9 -122.78 2 The last number indicates the dataset the reflection belongs to. Here, (1 -41 1), (41 -1 -1), (41, 1, 1), and (-1, -41, -1) are equivalent reflections, but (41, 1, -1) in the middle is not when anomalous data! Therefore, in phenix.merging_statistics, subsets calculation for (41,1,1) is performed twice (for first two and last two). When I changed the cctbx code to sort reflections after mapping to a.s.u., CC1/2 values by phenix.merging_statistics became closer to XSCALE. In merging_stats.__init__() in iotbx/merging_statistics.py, I inserted a line > array = array.sort("packed_indices") after > array = array.customized_copy(anomalous_flag=anomalous).map_to_asu() but I think there would be more correct way to fix this unexpected behavior. Best regards, Keitaro

2 3

correlation function for cctbx.miller.array
by Gabor Bunkoczi 24 Oct '13

24 Oct '13

Hi, I need to calculate the correlation function between two maps, both of them are stored as miller.array objects. There is a map_correlation method, which does something similar to what I want, but this calculates the total correlation coefficient, while what I need is another miller.array with amplitudes = f1*f2 and phases = p2-p1 (where f1, f2 are amplitudes and p1, p2 are phases of map1 and map2, respectively), i.e. not just the sum of these. Is this available somewhere? There is the additional complication that the hkl-list for map1 and map2 may not be fully identical, but the intersect of the two maps could be taken. Thanks, Gabor

2 5

Scipy / cctbx core dump
by Graeme.Winter＠diamond.ac.uk 15 Oct '13

15 Oct '13

Hi Folks, Just trying to use something here which depends on scipy but it seems scipy is not playing nice with flex: from scitbx.array_family import flex import scipy.linalg A tiny and inoffensive script however... cctbx.python boom.py show_stack(1): /home/gw56/svn/cctbx_base_15oct13/cctbx_plus_build/base/lib/python2.7/site-packages/scipy/linalg/blas.py(113) <module> show_stack(2): /home/gw56/svn/cctbx_base_15oct13/cctbx_plus_build/base/lib/python2.7/site-packages/scipy/linalg/misc.py(5) <module> show_stack(3): /home/gw56/svn/cctbx_base_15oct13/cctbx_plus_build/base/lib/python2.7/site-packages/scipy/linalg/__init__.py(147) <module> show_stack(4): /scratch/gw56/2013/OCT/15/multi-lattice/boom/boom.py(2) <module> libc backtrace (2 frames, most recent call last): /home/gw56/svn/cctbx_base_15oct13/cctbx_plus_build/base/lib/python2.7/site-packages/numpy/core/multiarray.so(PyArray_API+0) [0x7fb2064c9940] /lib64/libc.so.6() [0x37b3232920] Segmentation fault (Python and libc call stacks above) This crash may be due to a problem in any imported Python module, including modules which are not part of the cctbx project. To disable the traps leading to this message, define these environment variables (e.g. assign the value 1): BOOST_ADAPTBX_FPE_DEFAULT BOOST_ADAPTBX_SIGNALS_DEFAULT This will NOT solve the problem, just mask it, but may allow you to proceed in case it is not critical. I did a fresh nightly build install just to be sure :o( if I export those naughty env vars I get a core dump: cctbx.python boom.py Segmentation fault (core dumped) RHEL6.4 / scipy 0.12.0 and 0.12.1 any ideas? Thanks, Graeme Dr. Graeme Winter Senior Software Scientist Diamond Light Source +44 1235 778091 (work) +44 7786 662784 (work mobile) -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 1

Model based outlier calculation in mmtbx.scaling.outlier_rejection
by Keitaro Yamashita 09 Sep '13

09 Sep '13

Dear cctbx developers, I am interested in the implementation of model-based reflection outlier rejection. As I read the code mmtbx/scaling/outlier_rejection.py (lines 244-351), I noticed that maybe there was a discrepancy between what log_message explained and the actual code. The log_message in the code says: > Outliers are rejected on the basis of the assumption that a scaled > log likelihood differnce 2(log[P(Fobs)]-log[P(Fmode)])/Q\" is distributed > according to a Chi-square distribution (Q\" is equal to the second > derivative of the log likelihood function of the mode of the > distribution). > The outlier threshold of the p-value relates to the p-value of the > extreme value distribution of the chi-square distribution. while actual p_value is calculated for each hkl as p_value = 1 - erf(sqrt(LLG))**N, where LLG = log p(F=Fbar | Fmodel) - log p(F=Fobs | Fmodel), and N is the number of reflections. Here, Fbar is F which gives the maximum value of p(F | Fmodel). At least, Q (the second derivative of p(F=Fbar | Fmodel)) is not used in the actual calculation. Could someone please explain the meaning of the actual calculation? Why taking square-root and raising erf() result to the power of N? Thank you very much, Keitaro

4 4

map_to_asu - get the symop used to perform mapping
by Graeme.Winter＠diamond.ac.uk 24 Jun '13

24 Jun '13

Hi All, In the progress of trying to (for reasons of my own) reproduce scale pack unmerged format files from flex arrays of Miller indices. map_to_asu is great, but it would be great to get the symop # used to perform the mapping as a result of this (it is computed somewhere in asu.h but not easily pushed out…) Has anyone done this already? I mean the getting-of-symop-which-maps bit... Thanks, Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 3

build problem on ubuntu 12.04
by Alastair Fyfe 11 Jun '13

11 Jun '13

Hello, is the current cctbx build (2013_02_27_0005) known to be buildable on Ubuntu ? I tried "perl cctbx_bundle.selfx" without success. After some poking about I found an exception raised in the platform.architecture() call in env_config.py/is_64bit_architecture(). Modifying the function to simply return False allowed the build to complete (log attached).This is on 12.04 with stock python 2.7.3 and the "platform.architecture()" call works as expected in a simple test. thanks, Alastair

4 8

updated project docs
by Nathaniel Echols 05 Jun '13

05 Jun '13

FYI, I finally updated the online documentation for the first time since switching the project backend. Included in this is the corrected link for the SVN browser, which for the record is now: http://sourceforge.net/p/cctbx/code/HEAD/tree/trunk/ (I find this somewhat over-engineered, but the old URL only shows the frozen repository as of early March, so we don't have a choice.) If anyone notices broken stuff, please let me know. -Nat

2 1

Re: [cctbxbb] building sphinx documentation
by Jan Marten Simons 19 Apr '13

19 Apr '13

Am Donnerstag 18 April 2013 19:16:00 schrieben Sie: > Hi Jan, Hi Richard, > I have attempted to build the sphinx documentation using > dox.sphinx/build_docs.sh, and this appears to have run successfully and > produced some html which shows the table of contents etc. However, I can't > seem to find the documentation for any of the significant modules, e.g. > cctbx/xray/structure.py. When I navigate to the link for the structure > module within the xray package > (dox.sphinx/_build/html/cctbx/cctbx.xray.html#structure-module), I see > nothing other than a list of all the modules within the xray package. Am I > missing something here? Have you followed the instructions given in the comments at the top of dox.sphinx/build_docs.sh? Which version of Sphinx do you use? I'm using 1.1.3 regularly and get useful documentation (I've attached the "makescript" I use to build cctbx and the sphinx documentation. If you are interested in it I can also put the output of the sphinx run on my system with current svn on webserver over here.). Has your sphinx build completed successfully or did it abort (e.g. due to out of memory* or due to other problems)? Could you also check if build/dox.sphinx/cctbx/cctbx.xray.rst has all modules listed? > As a novice to using sphinx, I don't like to add significant documentation > in sphinx style without being able to check via generating the > documentation that what I have written is correct sphinx-speak. I can totally understand this ;). Cheers and hopefully we can get this working for you as well, Jan *) For me running sphinx with all those modules selected needs about 3.3 GiB of virtual memory (RAM) and takes a few minutes to actually finish. p.s.: Sorry didn't hit "reply all".

3 6

Finding neighbouring atoms using pair_asu_table
by Lukas Reck 19 Apr '13

19 Apr '13

Dear all, please forgive me if the following is trivial or if I've overlooked something simple, I'm new to cctbx. I'm trying to use the cctbx python libraries to read in a large number of crystal structures and extract some geometric information on the coordination environment of certain atom types in them. I've been following the instructions from the 8/2004 CompComm Newsletter No. 4 at <http://www.iucr.org/__data/assets/pdf_file/0003/6384/iucrcompcomm_aug2004.p…>, pages 22-28 (is that still up to date?) but I cannot seem to get at the correct symmetry elements for each neighbouring atom in the coordination environment of some atom of interest. As far as I understand, the following should be correct: assert(isinstance(structure, cctbx.xray.structure)) # yes, it is a structure asu_mappings = structure.asu_mappings(buffer_thickness = 3) # create the ASU mappings pair_asu_table = cctbx.crystal.pair_asu_table(asu_mappings = asu_mappings) pair_asu_table.add_all_pairs(distance_cutoff = 3) # create the pair ASU table raw_mappings = asu_mappings.mappings() raw_pair_table = pair_asu_table.table() # access to the raw structures for scatterer_index, scatterer in enumerate(structure.scatterers()): # iterate over all atoms in the structure label = scatterer.element_symbol() if label == atom_of_interest: # we are only interested in one atom type for j_seq, j_syms in raw_pair_table[scatterer_index].items(): # j_seq is the neighbouring scatterer's index, j_sym is a list of lists of symmetry indices for sym_group in j_syms: for j_sym in sym_group: # j_sym is an index into raw_mappings[j_seq] print "j_seq:", j_seq, "j_sym:", j_sym print "i_sym_op:", raw_mappings[j_seq][j_sym].i_sym_op() symmetry = asu_mappings.get_rt_mx(j_seq, j_sym) print "%s at %s" % (scatterer.element_symbol(), scatterer.site) print "Neighbour: %s at %s" % (structure.scatterers()[j_seq].element_symbol(), structure.scatterers()[j_seq].site) print "To be transformed by %s (symmetry #%s)" % (symmetry, j_sym) print "into %s" % (symmetry(structure.scatterers()[j_seq].site),) Here's a sample output for one pair of neighbouring atoms: j_seq: 10 j_sym: 1 i_sym_op: 10 V at (0.30204, 0.52919, 0.5) Neighbour: O at (0.36242, 0.49166, 0.5) To be transformed by y-1/2,-x+1/2,z-1/2 (symmetry #1) into (-0.00834, 0.13758, 0.0) As you can see from the coordinates, the neighbouring atom was correctly identified, but the corresponding symmetry as returned by get_rt_mx transforms it into a symmetry equivalent that is very far away. Clearly I misunderstand the function of one or more of the various indices here, but I am not quite clear on where my error lies... Thanks for any pointers! Best regards Lukas Reck -- School of Chemistry Faculty of Engineering, Mathematics and Science Trinity College Dublin Dublin 2 Tel: (+353) 1 896 3452

2 3

building sphinx documentation
by Richard Gildea 19 Apr '13

19 Apr '13

Hi Jan, I have attempted to build the sphinx documentation using dox.sphinx/build_docs.sh, and this appears to have run successfully and produced some html which shows the table of contents etc. However, I can't seem to find the documentation for any of the significant modules, e.g. cctbx/xray/structure.py. When I navigate to the link for the structure module within the xray package (dox.sphinx/_build/html/cctbx/cctbx.xray.html#structure-module), I see nothing other than a list of all the modules within the xray package. Am I missing something here? As a novice to using sphinx, I don't like to add significant documentation in sphinx style without being able to check via generating the documentation that what I have written is correct sphinx-speak. Cheers, Richard

3 2