September 2018 - cctbxbb - phenix-online.org

Allocators
by Graeme.Winter＠Diamond.ac.uk 25 Sep '18

25 Sep '18

Hello All, In keeping with the spirit of https://github.com/cctbx/cctbx_project/wiki/cctbx-contributors-guide we would like to revisit the allocator work now Phenix 1.14 is out in the wild - however since we now know that this needs to be done we are interested in other points of view people have on ways to do this, and options people would like to have. It is important to note that we have no intention of changing the default behaviour, but we do want to be able to use customised allocators with flex arrays e.g. for handing out memory of well known and repeatedly used sizes (say, 16 M pixel double arrays) or allocate in specific areas for NUMA systems such as xeon phi. We prototyped this a while back and stepped the changes back out again to enable the phenix release to proceed smoothly, so we have a pretty good idea of the sorts of things we would expect to do however will welcome input if you would like additional features. As a baseline, we are talking as a parameter that PHENIX are no longer supporting CENTOS 5 32 bit (since ISTR this was a constraint before) Any thoughts? Appreciate a number of folks will be travelling back after the PHENIX workshop. Many thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

Conda
by Graeme.Winter＠Diamond.ac.uk 24 Sep '18

24 Sep '18

Hello again, Trying to keep threads from getting warped :-) We’re interested in what the plans and timescales are for the move to Conda for a base to cctbx-related projects, and in particular - - what is planned? - when i.e. timescales (rough) - will bootstrap go away? I know from talking to Billy on Wednesday that this is back on the cards again now the PHENIX release is out, but I didn’t ask at the time fo the details… oops! Many thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

Fwd: [git/cctbx] master: Python 3 syntax changes (790d9e315)
by Graeme.Winter＠Diamond.ac.uk 21 Sep '18

21 Sep '18

Folks, I expressed some surprise at this & friends however - - this is necessary to make the bootstrap process Python 3 compatible - the timetable for which said this should have been finished 9 months ago (https://github.com/cctbx/cctbx_project/wiki/Python-3-roadmap) - these changes were discussed and agreed (https://github.com/cctbx/cctbx_project/pull/199) The only part of this which was not approved was reverted before the request was merged (I felt that this annotation was helpful as the changes themselves go back ~6 months so people have possibly forgotten the context) Best wishes Graeme Begin forwarded message: From: CCTBX commit <diamondlightsource.jenkins(a)GMAIL.COM<mailto:[email protected]>> Subject: [git/cctbx] master: Python 3 syntax changes (790d9e315) Date: 21 September 2018 at 14:25:29 BST To: <CCTBX-COMMIT(a)JISCMAIL.AC.UK<mailto:[email protected]>> Reply-To: <cctbxbb(a)PHENIX-ONLINE.ORG<mailto:[email protected]>> Repository : ssh://ws133.diamond.ac.uk/cctbx On branch : master ________________________________ commit 790d9e315a2a6ae5a875ba86b461ed570c77280d Author: Markus Gerstel <markus.gerstel(a)diamond.ac.uk<mailto:[email protected]>> Date: Thu May 24 12:16:38 2018 +0100 Python 3 syntax changes Replace the reduce(add, ...) function by sum(). reduce() is no longer in the global namespace in python 3. ________________________________ 790d9e315a2a6ae5a875ba86b461ed570c77280d scitbx/math/__init__.py | 34 ++++++++++++++++------------------ 1 file changed, 16 insertions(+), 18 deletions(-) diff --git a/scitbx/math/__init__.py b/scitbx/math/__init__.py index a13405f92..49e295eb0 100644 --- a/scitbx/math/__init__.py +++ b/scitbx/math/__init__.py @@ -1,11 +1,13 @@ -from __future__ import division, absolute_import +from __future__ import absolute_import, division, print_function + +import math + from scitbx.math.ext import * import scitbx.linalg.eigensystem import scitbx.math.gaussian # implicit import from scitbx import matrix from scitbx.array_family import flex from boost import rational # implicit import -import math gaussian_fit_1d_analytical.__doc__ = """ Fits a gaussian function to a list of points. @@ -88,16 +90,14 @@ def r3_rotation_average_rotation_matrix_from_matrices(*matrices): """ if not matrices: - raise TypeError, "Average of empty sequence" + raise TypeError("Average of empty sequence") - import operator import scitbx.matrix - average = reduce( - operator.add, - [ scitbx.matrix.col( r3_rotation_matrix_as_unit_quaternion( e ) ) - for e in matrices ] - ) + average = sum( + scitbx.matrix.col( r3_rotation_matrix_as_unit_quaternion( e ) ) + for e in matrices + ) return r3_rotation_unit_quaternion_as_matrix( average.normalize() ) @@ -111,13 +111,12 @@ def r3_rotation_average_rotation_via_lie_algebra(matrices, maxiter = 100, conver """ if not matrices: - raise TypeError, "Average of empty sequence" + raise TypeError("Average of empty sequence") if maxiter < 0 or convergence <= 0: - raise ValueError, "Invalid iteration parameters" + raise ValueError("Invalid iteration parameters") from scitbx.math import so3_lie_algebra - import operator conv_sq = convergence * convergence norm = 1.0 / len( matrices ) @@ -125,10 +124,9 @@ def r3_rotation_average_rotation_via_lie_algebra(matrices, maxiter = 100, conver for i in range( maxiter ): inverse = current.inverse() - log_diff = norm * reduce( - operator.add, - [ so3_lie_algebra.element.from_rotation_matrix( matrix = inverse * m ) - for m in matrices ] + log_diff = norm * sum( + so3_lie_algebra.element.from_rotation_matrix( matrix = inverse * m ) + for m in matrices ) if log_diff.norm_sq() < conv_sq: @@ -136,7 +134,7 @@ def r3_rotation_average_rotation_via_lie_algebra(matrices, maxiter = 100, conver current *= log_diff.exponential() - raise RuntimeError, "Iteration limit exceeded" + raise RuntimeError("Iteration limit exceeded") def euler_angles_as_matrix(angles, deg=False): if (deg): @@ -174,7 +172,7 @@ class erf_verification(object): if (self.max_delta < delta): self.max_delta = delta if (delta > self.tolerance): - print x, expected_result, result, delta + print(x, expected_result, result, delta) class minimum_covering_sphere_nd(object): ________________________________ To unsubscribe from the CCTBX-COMMIT list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCTBX-COMMIT&A=1 -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

Re: [cctbxbb] [git/cctbx] master: Python 3 syntax changes (790d9e315)
by Graeme Winter 21 Sep '18

21 Sep '18

Folks, I expressed some surprise at this & friends however - - this is necessary to make the bootstrap process Python 3 compatible - the timetable for which said this should have been finished 9 months ago (https://github.com/cctbx/cctbx_project/wiki/Python-3-roadmap <https://github.com/cctbx/cctbx_project/wiki/Python-3-roadmap>) - these changes were discussed and agreed (https://github.com/cctbx/cctbx_project/pull/199 <https://github.com/cctbx/cctbx_project/pull/199>) The only part of this which was not approved was reverted before the request was merged (I felt that this annotation was helpful as the changes themselves go back ~6 months so people have possibly forgotten the context) Best wishes Graeme > On 21 Sep 2018, at 14:25, CCTBX commit <diamondlightsource.jenkins(a)GMAIL.COM> wrote: > > Repository : ssh://ws133.diamond.ac.uk/cctbx > On branch : master > > > > commit 790d9e315a2a6ae5a875ba86b461ed570c77280d > Author: Markus Gerstel <markus.gerstel(a)diamond.ac.uk> > Date: Thu May 24 12:16:38 2018 +0100 > > Python 3 syntax changes > > Replace the reduce(add, ...) function by sum(). > reduce() is no longer in the global namespace in python 3. > > > > > 790d9e315a2a6ae5a875ba86b461ed570c77280d > scitbx/math/__init__.py | 34 ++++++++++++++++------------------ > 1 file changed, 16 insertions(+), 18 deletions(-) > > diff --git a/scitbx/math/__init__.py b/scitbx/math/__init__.py > index a13405f92..49e295eb0 100644 > --- a/scitbx/math/__init__.py > +++ b/scitbx/math/__init__.py > @@ -1,11 +1,13 @@ > -from __future__ import division, absolute_import > +from __future__ import absolute_import, division, print_function > + > +import math > + > from scitbx.math.ext import * > import scitbx.linalg.eigensystem > import scitbx.math.gaussian # implicit import > from scitbx import matrix > from scitbx.array_family import flex > from boost import rational # implicit import > -import math > > gaussian_fit_1d_analytical.__doc__ = """ > Fits a gaussian function to a list of points. > @@ -88,16 +90,14 @@ def r3_rotation_average_rotation_matrix_from_matrices(*matrices): > """ > > if not matrices: > - raise TypeError, "Average of empty sequence" > + raise TypeError("Average of empty sequence") > > - import operator > import scitbx.matrix > > - average = reduce( > - operator.add, > - [ scitbx.matrix.col( r3_rotation_matrix_as_unit_quaternion( e ) ) > - for e in matrices ] > - ) > + average = sum( > + scitbx.matrix.col( r3_rotation_matrix_as_unit_quaternion( e ) ) > + for e in matrices > + ) > > return r3_rotation_unit_quaternion_as_matrix( average.normalize() ) > > @@ -111,13 +111,12 @@ def r3_rotation_average_rotation_via_lie_algebra(matrices, maxiter = 100, conver > """ > > if not matrices: > - raise TypeError, "Average of empty sequence" > + raise TypeError("Average of empty sequence") > > if maxiter < 0 or convergence <= 0: > - raise ValueError, "Invalid iteration parameters" > + raise ValueError("Invalid iteration parameters") > > from scitbx.math import so3_lie_algebra > - import operator > > conv_sq = convergence * convergence > norm = 1.0 / len( matrices ) > @@ -125,10 +124,9 @@ def r3_rotation_average_rotation_via_lie_algebra(matrices, maxiter = 100, conver > > for i in range( maxiter ): > inverse = current.inverse() > - log_diff = norm * reduce( > - operator.add, > - [ so3_lie_algebra.element.from_rotation_matrix( matrix = inverse * m ) > - for m in matrices ] > + log_diff = norm * sum( > + so3_lie_algebra.element.from_rotation_matrix( matrix = inverse * m ) > + for m in matrices > ) > > if log_diff.norm_sq() < conv_sq: > @@ -136,7 +134,7 @@ def r3_rotation_average_rotation_via_lie_algebra(matrices, maxiter = 100, conver > > current *= log_diff.exponential() > > - raise RuntimeError, "Iteration limit exceeded" > + raise RuntimeError("Iteration limit exceeded") > > def euler_angles_as_matrix(angles, deg=False): > if (deg): > @@ -174,7 +172,7 @@ class erf_verification(object): > if (self.max_delta < delta): > self.max_delta = delta > if (delta > self.tolerance): > - print x, expected_result, result, delta > + print(x, expected_result, result, delta) > > class minimum_covering_sphere_nd(object): > > > To unsubscribe from the CCTBX-COMMIT list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCTBX-COMMIT&A=1 <https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCTBX-COMMIT&A=1>

1 0

Post-PHENIX changes
by Graeme.Winter＠Diamond.ac.uk 21 Sep '18

21 Sep '18

Morning All, I see from http://phenix-online.org/ That PHENIX 1.14 has been released Now, I see at https://github.com/cctbx/cctbx_project/pulls we have a few fairly hefty pull requests parked as “hold for phenix release” - I would certainly not suggest merging all these in one go however probably we should give some thought to stepping through some of them, deciding which ones we should look at first and starting to merge things? I for one would like Nick’s allocator code to come back into play (so we can make use of it for DIALS) Thoughts / opinions? For info: PHENIX folks did you tag the version which was released in PHENIX 1.14 to allow a release branch for .1 etc. releases? Many thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 1

repr for flex arrays
by Graeme.Winter＠Diamond.ac.uk 14 Sep '18

14 Sep '18

Hi Folks, Is there a reason for not having a helpful __repr__ for flex arrays i.e. Grey-Area:002-decide-when graeme$ dials.python Python 2.7.14 (default, Apr 17 2018, 15:02:10) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from dials.array_family import flex >>> a = flex.double([1, 2, 3]) >>> print a <scitbx_array_family_flex_ext.double object at 0x1095673c0> (I have a feeling I asked this before but cannot find reference) For example >>> print a.as_numpy_array() [ 1. 2. 3.] is more useful… but annoying to type I appreciate a could be “big” but numpy seems to handle this OK even - could __repr__ be punned to as_numpy_array(),__repr__()? Currently it prints the type of the array and its location in ram, which are adequately served by >>> type(a) <class 'scitbx_array_family_flex_ext.double'> >>> print id(a) 4451627968 If I print an array I more often want to see the content and less often the storage class and location (I humbly assert) Thoughts? I know this tends to confuse and annoy people who are new to flex arrays (and veterans who are busy debugging stuff…) Many thanks Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

4 9

Python 3 status
by Graeme.Winter＠Diamond.ac.uk 06 Sep '18

06 Sep '18

Hi Folks, Are there any updates on the progress to Python 3 compatibility? I know there is a project here https://github.com/cctbx/cctbx_project/projects/2 but I have lost the thread of activities since I have been out of the office much of the summer. At the very least I know no progress has been made from this side of the pond as we’ve put this work down for now. i would guess that a lot is on hold while the PHENIX release is done - however I thought I should check in case I have missed anything? Part of why I ask is that there is a PHENIX meeting in Cambridge real soon now (mid September) and I wonder if it is worth folks from DIALS and PHENIX getting together at that time to discuss this? Any thoughts? Thanks & best wishes Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

7 12

Re: [cctbxbb] [git/cctbx] master: re-defining the shift/est convergence limit in terms of crystallographic rather than independent parameters to make it easier to relate to the convergence reported in the CIF; also caching the shifts for further analysis and reporting; (9339fd5f7)
by Nicholas Sauter 05 Sep '18

05 Sep '18

I'm worried about making changes like this to a fundamental toolbox component like normal equations solving. We use this for processing xfel data, and would like the opportunity to test any changes through a pull request, prior to having them committed to the main branch. Nick Sauter Nicholas K. Sauter, Ph. D. Senior Scientist, Molecular Biophysics & Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720 (510) 486-5713 On Wed, Sep 5, 2018 at 5:29 AM, CCTBX commit < diamondlightsource.jenkins(a)gmail.com> wrote: > Repository : ssh://g18-sc-serv-04.diamond.ac.uk/cctbx > On branch : master > > ------------------------------ > > > commit 9339fd5f7d240f0d6f85dd57ce42e796d19608f1 > Author: pcxod <oleg_dolomanov(a)hotmail.com> > Date: Wed Sep 5 13:29:29 2018 +0100 > > re-defining the shift/est convergence limit in terms of > crystallographic rather than independent parameters to make it easier to > relate to the convergence reported in the CIF; also caching the shifts for > further analysis and reporting; > > > ------------------------------ > > > 9339fd5f7d240f0d6f85dd57ce42e796d19608f1 > scitbx/lstbx/normal_eqns_solving.py | 17 +++++++++++------ > 1 file changed, 11 insertions(+), 6 deletions(-) > > diff --git a/scitbx/lstbx/normal_eqns_solving.py > b/scitbx/lstbx/normal_eqns_solving.py > index 660304eab..45f070622 100644 > --- a/scitbx/lstbx/normal_eqns_solving.py > +++ b/scitbx/lstbx/normal_eqns_solving.py > @@ -140,15 +140,20 @@ class iterations(object): > a = self.non_linear_ls.normal_matrix_packed_u() > a.matrix_packed_u_diagonal_add_in_place(value*a. > matrix_packed_u_diagonal()) > > - def do_scale_shifts(self, max_shift_over_esd): > + def do_scale_shifts(self, limit_shift_over_su): > x = self.non_linear_ls.step() > esd = self.non_linear_ls.covariance_matrix().matrix_packed_u_ > diagonal() > - x_over_esd = flex.abs(x/flex.sqrt(esd)) > - max_val = flex.max(x_over_esd) > - if max_val < self.convergence_as_shift_over_esd: > + ls_shifts_over_su = flex.abs(x/flex.sqrt(esd)) > + #max shift for the LS > + self.max_ls_shift_over_su = flex.max(ls_shifts_over_su) > + jac_tr = self.non_linear_ls.actual.\ > + reparametrisation.jacobian_transpose_matching_grad_fc() > + self.shifts_over_su = jac_tr.transpose() * ls_shifts_over_su > + self.max_shift_over_su = flex.max(self.shifts_over_su) > + if self.max_shift_over_su < self.convergence_as_shift_over_esd: > return True > - if max_val > max_shift_over_esd: > - shift_scale = max_shift_over_esd/max_val > + if self.max_ls_shift_over_su > limit_shift_over_su: > + shift_scale = limit_shift_over_su/self.max_ls_shift_over_su > x *= shift_scale > return False > > > ------------------------------ > > To unsubscribe from the CCTBX-COMMIT list, click the following link: > https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCTBX-COMMIT&A=1 >

2 3