- cctbxbb - phenix-online.org

pickling nan values
by Robert Oeffner 13 Jun '19

13 Jun '19

Hi all, It seems that a flex.double array containing "nan" values doesn't survive pickling as in: oeffner@grove:~$ cctbx.python Python 2.7.15 (default, Sep 11 2018, 23:48:45) [GCC 4.4.7 20120313 (Red Hat 4.4.7-18)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> >>> from cctbx.array_family import flex >>> import pickle >>> flex_nans = flex.double(3, float('nan')) >>> list(flex_nans) [nan, nan, nan] >>> list(pickle.loads( pickle.dumps(flex_nans))) [0.0, 0.0, 0.0] >>> >>> If the flex double is cast into a python list then it does survive pickling as in: >>> >>> list(pickle.loads( pickle.dumps(list( flex_nans )))) [nan, nan, nan] >>> I would like to use nan values to indicate missing or invalid data values. Would anyone have a suggestion how to go about this? Many thanks, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus The Keith Peters Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

2 1

Fwd: Conda/DIALS 1.14.5 source build with --use-conda does not work
by Nicholas Devenish 27 May '19

27 May '19

(Forwarding this manually as apparently my Diamond address isn't on cctbxbb) Hi All, On 24 May 2019, at 14:12, David Waterman - UKRI STFC < david.waterman(a)stfc.ac.uk<mailto:[email protected]>> wrote: > In the process, I discovered that the source builds we point to from > https://dials.github.io/installation.html don't work with --use-conda > because the paths to various packages are out of date. This seems to be to be a fundamental problem with the conda implementation? Let’s try to reproduce an old build from, say, 1st march: the point in cctbx_project at which dials 1.14 split off. This is https://github.com/cctbx/cctbx_project/commit/35652e06564da7ff989500a6d929b… . Try reproducing this point: git clone git(a)github.com<mailto:[email protected]>:cctbx/cctbx_project.git modules/cctbx_project ( cd modules/cctbx_project/ && git checkout 35652e06564da7ff989500a6d929bc595f69040f ) ln -s modules/cctbx_project/libtbx/auto_build/bootstrap.py . ./bootstrap.py base --use-conda —> many "An HTTP error occurred when trying to retrieve this URL.” This means that with the current conda implementation it’s impossible to guarantee reproducibility for any build beyond the past few days, without lots and lots of manual work reconstructing the versioning out of the file list dump. Because the version constraints aren’t kept with the repository, this means that a source distribution is not nearly enough to recreate any given release. This seems like a problem? Nick

3 2

Python3 / scons / non-conda builds
by Graeme.Winter＠Diamond.ac.uk 13 May '19

13 May '19

Dear All, Back in the mists of time, one could build cctbx modules without bootstrap or conda by having the right modules available yourself - a so-called baseless install which I think is still important in X-fel land? Anyhow, trying this with Python3 suggests that we have some special constraints which are perhaps too tight regarding module locations - for example scons - Grey-Area p3_compile :( $ pip3 install scons Collecting scons Downloading https://files.pythonhosted.org/packages/65/41/ac58638421414acc64e26fc120c81… (831kB) 100% |████████████████████████████████| 839kB 14.3MB/s Installing collected packages: scons Successfully installed scons-3.0.5 You are using pip version 19.0.1, however version 19.1.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. Grey-Area p3_compile :) $ make ./bin/libtbx.scons -j "`./bin/libtbx.show_number_of_processors`" ModuleNotFoundError: No module named 'SCons' During handling of the above exception, another exception occurred: Sorry: SCons is not available. A possible solution is to unpack a SCons distribution in one of these directories: "/Users/graeme/svn/scr" "/Users/graeme/svn/scr/cctbx_project" SCons distributions are available at this location: http://www.scons.org/ It may be necessary to rename the unpacked distribution, e.g.: mv scons-0.96.1 scons make: *** [default] Error 1 Grey-Area p3_compile :( $ which scons /Library/Frameworks/Python.framework/Versions/3.7/bin/scons Should we require that scons is installed as part of cctbx when it is already installed on the system? If we *do* have to have this constraint, maybe that should be checked at configure time (where it complains about six and future) Thoughts? In other news - the syntax changing branch referenced in https://github.com/cctbx/cctbx_project/pull/325 now compiles correctly on Python3 (note: not run, just compile but this means that it’s purpose is done) for the modules which it intends to touch and has no conflicts right now… Best wishes Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

2 2

Overloading Python builtin names with local variables
by Graeme.Winter＠Diamond.ac.uk 03 May '19

03 May '19

Morning all, As a side-effect of looking at some of the Python3 refactoring I stumbled across a few places where e.g. range is used as a variable name. While this is fine and legal, when moving the code to Python3 xrange => range and from builtins import range is a back-port of Python3 range to Python2 which allows the existing behaviour of xrange generator to be maintained This is fine except for where range is a variable name and we then get a “variable referenced before assignment” type error As a general statement I would say overloading Python reserved names with local variables is untidy at best, and can easily cause confusion (as well as real problems as identified above) - I have therefore taken the liberty of adding a tool to libtbx which can be used to find such variables to give people a chance to avoid them: Grey-Area dxtbx :) [master] $ libtbx.find_reserved_names format/ Checking format/ format/FormatSMVRigakuSaturn.py:239 "format" format = self._scan_factory.format("SMV") format/FormatTIFFRayonixESRF.py:36 "format" format = {LITTLE_ENDIAN: "<", BIG_ENDIAN: ">"}[order] format/FormatTIFFRayonixESRF.py:29 "bytes" width, height, depth, order, bytes = FormatTIFFRayonix.get_tiff_header( format/FormatCBFMini.py:231 "format" format = self._scan_factory.format("CBF") format/FormatTIFFRayonix.py:176 "format" format = self._scan_factory.format("TIFF") format/FormatTIFFRayonix.py:31 "bytes" width, height, depth, order, bytes = FormatTIFF.get_tiff_header(image_file) format/FormatTIFFRayonix.py:74 "bytes" width, height, depth, order, bytes = FormatTIFF.get_tiff_header(image_file) format/FormatSMVRigakuEiger.py:243 "format" format = self._scan_factory.format("SMV") format/FormatSMVADSC.py:212 "format" format = self._scan_factory.format("SMV") format/Registry.py:88 "format" for format in sorted(self._formats, key=lambda x: x.__name__): format/FormatSMVJHSim.py:141 "format" format = self._scan_factory.format("SMV") format/FormatSMVRigakuPilatus.py:188 "format" format = self._scan_factory.format("SMV") format/FormatSMVADSCSN928.py:47 "format" format = self._scan_factory.format("SMV") format/FormatRAXISIVSpring8.py:134 "format" format = self._scan_factory.format("RAXIS") format/FormatTIFFRayonixSPring8.py:31 "bytes" width, height, depth, order, bytes = FormatTIFFRayonix.get_tiff_header( format/FormatSMVADSCmlfsom.py:37 "format" format = self._scan_factory.format("SMV") format/FormatTIFFBruker.py:174 "format" format = self._scan_factory.format("TIFF") format/FormatTIFFBruker.py:31 "bytes" width, height, depth, order, bytes = FormatTIFF.get_tiff_header(image_file) format/FormatTIFFBruker.py:71 "bytes" width, height, depth, order, bytes = FormatTIFF.get_tiff_header(image_file) format/FormatCBFMiniADSCHF4M.py:26 "format" for format in ["%a_%b_%d_%H:%M:%S_%Y"]: format/FormatCBFMiniADSCHF4M.py:169 "format" format = self._scan_factory.format("CBF") format/FormatSMVADSCNoDateStamp.py:54 "format" format = self._scan_factory.format("SMV") format/FormatSMVRigakuSaturnNoTS.py:228 "format" format = self._scan_factory.format("SMV") format/FormatPYunspecified.py:152 "format" format = "" format/FormatSMVRigakuA200.py:234 "format" format = self._scan_factory.format("SMV") format/FormatCBFMiniEigerPhotonFactory.py:131 "format" format = self._scan_factory.format("CBF") format/FormatSMVTimePix_SU.py:97 "format" format = self._scan_factory.format("SMV") format/FormatCBFMiniPilatus.py:83 "format" format = self._scan_factory.format("CBF") format/FormatCBFMiniPilatusHelpers.py:22 "format" for format in ["%Y-%b-%dT%H:%M:%S", "%Y-%m-%dT%H:%M:%S", "%Y/%b/%d %H:%M:%S"]: format/FormatSMVADSCSNAPSID19.py:91 "format" format = self._scan_factory.format("SMV") format/FormatCBFMiniEiger.py:216 "format" format = self._scan_factory.format("CBF") format/FormatSMVNOIR.py:218 "format" format = self._scan_factory.format("SMV") format/FormatSMVCMOS1.py:197 "format" format = self._scan_factory.format("SMV") format/FormatRAXIS.py:238 "format" format = self._scan_factory.format("RAXIS") is an example where “format” is scattered all over the place but is also a useful function - replacing with fmt will mean the same and avoid this overloading, as an example. This uses AST parsing to work through the code and find variable assignments - doing this for dials found lots of examples of “file” and “type” being used. How much people care about this is a local issue, but I thought the tool would be useful All the best Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

1 0

modulo operation on flex.double
by Robert Oeffner 11 Apr '19

11 Apr '19

Hi, I have a flex.double array of phase values between -360 and 360 degrees I would like to cast them all in one go to lie within 0 to 360. Does anybody know if there is a nice way of doing this with flex arrays? The modulo operator doesn't seem to be supported for flex arrays unlike the +, -, * and the / operators. This would otherwise be handy as I could then convert the array in one line of code like phases = phases % 360.0 If not I guess I'll have to numpy the problem :-) Many thanks, Rob

5 5

Re: [cctbxbb] cannot compile Phenix
by R. D. Oeffner 06 Apr '19

06 Apr '19

>From the messages on the cctbx github mailing list it seems that Dials people have been extricating dxtbx from cctbx_project. Rob -----Original Message----- From: Pavel Afonine Sent: Saturday, April 6, 2019 7:07 PM To: phenixdev Subject: cannot compile Phenix Hi, anyone knows why current Phenix code is broken: python bootstrap.py --builder=phenix --sfuser=afonine --cciuser=afonine update python bootstrap.py --builder=phenix --sfuser=afonine --cciuser=afonine build --nproc=12 leads to (...) AttributeError: 'empty' object has no attribute 'dxtbx_include': File "/Users/pafonine/phenix/build/SConstruct", line 31: SConscript("spotfinder/SConscript") File "/Users/pafonine/phenix/modules/scons/engine/SCons/Script/SConscript.py", line 607: return method(*args, **kw) File "/Users/pafonine/phenix/modules/scons/engine/SCons/Script/SConscript.py", line 544: return _SConscript(self.fs, *files, **subst_kw) File "/Users/pafonine/phenix/modules/scons/engine/SCons/Script/SConscript.py", line 253: call_stack[-1].globals) File "/Users/pafonine/phenix/modules/cctbx_project/spotfinder/SConscript", line 22: env_etc.dxtbx_include, usr+sys time: 1.94 seconds wall clock time: 4.51 seconds Process failed with return code 2 Any idea for a workaround? Pavel --- This email has been checked for viruses by AVG. https://www.avg.com

2 3

jupyter notebook
by wtempel 02 Apr '19

02 Apr '19

Hi all, how does one configure, and run inside a jupyter notebook, a cctbx-enabled kernel, beginning with a downloaded [cctbx build]( http://cci.lbl.gov/cctbx_build/)? [This page]( https://medium.com/@sljack1992/making-a-custom-ipython-notebook-kernel-c59e…) seems to describe the first part, but may no longer apply to more recent cctbx builds. Thanks. Wolfram Tempel

2 1

mtz2map gridding
by Elliot Nelson 21 Feb '19

21 Feb '19

Hi All, I've been trying to use Phenix.mtz2map to sample maps to a particular array size with the "gridding" option, but found that this seems to have no effect on the output of the program. I was wondering whether there were some restrictions to the integers that are permitted or otherwise what might be the cause of this behaviour. Having hunted through the code it seems to just set n_real in the crystal gridding, which is then fed into an fft - possibly the latter ignores n_reals that aren't sensible? Any idea? Thanks, Conor on behalf of Conor Wild: conor.wild(a)oriel.ox.ac.uk Elliot DPhil Student, Systems Approaches to Biomedical Science<http://www.sabsidc.ox.ac.uk/> Protein Crystallography<http://www.thesgc.org/groupprofile/9489>, Structural Genomics Consortium, NDM, University of Oxford Oxford Protein Informatics Group<http://opig.stats.ox.ac.uk/>, Department of Statistics, University of Oxford

2 1

OpenMP in fftpack
by Keitaro Yamashita 30 Jan '19

30 Jan '19

Dear CCTBX/PHENIX developers, According to PHENIX FAQ: https://www.phenix-online.org/documentation/faqs/index.html#general > In addition to these options, it is also possible to compile phenix.refine and Phaser with the OpenMP library, which automatically parallelizes specific instructions such as the FFT. This requires using the source installer for Phenix, and adding the argument "--openmp" to the install command. Because of threading conflicts, OpenMP is not compatible with the Phenix GUI. FFT in phenix.refine can be parallelized if OpenMP is enabled. However, it seems incorrect since 2017 due to this change: https://github.com/cctbx/cctbx_project/commit/3e932248e0491847a04ddeee429f8… The FAQ may need updating, but I am curious to know why it is disabled now. Best regards, Keitaro

1 0

This is a test
by Billy Poon 28 Jan '19

28 Jan '19

The Phenix website was down for a few hours, so this is a test to see if the bulletin board is working. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

1 0