Re: [cctbxbb] Tidy up CCTBX

Hi Aaron, As far as I understand it the issue is not being able to use iotbx to read structure model/reflection files without an enforced dependency on hdf5/cbflib via dxtbx which are not required in order to do such operations. Only a relatively small part of iotbx (i.e. iotbx.detectors) seems to be dependent on dxtbx, so perhaps the quickest and easiest solution would be to make iotbx optionally dependent on dxtbx, and disable the iotbx.detectors module if hdf5 or cbflib aren't available. In contrast, making dxtbx optionally dependent on hdf5 and cbflib sounds like it would be more involved, and the functionality of dxtbx would be dramatically reduced without the ability to read hdf5/cbf files (probably the majority of diffraction images currently/going forward). Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE ________________________________ From: [email protected] [[email protected]] on behalf of Aaron Brewster [[email protected]] Sent: 26 January 2018 19:52 To: cctbx mailing list Cc: [email protected] Subject: Re: [cctbxbb] Tidy up CCTBX @Graeme, replying to your points " - I am not sure that these address the biggest problem i.e. iotbx depends_on dxtbx" Luc's original problem with iotbx depending on dxtbx was bringing in the hdf5 dependency. ISo making hdf5 optional for dxtbx would be helpful. The secondary complaint of a cyclical dependency of iotbx on dxtbx and dxtbx on iotbx is valid. Merging iotbx and dxtbx has been discussed previously and is probably the right solution, but that would take a bit of doing... " - Will we end up getting bug reports where DIALS programs can’t read e.g. Eiger, CBF images?" This depends on distribution. We are distributing builds of phenix, labelit and dials for linux and mac created by bootstrap that include hdf5 and cbflib, so there are no issues there. I don't know how or if Windows builds will be affected as I thought they were distributing HDF5 and cbflib too. I believe this change only opens up possibilities for easier distribution for applications outside of phenix, labelit and dials. Please, if Rob or Luc want to correct me, feel free :) -Aaron On Fri, Jan 26, 2018 at 4:00 AM, mailto:[email protected]> wrote: This sounds like “pip install libtbx” “pip install scitbx” Since both of these are relatively static & self contained, would make sense ISTR there was a suggestion earlier in the thread to this effect, which perhaps got lost… Best wishes Graeme

On 25 Jan 2018, at 20:41, Tristan Croll mailto:[email protected]> wrote:

My 2 cents: when talking about creating PyPI CCTBX modules it may be worthwhile to start by thinking smaller. For instance, while planning out how to interface my (largely NumPy-based by necessity) application to PHENIX, it would be an enormous help if I could, say, 'pip install flex_array' and 'pip install phil'.

On 2018-01-25 19:20, Aaron Brewster wrote:

...

I've created the issues: https://github.com/cctbx/cctbx_project/issues/125 [6] https://github.com/cctbx/cctbx_project/issues/126 [7] -Aaron On Tue, Jan 23, 2018 at 10:50 AM, Paul Adams mailto:[email protected]> wrote:

...

...

On Jan 23, 2018, at 9:53 AM, Aaron Brewster mailto:[email protected]> wrote: Hi folks, There are lots of threads in this email. Pulling a couple of

...

• The LBL team runs the libtbx/phil/tst_interface.py test nightly in buildbot as part of testing phenix, so the test is exercised (and it passed last night). Whether it should be moved to

...

• I think there are many projects that shouldn't be moved out of cctbx_project. IOTA, xfel, prime, simtbx, and dxtbx are all under active development and are useful to many crystallographers. cctbx is a toolbox after all. It just has a lot of tools. • That said, It may be worth having a discussion about fable, and maybe one or two of the others. That can be done in separate threads for each project? • I would like to break compile-time dependencies on external packages in dxtbx as identified by Rob. If the C++ code in dxtbx/iotbx that imports cbf.h and hdf5.h is moved to separate boost-python extension modules (some of this is done already), then

Hi Aaron, thanks for the email. I think your proposal to remove the compile time dependencies by moving code to extension modules is great. I think this would be greatly appreciated by Rob and others. Fable, contrary to recent rumors, is actually often used by others outside of the cctbx community. While it is true that we wrote it for mainly one task, it turns out there are lots of people with old f77 code they’d like to translate. Given that we don’t seem to have any dependency or support issues with the code I’d recommend it stays - making and supporting a separate package would be too much. Cheers, Paul them... phenix_regression is another issue (I don't think it's necessary, the has_module call seems clear to me). the SConscript can be made smarter and compile only the libraries it can find headers for. I'd be in favor of making two issues to track this work, one for cbflib and one for hdf5. That would be a step in the direction restoring the 'standaloneness' of the cctbx package.

...

-Aaron On Tue, Jan 23, 2018 at 3:16 AM, mailto:[email protected]> wrote: Hi Marcin Hmm, so in other words "his free / open source software cannot be fully tested without agreeing to this license"? No, still doesn't make sense to me. It only raises more questions. Such as: does Travis have a phenix license? Currently we don't run these tests at all - although we *should* run all tests on libtbx - because we don't have phenix installed during cctbx tests. Either the required test data should fully go into libtbx or the test should fully go into phenix. Distributing a test in two parts is no good to anyone. -Markus ________________________________________ From: [email protected]mailto:[email protected] [[email protected]mailto:[email protected]] on behalf of Marcin Wojdyr [[email protected]mailto:[email protected]] Sent: Tuesday, January 23, 2018 10:55 To: cctbx mailing list Cc: Randy Read Subject: Re: [cctbxbb] Tidy up CCTBX Hi Graeme,

...

Turns out that the assert that libtbx has no dependencies is not strictly true:

if (not libtbx.env.has_module(name="phenix")): print "phenix module not available: skipping advanced tests" return This is optional dependency (cctbx optionally uses Phenix for testing in various places) and doesn't cause problems. If you'd like to do disentangling, dxtbx-iotbx would be a good starting point as it really causes problems.

---

cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1] -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1] _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1] -- Paul Adams Division Director, Molecular Biophysics & Integrated Bioimaging, Lawrence Berkeley Lab Division Deputy for Biosciences, Advanced Light Source, Lawrence Berkeley Lab Adjunct Professor, Department of Bioengineering, U.C. Berkeley Vice President for Technology, the Joint BioEnergy Institute Laboratory Research Manager, ENIGMA Science Focus Area Building 33, Room 347 Building 978, Room 4126 Tel: 1-510-486-4225tel:1-510-486-4225 [2], Fax: 1-510-486-5909tel:1-510-486-5909 [3] http://cci.lbl.gov/paul [4] Lawrence Berkeley Laboratory 1 Cyclotron Road BLDG 33R0345 Berkeley, CA 94720, USA. Executive Assistant: Louise Benvenue [ [email protected]mailto:[email protected] ][ 1-510-495-2506tel:1-510-495-2506 [5] ] --

---

cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1] Links:

---

[1] http://phenix-online.org/mailman/listinfo/cctbxbb [2] tel:1-510-486-4225tel:1-510-486-4225 [3] tel:1-510-486-5909tel:1-510-486-5909 [4] http://cci.lbl.gov/paul [5] tel:%5B%201-510-495-2506 [6] https://github.com/cctbx/cctbx_project/issues/125 [7] https://github.com/cctbx/cctbx_project/issues/126 _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb