Tidy up CCTBX

older
libtbx.configure crashes RE: new...

Dr. Robert Oeffner

18 Jan 2018 18 Jan '18

7:56 a.m.

Hi all, I'm wondering if we could make efforts to refactor code modules in the cctbx_project so as making it have the minimum number of dependencies. This would be helpful to other users of the CCTBX that wishes to develop code on top of it. In the longer term it would also make it easier for us to deploy CCTBX as a Python wheel installer and by implication, spread the gospel of the wonders of the CCTBX. What is currently complicating this is the inclusion of the dxtbx module within the cctbx_project. It is being used by Dials. So my question is if it would be possible to move this project into the Dials code? If the dxtbx was just another small module it wouldn't be much of a problem. But the dxtbx depends on the cbflib and HDF5 components. If you want to build CCTBX from sources that's cumbersome. Why? Because CCTBX as deployed from CCI on http://cci.lbl.gov/cctbx_build/ is only built for a few select platforms. Building CCTBX on a different platform from those with bootstrap compels us to also build cbflib and HDF5 from scratch. Although this is not impossible it does increase the entry barrier for other potential users of the CCTBX who wish to incorporate the CCTBX in their programs: If all a user wants is to calculate allowed origins and symmetry sites for a given space group then an alternative to CCTBX might be preferable rather than having to massively expand their project with the seemingly unrelated HDF5 component. In other words, as it stands now the CCTBX is not straightforward for others to adopt. What do other people think? Regards, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234

Show replies by date

Graeme.Winter＠diamond.ac.uk

19 Jan 19 Jan

6:10 a.m.

Hi Rob Interesting question. I have sympathy for your point of view. Some thoughts, from a conversation here in DIALS land… (i) dxtbx is not only there to support DIALS - cctbx image viewer, bits of iotbx, cctbx.xfel also use it (I think) (ii) from where I am sat this is a wider question - if I have a look in cctbx_project I see more “non core code" than just dxtbx: Graemes-MBP-5:cctbx_project graeme$ ls README.md cootbx fftw3tbx omptbx sphinx boost_adaptbx crys3d gltbx prime spotfinder cbflib_adaptbx cudatbx iota rstbx tbxx cctbx dox iotbx scitbx ucif chiltbx dox.sphinx libtbx simtbx wxtbx clipper_adaptbx dxtbx mmtbx site.pyc xfel cma_es fable msvc9.0_include smtbx So, in here we have several things which probably do not “belong” in the world view you present below, for example fable, prime, simtbx, xfel etc. (iii) If we are visiting this stuff, should we be visiting the wider build system which I understand is one of the “blockers” for e.g. pip install cctbx I appreciate that this is a wider question than can we pull dxtbx out, but is essentially derived… Best wishes Graeme On 18 Jan 2018, at 15:56, Dr. Robert Oeffner mailto:[email protected]> wrote: Hi all, I'm wondering if we could make efforts to refactor code modules in the cctbx_project so as making it have the minimum number of dependencies. This would be helpful to other users of the CCTBX that wishes to develop code on top of it. In the longer term it would also make it easier for us to deploy CCTBX as a Python wheel installer and by implication, spread the gospel of the wonders of the CCTBX. What is currently complicating this is the inclusion of the dxtbx module within the cctbx_project. It is being used by Dials. So my question is if it would be possible to move this project into the Dials code? If the dxtbx was just another small module it wouldn't be much of a problem. But the dxtbx depends on the cbflib and HDF5 components. If you want to build CCTBX from sources that's cumbersome. Why? Because CCTBX as deployed from CCI on http://cci.lbl.gov/cctbx_build/ is only built for a few select platforms. Building CCTBX on a different platform from those with bootstrap compels us to also build cbflib and HDF5 from scratch. Although this is not impossible it does increase the entry barrier for other potential users of the CCTBX who wish to incorporate the CCTBX in their programs: If all a user wants is to calculate allowed origins and symmetry sites for a given space group then an alternative to CCTBX might be preferable rather than having to massively expand their project with the seemingly unrelated HDF5 component. In other words, as it stands now the CCTBX is not straightforward for others to adopt. What do other people think? Regards, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.htmlhttp://www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

Graeme.Winter＠diamond.ac.uk

22 Jan 22 Jan

12:58 a.m.

Hi Rob More thinking on this one, following from a little conversation over the weekend at this end If we were looking to be able to package cctbx in a more Pythonic manner - i.e. to allow pip install - would it make sense to start at the bottom rather than at the top? i.e. libtbx - which is pure Python with deliberately no dependencies - would be a good place to start? The lower level code is also more static so a safer candidate for pip installing compared with some of the more rapid changing things…

...

From a diamond perspective, there are a bunch of people here who want cctbx-without-dials-but-with-their-own-packages so being able to pip install cctbx and it's dependencies would be pretty cool…

Another happy side effect is this would allow e.g. libtbx.bootstrap to exist... Cheers Graeme On 19 Jan 2018, at 14:10, Winter, Graeme (DLSLtd,RAL,LSCI) mailto:[email protected]> wrote: Hi Rob Interesting question. I have sympathy for your point of view. Some thoughts, from a conversation here in DIALS land… (i) dxtbx is not only there to support DIALS - cctbx image viewer, bits of iotbx, cctbx.xfel also use it (I think) (ii) from where I am sat this is a wider question - if I have a look in cctbx_project I see more “non core code" than just dxtbx: Graemes-MBP-5:cctbx_project graeme$ ls README.md cootbx fftw3tbx omptbx sphinx boost_adaptbx crys3d gltbx prime spotfinder cbflib_adaptbx cudatbx iota rstbx tbxx cctbx dox iotbx scitbx ucif chiltbx dox.sphinx libtbx simtbx wxtbx clipper_adaptbx dxtbx mmtbx site.pyc xfel cma_es fable msvc9.0_include smtbx So, in here we have several things which probably do not “belong” in the world view you present below, for example fable, prime, simtbx, xfel etc. (iii) If we are visiting this stuff, should we be visiting the wider build system which I understand is one of the “blockers” for e.g. pip install cctbx I appreciate that this is a wider question than can we pull dxtbx out, but is essentially derived… Best wishes Graeme On 18 Jan 2018, at 15:56, Dr. Robert Oeffner mailto:[email protected]> wrote: Hi all, I'm wondering if we could make efforts to refactor code modules in the cctbx_project so as making it have the minimum number of dependencies. This would be helpful to other users of the CCTBX that wishes to develop code on top of it. In the longer term it would also make it easier for us to deploy CCTBX as a Python wheel installer and by implication, spread the gospel of the wonders of the CCTBX. What is currently complicating this is the inclusion of the dxtbx module within the cctbx_project. It is being used by Dials. So my question is if it would be possible to move this project into the Dials code? If the dxtbx was just another small module it wouldn't be much of a problem. But the dxtbx depends on the cbflib and HDF5 components. If you want to build CCTBX from sources that's cumbersome. Why? Because CCTBX as deployed from CCI on http://cci.lbl.gov/cctbx_build/ is only built for a few select platforms. Building CCTBX on a different platform from those with bootstrap compels us to also build cbflib and HDF5 from scratch. Although this is not impossible it does increase the entry barrier for other potential users of the CCTBX who wish to incorporate the CCTBX in their programs: If all a user wants is to calculate allowed origins and symmetry sites for a given space group then an alternative to CCTBX might be preferable rather than having to massively expand their project with the seemingly unrelated HDF5 component. In other words, as it stands now the CCTBX is not straightforward for others to adopt. What do other people think? Regards, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.htmlhttp://www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

Dr. Robert Oeffner

10:29 a.m.

Hi Graeme and others, Thank you for your replies. I was not aware of all the dependencies between the various modules within the cctbx. As I don't know much more about the cctbx than other developers I am not sure what the best course of action would be. Here are some random thoughts. Probably never gonna happen but, ideally a code freeze for 2 or 3 weeks stopping anyone from amending or adding code to the cctbx could be spent on disentangling modules from one another and the cctbx. I agree that there appear to be some other projects that do not belong to the cctbx. For instance, I think fable was conceived by Ralf as a one time conversion tool of F77 code to C++ code. Not sure if it is used anymore. As for pip install ability, yes that sounds like a good idea to achieve if it is done from the lowest level first, i.e. the libtbx submodule. Documenting how to build derived projects using the CCTBX should encourage people not to add unrelated code into the cctbx. Although SConscripts are familiar to many of us, to others SConscript or bootstrap are quite alien and opaque ways of building from sources. cheers, Rob -----Original Message----- From: [email protected] Sent: Monday, January 22, 2018 8:58 AM To: [email protected] Cc: [email protected] Subject: Re: [cctbxbb] Tidy up CCTBX Hi Rob More thinking on this one, following from a little conversation over the weekend at this end If we were looking to be able to package cctbx in a more Pythonic manner - i.e. to allow pip install - would it make sense to start at the bottom rather than at the top? i.e. libtbx - which is pure Python with deliberately no dependencies - would be a good place to start? The lower level code is also more static so a safer candidate for pip installing compared with some of the more rapid changing things…

...

From a diamond perspective, there are a bunch of people here who want cctbx-without-dials-but-with-their-own-packages so being able to pip install cctbx and it's dependencies would be pretty cool…

Graeme.Winter＠diamond.ac.uk

23 Jan 23 Jan

12:46 a.m.

Dear All, Turns out that the assert that libtbx has no dependencies is not strictly true: https://github.com/cctbx/cctbx_project/blob/master/libtbx/phil/tst_interface... # XXX sorry about the cross-import here, but I really need to test this on # something large and complex def exercise_2 (verbose=False) : if (not libtbx.env.has_module(name="phenix")): print "phenix module not available: skipping advanced tests" return from phenix.refinement import runtime import iotbx.phil from time import time phil_str = """ refinement.pdb_interpretation.secondary_structure.protein { helix { selection = "chain A and resseq 10:20" } helix { selection = "chain A and resseq 30:40" } helix { selection = "chain A and resseq 50:60" } } “"" around line 190 Maybe the down tools and disentangle has more merit than appears at first glance… Best wishes Graeme On 22 Jan 2018, at 18:29, Dr. Robert Oeffner mailto:[email protected]> wrote: Hi Graeme and others, Thank you for your replies. I was not aware of all the dependencies between the various modules within the cctbx. As I don't know much more about the cctbx than other developers I am not sure what the best course of action would be. Here are some random thoughts. Probably never gonna happen but, ideally a code freeze for 2 or 3 weeks stopping anyone from amending or adding code to the cctbx could be spent on disentangling modules from one another and the cctbx. I agree that there appear to be some other projects that do not belong to the cctbx. For instance, I think fable was conceived by Ralf as a one time conversion tool of F77 code to C++ code. Not sure if it is used anymore. As for pip install ability, yes that sounds like a good idea to achieve if it is done from the lowest level first, i.e. the libtbx submodule. Documenting how to build derived projects using the CCTBX should encourage people not to add unrelated code into the cctbx. Although SConscripts are familiar to many of us, to others SConscript or bootstrap are quite alien and opaque ways of building from sources. cheers, Rob -----Original Message----- From: [email protected]mailto:[email protected] Sent: Monday, January 22, 2018 8:58 AM To: [email protected]mailto:[email protected] Cc: [email protected]mailto:[email protected] Subject: Re: [cctbxbb] Tidy up CCTBX Hi Rob More thinking on this one, following from a little conversation over the weekend at this end If we were looking to be able to package cctbx in a more Pythonic manner - i.e. to allow pip install - would it make sense to start at the bottom rather than at the top? i.e. libtbx - which is pure Python with deliberately no dependencies - would be a good place to start? The lower level code is also more static so a safer candidate for pip installing compared with some of the more rapid changing things…

...

From a diamond perspective, there are a bunch of people here who want cctbx-without-dials-but-with-their-own-packages so being able to pip install cctbx and it's dependencies would be pretty cool…

Another happy side effect is this would allow e.g. libtbx.bootstrap to exist... Cheers Graeme On 19 Jan 2018, at 14:10, Winter, Graeme (DLSLtd,RAL,LSCI) mailto:[email protected]mailto:[email protected]> wrote: Hi Rob Interesting question. I have sympathy for your point of view. Some thoughts, from a conversation here in DIALS land… (i) dxtbx is not only there to support DIALS - cctbx image viewer, bits of iotbx, cctbx.xfel also use it (I think) (ii) from where I am sat this is a wider question - if I have a look in cctbx_project I see more “non core code" than just dxtbx: Graemes-MBP-5:cctbx_project graeme$ ls README.md cootbx fftw3tbx omptbx sphinx boost_adaptbx crys3d gltbx prime spotfinder cbflib_adaptbx cudatbx iota rstbx tbxx cctbx dox iotbx scitbx ucif chiltbx dox.sphinx libtbx simtbx wxtbx clipper_adaptbx dxtbx mmtbx site.pyc xfel cma_es fable msvc9.0_include smtbx So, in here we have several things which probably do not “belong” in the world view you present below, for example fable, prime, simtbx, xfel etc. (iii) If we are visiting this stuff, should we be visiting the wider build system which I understand is one of the “blockers” for e.g. pip install cctbx I appreciate that this is a wider question than can we pull dxtbx out, but is essentially derived… Best wishes Graeme On 18 Jan 2018, at 15:56, Dr. Robert Oeffner mailto:[email protected]mailto:[email protected]> wrote: Hi all, I'm wondering if we could make efforts to refactor code modules in the cctbx_project so as making it have the minimum number of dependencies. This would be helpful to other users of the CCTBX that wishes to develop code on top of it. In the longer term it would also make it easier for us to deploy CCTBX as a Python wheel installer and by implication, spread the gospel of the wonders of the CCTBX. What is currently complicating this is the inclusion of the dxtbx module within the cctbx_project. It is being used by Dials. So my question is if it would be possible to move this project into the Dials code? If the dxtbx was just another small module it wouldn't be much of a problem. But the dxtbx depends on the cbflib and HDF5 components. If you want to build CCTBX from sources that's cumbersome. Why? Because CCTBX as deployed from CCI on http://cci.lbl.gov/cctbx_build/ is only built for a few select platforms. Building CCTBX on a different platform from those with bootstrap compels us to also build cbflib and HDF5 from scratch. Although this is not impossible it does increase the entry barrier for other potential users of the CCTBX who wish to incorporate the CCTBX in their programs: If all a user wants is to calculate allowed origins and symmetry sites for a given space group then an alternative to CCTBX might be preferable rather than having to massively expand their project with the seemingly unrelated HDF5 component. In other words, as it stands now the CCTBX is not straightforward for others to adopt. What do other people think? Regards, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.htmlhttp://www.cimr.cam.ac.uk/investigators/read/index.html http://www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb --- This email has been checked for viruses by AVG. http://www.avg.com http://www.avg.com/ _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Marcin Wojdyr

2:55 a.m.

Hi Graeme,

...

Turns out that the assert that libtbx has no dependencies is not strictly true:

...

if (not libtbx.env.has_module(name="phenix")): print "phenix module not available: skipping advanced tests" return

markus.gerstel＠diamond.ac.uk

3:16 a.m.

Hi Marcin Hmm, so in other words "his free / open source software cannot be fully tested without agreeing to this license"? No, still doesn't make sense to me. It only raises more questions. Such as: does Travis have a phenix license? Currently we don't run these tests at all - although we *should* run all tests on libtbx - because we don't have phenix installed during cctbx tests. Either the required test data should fully go into libtbx or the test should fully go into phenix. Distributing a test in two parts is no good to anyone. -Markus ________________________________________ From: [email protected] [[email protected]] on behalf of Marcin Wojdyr [[email protected]] Sent: Tuesday, January 23, 2018 10:55 To: cctbx mailing list Cc: Randy Read Subject: Re: [cctbxbb] Tidy up CCTBX Hi Graeme,

...

Turns out that the assert that libtbx has no dependencies is not strictly true:

...

if (not libtbx.env.has_module(name="phenix")): print "phenix module not available: skipping advanced tests" return

This is optional dependency (cctbx optionally uses Phenix for testing in various places) and doesn't cause problems. If you'd like to do disentangling, dxtbx-iotbx would be a good starting point as it really causes problems. _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

Aaron Brewster

9:53 a.m.

Hi folks, There are lots of threads in this email. Pulling a couple of them... 1. The LBL team runs the libtbx/phil/tst_interface.py test nightly in buildbot as part of testing phenix, so the test is exercised (and it passed last night). Whether it should be moved to phenix_regression is another issue (I don't think it's necessary, the has_module call seems clear to me). 2. I think there are many projects that shouldn't be moved out of cctbx_project. IOTA, xfel, prime, simtbx, and dxtbx are all under active development and are useful to many crystallographers. cctbx is a toolbox after all. It just has a lot of tools. 3. That said, It may be worth having a discussion about fable, and maybe one or two of the others. That can be done in separate threads for each project? 4. I would like to break compile-time dependencies on external packages in dxtbx as identified by Rob. If the C++ code in dxtbx/iotbx that imports cbf.h and hdf5.h is moved to separate boost-python extension modules (some of this is done already), then the SConscript can be made smarter and compile only the libraries it can find headers for. I'd be in favor of making two issues to track this work, one for cbflib and one for hdf5. That would be a step in the direction restoring the 'standaloneness' of the cctbx package. -Aaron On Tue, Jan 23, 2018 at 3:16 AM, wrote:

...

Hi Marcin

Hmm, so in other words "his free / open source software cannot be fully tested without agreeing to this license"?

No, still doesn't make sense to me. It only raises more questions. Such as: does Travis have a phenix license?

Currently we don't run these tests at all - although we *should* run all tests on libtbx - because we don't have phenix installed during cctbx tests. Either the required test data should fully go into libtbx or the test should fully go into phenix. Distributing a test in two parts is no good to anyone.

-Markus

________________________________________ From: [email protected] [[email protected]] on behalf of Marcin Wojdyr [[email protected]] Sent: Tuesday, January 23, 2018 10:55 To: cctbx mailing list Cc: Randy Read Subject: Re: [cctbxbb] Tidy up CCTBX

Hi Graeme,

...
Turns out that the assert that libtbx has no dependencies is not strictly true:

...
if (not libtbx.env.has_module(name="phenix")): print "phenix module not available: skipping advanced tests" return

This is optional dependency (cctbx optionally uses Phenix for testing in various places) and doesn't cause problems. If you'd like to do disentangling, dxtbx-iotbx would be a good starting point as it really causes problems. _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Paul Adams

10:50 a.m.

Hi Aaron, thanks for the email. I think your proposal to remove the compile time dependencies by moving code to extension modules is great. I think this would be greatly appreciated by Rob and others. Fable, contrary to recent rumors, is actually often used by others outside of the cctbx community. While it is true that we wrote it for mainly one task, it turns out there are lots of people with old f77 code they’d like to translate. Given that we don’t seem to have any dependency or support issues with the code I’d recommend it stays - making and supporting a separate package would be too much. Cheers, Paul

...

On Jan 23, 2018, at 9:53 AM, Aaron Brewster wrote:

Hi folks,

There are lots of threads in this email. Pulling a couple of them... • The LBL team runs the libtbx/phil/tst_interface.py test nightly in buildbot as part of testing phenix, so the test is exercised (and it passed last night). Whether it should be moved to phenix_regression is another issue (I don't think it's necessary, the has_module call seems clear to me). • I think there are many projects that shouldn't be moved out of cctbx_project. IOTA, xfel, prime, simtbx, and dxtbx are all under active development and are useful to many crystallographers. cctbx is a toolbox after all. It just has a lot of tools. • That said, It may be worth having a discussion about fable, and maybe one or two of the others. That can be done in separate threads for each project? • I would like to break compile-time dependencies on external packages in dxtbx as identified by Rob. If the C++ code in dxtbx/iotbx that imports cbf.h and hdf5.h is moved to separate boost-python extension modules (some of this is done already), then the SConscript can be made smarter and compile only the libraries it can find headers for. I'd be in favor of making two issues to track this work, one for cbflib and one for hdf5. That would be a step in the direction restoring the 'standaloneness' of the cctbx package. -Aaron

On Tue, Jan 23, 2018 at 3:16 AM, wrote: Hi Marcin

Hmm, so in other words "his free / open source software cannot be fully tested without agreeing to this license"?

No, still doesn't make sense to me. It only raises more questions. Such as: does Travis have a phenix license?

Currently we don't run these tests at all - although we *should* run all tests on libtbx - because we don't have phenix installed during cctbx tests. Either the required test data should fully go into libtbx or the test should fully go into phenix. Distributing a test in two parts is no good to anyone.

-Markus

________________________________________ From: [email protected] [[email protected]] on behalf of Marcin Wojdyr [[email protected]] Sent: Tuesday, January 23, 2018 10:55 To: cctbx mailing list Cc: Randy Read Subject: Re: [cctbxbb] Tidy up CCTBX

Hi Graeme,

...
Turns out that the assert that libtbx has no dependencies is not strictly true:

...
if (not libtbx.env.has_module(name="phenix")): print "phenix module not available: skipping advanced tests" return

This is optional dependency (cctbx optionally uses Phenix for testing in various places) and doesn't cause problems. If you'd like to do disentangling, dxtbx-iotbx would be a good starting point as it really causes problems. _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Aaron Brewster

25 Jan 25 Jan

11:20 a.m.

I've created the issues: https://github.com/cctbx/cctbx_project/issues/125 https://github.com/cctbx/cctbx_project/issues/126 -Aaron On Tue, Jan 23, 2018 at 10:50 AM, Paul Adams wrote:

...

Hi Aaron,

thanks for the email. I think your proposal to remove the compile time dependencies by moving code to extension modules is great. I think this would be greatly appreciated by Rob and others. Fable, contrary to recent rumors, is actually often used by others outside of the cctbx community. While it is true that we wrote it for mainly one task, it turns out there are lots of people with old f77 code they’d like to translate. Given that we don’t seem to have any dependency or support issues with the code I’d recommend it stays - making and supporting a separate package would be too much.

Cheers, Paul

...
On Jan 23, 2018, at 9:53 AM, Aaron Brewster wrote:

Hi folks,

There are lots of threads in this email. Pulling a couple of them... • The LBL team runs the libtbx/phil/tst_interface.py test nightly in buildbot as part of testing phenix, so the test is exercised (and it passed last night). Whether it should be moved to phenix_regression is another issue (I don't think it's necessary, the has_module call seems clear to me). • I think there are many projects that shouldn't be moved out of cctbx_project. IOTA, xfel, prime, simtbx, and dxtbx are all under active development and are useful to many crystallographers. cctbx is a toolbox after all. It just has a lot of tools. • That said, It may be worth having a discussion about fable, and maybe one or two of the others. That can be done in separate threads for each project? • I would like to break compile-time dependencies on external packages in dxtbx as identified by Rob. If the C++ code in dxtbx/iotbx that imports cbf.h and hdf5.h is moved to separate boost-python extension modules (some of this is done already), then the SConscript can be made smarter and compile only the libraries it can find headers for. I'd be in favor of making two issues to track this work, one for cbflib and one for hdf5. That would be a step in the direction restoring the 'standaloneness' of the cctbx package. -Aaron

On Tue, Jan 23, 2018 at 3:16 AM, wrote: Hi Marcin

Hmm, so in other words "his free / open source software cannot be fully tested without agreeing to this license"?

No, still doesn't make sense to me. It only raises more questions. Such as: does Travis have a phenix license?

Currently we don't run these tests at all - although we *should* run all tests on libtbx - because we don't have phenix installed during cctbx tests. Either the required test data should fully go into libtbx or the test should fully go into phenix. Distributing a test in two parts is no good to anyone.

-Markus

________________________________________ From: [email protected] [cctbxbb-bounces@phenix- online.org] on behalf of Marcin Wojdyr [[email protected]] Sent: Tuesday, January 23, 2018 10:55 To: cctbx mailing list Cc: Randy Read Subject: Re: [cctbxbb] Tidy up CCTBX

Hi Graeme,

...
Turns out that the assert that libtbx has no dependencies is not strictly true:

...
if (not libtbx.env.has_module(name="phenix")): print "phenix module not available: skipping advanced tests" return

This is optional dependency (cctbx optionally uses Phenix for testing in various places) and doesn't cause problems. If you'd like to do disentangling, dxtbx-iotbx would be a good starting point as it really causes problems. _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Paul Adams Division Director, Molecular Biophysics & Integrated Bioimaging, Lawrence Berkeley Lab Division Deputy for Biosciences, Advanced Light Source, Lawrence Berkeley Lab Adjunct Professor, Department of Bioengineering, U.C. Berkeley Vice President for Technology, the Joint BioEnergy Institute Laboratory Research Manager, ENIGMA Science Focus Area

Building 33, Room 347 Building 978, Room 4126 Tel: 1-510-486-4225, Fax: 1-510-486-5909 http://cci.lbl.gov/paul

Lawrence Berkeley Laboratory 1 Cyclotron Road BLDG 33R0345 Berkeley, CA 94720, USA.

Executive Assistant: Louise Benvenue [ [email protected] ][ 1-510-495-2506 ] --

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Tristan Croll

12:41 p.m.

My 2 cents: when talking about creating PyPI CCTBX modules it may be worthwhile to start by thinking smaller. For instance, while planning out how to interface my (largely NumPy-based by necessity) application to PHENIX, it would be an enormous help if I could, say, 'pip install flex_array' and 'pip install phil'. On 2018-01-25 19:20, Aaron Brewster wrote:

...

I've created the issues:

https://github.com/cctbx/cctbx_project/issues/125 [6]

https://github.com/cctbx/cctbx_project/issues/126 [7]

-Aaron

On Tue, Jan 23, 2018 at 10:50 AM, Paul Adams wrote:

...
Hi Aaron,

thanks for the email. I think your proposal to remove the compile time dependencies by moving code to extension modules is great. I think this would be greatly appreciated by Rob and others. Fable, contrary to recent rumors, is actually often used by others outside of the cctbx community. While it is true that we wrote it for mainly one task, it turns out there are lots of people with old f77 code they’d like to translate. Given that we don’t seem to have any dependency or support issues with the code I’d recommend it stays - making and supporting a separate package would be too much.

Cheers, Paul

...
On Jan 23, 2018, at 9:53 AM, Aaron Brewster wrote:

Hi folks,

There are lots of threads in this email. Pulling a couple of them... • The LBL team runs the libtbx/phil/tst_interface.py test nightly in buildbot as part of testing phenix, so the test is exercised (and it passed last night). Whether it should be moved to phenix_regression is another issue (I don't think it's necessary, the has_module call seems clear to me). • I think there are many projects that shouldn't be moved out of cctbx_project. IOTA, xfel, prime, simtbx, and dxtbx are all under active development and are useful to many crystallographers. cctbx is a toolbox after all. It just has a lot of tools. • That said, It may be worth having a discussion about fable, and maybe one or two of the others. That can be done in separate threads for each project? • I would like to break compile-time dependencies on external packages in dxtbx as identified by Rob. If the C++ code in dxtbx/iotbx that imports cbf.h and hdf5.h is moved to separate boost-python extension modules (some of this is done already), then the SConscript can be made smarter and compile only the libraries it can find headers for. I'd be in favor of making two issues to track this work, one for cbflib and one for hdf5. That would be a step in the direction restoring the 'standaloneness' of the cctbx package. -Aaron

On Tue, Jan 23, 2018 at 3:16 AM, wrote: Hi Marcin

Hmm, so in other words "his free / open source software cannot be fully tested without agreeing to this license"?

No, still doesn't make sense to me. It only raises more questions. Such as: does Travis have a phenix license?

Currently we don't run these tests at all - although we *should* run all tests on libtbx - because we don't have phenix installed during cctbx tests. Either the required test data should fully go into libtbx or the test should fully go into phenix. Distributing a test in two parts is no good to anyone.

-Markus

________________________________________ From: [email protected] [[email protected]] on behalf of Marcin Wojdyr [[email protected]] Sent: Tuesday, January 23, 2018 10:55 To: cctbx mailing list Cc: Randy Read Subject: Re: [cctbxbb] Tidy up CCTBX

Hi Graeme,

...
Turns out that the assert that libtbx has no dependencies is not strictly true:

...
if (not libtbx.env.has_module(name="phenix")): print "phenix module not available: skipping advanced tests" return

This is optional dependency (cctbx optionally uses Phenix for testing in various places) and doesn't cause problems. If you'd like to do disentangling, dxtbx-iotbx would be a good starting point as it really causes problems. _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1]

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1]

-- Paul Adams Division Director, Molecular Biophysics & Integrated Bioimaging, Lawrence Berkeley Lab Division Deputy for Biosciences, Advanced Light Source, Lawrence Berkeley Lab Adjunct Professor, Department of Bioengineering, U.C. Berkeley Vice President for Technology, the Joint BioEnergy Institute Laboratory Research Manager, ENIGMA Science Focus Area

Building 33, Room 347 Building 978, Room 4126 Tel: 1-510-486-4225 [2], Fax: 1-510-486-5909 [3] http://cci.lbl.gov/paul [4]

Lawrence Berkeley Laboratory 1 Cyclotron Road BLDG 33R0345 Berkeley, CA 94720, USA.

Executive Assistant: Louise Benvenue [ [email protected] ][ 1-510-495-2506 [5] ] --

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1]

Links: ------ [1] http://phenix-online.org/mailman/listinfo/cctbxbb [2] tel:1-510-486-4225 [3] tel:1-510-486-5909 [4] http://cci.lbl.gov/paul [5] tel:%5B%201-510-495-2506 [6] https://github.com/cctbx/cctbx_project/issues/125 [7] https://github.com/cctbx/cctbx_project/issues/126

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Graeme.Winter＠diamond.ac.uk

26 Jan 26 Jan

4 a.m.

This sounds like “pip install libtbx” “pip install scitbx” Since both of these are relatively static & self contained, would make sense ISTR there was a suggestion earlier in the thread to this effect, which perhaps got lost… Best wishes Graeme

...

On 25 Jan 2018, at 20:41, Tristan Croll wrote:

My 2 cents: when talking about creating PyPI CCTBX modules it may be worthwhile to start by thinking smaller. For instance, while planning out how to interface my (largely NumPy-based by necessity) application to PHENIX, it would be an enormous help if I could, say, 'pip install flex_array' and 'pip install phil'.

On 2018-01-25 19:20, Aaron Brewster wrote:

...
I've created the issues: https://github.com/cctbx/cctbx_project/issues/125 [6] https://github.com/cctbx/cctbx_project/issues/126 [7] -Aaron On Tue, Jan 23, 2018 at 10:50 AM, Paul Adams wrote:

...
...
On Jan 23, 2018, at 9:53 AM, Aaron Brewster wrote: Hi folks, There are lots of threads in this email. Pulling a couple of

...
• The LBL team runs the libtbx/phil/tst_interface.py test nightly in buildbot as part of testing phenix, so the test is exercised (and it passed last night). Whether it should be moved to

...
• I think there are many projects that shouldn't be moved out of cctbx_project. IOTA, xfel, prime, simtbx, and dxtbx are all under active development and are useful to many crystallographers. cctbx is a toolbox after all. It just has a lot of tools. • That said, It may be worth having a discussion about fable, and maybe one or two of the others. That can be done in separate threads for each project? • I would like to break compile-time dependencies on external packages in dxtbx as identified by Rob. If the C++ code in dxtbx/iotbx that imports cbf.h and hdf5.h is moved to separate boost-python extension modules (some of this is done already), then

Hi Aaron, thanks for the email. I think your proposal to remove the compile time dependencies by moving code to extension modules is great. I think this would be greatly appreciated by Rob and others. Fable, contrary to recent rumors, is actually often used by others outside of the cctbx community. While it is true that we wrote it for mainly one task, it turns out there are lots of people with old f77 code they’d like to translate. Given that we don’t seem to have any dependency or support issues with the code I’d recommend it stays - making and supporting a separate package would be too much. Cheers, Paul them... phenix_regression is another issue (I don't think it's necessary, the has_module call seems clear to me). the SConscript can be made smarter and compile only the libraries it can find headers for. I'd be in favor of making two issues to track this work, one for cbflib and one for hdf5. That would be a step in the direction restoring the 'standaloneness' of the cctbx package.

...
-Aaron On Tue, Jan 23, 2018 at 3:16 AM, wrote: Hi Marcin Hmm, so in other words "his free / open source software cannot be fully tested without agreeing to this license"? No, still doesn't make sense to me. It only raises more questions. Such as: does Travis have a phenix license? Currently we don't run these tests at all - although we *should* run all tests on libtbx - because we don't have phenix installed during cctbx tests. Either the required test data should fully go into libtbx or the test should fully go into phenix. Distributing a test in two parts is no good to anyone. -Markus ________________________________________ From: [email protected] [[email protected]] on behalf of Marcin Wojdyr [[email protected]] Sent: Tuesday, January 23, 2018 10:55 To: cctbx mailing list Cc: Randy Read Subject: Re: [cctbxbb] Tidy up CCTBX Hi Graeme,

...
Turns out that the assert that libtbx has no dependencies is not strictly true:

if (not libtbx.env.has_module(name="phenix")): print "phenix module not available: skipping advanced tests" return This is optional dependency (cctbx optionally uses Phenix for testing in various places) and doesn't cause problems. If you'd like to do disentangling, dxtbx-iotbx would be a good starting point as it really causes problems.

cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1] -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1] _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1] -- Paul Adams Division Director, Molecular Biophysics & Integrated Bioimaging, Lawrence Berkeley Lab Division Deputy for Biosciences, Advanced Light Source, Lawrence Berkeley Lab Adjunct Professor, Department of Bioengineering, U.C. Berkeley Vice President for Technology, the Joint BioEnergy Institute Laboratory Research Manager, ENIGMA Science Focus Area Building 33, Room 347 Building 978, Room 4126 Tel: 1-510-486-4225 [2], Fax: 1-510-486-5909 [3] http://cci.lbl.gov/paul [4] Lawrence Berkeley Laboratory 1 Cyclotron Road BLDG 33R0345 Berkeley, CA 94720, USA. Executive Assistant: Louise Benvenue [ [email protected] ][ 1-510-495-2506 [5] ] --

cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1] Links:

[1] http://phenix-online.org/mailman/listinfo/cctbxbb [2] tel:1-510-486-4225 [3] tel:1-510-486-5909 [4] http://cci.lbl.gov/paul [5] tel:%5B%201-510-495-2506 [6] https://github.com/cctbx/cctbx_project/issues/125 [7] https://github.com/cctbx/cctbx_project/issues/126 _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

Aaron Brewster

11:52 a.m.

@Graeme, replying to your points " - I am not sure that these address the biggest problem i.e. iotbx depends_on dxtbx" Luc's original problem with iotbx depending on dxtbx was bringing in the hdf5 dependency. ISo making hdf5 optional for dxtbx would be helpful. The secondary complaint of a cyclical dependency of iotbx on dxtbx and dxtbx on iotbx is valid. Merging iotbx and dxtbx has been discussed previously and is probably the right solution, but that would take a bit of doing... " - Will we end up getting bug reports where DIALS programs can’t read e.g. Eiger, CBF images?" This depends on distribution. We are distributing builds of phenix, labelit and dials for linux and mac created by bootstrap that include hdf5 and cbflib, so there are no issues there. I don't know how or if Windows builds will be affected as I thought they were distributing HDF5 and cbflib too. I believe this change only opens up possibilities for easier distribution for applications outside of phenix, labelit and dials. Please, if Rob or Luc want to correct me, feel free :) -Aaron On Fri, Jan 26, 2018 at 4:00 AM, wrote:

...

This sounds like “pip install libtbx” “pip install scitbx”

Since both of these are relatively static & self contained, would make sense

ISTR there was a suggestion earlier in the thread to this effect, which perhaps got lost…

Best wishes Graeme

...
On 25 Jan 2018, at 20:41, Tristan Croll wrote:

My 2 cents: when talking about creating PyPI CCTBX modules it may be worthwhile to start by thinking smaller. For instance, while planning out how to interface my (largely NumPy-based by necessity) application to PHENIX, it would be an enormous help if I could, say, 'pip install flex_array' and 'pip install phil'.

On 2018-01-25 19:20, Aaron Brewster wrote:

...
I've created the issues: https://github.com/cctbx/cctbx_project/issues/125 [6] https://github.com/cctbx/cctbx_project/issues/126 [7] -Aaron On Tue, Jan 23, 2018 at 10:50 AM, Paul Adams wrote:

...
...
On Jan 23, 2018, at 9:53 AM, Aaron Brewster wrote: Hi folks, There are lots of threads in this email. Pulling a couple of

...
• The LBL team runs the libtbx/phil/tst_interface.py test nightly in buildbot as part of testing phenix, so the test is exercised (and it passed last night). Whether it should be moved to

...
• I think there are many projects that shouldn't be moved out of cctbx_project. IOTA, xfel, prime, simtbx, and dxtbx are all under active development and are useful to many crystallographers. cctbx is a toolbox after all. It just has a lot of tools. • That said, It may be worth having a discussion about fable, and maybe one or two of the others. That can be done in separate threads for each project? • I would like to break compile-time dependencies on external packages in dxtbx as identified by Rob. If the C++ code in dxtbx/iotbx that imports cbf.h and hdf5.h is moved to separate boost-python extension modules (some of this is done already), then

Hi Aaron, thanks for the email. I think your proposal to remove the compile time dependencies by moving code to extension modules is great. I think this would be greatly appreciated by Rob and others. Fable, contrary to recent rumors, is actually often used by others outside of the cctbx community. While it is true that we wrote it for mainly one task, it turns out there are lots of people with old f77 code they’d like to translate. Given that we don’t seem to have any dependency or support issues with the code I’d recommend it stays - making and supporting a separate package would be too much. Cheers, Paul them... phenix_regression is another issue (I don't think it's necessary, the has_module call seems clear to me). the SConscript can be made smarter and compile only the libraries it can find headers for. I'd be in favor of making two issues to track this work, one for cbflib and one for hdf5. That would be a step in the direction restoring the 'standaloneness' of the cctbx package.

...
-Aaron On Tue, Jan 23, 2018 at 3:16 AM, wrote: Hi Marcin Hmm, so in other words "his free / open source software cannot be fully tested without agreeing to this license"? No, still doesn't make sense to me. It only raises more questions. Such as: does Travis have a phenix license? Currently we don't run these tests at all - although we *should* run all tests on libtbx - because we don't have phenix installed during cctbx tests. Either the required test data should fully go into libtbx or the test should fully go into phenix. Distributing a test in two parts is no good to anyone. -Markus ________________________________________ From: [email protected] [[email protected]] on behalf of Marcin Wojdyr [[email protected]] Sent: Tuesday, January 23, 2018 10:55 To: cctbx mailing list Cc: Randy Read Subject: Re: [cctbxbb] Tidy up CCTBX Hi Graeme,

...
Turns out that the assert that libtbx has no dependencies is not strictly true:

if (not libtbx.env.has_module(name="phenix")): print "phenix module not available: skipping advanced tests" return This is optional dependency (cctbx optionally uses Phenix for testing in various places) and doesn't cause problems. If you'd like to do disentangling, dxtbx-iotbx would be a good starting point as it really causes problems.

cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1] -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1] _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1] -- Paul Adams Division Director, Molecular Biophysics & Integrated Bioimaging, Lawrence Berkeley Lab Division Deputy for Biosciences, Advanced Light Source, Lawrence Berkeley Lab Adjunct Professor, Department of Bioengineering, U.C. Berkeley Vice President for Technology, the Joint BioEnergy Institute Laboratory Research Manager, ENIGMA Science Focus Area Building 33, Room 347 Building 978, Room 4126 Tel: 1-510-486-4225 [2], Fax: 1-510-486-5909 [3] http://cci.lbl.gov/paul [4] Lawrence Berkeley Laboratory 1 Cyclotron Road BLDG 33R0345 Berkeley, CA 94720, USA. Executive Assistant: Louise Benvenue [ [email protected] ][ 1-510-495-2506 [5] ] --

cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1] Links:

[1] http://phenix-online.org/mailman/listinfo/cctbxbb [2] tel:1-510-486-4225 [3] tel:1-510-486-5909 [4] http://cci.lbl.gov/paul [5] tel:%5B%201-510-495-2506 [6] https://github.com/cctbx/cctbx_project/issues/125 [7] https://github.com/cctbx/cctbx_project/issues/126 _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

richard.gildea＠diamond.ac.uk

3:08 p.m.

Hi Aaron, As far as I understand it the issue is not being able to use iotbx to read structure model/reflection files without an enforced dependency on hdf5/cbflib via dxtbx which are not required in order to do such operations. Only a relatively small part of iotbx (i.e. iotbx.detectors) seems to be dependent on dxtbx, so perhaps the quickest and easiest solution would be to make iotbx optionally dependent on dxtbx, and disable the iotbx.detectors module if hdf5 or cbflib aren't available. In contrast, making dxtbx optionally dependent on hdf5 and cbflib sounds like it would be more involved, and the functionality of dxtbx would be dramatically reduced without the ability to read hdf5/cbf files (probably the majority of diffraction images currently/going forward). Cheers, Richard Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078 Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE ________________________________ From: [email protected] [[email protected]] on behalf of Aaron Brewster [[email protected]] Sent: 26 January 2018 19:52 To: cctbx mailing list Cc: [email protected] Subject: Re: [cctbxbb] Tidy up CCTBX @Graeme, replying to your points " - I am not sure that these address the biggest problem i.e. iotbx depends_on dxtbx" Luc's original problem with iotbx depending on dxtbx was bringing in the hdf5 dependency. ISo making hdf5 optional for dxtbx would be helpful. The secondary complaint of a cyclical dependency of iotbx on dxtbx and dxtbx on iotbx is valid. Merging iotbx and dxtbx has been discussed previously and is probably the right solution, but that would take a bit of doing... " - Will we end up getting bug reports where DIALS programs can’t read e.g. Eiger, CBF images?" This depends on distribution. We are distributing builds of phenix, labelit and dials for linux and mac created by bootstrap that include hdf5 and cbflib, so there are no issues there. I don't know how or if Windows builds will be affected as I thought they were distributing HDF5 and cbflib too. I believe this change only opens up possibilities for easier distribution for applications outside of phenix, labelit and dials. Please, if Rob or Luc want to correct me, feel free :) -Aaron On Fri, Jan 26, 2018 at 4:00 AM, mailto:[email protected]> wrote: This sounds like “pip install libtbx” “pip install scitbx” Since both of these are relatively static & self contained, would make sense ISTR there was a suggestion earlier in the thread to this effect, which perhaps got lost… Best wishes Graeme

...

On 25 Jan 2018, at 20:41, Tristan Croll mailto:[email protected]> wrote:

My 2 cents: when talking about creating PyPI CCTBX modules it may be worthwhile to start by thinking smaller. For instance, while planning out how to interface my (largely NumPy-based by necessity) application to PHENIX, it would be an enormous help if I could, say, 'pip install flex_array' and 'pip install phil'.

On 2018-01-25 19:20, Aaron Brewster wrote:

...
I've created the issues: https://github.com/cctbx/cctbx_project/issues/125 [6] https://github.com/cctbx/cctbx_project/issues/126 [7] -Aaron On Tue, Jan 23, 2018 at 10:50 AM, Paul Adams mailto:[email protected]> wrote:

...
...
On Jan 23, 2018, at 9:53 AM, Aaron Brewster mailto:[email protected]> wrote: Hi folks, There are lots of threads in this email. Pulling a couple of

...
• The LBL team runs the libtbx/phil/tst_interface.py test nightly in buildbot as part of testing phenix, so the test is exercised (and it passed last night). Whether it should be moved to

...
• I think there are many projects that shouldn't be moved out of cctbx_project. IOTA, xfel, prime, simtbx, and dxtbx are all under active development and are useful to many crystallographers. cctbx is a toolbox after all. It just has a lot of tools. • That said, It may be worth having a discussion about fable, and maybe one or two of the others. That can be done in separate threads for each project? • I would like to break compile-time dependencies on external packages in dxtbx as identified by Rob. If the C++ code in dxtbx/iotbx that imports cbf.h and hdf5.h is moved to separate boost-python extension modules (some of this is done already), then

Hi Aaron, thanks for the email. I think your proposal to remove the compile time dependencies by moving code to extension modules is great. I think this would be greatly appreciated by Rob and others. Fable, contrary to recent rumors, is actually often used by others outside of the cctbx community. While it is true that we wrote it for mainly one task, it turns out there are lots of people with old f77 code they’d like to translate. Given that we don’t seem to have any dependency or support issues with the code I’d recommend it stays - making and supporting a separate package would be too much. Cheers, Paul them... phenix_regression is another issue (I don't think it's necessary, the has_module call seems clear to me). the SConscript can be made smarter and compile only the libraries it can find headers for. I'd be in favor of making two issues to track this work, one for cbflib and one for hdf5. That would be a step in the direction restoring the 'standaloneness' of the cctbx package.

...
-Aaron On Tue, Jan 23, 2018 at 3:16 AM, mailto:[email protected]> wrote: Hi Marcin Hmm, so in other words "his free / open source software cannot be fully tested without agreeing to this license"? No, still doesn't make sense to me. It only raises more questions. Such as: does Travis have a phenix license? Currently we don't run these tests at all - although we *should* run all tests on libtbx - because we don't have phenix installed during cctbx tests. Either the required test data should fully go into libtbx or the test should fully go into phenix. Distributing a test in two parts is no good to anyone. -Markus ________________________________________ From: [email protected]mailto:[email protected] [[email protected]mailto:[email protected]] on behalf of Marcin Wojdyr [[email protected]mailto:[email protected]] Sent: Tuesday, January 23, 2018 10:55 To: cctbx mailing list Cc: Randy Read Subject: Re: [cctbxbb] Tidy up CCTBX Hi Graeme,

...
Turns out that the assert that libtbx has no dependencies is not strictly true:

if (not libtbx.env.has_module(name="phenix")): print "phenix module not available: skipping advanced tests" return This is optional dependency (cctbx optionally uses Phenix for testing in various places) and doesn't cause problems. If you'd like to do disentangling, dxtbx-iotbx would be a good starting point as it really causes problems.

cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1] -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1] _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1] -- Paul Adams Division Director, Molecular Biophysics & Integrated Bioimaging, Lawrence Berkeley Lab Division Deputy for Biosciences, Advanced Light Source, Lawrence Berkeley Lab Adjunct Professor, Department of Bioengineering, U.C. Berkeley Vice President for Technology, the Joint BioEnergy Institute Laboratory Research Manager, ENIGMA Science Focus Area Building 33, Room 347 Building 978, Room 4126 Tel: 1-510-486-4225tel:1-510-486-4225 [2], Fax: 1-510-486-5909tel:1-510-486-5909 [3] http://cci.lbl.gov/paul [4] Lawrence Berkeley Laboratory 1 Cyclotron Road BLDG 33R0345 Berkeley, CA 94720, USA. Executive Assistant: Louise Benvenue [ [email protected]mailto:[email protected] ][ 1-510-495-2506tel:1-510-495-2506 [5] ] --

cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1] Links:

[1] http://phenix-online.org/mailman/listinfo/cctbxbb [2] tel:1-510-486-4225tel:1-510-486-4225 [3] tel:1-510-486-5909tel:1-510-486-5909 [4] http://cci.lbl.gov/paul [5] tel:%5B%201-510-495-2506 [6] https://github.com/cctbx/cctbx_project/issues/125 [7] https://github.com/cctbx/cctbx_project/issues/126 _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Aaron Brewster

5:05 p.m.

Hi Richard, Making dxtbx an optional dependency (as Nick just did, see his commit) doesn't help us. If iotbx is configured, then dxtbx will be compiled since dxtbx is always available in cctbx_project. dxtbx would only truly be optional if it were moved out of cctbx_project, which we aren't willing to do. Again all of our distributions (phenix, labelit, dials) include cbflib and hdf5. If we made these packages optional in dxtbx, then any application who wants to use cctbx without those packages would be able to. Those applications won't be reading image data anyway (otherwise they would solve the hdf5/cbflib problem themselves). -Aaron On Fri, Jan 26, 2018 at 3:08 PM, wrote:

...

Hi Aaron,

As far as I understand it the issue is not being able to use iotbx to read structure model/reflection files without an enforced dependency on hdf5/cbflib via dxtbx which are not required in order to do such operations. Only a relatively small part of iotbx (i.e. iotbx.detectors) seems to be dependent on dxtbx, so perhaps the quickest and easiest solution would be to make iotbx optionally dependent on dxtbx, and disable the iotbx.detectors module if hdf5 or cbflib aren't available. In contrast, making dxtbx optionally dependent on hdf5 and cbflib sounds like it would be more involved, and the functionality of dxtbx would be dramatically reduced without the ability to read hdf5/cbf files (probably the majority of diffraction images currently/going forward).

Cheers,

Richard

Dr Richard Gildea Data Analysis Scientist Tel: +441235 77 8078

Diamond Light Source Ltd. Diamond House Harwell Science & Innovation Campus Didcot Oxfordshire OX11 0DE ________________________________ From: [email protected] [[email protected]] on behalf of Aaron Brewster [[email protected]] Sent: 26 January 2018 19:52 To: cctbx mailing list Cc: [email protected] Subject: Re: [cctbxbb] Tidy up CCTBX

@Graeme, replying to your points

" - I am not sure that these address the biggest problem i.e. iotbx depends_on dxtbx"

Luc's original problem with iotbx depending on dxtbx was bringing in the hdf5 dependency. ISo making hdf5 optional for dxtbx would be helpful. The secondary complaint of a cyclical dependency of iotbx on dxtbx and dxtbx on iotbx is valid. Merging iotbx and dxtbx has been discussed previously and is probably the right solution, but that would take a bit of doing...

" - Will we end up getting bug reports where DIALS programs can’t read e.g. Eiger, CBF images?"

This depends on distribution. We are distributing builds of phenix, labelit and dials for linux and mac created by bootstrap that include hdf5 and cbflib, so there are no issues there. I don't know how or if Windows builds will be affected as I thought they were distributing HDF5 and cbflib too. I believe this change only opens up possibilities for easier distribution for applications outside of phenix, labelit and dials. Please, if Rob or Luc want to correct me, feel free :)

-Aaron

On Fri, Jan 26, 2018 at 4:00 AM, > wrote: This sounds like “pip install libtbx” “pip install scitbx”

Since both of these are relatively static & self contained, would make sense

ISTR there was a suggestion earlier in the thread to this effect, which perhaps got lost…

Best wishes Graeme

...
On 25 Jan 2018, at 20:41, Tristan Croll > wrote:

My 2 cents: when talking about creating PyPI CCTBX modules it may be worthwhile to start by thinking smaller. For instance, while planning out how to interface my (largely NumPy-based by necessity) application to PHENIX, it would be an enormous help if I could, say, 'pip install flex_array' and 'pip install phil'.

...
I've created the issues: https://github.com/cctbx/cctbx_project/issues/125 [6] https://github.com/cctbx/cctbx_project/issues/126 [7] -Aaron On Tue, Jan 23, 2018 at 10:50 AM, Paul Adams
...
...
...
On Jan 23, 2018, at 9:53 AM, Aaron Brewster mailto:[email protected]> wrote: Hi folks, There are lots of threads in this email. Pulling a couple of

...
• The LBL team runs the libtbx/phil/tst_interface.py test nightly in buildbot as part of testing phenix, so the test is exercised (and it passed last night). Whether it should be moved to

...
• I think there are many projects that shouldn't be moved out of cctbx_project. IOTA, xfel, prime, simtbx, and dxtbx are all under active development and are useful to many crystallographers. cctbx is a toolbox after all. It just has a lot of tools. • That said, It may be worth having a discussion about fable, and maybe one or two of the others. That can be done in separate threads for each project? • I would like to break compile-time dependencies on external packages in dxtbx as identified by Rob. If the C++ code in dxtbx/iotbx that imports cbf.h and hdf5.h is moved to separate boost-python extension modules (some of this is done already), then

Hi Aaron, thanks for the email. I think your proposal to remove the compile time dependencies by moving code to extension modules is great. I think this would be greatly appreciated by Rob and others. Fable, contrary to recent rumors, is actually often used by others outside of the cctbx community. While it is true that we wrote it for mainly one task, it turns out there are lots of people with old f77 code they’d like to translate. Given that we don’t seem to have any dependency or support issues with the code I’d recommend it stays - making and supporting a separate package would be too much. Cheers, Paul them... phenix_regression is another issue (I don't think it's necessary, the has_module call seems clear to me). the SConscript can be made smarter and compile only the libraries it can find headers for. I'd be in favor of making two issues to track this work, one for cbflib and one for hdf5. That would be a step in the direction restoring the 'standaloneness' of the cctbx package.

...
-Aaron On Tue, Jan 23, 2018 at 3:16 AM, > wrote: Hi Marcin Hmm, so in other words "his free / open source software cannot be fully tested without agreeing to this license"? No, still doesn't make sense to me. It only raises more questions. Such as: does Travis have a phenix license? Currently we don't run these tests at all - although we *should* run all tests on libtbx - because we don't have phenix installed during cctbx tests. Either the required test data should fully go into libtbx or the test should fully go into phenix. Distributing a test in two parts is no good to anyone. -Markus ________________________________________ From: [email protected]
On 2018-01-25 19:20, Aaron Brewster wrote: [email protected]>> wrote: phenix-online.org>

...
...
[[email protected]] on behalf of Marcin Wojdyr [[email protected]mailto:[email protected]]

...
Sent: Tuesday, January 23, 2018 10:55 To: cctbx mailing list Cc: Randy Read Subject: Re: [cctbxbb] Tidy up CCTBX Hi Graeme,

...
Turns out that the assert that libtbx has no dependencies is not strictly true:

if (not libtbx.env.has_module(name="phenix")): print "phenix module not available: skipping advanced tests" return This is optional dependency (cctbx optionally uses Phenix for testing in various places) and doesn't cause problems. If you'd like to do disentangling, dxtbx-iotbx would be a good starting point as it really causes problems.

cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1] -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1] _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1] -- Paul Adams Division Director, Molecular Biophysics & Integrated Bioimaging, Lawrence Berkeley Lab Division Deputy for Biosciences, Advanced Light Source, Lawrence Berkeley Lab Adjunct Professor, Department of Bioengineering, U.C. Berkeley Vice President for Technology, the Joint BioEnergy Institute Laboratory Research Manager, ENIGMA Science Focus Area Building 33, Room 347 Building 978, Room 4126 Tel: 1-510-486-4225tel:1-510-486-4225 [2], Fax: 1-510-486-5909 [3] http://cci.lbl.gov/paul [4] Lawrence Berkeley Laboratory 1 Cyclotron Road BLDG 33R0345 Berkeley, CA 94720, USA. Executive Assistant: Louise Benvenue [ [email protected] ][ 1-510-495-2506tel:1-510-495-2506 [5] ] --

cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb [1] Links:

[1] http://phenix-online.org/mailman/listinfo/cctbxbb [2] tel:1-510-486-4225tel:1-510-486-4225 [3] tel:1-510-486-5909tel:1-510-486-5909 [4] http://cci.lbl.gov/paul [5] tel:%5B%201-510-495-2506 [6] https://github.com/cctbx/cctbx_project/issues/125 [7] https://github.com/cctbx/cctbx_project/issues/126 _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Graeme.Winter＠diamond.ac.uk

3:58 a.m.

Folks, Two concerns with these: - I am not sure that these address the biggest problem i.e. iotbx depends_on dxtbx - Will we end up getting bug reports where DIALS programs can’t read e.g. Eiger, CBF images? The latter worries me more, as it could be a tricky one to diagnose… Best wishes Graeme On 25 Jan 2018, at 19:20, Aaron Brewster mailto:[email protected]> wrote: I've created the issues: https://github.com/cctbx/cctbx_project/issues/125 https://github.com/cctbx/cctbx_project/issues/126 -Aaron On Tue, Jan 23, 2018 at 10:50 AM, Paul Adams mailto:[email protected]> wrote: Hi Aaron, thanks for the email. I think your proposal to remove the compile time dependencies by moving code to extension modules is great. I think this would be greatly appreciated by Rob and others. Fable, contrary to recent rumors, is actually often used by others outside of the cctbx community. While it is true that we wrote it for mainly one task, it turns out there are lots of people with old f77 code they’d like to translate. Given that we don’t seem to have any dependency or support issues with the code I’d recommend it stays - making and supporting a separate package would be too much. Cheers, Paul

...

On Jan 23, 2018, at 9:53 AM, Aaron Brewster mailto:[email protected]> wrote:

Hi folks,

There are lots of threads in this email. Pulling a couple of them... • The LBL team runs the libtbx/phil/tst_interface.py test nightly in buildbot as part of testing phenix, so the test is exercised (and it passed last night). Whether it should be moved to phenix_regression is another issue (I don't think it's necessary, the has_module call seems clear to me). • I think there are many projects that shouldn't be moved out of cctbx_project. IOTA, xfel, prime, simtbx, and dxtbx are all under active development and are useful to many crystallographers. cctbx is a toolbox after all. It just has a lot of tools. • That said, It may be worth having a discussion about fable, and maybe one or two of the others. That can be done in separate threads for each project? • I would like to break compile-time dependencies on external packages in dxtbx as identified by Rob. If the C++ code in dxtbx/iotbx that imports cbf.h and hdf5.h is moved to separate boost-python extension modules (some of this is done already), then the SConscript can be made smarter and compile only the libraries it can find headers for. I'd be in favor of making two issues to track this work, one for cbflib and one for hdf5. That would be a step in the direction restoring the 'standaloneness' of the cctbx package. -Aaron

On Tue, Jan 23, 2018 at 3:16 AM, mailto:[email protected]> wrote: Hi Marcin

Hmm, so in other words "his free / open source software cannot be fully tested without agreeing to this license"?

No, still doesn't make sense to me. It only raises more questions. Such as: does Travis have a phenix license?

Currently we don't run these tests at all - although we *should* run all tests on libtbx - because we don't have phenix installed during cctbx tests. Either the required test data should fully go into libtbx or the test should fully go into phenix. Distributing a test in two parts is no good to anyone.

-Markus

________________________________________ From: [email protected]mailto:[email protected] [[email protected]mailto:[email protected]] on behalf of Marcin Wojdyr [[email protected]mailto:[email protected]] Sent: Tuesday, January 23, 2018 10:55 To: cctbx mailing list Cc: Randy Read Subject: Re: [cctbxbb] Tidy up CCTBX

Hi Graeme,

...
Turns out that the assert that libtbx has no dependencies is not strictly true:

...
if (not libtbx.env.has_module(name="phenix")): print "phenix module not available: skipping advanced tests" return

This is optional dependency (cctbx optionally uses Phenix for testing in various places) and doesn't cause problems. If you'd like to do disentangling, dxtbx-iotbx would be a good starting point as it really causes problems. _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Paul Adams Division Director, Molecular Biophysics & Integrated Bioimaging, Lawrence Berkeley Lab Division Deputy for Biosciences, Advanced Light Source, Lawrence Berkeley Lab Adjunct Professor, Department of Bioengineering, U.C. Berkeley Vice President for Technology, the Joint BioEnergy Institute Laboratory Research Manager, ENIGMA Science Focus Area Building 33, Room 347 Building 978, Room 4126 Tel: 1-510-486-4225tel:1-510-486-4225, Fax: 1-510-486-5909tel:1-510-486-5909 http://cci.lbl.gov/paul Lawrence Berkeley Laboratory 1 Cyclotron Road BLDG 33R0345 Berkeley, CA 94720, USA. Executive Assistant: Louise Benvenue [ [email protected]mailto:[email protected] ][ 1-510-495-2506tel:%5B%201-510-495-2506 ] -- _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Marcin Wojdyr

19 Jan 19 Jan

8:02 a.m.

Hi Rob,

...

What is currently complicating this is the inclusion of the dxtbx module within the cctbx_project.

and in particular that iotbx depends on it, as it was discussed half a year ago in a thread: "shall iotbx really depends on dxtbx?"

...

If all a user wants is to calculate allowed origins and symmetry sites for a given space group then an alternative to CCTBX might be preferable rather than having to massively expand their project [...]

Yes! Please don't change it :-) It's one of the selling points for a library I'm working on ( https://project-gemmi.github.io/ ). Cheers, Marcin

2561

Age (days ago)

2570

Last active (days ago)

List overview

Download

16 comments

8 participants

participants (8)

Aaron Brewster
Dr. Robert Oeffner
Graeme.Winter＠diamond.ac.uk
Marcin Wojdyr
markus.gerstel＠diamond.ac.uk
Paul Adams
richard.gildea＠diamond.ac.uk
Tristan Croll

Tidy up CCTBX

Dr. Robert Oeffner

Dr. Robert Oeffner

markus.gerstel＠diamond.ac.uk

tags

participants (8)