Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip ( https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-...). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not. All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support. There is also a manylinux2010 standard, which is based on Centos 6 ( https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6). If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4. Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :) Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users. Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort. Thanks, -Aaron

Hi Luc, thanks. I did recall someone working on this a while back. For conda, there are a couple more things to finish and then we hope to have cctbx available through conda. 1) There is work being done in a branch to make cctbx use boost in standard locations (e.g. the system boost, or a conda boost). That will allow us to use install boost by conda and not build it ourselves. ( https://github.com/cctbx/cctbx_project/tree/conda_boost). Also, newer versions of boost are being tested (up through 1.70). (This will also enable python 3.7 support.) 2) We need to work on getting a make install step in place so that we can build the conda package and upload it. 3) We want to split the dependencies up by builder (cctbx, cctbx-lite, dials, phenix, etc.) into meta packages and their associated manifests. I can talk more about this if there is interest. Thanks, -Aaron On Fri, Aug 16, 2019 at 1:48 PM Luc Bourhis wrote:

Hi,

I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.org, and then to advertise it hard to every potential user out there.

Best wishes,

Luc

On 16 Aug 2019, at 21:45, Aaron Brewster wrote:

Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip ( https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-...). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not.

All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support.

There is also a manylinux2010 standard, which is based on Centos 6 ( https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6).

If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4.

Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :)

Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users.

Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort.

Thanks, -Aaron

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Hi Aaron Re: talk about if interest I think it would be very useful to have a roadmap of where this is going, the intentions and how we expect it to play with other cctbx-dependent projects - have I missed this? Having it somewhere e.g. on the wiki would be a big help Thanks Graeme On 17 Aug 2019, at 00:36, Aaron Brewster mailto:[email protected]> wrote: Hi Luc, thanks. I did recall someone working on this a while back. For conda, there are a couple more things to finish and then we hope to have cctbx available through conda. 1) There is work being done in a branch to make cctbx use boost in standard locations (e.g. the system boost, or a conda boost). That will allow us to use install boost by conda and not build it ourselves. (https://github.com/cctbx/cctbx_project/tree/conda_boost). Also, newer versions of boost are being tested (up through 1.70). (This will also enable python 3.7 support.) 2) We need to work on getting a make install step in place so that we can build the conda package and upload it. 3) We want to split the dependencies up by builder (cctbx, cctbx-lite, dials, phenix, etc.) into meta packages and their associated manifests. I can talk more about this if there is interest. Thanks, -Aaron On Fri, Aug 16, 2019 at 1:48 PM Luc Bourhis mailto:[email protected]> wrote: Hi, I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.orghttp://anaconda.org/, and then to advertise it hard to every potential user out there. Best wishes, Luc On 16 Aug 2019, at 21:45, Aaron Brewster mailto:[email protected]> wrote: Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-...). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not. All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support. There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6). If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4. Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :) Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users. Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort. Thanks, -Aaron _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

Agree. I talked with a small molecules software developer here at the ECM32 yesterday. He would love to be able to do “pip install cctbx” when he develops his program, provided it doesn’t take up much file space. Rob Sent from my Windows 10 phone -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 From: Derek Mendez Sent: 20 August 2019 18:06 To: cctbx mailing list Subject: Re: [cctbxbb] some thoughts on cctbx and pip I think its worth getting a cctbx-light pip build..  I think modules like cctbx miller, sgtbx are extremely  useful. Also simtbx.nanoBragg. -Derek On Sun, Aug 18, 2019 at 10:48 PM [email protected] wrote: Hi Aaron Re: talk about if interest I think it would be very useful to have a roadmap of where this is going, the intentions and how we expect it to play with other cctbx-dependent projects - have I missed this? Having it somewhere e.g. on the wiki would be a big help Thanks Graeme On 17 Aug 2019, at 00:36, Aaron Brewster mailto:[email protected]> wrote: Hi Luc, thanks.  I did recall someone working on this a while back. For conda, there are a couple more things to finish and then we hope to have cctbx available through conda. 1) There is work being done in a branch to make cctbx use boost in standard locations (e.g. the system boost, or a conda boost).  That will allow us to use install boost by conda and not build it ourselves.  (https://github.com/cctbx/cctbx_project/tree/conda_boost).  Also, newer versions of boost are being tested (up through 1.70).  (This will also enable python 3.7 support.) 2) We need to work on getting a make install step in place so that we can build the conda package and upload it. 3) We want to split the dependencies up by builder (cctbx, cctbx-lite, dials, phenix, etc.) into meta packages and their associated manifests.  I can talk more about this if there is interest. Thanks, -Aaron On Fri, Aug 16, 2019 at 1:48 PM Luc Bourhis mailto:[email protected]> wrote: Hi, I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.orghttp://anaconda.org/, and then to advertise it hard to every potential user out there. Best wishes, Luc On 16 Aug 2019, at 21:45, Aaron Brewster mailto:[email protected]> wrote: Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip.  Pip doesn't install non-python dependencies well.  I don't think boost is available as a package on pip (at least the package version we use).  wxPython4 isn't portable through pip (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-...).  MPI libraries are system dependent.  If cctbx were a pure python package, pip would be fine, but cctbx is not. All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this).  For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/).  For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel).  Note, the manylinux1 standard is built with Centos 5.11 which we no longer support. There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/).  This is likely a more attainable target (note though by default C++11 is not supported on Centos 6). If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies.  There's no way to specify these in pip.  For example, cctbx requires boost 1.63 or better.  The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library.  We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4. Further, we need to think about how dials and other cctbx-based packages interact.  If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries?  Can shared libraries from one pip package link against libraries in another pip package?  Would each package need to supply its own boost?  Possibly this is well understood in the pip field, but not by me :) Finally, there's the option of providing a source pip package.  This would require the full compiler toolchain for any given platform (macOS, linux, windows).  These are likely available for developers, but not for general users. Anyway, these are some of the obstacles.  Not saying it isn't possible, it's just a lot of effort. Thanks, -Aaron _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Hi, Even if we managed to ship our the boost dynamic libraries with pip, it would still not be pip-like, as we would still need our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work with the standard python exe. LD_LIBRARY_PATH, we could get around that by changing the way we compile, using -Wl,-R, which is the runtime equivalent of build time -L. That’s a significant change that would need to be tested. But there is no way around setting LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps there is a way to hack libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip installs? Best wishes, Luc

On 16 Aug 2019, at 22:47, Luc Bourhis wrote:

Hi,

I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.org http://anaconda.org/, and then to advertise it hard to every potential user out there.

Best wishes,

Luc

...

On 16 Aug 2019, at 21:45, Aaron Brewster mailto:[email protected]> wrote:

Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-... https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-...). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not.

All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/ https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support.

There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/ https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6).

If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4.

Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :)

Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users.

Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort.

Thanks, -Aaron

_______________________________________________ cctbxbb mailing list [email protected] mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Without discussing the merits of this or whether we _choose_ to make the move to supporting PIP, I am certain it would be _possible_ - many other packages make dispatcher scripts when you pip install them e.g. Silver-Surfer rescale_f2 :) $ which black; cat $(which black) /Library/Frameworks/Python.framework/Versions/3.6/bin/black #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 # -*- coding: utf-8 -*- import re import sys from black import main if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) sys.exit(main()) So we _could_ work around the absence of LIBTBX_BUILD etc. in the system. Whether or not we elect to do the work is a different question, and it seems clear that here are very mixed opinions on this. Best wishes Graeme On 23 Aug 2019, at 01:21, Luc Bourhis mailto:[email protected]> wrote: Hi, Even if we managed to ship our the boost dynamic libraries with pip, it would still not be pip-like, as we would still need our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work with the standard python exe. LD_LIBRARY_PATH, we could get around that by changing the way we compile, using -Wl,-R, which is the runtime equivalent of build time -L. That’s a significant change that would need to be tested. But there is no way around setting LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps there is a way to hack libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip installs? Best wishes, Luc On 16 Aug 2019, at 22:47, Luc Bourhis mailto:[email protected]> wrote: Hi, I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.orghttp://anaconda.org/, and then to advertise it hard to every potential user out there. Best wishes, Luc On 16 Aug 2019, at 21:45, Aaron Brewster mailto:[email protected]> wrote: Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-...). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not. All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support. There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6). If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4. Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :) Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users. Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort. Thanks, -Aaron _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

To add my two cents on this: probably the second-most common question I've had about ISOLDE's implementation is, "why didn't you use CCTBX?". The honest answer to that is, "I didn't know how." Still don't, really - although the current developments are rather promising. The problem I've faced is that CCTBX was designed as its own self-contained Python (2.7, until very recently) environment, with its own interpreter and a lot of very specific environment setup. Meanwhile I'm developing ISOLDE in ChimeraX, which is *also* its own self-contained Python (3.7) environment. To plug one into the other in that form... well, I don't think I'm a good enough programmer to really know where to start. The move to Conda and a more modular CCTBX architecture should make a lot more possible in that direction. Pip would be even better for me personally (ChimeraX can install directly from the PyPI, but doesn't interact with Conda) - but I understand pretty well the substantial challenge that would amount to (not least being that the PyPI imposes a limit - around 100MB from memory? - on the size of an individual package). Best regards, Tristan On 2019-08-23 09:28, Luc Bourhis wrote:

Hi Graeme,

Yes, I know. But “black" is a program doing a very particular task (code formatting from the top of my head). Requiring to use a wrapper for python itself is another level. But ok, I think I am mellowing to the idea after all! Talking with people around me, and extrapolating, I would bet that, right now, a great majority of people interested by cctbx in pip have already used the cctbx, so they know about the Python wrapper, and they would not be too sanguine about that. My concern is for the future, when pip will be the first time some people use cctbx. Big fat warning notices on PyPI page and a better error message when cctbx fails because LIBTBX_BUILD is not set would be needed but that could be all right.

If we do a pip installer, we should aim at a minimal install: cctbx, iotbx and their dependencies, and that’s it.

Best wishes,

Luc

...

On 23 Aug 2019, at 07:17, [email protected] wrote:

Without discussing the merits of this or whether we _choose_ to make the move to supporting PIP, I am certain it would be _possible_ - many other packages make dispatcher scripts when you pip install them e.g.

Silver-Surfer rescale_f2 :) $ which black; cat $(which black) /Library/Frameworks/Python.framework/Versions/3.6/bin/black #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6

# -*- coding: utf-8 -*- import re import sys

from black import main

if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) sys.exit(main())

So we _could_ work around the absence of LIBTBX_BUILD etc. in the system. Whether or not we elect to do the work is a different question, and it seems clear that here are very mixed opinions on this.

Best wishes Graeme

On 23 Aug 2019, at 01:21, Luc Bourhis mailto:[email protected]> wrote:

Hi,

Even if we managed to ship our the boost dynamic libraries with pip, it would still not be pip-like, as we would still need our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work with the standard python exe. LD_LIBRARY_PATH, we could get around that by changing the way we compile, using -Wl,-R, which is the runtime equivalent of build time -L. That’s a significant change that would need to be tested. But there is no way around setting LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps there is a way to hack libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip installs?

Best wishes,

Luc

On 16 Aug 2019, at 22:47, Luc Bourhis mailto:[email protected]> wrote:

Hi,

I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.orghttp://anaconda.org/, and then to advertise it hard to every potential user out there.

Best wishes,

Luc

On 16 Aug 2019, at 21:45, Aaron Brewster mailto:[email protected]> wrote:

Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-...). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not.

All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support.

There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6).

If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4.

Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :)

Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users.

Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort.

Thanks, -Aaron

_______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Hi Billy, Sorry, I wasn’t clear: ChimeraX can install packages *from* Pip, but isn’t installable that way itself. It’s built and distributed as a self-contained application. Would certainly make my life easier if they were to migrate to Conda, but I don’t see that happening in the near future. Best regards, Tristan

On 24 Aug 2019, at 02:16, Billy Poon wrote:

Hi all,

Sorry for the delayed response, I was getting the message digests to test some bulletin board settings, so I wasn't getting the individual messages for replying.

For a brief update on conda,

1) External Boost libraries can now be used to build on all platforms (linux, macOS, Windows) and I'm testing updated environments with Boost 1.68 and 1.70. This is for Python 2.7, 3.6, and some initial testing on 3.7. Once 3.7 is ready, Azure Pipelines will be updated to also build 3.7. I'll do the updated dependencies as a pull request since there will be changes that stop the downloading of modules/boost. Mixing Boost versions can cause seg faults or unexpected behavior. I also did some brief testing with 1.65 since that is the minimum version that the latest versions of CUDA require. 2) The next step, like Aaron mentioned, is to get our build system to put things in more standard locations (e.g. dispatchers/binaries in $PREFIX/bin, shared libraries in $PREFIX/lib, Python source/extension modules in $PREFIX/lib/python<version>/site-packages) as a "make install" command. conda on Windows has a similar hierarchy that's specific to conda, so for Windows, this will be conda-specific. 3) Update the conda_compiler branch (https://github.com/cctbx/cctbx_project/tree/conda_compiler). This lets the build process use the standard compilers in conda's build system.

Then a CCTBX conda package can be built and added to the conda-forge channel. The conda environments for building CCTBX are already based on conda-forge dependencies.

4) With the ability to build a conda package, conda constructor (https://github.com/conda/constructor) can be used for building our installers. This would be the more official way of packaging CCTBX with conda dependencies as a stand-alone installer. One of the roadblocks to using conda constructor was supporting noarch packages. It looks like that has been fixed (https://www.anaconda.com/announcing-the-anaconda-2019-07-release/).

So a goal for this work with conda is to make CCTBX-based programs installable as conda packages so that installing and running the software is consistent across platforms and operating system versions while also making it easier for users to add their tools to the software stack.

We do have a Phenix release coming up, so I won't be able to spend much effort on this until later in September.

Tristan,

If ChimeraX is installable via pip, it can be installed into a conda environment. And once 2) is complete, you would be able to access CCTBX from ChimeraX. Also, this would make installing ChimeraX pretty easy for Phenix. Interestingly, the latest open source PyMOL can be built with dependencies from Schrodinger's conda channel, so it should be possible to build a conda package for PyMOL that integrates with our conda environment.

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

...

On Fri, Aug 23, 2019 at 10:03 AM Tristan Croll wrote: To add my two cents on this: probably the second-most common question I've had about ISOLDE's implementation is, "why didn't you use CCTBX?". The honest answer to that is, "I didn't know how."

Still don't, really - although the current developments are rather promising. The problem I've faced is that CCTBX was designed as its own self-contained Python (2.7, until very recently) environment, with its own interpreter and a lot of very specific environment setup. Meanwhile I'm developing ISOLDE in ChimeraX, which is *also* its own self-contained Python (3.7) environment. To plug one into the other in that form... well, I don't think I'm a good enough programmer to really know where to start.

The move to Conda and a more modular CCTBX architecture should make a lot more possible in that direction. Pip would be even better for me personally (ChimeraX can install directly from the PyPI, but doesn't interact with Conda) - but I understand pretty well the substantial challenge that would amount to (not least being that the PyPI imposes a limit - around 100MB from memory? - on the size of an individual package).

Best regards,

Tristan

On 2019-08-23 09:28, Luc Bourhis wrote:

...

Hi Graeme,

Yes, I know. But “black" is a program doing a very particular task (code formatting from the top of my head). Requiring to use a wrapper for python itself is another level. But ok, I think I am mellowing to the idea after all! Talking with people around me, and extrapolating, I would bet that, right now, a great majority of people interested by cctbx in pip have already used the cctbx, so they know about the Python wrapper, and they would not be too sanguine about that. My concern is for the future, when pip will be the first time some people use cctbx. Big fat warning notices on PyPI page and a better error message when cctbx fails because LIBTBX_BUILD is not set would be needed but that could be all right.

If we do a pip installer, we should aim at a minimal install: cctbx, iotbx and their dependencies, and that’s it.

Best wishes,

Luc

...

On 23 Aug 2019, at 07:17, [email protected] wrote:

Without discussing the merits of this or whether we _choose_ to make the move to supporting PIP, I am certain it would be _possible_ - many other packages make dispatcher scripts when you pip install them e.g.

Silver-Surfer rescale_f2 :) $ which black; cat $(which black) /Library/Frameworks/Python.framework/Versions/3.6/bin/black #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6

# -*- coding: utf-8 -*- import re import sys

from black import main

if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) sys.exit(main())

So we _could_ work around the absence of LIBTBX_BUILD etc. in the system. Whether or not we elect to do the work is a different question, and it seems clear that here are very mixed opinions on this.

Best wishes Graeme

On 23 Aug 2019, at 01:21, Luc Bourhis mailto:[email protected]> wrote:

Hi,

Even if we managed to ship our the boost dynamic libraries with pip, it would still not be pip-like, as we would still need our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work with the standard python exe. LD_LIBRARY_PATH, we could get around that by changing the way we compile, using -Wl,-R, which is the runtime equivalent of build time -L. That’s a significant change that would need to be tested. But there is no way around setting LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps there is a way to hack libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip installs?

Best wishes,

Luc

On 16 Aug 2019, at 22:47, Luc Bourhis mailto:[email protected]> wrote:

Hi,

I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.orghttp://anaconda.org/, and then to advertise it hard to every potential user out there.

Best wishes,

Luc

On 16 Aug 2019, at 21:45, Aaron Brewster mailto:[email protected]> wrote:

Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-...). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not.

All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support.

There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6).

If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4.

Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :)

Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users.

Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort.

Thanks, -Aaron

_______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

---

cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Hi Luc, That sounds promising. From there, I’d need to work out how to make a fully-packaged installer (basically a modified wheel file) for the ChimeraX ToolShed - the aim is for the end user to not have to worry about any of this. That adds a couple of complications - e.g. $LIBTBX_BUILD would need to be set dynamically before first import - but doesn’t seem insurmountable. Thanks, Tristan Tristan Croll Research Fellow Cambridge Institute for Medical Research University of Cambridge CB2 0XY

On 25 Aug 2019, at 18:31, Luc Bourhis wrote:

Hi Tristan,

cctbx could be built to use your ChimeraX python, now that cctbx is moving to Python 3. The option —with-python is there for that with the bootstrap script. The specific environment setup boil down to setting two environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on Linux, PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a framework such as ChimeraX, that should not be difficult to ensure those two variables are set.

Best wishes,

Luc

...

On 23 Aug 2019, at 19:02, Tristan Croll wrote:

To add my two cents on this: probably the second-most common question I've had about ISOLDE's implementation is, "why didn't you use CCTBX?". The honest answer to that is, "I didn't know how."

Still don't, really - although the current developments are rather promising. The problem I've faced is that CCTBX was designed as its own self-contained Python (2.7, until very recently) environment, with its own interpreter and a lot of very specific environment setup. Meanwhile I'm developing ISOLDE in ChimeraX, which is *also* its own self-contained Python (3.7) environment. To plug one into the other in that form... well, I don't think I'm a good enough programmer to really know where to start.

The move to Conda and a more modular CCTBX architecture should make a lot more possible in that direction. Pip would be even better for me personally (ChimeraX can install directly from the PyPI, but doesn't interact with Conda) - but I understand pretty well the substantial challenge that would amount to (not least being that the PyPI imposes a limit - around 100MB from memory? - on the size of an individual package).

Best regards,

Tristan

...

On 2019-08-23 09:28, Luc Bourhis wrote: Hi Graeme, Yes, I know. But “black" is a program doing a very particular task (code formatting from the top of my head). Requiring to use a wrapper for python itself is another level. But ok, I think I am mellowing to the idea after all! Talking with people around me, and extrapolating, I would bet that, right now, a great majority of people interested by cctbx in pip have already used the cctbx, so they know about the Python wrapper, and they would not be too sanguine about that. My concern is for the future, when pip will be the first time some people use cctbx. Big fat warning notices on PyPI page and a better error message when cctbx fails because LIBTBX_BUILD is not set would be needed but that could be all right. If we do a pip installer, we should aim at a minimal install: cctbx, iotbx and their dependencies, and that’s it. Best wishes, Luc

...

On 23 Aug 2019, at 07:17, [email protected] wrote: Without discussing the merits of this or whether we _choose_ to make the move to supporting PIP, I am certain it would be _possible_ - many other packages make dispatcher scripts when you pip install them e.g. Silver-Surfer rescale_f2 :) $ which black; cat $(which black) /Library/Frameworks/Python.framework/Versions/3.6/bin/black #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 # -*- coding: utf-8 -*- import re import sys from black import main if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) sys.exit(main()) So we _could_ work around the absence of LIBTBX_BUILD etc. in the system. Whether or not we elect to do the work is a different question, and it seems clear that here are very mixed opinions on this. Best wishes Graeme On 23 Aug 2019, at 01:21, Luc Bourhis mailto:[email protected]> wrote: Hi, Even if we managed to ship our the boost dynamic libraries with pip, it would still not be pip-like, as we would still need our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work with the standard python exe. LD_LIBRARY_PATH, we could get around that by changing the way we compile, using -Wl,-R, which is the runtime equivalent of build time -L. That’s a significant change that would need to be tested. But there is no way around setting LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps there is a way to hack libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip installs? Best wishes, Luc On 16 Aug 2019, at 22:47, Luc Bourhis mailto:[email protected]> wrote: Hi, I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.orghttp://anaconda.org/, and then to advertise it hard to every potential user out there. Best wishes, Luc On 16 Aug 2019, at 21:45, Aaron Brewster mailto:[email protected]> wrote: Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-...). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not. All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support. There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6). If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4. Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :) Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users. Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort. Thanks, -Aaron _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

---

cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Dear Billy, This sounds very promising and exciting. I am not sure if cctbx is already functional as a conda package in anaconda3 (Linux) or this is still work in progress. My technical expertise does not allow me to tell the difference. What I tried: Fresh install of anaconda3. Adding - cctbx and - conda-forge to .condarc . Installing pyside2 with conda. Running conda install conda_dependencies . I get a lot package version conflict, and I cannot import cctbx or iotbx to anaconda python. Am I following the right instructions? Or it is too early to expect that cctbx works when installed through conda? Best wishes, Gergely On Wed, Nov 27, 2019 at 3:56 PM Billy Poon wrote:

Hi all,

For a brief update, I have submitted a recipe for cctbxlite to conda-forge (https://github.com/conda-forge/staged-recipes/pull/10021) and support for Python 3.7 and 3.8 is being added (https://github.com/cctbx/cctbx_project/pull/409). With some fixes for Windows (https://github.com/cctbx/cctbx_project/pull/416), all platforms (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and 3.8. Some additional changes will be needed to get Windows to work with Python 3 and for tests to pass with Boost 1.70.0. That will enable the conda-forge recipe to build for all platforms and for all supported versions of Python.

Currently, the conda-forge recipe will install into the conda python and cctbx imports can be done without sourcing the environment scripts. It looks like a lot of the environment variables being set in the dispatchers can be removed since the Python files and C++ extensions are in the right places. I'll update the libtbx_env file so that commands that load the environment can work correctly.

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll wrote:

...

Hi Luc,

That sounds promising. From there, I’d need to work out how to make a fully-packaged installer (basically a modified wheel file) for the ChimeraX ToolShed - the aim is for the end user to not have to worry about any of this. That adds a couple of complications - e.g. $LIBTBX_BUILD would need to be set dynamically before first import - but doesn’t seem insurmountable.

Thanks,

Tristan

Tristan Croll Research Fellow Cambridge Institute for Medical Research University of Cambridge CB2 0XY

...

On 25 Aug 2019, at 18:31, Luc Bourhis wrote:

Hi Tristan,

cctbx could be built to use your ChimeraX python, now that cctbx is moving to Python 3. The option —with-python is there for that with the bootstrap script. The specific environment setup boil down to setting two environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on Linux, PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a framework such as ChimeraX, that should not be difficult to ensure those two variables are set.

Best wishes,

Luc

...

On 23 Aug 2019, at 19:02, Tristan Croll wrote:

To add my two cents on this: probably the second-most common question I've had about ISOLDE's implementation is, "why didn't you use CCTBX?". The honest answer to that is, "I didn't know how."

Still don't, really - although the current developments are rather promising. The problem I've faced is that CCTBX was designed as its own self-contained Python (2.7, until very recently) environment, with its own interpreter and a lot of very specific environment setup. Meanwhile I'm developing ISOLDE in ChimeraX, which is *also* its own self-contained Python (3.7) environment. To plug one into the other in that form... well, I don't think I'm a good enough programmer to really know where to start.

The move to Conda and a more modular CCTBX architecture should make a lot more possible in that direction. Pip would be even better for me personally (ChimeraX can install directly from the PyPI, but doesn't interact with Conda) - but I understand pretty well the substantial challenge that would amount to (not least being that the PyPI imposes a limit - around 100MB from memory? - on the size of an individual package).

Best regards,

Tristan

...

On 2019-08-23 09:28, Luc Bourhis wrote: Hi Graeme, Yes, I know. But “black" is a program doing a very particular task (code formatting from the top of my head). Requiring to use a wrapper for python itself is another level. But ok, I think I am mellowing to the idea after all! Talking with people around me, and extrapolating, I would bet that, right now, a great majority of people interested by cctbx in pip have already used the cctbx, so they know about the Python wrapper, and they would not be too sanguine about that. My concern is for the future, when pip will be the first time some people use cctbx. Big fat warning notices on PyPI page and a better error message when cctbx fails because LIBTBX_BUILD is not set would be needed but that could be all right. If we do a pip installer, we should aim at a minimal install: cctbx, iotbx and their dependencies, and that’s it. Best wishes, Luc

...

On 23 Aug 2019, at 07:17, [email protected] wrote: Without discussing the merits of this or whether we _choose_ to make the move to supporting PIP, I am certain it would be _possible_ - many other packages make dispatcher scripts when you pip install them e.g. Silver-Surfer rescale_f2 :) $ which black; cat $(which black) /Library/Frameworks/Python.framework/Versions/3.6/bin/black #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 # -*- coding: utf-8 -*- import re import sys from black import main if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) sys.exit(main()) So we _could_ work around the absence of LIBTBX_BUILD etc. in the system. Whether or not we elect to do the work is a different question, and it seems clear that here are very mixed opinions on this. Best wishes Graeme On 23 Aug 2019, at 01:21, Luc Bourhis mailto:[email protected]> wrote: Hi, Even if we managed to ship our the boost dynamic libraries with pip, it would still not be pip-like, as we would still need our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work with the standard python exe. LD_LIBRARY_PATH, we could get around that by changing the way we compile, using -Wl,-R, which is the runtime equivalent of build time -L. That’s a significant change that would need to be tested. But there is no way around setting LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps there is a way to hack libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip installs? Best wishes, Luc On 16 Aug 2019, at 22:47, Luc Bourhis mailto:[email protected]> wrote: Hi, I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.orghttp://anaconda.org/, and then to advertise it hard to every potential user out there. Best wishes, Luc On 16 Aug 2019, at 21:45, Aaron Brewster mailto:[email protected]> wrote: Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-...). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not. All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support. There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6). If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4. Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :) Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users. Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort. Thanks, -Aaron _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

---

cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, PhD Associate Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

Dear Billy, Thank you for this update and for your efforts! I found a solution and indeed most things already work in anaconda3. The steps I took (even if these instructions will have short expiry date): Modified .condarc with: channels: - cctbx - conda-forge - defaults run conda conda install cctbx_dependencies python=3.6 I expect this will work without python=3.6 in the near future. Then compiling cctbx with anaconda3 python went without any problem when using these flags: python bootstrap.py --use-conda --python3 --nproc=12 Then I probably did the unorthodox thing and sourced these directories: setenv LIBTBX_BUILD /home/gergely/cctbx/build setenv PATH ${PATH}:/home/gergely/cctbx/build/bin setenv PYTHONPATH /home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH setenv LD_LIBRARY_PATH /home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH There is probably a better way to put this into conda environment. With these steps I could run one of my scripts depending on cctbx. The only problem I found is that hdf5 library had conflict with another package in conda, but I does not find this as a showstopper. Best wishes, Gergely On Fri, Dec 13, 2019 at 7:23 PM Billy Poon wrote:

Hi Gergely,

It's still a work in progress. I'm sorting out some Windows issues right now. I can probably build a test package on a separate channel for people that want to test it (let's say next week?). I'll provide instructions, but basically, the test conda package will be in its own separate channel and the dependencies will be pulled from the conda-forge channel. I want most things to be working correctly on Python 2.7, 3.6, 3.7, and 3.8 on all 3 platforms.

Thanks!

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona wrote:

...

Dear Billy,

This sounds very promising and exciting. I am not sure if cctbx is already functional as a conda package in anaconda3 (Linux) or this is still work in progress. My technical expertise does not allow me to tell the difference. What I tried:

Fresh install of anaconda3. Adding - cctbx and - conda-forge to .condarc . Installing pyside2 with conda. Running conda install conda_dependencies . I get a lot package version conflict, and I cannot import cctbx or iotbx to anaconda python. Am I following the right instructions? Or it is too early to expect that cctbx works when installed through conda?

Best wishes,

Gergely

On Wed, Nov 27, 2019 at 3:56 PM Billy Poon wrote:

...

Hi all,

For a brief update, I have submitted a recipe for cctbxlite to conda-forge (https://github.com/conda-forge/staged-recipes/pull/10021) and support for Python 3.7 and 3.8 is being added (https://github.com/cctbx/cctbx_project/pull/409). With some fixes for Windows (https://github.com/cctbx/cctbx_project/pull/416), all platforms (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and 3.8. Some additional changes will be needed to get Windows to work with Python 3 and for tests to pass with Boost 1.70.0. That will enable the conda-forge recipe to build for all platforms and for all supported versions of Python.

Currently, the conda-forge recipe will install into the conda python and cctbx imports can be done without sourcing the environment scripts. It looks like a lot of the environment variables being set in the dispatchers can be removed since the Python files and C++ extensions are in the right places. I'll update the libtbx_env file so that commands that load the environment can work correctly.

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll wrote:

...

Hi Luc,

That sounds promising. From there, I’d need to work out how to make a fully-packaged installer (basically a modified wheel file) for the ChimeraX ToolShed - the aim is for the end user to not have to worry about any of this. That adds a couple of complications - e.g. $LIBTBX_BUILD would need to be set dynamically before first import - but doesn’t seem insurmountable.

Thanks,

Tristan

Tristan Croll Research Fellow Cambridge Institute for Medical Research University of Cambridge CB2 0XY

...

On 25 Aug 2019, at 18:31, Luc Bourhis wrote:

Hi Tristan,

cctbx could be built to use your ChimeraX python, now that cctbx is moving to Python 3. The option —with-python is there for that with the bootstrap script. The specific environment setup boil down to setting two environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on Linux, PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a framework such as ChimeraX, that should not be difficult to ensure those two variables are set.

Best wishes,

Luc

...

On 23 Aug 2019, at 19:02, Tristan Croll wrote:

To add my two cents on this: probably the second-most common question I've had about ISOLDE's implementation is, "why didn't you use CCTBX?". The honest answer to that is, "I didn't know how."

Still don't, really - although the current developments are rather promising. The problem I've faced is that CCTBX was designed as its own self-contained Python (2.7, until very recently) environment, with its own interpreter and a lot of very specific environment setup. Meanwhile I'm developing ISOLDE in ChimeraX, which is *also* its own self-contained Python (3.7) environment. To plug one into the other in that form... well, I don't think I'm a good enough programmer to really know where to start.

The move to Conda and a more modular CCTBX architecture should make a lot more possible in that direction. Pip would be even better for me personally (ChimeraX can install directly from the PyPI, but doesn't interact with Conda) - but I understand pretty well the substantial challenge that would amount to (not least being that the PyPI imposes a limit - around 100MB from memory? - on the size of an individual package).

Best regards,

Tristan

> On 2019-08-23 09:28, Luc Bourhis wrote: > Hi Graeme, > Yes, I know. But “black" is a program doing a very particular task > (code formatting from the top of my head). Requiring to use a wrapper > for python itself is another level. But ok, I think I am mellowing to > the idea after all! Talking with people around me, and extrapolating, > I would bet that, right now, a great majority of people interested by > cctbx in pip have already used the cctbx, so they know about the > Python wrapper, and they would not be too sanguine about that. My > concern is for the future, when pip will be the first time some people > use cctbx. Big fat warning notices on PyPI page and a better error > message when cctbx fails because LIBTBX_BUILD is not set would be > needed but that could be all right. > If we do a pip installer, we should aim at a minimal install: cctbx, > iotbx and their dependencies, and that’s it. > Best wishes, > Luc >> On 23 Aug 2019, at 07:17, [email protected] wrote: >> Without discussing the merits of this or whether we _choose_ to make the move to supporting PIP, I am certain it would be _possible_ - many other packages make dispatcher scripts when you pip install them e.g. >> Silver-Surfer rescale_f2 :) $ which black; cat $(which black) >> /Library/Frameworks/Python.framework/Versions/3.6/bin/black >> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 >> # -*- coding: utf-8 -*- >> import re >> import sys >> from black import main >> if __name__ == '__main__': >> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) >> sys.exit(main()) >> So we _could_ work around the absence of LIBTBX_BUILD etc. in the system. Whether or not we elect to do the work is a different question, and it seems clear that here are very mixed opinions on this. >> Best wishes Graeme >> On 23 Aug 2019, at 01:21, Luc Bourhis mailto:[email protected]> wrote: >> Hi, >> Even if we managed to ship our the boost dynamic libraries with pip, it would still not be pip-like, as we would still need our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work with the standard python exe. LD_LIBRARY_PATH, we could get around that by changing the way we compile, using -Wl,-R, which is the runtime equivalent of build time -L. That’s a significant change that would need to be tested. But there is no way around setting LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps there is a way to hack libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip installs? >> Best wishes, >> Luc >> On 16 Aug 2019, at 22:47, Luc Bourhis mailto:[email protected]> wrote: >> Hi, >> I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.orghttp://anaconda.org/, and then to advertise it hard to every potential user out there. >> Best wishes, >> Luc >> On 16 Aug 2019, at 21:45, Aaron Brewster mailto:[email protected]> wrote: >> Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-...). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not. >> All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support. >> There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6). >> If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4. >> Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :) >> Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users. >> Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort. >> Thanks, >> -Aaron >> _______________________________________________ >> cctbxbb mailing list >> [email protected]mailto:[email protected] >> http://phenix-online.org/mailman/listinfo/cctbxbb >> _______________________________________________ >> cctbxbb mailing list >> [email protected]mailto:[email protected] >> http://phenix-online.org/mailman/listinfo/cctbxbb >> _______________________________________________ >> cctbxbb mailing list >> [email protected]mailto:[email protected] >> http://phenix-online.org/mailman/listinfo/cctbxbb >> -- >> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. >> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. >> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. >> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom >> _______________________________________________ >> cctbxbb mailing list >> [email protected] >> http://phenix-online.org/mailman/listinfo/cctbxbb > _______________________________________________ > cctbxbb mailing list > [email protected] > http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, PhD Associate Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, PhD Associate Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

Dear Billy, Thank you for the detailed explanation, I look forward to do further testing! I aim to pool all modules under the same environment, if there are conflicts then I just try to reshuffle the order of imports. So far this did not cause problems for me even when I was using system python, but of course it is not the most prudent thing to do. With anaconda everything is much more standardized and isolated already and without being superuser I can have my familiar environment at any synchrotron based cluster. It is great that cctbx will be an integral part of this ecosystem and this was also the last thing holding me back from adopting python3. About hdf5, pymc3 requires it and importing causes a kernel restart with the following error messages, curiously if I import h5py first this can be avoided. Best wishes, Gergely import pymc3 as pm Warning! ***HDF5 library version mismatched error*** The HDF5 header files used to compile this application do not match the version used by the HDF5 library to which this application is linked. Data corruption or segmentation faults may occur if the application continues. This can happen when an application was compiled by one version of HDF5 but linked with a different version of static or shared HDF5 library. You should recompile the application or check your shared library related settings such as 'LD_LIBRARY_PATH'. You can, at your own risk, disable this warning by setting the environment variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'. Setting it to 2 or higher will suppress the warning messages totally. Headers are 1.10.4, library is 1.10.5 SUMMARY OF THE HDF5 CONFIGURATION ================================= General Information: ------------------- HDF5 Version: 1.10.5 Configured on: Tue Oct 22 12:02:13 UTC 2019 Configured by: conda@16247e67ecd5 Host system: x86_64-conda_cos6-linux-gnu Uname information: Linux 16247e67ecd5 4.15.0-1059-azure #64-Ubuntu SMP Fri Sep 13 17:02:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Byte sex: little-endian Installation point: /home/gergely/anaconda3 Compiling Options: ------------------ Build Mode: production Debugging Symbols: no Asserts: no Profiling: no Optimization Level: high Linking Options: ---------------- Libraries: static, shared Statically Linked Executables: LDFLAGS: -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/gergely/anaconda3/lib -Wl,-rpath-link,/home/gergely/anaconda3/lib -L/home/gergely/anaconda3/lib H5_LDFLAGS: AM_LDFLAGS: -L/home/gergely/anaconda3/lib Extra libraries: -lrt -lpthread -lz -ldl -lm Archiver: /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar AR_FLAGS: cr Ranlib: /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib Languages: ---------- C: yes C Compiler: /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc CPPFLAGS: -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/home/gergely/anaconda3/include H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200809L -DNDEBUG -UH5_DEBUG_API AM_CPPFLAGS: -I/home/gergely/anaconda3/include C Flags: -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/home/gergely/anaconda3/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5 -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix H5 C Flags: -std=c99 -pedantic -Wall -Wextra -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual -Wconversion -Wdeclaration-after-statement -Wdisabled-optimization -Wfloat-equal -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum -Wundef -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings -finline-functions -s -Wno-inline -Wno-aggregate-return -Wno-missing-format-attribute -Wno-missing-noreturn -O AM C Flags: Shared C Library: yes Static C Library: yes Fortran: yes Fortran Compiler: /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran Fortran Flags: H5 Fortran Flags: -pedantic -Wall -Wextra -Wunderflow -Wimplicit-interface -Wsurprising -Wno-c-binding-type -s -O2 AM Fortran Flags: Shared Fortran Library: yes Static Fortran Library: yes C++: yes C++ Compiler: /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++ C++ Flags: -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/home/gergely/anaconda3/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5 -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix H5 C++ Flags: -pedantic -Wall -W -Wundef -Wshadow -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Wconversion -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor -Wctor-dtor-privacy -Wabi -finline-functions -s -O AM C++ Flags: Shared C++ Library: yes Static C++ Library: yes Java: no Features: --------- Parallel HDF5: no Parallel Filtered Dataset Writes: no Large Parallel I/O: no High-level library: yes Threadsafety: yes Default API mapping: v110 With deprecated public symbols: yes I/O filters (external): deflate(zlib) MPE: no Direct VFD: no dmalloc: no Packages w/ extra debug output: none API tracing: no Using memory checker: yes Memory allocation sanity checks: no Function stack tracing: no Strict file format checks: no Optimization instrumentation: no On Tue, Dec 17, 2019 at 8:35 AM Billy Poon wrote:

Hi Gergely,

Let me build the test package tomorrow. All the gritty details for building with conda is being finalized and the official documentation will be updated to describe the steps. It would be too confusing to keep changing the documentation as the process evolves. You do have the general process, though, which is summarized as follows.

1) Installing dependencies. The cctbx_dependencies metapackage was an initial approach for managing the CCTBX dependencies, but after contacting the conda-forge folks, they recommended using the --only-deps flag. So when the CCTBX conda package is available, you'll be able to get a set of dependencies with,

conda install -c conda-forge --only-deps cctbx

By default, the bootstrap.py file will automatically install a set of dependencies in the "conda_base" directory (and a conda installation if one is not found). It just uses standard conda environment files located in libtbx/auto_build/conda_envs, so you do not need to install cctbx_dependencies as a separate step. The environment files avoid channel issues by explicitly defining the channel to pull the packages from and the cctbx channel just stores copies of packages from conda-forge. There were issues earlier where the conda-forge packages would sometimes be moved to a different label. The --use-conda flag also accepts a path to $CONDA_PREFIX if you want to use a specific environment for testing.

2) Building. The bootstrap.py file handles that with SCons.

3) Running. After building, there should be a setpaths.sh (and .csh) file that adds build/bin to your path. The build/bin directory has our dispatchers, which are just scripts that set up the environment variables for you. This prevents other programs from loading our libraries, whose versions may conflict. You should see that there is a "python" dispatcher, which is a convenience for developers. Otherwise, you can use libtbx.python, which will be able to import CCTBX modules.

The running part is where the conda package for CCTBX will be different than this build. Since our Python files and extensions modules will be in the "site-packages" directory for Python, the PYTHONPATH variable will not be needed (and conda suggests that that variable not be set). The other CCTBX libraries will be in $CONDA_PREFIX/lib, so LD_LIBRARY_PATH is not needed. In an active environment, PATH will already be modified. And then our LIBTBX_BUILD directory can be set to sys.prefix.

So with the conda package, you would only need to activate your conda environment and CCTBX should integrate with other conda packages. What is the conflict with hdf5? That's something that should be fixed. Thanks!

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Mon, Dec 16, 2019 at 2:53 AM Gergely Katona wrote:

...

Dear Billy,

Thank you for this update and for your efforts! I found a solution and indeed most things already work in anaconda3. The steps I took (even if these instructions will have short expiry date):

Modified .condarc with: channels: - cctbx - conda-forge - defaults

run conda conda install cctbx_dependencies python=3.6

I expect this will work without python=3.6 in the near future.

Then compiling cctbx with anaconda3 python went without any problem when using these flags: python bootstrap.py --use-conda --python3 --nproc=12

Then I probably did the unorthodox thing and sourced these directories:

setenv LIBTBX_BUILD /home/gergely/cctbx/build setenv PATH ${PATH}:/home/gergely/cctbx/build/bin setenv PYTHONPATH /home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH setenv LD_LIBRARY_PATH /home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH

There is probably a better way to put this into conda environment. With these steps I could run one of my scripts depending on cctbx. The only problem I found is that hdf5 library had conflict with another package in conda, but I does not find this as a showstopper.

Best wishes,

Gergely

On Fri, Dec 13, 2019 at 7:23 PM Billy Poon wrote:

...

Hi Gergely,

It's still a work in progress. I'm sorting out some Windows issues right now. I can probably build a test package on a separate channel for people that want to test it (let's say next week?). I'll provide instructions, but basically, the test conda package will be in its own separate channel and the dependencies will be pulled from the conda-forge channel. I want most things to be working correctly on Python 2.7, 3.6, 3.7, and 3.8 on all 3 platforms.

Thanks!

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona wrote:

...

Dear Billy,

This sounds very promising and exciting. I am not sure if cctbx is already functional as a conda package in anaconda3 (Linux) or this is still work in progress. My technical expertise does not allow me to tell the difference. What I tried:

Fresh install of anaconda3. Adding - cctbx and - conda-forge to .condarc . Installing pyside2 with conda. Running conda install conda_dependencies . I get a lot package version conflict, and I cannot import cctbx or iotbx to anaconda python. Am I following the right instructions? Or it is too early to expect that cctbx works when installed through conda?

Best wishes,

Gergely

On Wed, Nov 27, 2019 at 3:56 PM Billy Poon wrote:

...

Hi all,

For a brief update, I have submitted a recipe for cctbxlite to conda-forge (https://github.com/conda-forge/staged-recipes/pull/10021) and support for Python 3.7 and 3.8 is being added (https://github.com/cctbx/cctbx_project/pull/409). With some fixes for Windows (https://github.com/cctbx/cctbx_project/pull/416), all platforms (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and 3.8. Some additional changes will be needed to get Windows to work with Python 3 and for tests to pass with Boost 1.70.0. That will enable the conda-forge recipe to build for all platforms and for all supported versions of Python.

Currently, the conda-forge recipe will install into the conda python and cctbx imports can be done without sourcing the environment scripts. It looks like a lot of the environment variables being set in the dispatchers can be removed since the Python files and C++ extensions are in the right places. I'll update the libtbx_env file so that commands that load the environment can work correctly.

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll wrote:

...

Hi Luc,

That sounds promising. From there, I’d need to work out how to make a fully-packaged installer (basically a modified wheel file) for the ChimeraX ToolShed - the aim is for the end user to not have to worry about any of this. That adds a couple of complications - e.g. $LIBTBX_BUILD would need to be set dynamically before first import - but doesn’t seem insurmountable.

Thanks,

Tristan

Tristan Croll Research Fellow Cambridge Institute for Medical Research University of Cambridge CB2 0XY

> On 25 Aug 2019, at 18:31, Luc Bourhis wrote: > > Hi Tristan, > > cctbx could be built to use your ChimeraX python, now that cctbx is moving to Python 3. The option —with-python is there for that with the bootstrap script. The specific environment setup boil down to setting two environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on Linux, PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a framework such as ChimeraX, that should not be difficult to ensure those two variables are set. > > Best wishes, > > Luc > > >> On 23 Aug 2019, at 19:02, Tristan Croll wrote: >> >> To add my two cents on this: probably the second-most common question I've had about ISOLDE's implementation is, "why didn't you use CCTBX?". The honest answer to that is, "I didn't know how." >> >> Still don't, really - although the current developments are rather promising. The problem I've faced is that CCTBX was designed as its own self-contained Python (2.7, until very recently) environment, with its own interpreter and a lot of very specific environment setup. Meanwhile I'm developing ISOLDE in ChimeraX, which is *also* its own self-contained Python (3.7) environment. To plug one into the other in that form... well, I don't think I'm a good enough programmer to really know where to start. >> >> The move to Conda and a more modular CCTBX architecture should make a lot more possible in that direction. Pip would be even better for me personally (ChimeraX can install directly from the PyPI, but doesn't interact with Conda) - but I understand pretty well the substantial challenge that would amount to (not least being that the PyPI imposes a limit - around 100MB from memory? - on the size of an individual package). >> >> Best regards, >> >> Tristan >> >>> On 2019-08-23 09:28, Luc Bourhis wrote: >>> Hi Graeme, >>> Yes, I know. But “black" is a program doing a very particular task >>> (code formatting from the top of my head). Requiring to use a wrapper >>> for python itself is another level. But ok, I think I am mellowing to >>> the idea after all! Talking with people around me, and extrapolating, >>> I would bet that, right now, a great majority of people interested by >>> cctbx in pip have already used the cctbx, so they know about the >>> Python wrapper, and they would not be too sanguine about that. My >>> concern is for the future, when pip will be the first time some people >>> use cctbx. Big fat warning notices on PyPI page and a better error >>> message when cctbx fails because LIBTBX_BUILD is not set would be >>> needed but that could be all right. >>> If we do a pip installer, we should aim at a minimal install: cctbx, >>> iotbx and their dependencies, and that’s it. >>> Best wishes, >>> Luc >>>> On 23 Aug 2019, at 07:17, [email protected] wrote: >>>> Without discussing the merits of this or whether we _choose_ to make the move to supporting PIP, I am certain it would be _possible_ - many other packages make dispatcher scripts when you pip install them e.g. >>>> Silver-Surfer rescale_f2 :) $ which black; cat $(which black) >>>> /Library/Frameworks/Python.framework/Versions/3.6/bin/black >>>> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 >>>> # -*- coding: utf-8 -*- >>>> import re >>>> import sys >>>> from black import main >>>> if __name__ == '__main__': >>>> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) >>>> sys.exit(main()) >>>> So we _could_ work around the absence of LIBTBX_BUILD etc. in the system. Whether or not we elect to do the work is a different question, and it seems clear that here are very mixed opinions on this. >>>> Best wishes Graeme >>>> On 23 Aug 2019, at 01:21, Luc Bourhis mailto:[email protected]> wrote: >>>> Hi, >>>> Even if we managed to ship our the boost dynamic libraries with pip, it would still not be pip-like, as we would still need our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work with the standard python exe. LD_LIBRARY_PATH, we could get around that by changing the way we compile, using -Wl,-R, which is the runtime equivalent of build time -L. That’s a significant change that would need to be tested. But there is no way around setting LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps there is a way to hack libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip installs? >>>> Best wishes, >>>> Luc >>>> On 16 Aug 2019, at 22:47, Luc Bourhis mailto:[email protected]> wrote: >>>> Hi, >>>> I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.orghttp://anaconda.org/, and then to advertise it hard to every potential user out there. >>>> Best wishes, >>>> Luc >>>> On 16 Aug 2019, at 21:45, Aaron Brewster mailto:[email protected]> wrote: >>>> Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-...). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not. >>>> All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support. >>>> There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6). >>>> If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4. >>>> Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :) >>>> Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users. >>>> Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort. >>>> Thanks, >>>> -Aaron >>>> _______________________________________________ >>>> cctbxbb mailing list >>>> [email protected]mailto:[email protected] >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >>>> _______________________________________________ >>>> cctbxbb mailing list >>>> [email protected]mailto:[email protected] >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >>>> _______________________________________________ >>>> cctbxbb mailing list >>>> [email protected]mailto:[email protected] >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >>>> -- >>>> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. >>>> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. >>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. >>>> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom >>>> _______________________________________________ >>>> cctbxbb mailing list >>>> [email protected] >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >>> _______________________________________________ >>> cctbxbb mailing list >>> [email protected] >>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> _______________________________________________ >> cctbxbb mailing list >> [email protected] >> http://phenix-online.org/mailman/listinfo/cctbxbb > > > _______________________________________________ > cctbxbb mailing list > [email protected] > http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, PhD Associate Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, PhD Associate Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, PhD Associate Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

Hi Billy, Conda install went fine with your instructions after rearranging the channels by putting cctbx last. I removed all environmental variables and previous build of cctbx. import sys print (sys.path, sys.prefix) ['/home/gergely/anaconda3/lib/python36.zip', '/home/gergely/anaconda3/lib/python3.6', '/home/gergely/anaconda3/lib/python3.6/lib-dynload', '', '/home/gergely/anaconda3/lib/python3.6/site-packages', '/home/gergely/anaconda3/lib/python3.6/site-packages/IPython/extensions', '/home/gergely/.ipython'] /home/gergely/anaconda3 Many imports went fine including pymc3, but I encountered problems with these three: from cctbx import miller import iotbx.pdb from iotbx import reflection_file_reader, mtz I also got type error when handling space groups. --------------------------------------------------------------------------- KeyError Traceback (most recent call last) <ipython-input-5-cedbd4f84f3d> in <module> 25 from cctbx import uctbx 26 from cctbx import sgtbx ---> 27 from cctbx import miller 28 #from iotbx import reflection_file_reader, mtz 29 sns.set_context("poster") ~/anaconda3/lib/python3.6/site-packages/cctbx/miller/__init__.py in <module> 11 12 from cctbx import crystal ---> 13 from cctbx import maptbx 14 from cctbx import sgtbx 15 from cctbx.sgtbx import lattice_symmetry ~/anaconda3/lib/python3.6/site-packages/cctbx/maptbx/__init__.py in <module> 15 from libtbx import adopt_init_args 16 from libtbx.utils import Sorry ---> 17 import libtbx.load_env 18 import math 19 import sys, os ~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module> 3 import libtbx.env_config 4 import os ----> 5 libtbx.env = libtbx.env_config.unpickle() 6 libtbx.env.set_os_environ_all_dist() 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME") ~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle() 2603 2604 def unpickle(): -> 2605 build_path = os.environ["LIBTBX_BUILD"] 2606 set_preferred_sys_prefix_and_sys_executable(build_path=build_path) 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb") ~/anaconda3/lib/python3.6/os.py in __getitem__(self, key) 667 except KeyError: 668 # raise KeyError with the original key value --> 669 raise KeyError(key) from None 670 return self.decodevalue(value) 671 KeyError: 'LIBTBX_BUILD' --------------------------------------------------------------------------- ArgumentError Traceback (most recent call last) <ipython-input-7-2acd3a9ce26a> in <module> 26 return ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar 27 ---> 28 ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar=initializecrystal() <ipython-input-7-2acd3a9ce26a> in initializecrystal() 11 uc = uctbx.unit_cell(unit_cell) 12 wavelength = 1.54980 ---> 13 xtal_symm = crystal.symmetry(unit_cell=unit_cell, space_group_symbol="P 43 21 2") 14 15 ms = miller.build_set(crystal_symmetry=xtal_symm,anomalous_flag=True, d_min=1.61) ~/anaconda3/lib/python3.6/site-packages/cctbx/crystal/__init__.py in __init__(self, unit_cell, space_group_symbol, space_group_info, space_group, correct_rhombohedral_setting_if_necessary, assert_is_compatible_unit_cell, raise_sorry_if_incompatible_unit_cell, force_compatible_unit_cell) 74 if (space_group_symbol is not None): 75 self._space_group_info = sgtbx.space_group_info( ---> 76 symbol=space_group_symbol) 77 elif (space_group is not None): 78 if (isinstance(space_group, sgtbx.space_group)): ~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/__init__.py in __init__(self, symbol, table_id, group, number, space_group_t_den) 100 assert group is None 101 if (table_id is None): --> 102 symbols = space_group_symbols(symbol) 103 else: 104 if (isinstance(symbol, int)): symbol = str(symbol) ArgumentError: Python argument types in space_group_symbols.__init__(space_group_symbols, str) did not match C++ signature: __init__(_object*, int space_group_number) __init__(_object*, int space_group_number, std::__cxx11::basic_string extension='') __init__(_object*, int space_group_number, std::__cxx11::basic_string extension='', std::__cxx11::basic_string table_id='') __init__(_object*, std::__cxx11::basic_string symbol) __init__(_object*, std::__cxx11::basic_string symbol, std::__cxx11::basic_string table_id='') When I tried to set LIBTBX_BUILD to /home/gergely/anaconda3 (this is sys.prefix on my system), I got other problems when importing. Can LIBTBX_BUILD be set in the conda package? --------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) <ipython-input-1-7e2b144826cc> in <module> 26 from cctbx import sgtbx 27 #from cctbx import miller ---> 28 from iotbx import reflection_file_reader, mtz 29 sns.set_context("poster") 30 plt.rcParams.update({'figure.autolayout': True}) ~/anaconda3/lib/python3.6/site-packages/iotbx/reflection_file_reader.py in <module> 53 54 from __future__ import absolute_import, division, print_function ---> 55 from iotbx import mtz 56 from iotbx.scalepack import merge as scalepack_merge 57 from iotbx.scalepack import no_merge_original_index as scalepack_no_merge ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/__init__.py in <module> 9 import iotbx_mtz_ext as ext 10 ---> 11 from iotbx.mtz import extract_from_symmetry_lib 12 from cctbx import xray 13 import cctbx.xray.observation_types ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/extract_from_symmetry_lib.py in <module> 1 from __future__ import absolute_import, division, print_function 2 from cctbx import sgtbx ----> 3 import libtbx.load_env 4 import os.path as op 5 from six.moves import range ~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module> 3 import libtbx.env_config 4 import os ----> 5 libtbx.env = libtbx.env_config.unpickle() 6 libtbx.env.set_os_environ_all_dist() 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME") ~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle() 2605 build_path = os.environ["LIBTBX_BUILD"] 2606 set_preferred_sys_prefix_and_sys_executable(build_path=build_path) -> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb") 2608 env = pickle.load(libtbx_env) 2609 if (env.python_version_major_minor != sys.version_info[:2]): FileNotFoundError: [Errno 2] No such file or directory: '/home/gergely/anaconda3/libtbx_env' On Wed, Dec 18, 2019 at 9:10 AM Billy Poon wrote:

Hi Gergely,

I've uploaded linux packages to a new channel, cctbx-dev, and you can install it with

conda install -c cctbx-dev cctbx

in your current environment. You should set your ~/.condarc file to pull the other dependencies from the conda-forge channel first, so put conda-forge above cctbx. Mine looks like

channels: - conda-forge - defaults - cctbx

Do you need dxtbx for your scripts? This package does not build that part. I think the plan is to build a separate conda package for dxtbx so that it can be updated more frequently. I can rebuild the packages to include it for testing, but the one being submitted to conda-forge will not have it.

Also, your error message is probably due to version of HDF5 that the development build installs. The bootstrap.py script will install 1.10.4, but your other dependency is looking for 1.10.5. Installing this cctbx conda package should install 1.10.5, which should fix the issue. Also, I'm updating those environments and 1.10.5 will be the new default version.

Lastly, the dispatchers will not work in these packages because the old paths during the build process are still in them and many of them expect some additional information that has not been updated in the packages yet. I'm in the process of doing that and will update the cctbx-dev channel when that's done. However, by starting python, you can import cctbx modules. So you can run commands like

from scitbx.array_family import flex a = flex.random_double(1000000) b = flex.min_max_mean_double(a) b.min b.max b.mean

Thanks!

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Tue, Dec 17, 2019 at 5:39 AM Gergely Katona wrote:

...

Dear Billy,

Thank you for the detailed explanation, I look forward to do further testing! I aim to pool all modules under the same environment, if there are conflicts then I just try to reshuffle the order of imports. So far this did not cause problems for me even when I was using system python, but of course it is not the most prudent thing to do. With anaconda everything is much more standardized and isolated already and without being superuser I can have my familiar environment at any synchrotron based cluster. It is great that cctbx will be an integral part of this ecosystem and this was also the last thing holding me back from adopting python3. About hdf5, pymc3 requires it and importing causes a kernel restart with the following error messages, curiously if I import h5py first this can be avoided.

Best wishes,

Gergely

import pymc3 as pm

Warning! ***HDF5 library version mismatched error*** The HDF5 header files used to compile this application do not match the version used by the HDF5 library to which this application is linked. Data corruption or segmentation faults may occur if the application continues. This can happen when an application was compiled by one version of HDF5 but linked with a different version of static or shared HDF5 library. You should recompile the application or check your shared library related settings such as 'LD_LIBRARY_PATH'. You can, at your own risk, disable this warning by setting the environment variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'. Setting it to 2 or higher will suppress the warning messages totally. Headers are 1.10.4, library is 1.10.5 SUMMARY OF THE HDF5 CONFIGURATION =================================

General Information: ------------------- HDF5 Version: 1.10.5 Configured on: Tue Oct 22 12:02:13 UTC 2019 Configured by: conda@16247e67ecd5 Host system: x86_64-conda_cos6-linux-gnu Uname information: Linux 16247e67ecd5 4.15.0-1059-azure #64-Ubuntu SMP Fri Sep 13 17:02:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Byte sex: little-endian Installation point: /home/gergely/anaconda3

Compiling Options: ------------------ Build Mode: production Debugging Symbols: no Asserts: no Profiling: no Optimization Level: high

Linking Options: ---------------- Libraries: static, shared Statically Linked Executables: LDFLAGS: -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/gergely/anaconda3/lib -Wl,-rpath-link,/home/gergely/anaconda3/lib -L/home/gergely/anaconda3/lib H5_LDFLAGS: AM_LDFLAGS: -L/home/gergely/anaconda3/lib Extra libraries: -lrt -lpthread -lz -ldl -lm Archiver: /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar AR_FLAGS: cr Ranlib: /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib

Languages: ---------- C: yes C Compiler: /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc CPPFLAGS: -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/home/gergely/anaconda3/include H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200809L -DNDEBUG -UH5_DEBUG_API AM_CPPFLAGS: -I/home/gergely/anaconda3/include C Flags: -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/home/gergely/anaconda3/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5 -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix H5 C Flags: -std=c99 -pedantic -Wall -Wextra -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual -Wconversion -Wdeclaration-after-statement -Wdisabled-optimization -Wfloat-equal -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum -Wundef -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings -finline-functions -s -Wno-inline -Wno-aggregate-return -Wno-missing-format-attribute -Wno-missing-noreturn -O AM C Flags: Shared C Library: yes Static C Library: yes

Fortran: yes Fortran Compiler: /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran Fortran Flags: H5 Fortran Flags: -pedantic -Wall -Wextra -Wunderflow -Wimplicit-interface -Wsurprising -Wno-c-binding-type -s -O2 AM Fortran Flags: Shared Fortran Library: yes Static Fortran Library: yes

C++: yes C++ Compiler: /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++ C++ Flags: -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/home/gergely/anaconda3/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5 -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix H5 C++ Flags: -pedantic -Wall -W -Wundef -Wshadow -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Wconversion -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor -Wctor-dtor-privacy -Wabi -finline-functions -s -O AM C++ Flags: Shared C++ Library: yes Static C++ Library: yes

Java: no

Features: --------- Parallel HDF5: no Parallel Filtered Dataset Writes: no Large Parallel I/O: no High-level library: yes Threadsafety: yes Default API mapping: v110 With deprecated public symbols: yes I/O filters (external): deflate(zlib) MPE: no Direct VFD: no dmalloc: no Packages w/ extra debug output: none API tracing: no Using memory checker: yes Memory allocation sanity checks: no Function stack tracing: no Strict file format checks: no Optimization instrumentation: no

On Tue, Dec 17, 2019 at 8:35 AM Billy Poon wrote:

...

Hi Gergely,

Let me build the test package tomorrow. All the gritty details for building with conda is being finalized and the official documentation will be updated to describe the steps. It would be too confusing to keep changing the documentation as the process evolves. You do have the general process, though, which is summarized as follows.

1) Installing dependencies. The cctbx_dependencies metapackage was an initial approach for managing the CCTBX dependencies, but after contacting the conda-forge folks, they recommended using the --only-deps flag. So when the CCTBX conda package is available, you'll be able to get a set of dependencies with,

conda install -c conda-forge --only-deps cctbx

By default, the bootstrap.py file will automatically install a set of dependencies in the "conda_base" directory (and a conda installation if one is not found). It just uses standard conda environment files located in libtbx/auto_build/conda_envs, so you do not need to install cctbx_dependencies as a separate step. The environment files avoid channel issues by explicitly defining the channel to pull the packages from and the cctbx channel just stores copies of packages from conda-forge. There were issues earlier where the conda-forge packages would sometimes be moved to a different label. The --use-conda flag also accepts a path to $CONDA_PREFIX if you want to use a specific environment for testing.

2) Building. The bootstrap.py file handles that with SCons.

3) Running. After building, there should be a setpaths.sh (and .csh) file that adds build/bin to your path. The build/bin directory has our dispatchers, which are just scripts that set up the environment variables for you. This prevents other programs from loading our libraries, whose versions may conflict. You should see that there is a "python" dispatcher, which is a convenience for developers. Otherwise, you can use libtbx.python, which will be able to import CCTBX modules.

The running part is where the conda package for CCTBX will be different than this build. Since our Python files and extensions modules will be in the "site-packages" directory for Python, the PYTHONPATH variable will not be needed (and conda suggests that that variable not be set). The other CCTBX libraries will be in $CONDA_PREFIX/lib, so LD_LIBRARY_PATH is not needed. In an active environment, PATH will already be modified. And then our LIBTBX_BUILD directory can be set to sys.prefix.

So with the conda package, you would only need to activate your conda environment and CCTBX should integrate with other conda packages. What is the conflict with hdf5? That's something that should be fixed. Thanks!

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Mon, Dec 16, 2019 at 2:53 AM Gergely Katona wrote:

...

Dear Billy,

Thank you for this update and for your efforts! I found a solution and indeed most things already work in anaconda3. The steps I took (even if these instructions will have short expiry date):

Modified .condarc with: channels: - cctbx - conda-forge - defaults

run conda conda install cctbx_dependencies python=3.6

I expect this will work without python=3.6 in the near future.

Then compiling cctbx with anaconda3 python went without any problem when using these flags: python bootstrap.py --use-conda --python3 --nproc=12

Then I probably did the unorthodox thing and sourced these directories:

setenv LIBTBX_BUILD /home/gergely/cctbx/build setenv PATH ${PATH}:/home/gergely/cctbx/build/bin setenv PYTHONPATH /home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH setenv LD_LIBRARY_PATH /home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH

There is probably a better way to put this into conda environment. With these steps I could run one of my scripts depending on cctbx. The only problem I found is that hdf5 library had conflict with another package in conda, but I does not find this as a showstopper.

Best wishes,

Gergely

On Fri, Dec 13, 2019 at 7:23 PM Billy Poon wrote:

...

Hi Gergely,

It's still a work in progress. I'm sorting out some Windows issues right now. I can probably build a test package on a separate channel for people that want to test it (let's say next week?). I'll provide instructions, but basically, the test conda package will be in its own separate channel and the dependencies will be pulled from the conda-forge channel. I want most things to be working correctly on Python 2.7, 3.6, 3.7, and 3.8 on all 3 platforms.

Thanks!

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona wrote:

...

Dear Billy,

This sounds very promising and exciting. I am not sure if cctbx is already functional as a conda package in anaconda3 (Linux) or this is still work in progress. My technical expertise does not allow me to tell the difference. What I tried:

Fresh install of anaconda3. Adding - cctbx and - conda-forge to .condarc . Installing pyside2 with conda. Running conda install conda_dependencies . I get a lot package version conflict, and I cannot import cctbx or iotbx to anaconda python. Am I following the right instructions? Or it is too early to expect that cctbx works when installed through conda?

Best wishes,

Gergely

On Wed, Nov 27, 2019 at 3:56 PM Billy Poon wrote: > > Hi all, > > For a brief update, I have submitted a recipe for cctbxlite to conda-forge (https://github.com/conda-forge/staged-recipes/pull/10021) and support for Python 3.7 and 3.8 is being added (https://github.com/cctbx/cctbx_project/pull/409). With some fixes for Windows (https://github.com/cctbx/cctbx_project/pull/416), all platforms (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and 3.8. Some additional changes will be needed to get Windows to work with Python 3 and for tests to pass with Boost 1.70.0. That will enable the conda-forge recipe to build for all platforms and for all supported versions of Python. > > Currently, the conda-forge recipe will install into the conda python and cctbx imports can be done without sourcing the environment scripts. It looks like a lot of the environment variables being set in the dispatchers can be removed since the Python files and C++ extensions are in the right places. I'll update the libtbx_env file so that commands that load the environment can work correctly. > > -- > Billy K. Poon > Research Scientist, Molecular Biophysics and Integrated Bioimaging > Lawrence Berkeley National Laboratory > 1 Cyclotron Road, M/S 33R0345 > Berkeley, CA 94720 > Tel: (510) 486-5709 > Fax: (510) 486-5909 > Web: https://phenix-online.org > > > On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll wrote: >> >> Hi Luc, >> >> That sounds promising. From there, I’d need to work out how to make a fully-packaged installer (basically a modified wheel file) for the ChimeraX ToolShed - the aim is for the end user to not have to worry about any of this. That adds a couple of complications - e.g. $LIBTBX_BUILD would need to be set dynamically before first import - but doesn’t seem insurmountable. >> >> Thanks, >> >> Tristan >> >> >> >> Tristan Croll >> Research Fellow >> Cambridge Institute for Medical Research >> University of Cambridge CB2 0XY >> >> >> >> >> > On 25 Aug 2019, at 18:31, Luc Bourhis wrote: >> > >> > Hi Tristan, >> > >> > cctbx could be built to use your ChimeraX python, now that cctbx is moving to Python 3. The option —with-python is there for that with the bootstrap script. The specific environment setup boil down to setting two environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on Linux, PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a framework such as ChimeraX, that should not be difficult to ensure those two variables are set. >> > >> > Best wishes, >> > >> > Luc >> > >> > >> >> On 23 Aug 2019, at 19:02, Tristan Croll wrote: >> >> >> >> To add my two cents on this: probably the second-most common question I've had about ISOLDE's implementation is, "why didn't you use CCTBX?". The honest answer to that is, "I didn't know how." >> >> >> >> Still don't, really - although the current developments are rather promising. The problem I've faced is that CCTBX was designed as its own self-contained Python (2.7, until very recently) environment, with its own interpreter and a lot of very specific environment setup. Meanwhile I'm developing ISOLDE in ChimeraX, which is *also* its own self-contained Python (3.7) environment. To plug one into the other in that form... well, I don't think I'm a good enough programmer to really know where to start. >> >> >> >> The move to Conda and a more modular CCTBX architecture should make a lot more possible in that direction. Pip would be even better for me personally (ChimeraX can install directly from the PyPI, but doesn't interact with Conda) - but I understand pretty well the substantial challenge that would amount to (not least being that the PyPI imposes a limit - around 100MB from memory? - on the size of an individual package). >> >> >> >> Best regards, >> >> >> >> Tristan >> >> >> >>> On 2019-08-23 09:28, Luc Bourhis wrote: >> >>> Hi Graeme, >> >>> Yes, I know. But “black" is a program doing a very particular task >> >>> (code formatting from the top of my head). Requiring to use a wrapper >> >>> for python itself is another level. But ok, I think I am mellowing to >> >>> the idea after all! Talking with people around me, and extrapolating, >> >>> I would bet that, right now, a great majority of people interested by >> >>> cctbx in pip have already used the cctbx, so they know about the >> >>> Python wrapper, and they would not be too sanguine about that. My >> >>> concern is for the future, when pip will be the first time some people >> >>> use cctbx. Big fat warning notices on PyPI page and a better error >> >>> message when cctbx fails because LIBTBX_BUILD is not set would be >> >>> needed but that could be all right. >> >>> If we do a pip installer, we should aim at a minimal install: cctbx, >> >>> iotbx and their dependencies, and that’s it. >> >>> Best wishes, >> >>> Luc >> >>>> On 23 Aug 2019, at 07:17, [email protected] wrote: >> >>>> Without discussing the merits of this or whether we _choose_ to make the move to supporting PIP, I am certain it would be _possible_ - many other packages make dispatcher scripts when you pip install them e.g. >> >>>> Silver-Surfer rescale_f2 :) $ which black; cat $(which black) >> >>>> /Library/Frameworks/Python.framework/Versions/3.6/bin/black >> >>>> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 >> >>>> # -*- coding: utf-8 -*- >> >>>> import re >> >>>> import sys >> >>>> from black import main >> >>>> if __name__ == '__main__': >> >>>> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) >> >>>> sys.exit(main()) >> >>>> So we _could_ work around the absence of LIBTBX_BUILD etc. in the system. Whether or not we elect to do the work is a different question, and it seems clear that here are very mixed opinions on this. >> >>>> Best wishes Graeme >> >>>> On 23 Aug 2019, at 01:21, Luc Bourhis mailto:[email protected]> wrote: >> >>>> Hi, >> >>>> Even if we managed to ship our the boost dynamic libraries with pip, it would still not be pip-like, as we would still need our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work with the standard python exe. LD_LIBRARY_PATH, we could get around that by changing the way we compile, using -Wl,-R, which is the runtime equivalent of build time -L. That’s a significant change that would need to be tested. But there is no way around setting LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps there is a way to hack libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip installs? >> >>>> Best wishes, >> >>>> Luc >> >>>> On 16 Aug 2019, at 22:47, Luc Bourhis mailto:[email protected]> wrote: >> >>>> Hi, >> >>>> I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.orghttp://anaconda.org/, and then to advertise it hard to every potential user out there. >> >>>> Best wishes, >> >>>> Luc >> >>>> On 16 Aug 2019, at 21:45, Aaron Brewster mailto:[email protected]> wrote: >> >>>> Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-...). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not. >> >>>> All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support. >> >>>> There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6). >> >>>> If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4. >> >>>> Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :) >> >>>> Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users. >> >>>> Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort. >> >>>> Thanks, >> >>>> -Aaron >> >>>> _______________________________________________ >> >>>> cctbxbb mailing list >> >>>> [email protected]mailto:[email protected] >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >>>> _______________________________________________ >> >>>> cctbxbb mailing list >> >>>> [email protected]mailto:[email protected] >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >>>> _______________________________________________ >> >>>> cctbxbb mailing list >> >>>> [email protected]mailto:[email protected] >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >>>> -- >> >>>> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. >> >>>> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. >> >>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. >> >>>> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom >> >>>> _______________________________________________ >> >>>> cctbxbb mailing list >> >>>> [email protected] >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >>> _______________________________________________ >> >>> cctbxbb mailing list >> >>> [email protected] >> >>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> _______________________________________________ >> >> cctbxbb mailing list >> >> [email protected] >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> > >> > >> > _______________________________________________ >> > cctbxbb mailing list >> > [email protected] >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> _______________________________________________ >> cctbxbb mailing list >> [email protected] >> http://phenix-online.org/mailman/listinfo/cctbxbb > > _______________________________________________ > cctbxbb mailing list > [email protected] > http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, PhD Associate Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, PhD Associate Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, PhD Associate Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, PhD Associate Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

Hi everyone, I updated the conda packages on the cctbx-dev channel and there are also now packages for macOS. The old packages have been removed so you can create a new environment with conda create -n test -c cctbx-dev cctbx python=2.7 conda activate test where "test" is the environment name and 2.7 is the python version. For the python version, you can also pick 3.6, 3.7, or 3.8. I think the latest conda will install 3.8 as the default python if the python argument is not provided. Or install into an existing environment (with python already installed) with the same command as before, conda install -c cctbx-dev cctbx Again, your .condarc file should have conda-forge as the first channel so that dependencies can be pulled correctly. The dispatchers should work now and $LIBTBX_BUILD will be set on activation of an environment (and unset on environment deactivation) or after installing into an existing environment, so you can do more interesting things like iotbx.fetch_pdb --all --mtz 1kp8 However, this package does not install databases, so molprobity.molprobity 1kp8.pdb 1kp8.mtz will fail because the database is not available (also probe and reduce are not built). Some known issues in no particular order, 1) You can run "libtbx.run_tests_parallel module=cctbx nproc=Auto" to run tests, but not all the test files were copied, so I got 30 failures here. The final package will probably not have the test files. 2) The other python dispatchers, like "cctbx.python", were not updated, so those will fail. But you can just run "python" in the active environment. 3) Parts of mmtbx still need to be updated for Python 3. 4) The LIBTBX_BUILD variable should be set for sh and csh shells. Let me know if it's not working for your shell (i.e. "echo $LIBTBX_BUILD" should show the same thing as $CONDA_PREFIX). 5) One future step is to enable building other CCTBX-like modules with this conda package. This will require some bookkeeping updates so that modules in the conda environment and the modules being built are tracked properly. Also, making sure that all the necessary headers exist in $PREFIX/include. Gergely, let me know if your scripts are working. Happy holidays! -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org On Wed, Dec 18, 2019 at 11:25 AM Billy Poon wrote:

Hi Gergely,

That's the other information that hasn't been updated yet in the conda package. :)

For LIBTBX_BUILD, I can set up environment variables sort of how the conda gcc package sets up environment variables. I have to double check to see if they're set after installation of the package and after activation of an environment. The variable will just be a copy of $CONDA_PREFIX.

The libtbx_env file is a file that exists in $LIBTBX_BUILD that basically stores configuration information. I'll have to modify the contents to update the locations of the different modules since the conda package will not have a "modules" directory.

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Wed, Dec 18, 2019 at 2:08 AM Gergely Katona wrote:

...

Hi Billy,

Conda install went fine with your instructions after rearranging the channels by putting cctbx last. I removed all environmental variables and previous build of cctbx.

import sys print (sys.path, sys.prefix)

['/home/gergely/anaconda3/lib/python36.zip', '/home/gergely/anaconda3/lib/python3.6', '/home/gergely/anaconda3/lib/python3.6/lib-dynload', '', '/home/gergely/anaconda3/lib/python3.6/site-packages', '/home/gergely/anaconda3/lib/python3.6/site-packages/IPython/extensions', '/home/gergely/.ipython'] /home/gergely/anaconda3

Many imports went fine including pymc3, but I encountered problems with these three:

from cctbx import miller import iotbx.pdb from iotbx import reflection_file_reader, mtz

I also got type error when handling space groups.

--------------------------------------------------------------------------- KeyError Traceback (most recent call last) <ipython-input-5-cedbd4f84f3d> in <module> 25 from cctbx import uctbx 26 from cctbx import sgtbx ---> 27 from cctbx import miller 28 #from iotbx import reflection_file_reader, mtz 29 sns.set_context("poster")

~/anaconda3/lib/python3.6/site-packages/cctbx/miller/__init__.py in <module> 11 12 from cctbx import crystal ---> 13 from cctbx import maptbx 14 from cctbx import sgtbx 15 from cctbx.sgtbx import lattice_symmetry

~/anaconda3/lib/python3.6/site-packages/cctbx/maptbx/__init__.py in <module> 15 from libtbx import adopt_init_args 16 from libtbx.utils import Sorry ---> 17 import libtbx.load_env 18 import math 19 import sys, os

~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module> 3 import libtbx.env_config 4 import os ----> 5 libtbx.env = libtbx.env_config.unpickle() 6 libtbx.env.set_os_environ_all_dist() 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME")

~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle() 2603 2604 def unpickle(): -> 2605 build_path = os.environ["LIBTBX_BUILD"] 2606 set_preferred_sys_prefix_and_sys_executable(build_path=build_path) 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb")

~/anaconda3/lib/python3.6/os.py in __getitem__(self, key) 667 except KeyError: 668 # raise KeyError with the original key value --> 669 raise KeyError(key) from None 670 return self.decodevalue(value) 671

KeyError: 'LIBTBX_BUILD'

--------------------------------------------------------------------------- ArgumentError Traceback (most recent call last) <ipython-input-7-2acd3a9ce26a> in <module> 26 return ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar 27 ---> 28 ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar=initializecrystal()

<ipython-input-7-2acd3a9ce26a> in initializecrystal() 11 uc = uctbx.unit_cell(unit_cell) 12 wavelength = 1.54980 ---> 13 xtal_symm = crystal.symmetry(unit_cell=unit_cell, space_group_symbol="P 43 21 2") 14 15 ms = miller.build_set(crystal_symmetry=xtal_symm,anomalous_flag=True, d_min=1.61)

~/anaconda3/lib/python3.6/site-packages/cctbx/crystal/__init__.py in __init__(self, unit_cell, space_group_symbol, space_group_info, space_group, correct_rhombohedral_setting_if_necessary, assert_is_compatible_unit_cell, raise_sorry_if_incompatible_unit_cell, force_compatible_unit_cell) 74 if (space_group_symbol is not None): 75 self._space_group_info = sgtbx.space_group_info( ---> 76 symbol=space_group_symbol) 77 elif (space_group is not None): 78 if (isinstance(space_group, sgtbx.space_group)):

~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/__init__.py in __init__(self, symbol, table_id, group, number, space_group_t_den) 100 assert group is None 101 if (table_id is None): --> 102 symbols = space_group_symbols(symbol) 103 else: 104 if (isinstance(symbol, int)): symbol = str(symbol)

ArgumentError: Python argument types in space_group_symbols.__init__(space_group_symbols, str) did not match C++ signature: __init__(_object*, int space_group_number) __init__(_object*, int space_group_number, std::__cxx11::basic_string extension='') __init__(_object*, int space_group_number, std::__cxx11::basic_string extension='', std::__cxx11::basic_string table_id='') __init__(_object*, std::__cxx11::basic_string symbol) __init__(_object*, std::__cxx11::basic_string symbol, std::__cxx11::basic_string table_id='')

When I tried to set LIBTBX_BUILD to /home/gergely/anaconda3 (this is sys.prefix on my system), I got other problems when importing. Can LIBTBX_BUILD be set in the conda package?

--------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) <ipython-input-1-7e2b144826cc> in <module> 26 from cctbx import sgtbx 27 #from cctbx import miller ---> 28 from iotbx import reflection_file_reader, mtz 29 sns.set_context("poster") 30 plt.rcParams.update({'figure.autolayout': True})

~/anaconda3/lib/python3.6/site-packages/iotbx/reflection_file_reader.py in <module> 53 54 from __future__ import absolute_import, division, print_function ---> 55 from iotbx import mtz 56 from iotbx.scalepack import merge as scalepack_merge 57 from iotbx.scalepack import no_merge_original_index as scalepack_no_merge

~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/__init__.py in <module> 9 import iotbx_mtz_ext as ext 10 ---> 11 from iotbx.mtz import extract_from_symmetry_lib 12 from cctbx import xray 13 import cctbx.xray.observation_types

~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/extract_from_symmetry_lib.py in <module> 1 from __future__ import absolute_import, division, print_function 2 from cctbx import sgtbx ----> 3 import libtbx.load_env 4 import os.path as op 5 from six.moves import range

~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module> 3 import libtbx.env_config 4 import os ----> 5 libtbx.env = libtbx.env_config.unpickle() 6 libtbx.env.set_os_environ_all_dist() 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME")

~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle() 2605 build_path = os.environ["LIBTBX_BUILD"] 2606 set_preferred_sys_prefix_and_sys_executable(build_path=build_path) -> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb") 2608 env = pickle.load(libtbx_env) 2609 if (env.python_version_major_minor != sys.version_info[:2]):

FileNotFoundError: [Errno 2] No such file or directory: '/home/gergely/anaconda3/libtbx_env'

On Wed, Dec 18, 2019 at 9:10 AM Billy Poon wrote:

...

Hi Gergely,

I've uploaded linux packages to a new channel, cctbx-dev, and you can

install it with

...

conda install -c cctbx-dev cctbx

in your current environment. You should set your ~/.condarc file to

pull the other dependencies from the conda-forge channel first, so put conda-forge above cctbx. Mine looks like

...

channels: - conda-forge - defaults - cctbx

Do you need dxtbx for your scripts? This package does not build that

part. I think the plan is to build a separate conda package for dxtbx so that it can be updated more frequently. I can rebuild the packages to include it for testing, but the one being submitted to conda-forge will not have it.

...

Also, your error message is probably due to version of HDF5 that the

development build installs. The bootstrap.py script will install 1.10.4, but your other dependency is looking for 1.10.5. Installing this cctbx conda package should install 1.10.5, which should fix the issue. Also, I'm updating those environments and 1.10.5 will be the new default version.

...

Lastly, the dispatchers will not work in these packages because the old

paths during the build process are still in them and many of them expect some additional information that has not been updated in the packages yet. I'm in the process of doing that and will update the cctbx-dev channel when that's done. However, by starting python, you can import cctbx modules. So you can run commands like

...

from scitbx.array_family import flex a = flex.random_double(1000000) b = flex.min_max_mean_double(a) b.min b.max b.mean

Thanks!

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Tue, Dec 17, 2019 at 5:39 AM Gergely Katona

...

...

Dear Billy,

Thank you for the detailed explanation, I look forward to do further testing! I aim to pool all modules under the same environment, if there are conflicts then I just try to reshuffle the order of imports. So far this did not cause problems for me even when I was using system python, but of course it is not the most prudent thing to do. With anaconda everything is much more standardized and isolated already and without being superuser I can have my familiar environment at any synchrotron based cluster. It is great that cctbx will be an integral part of this ecosystem and this was also the last thing holding me back from adopting python3. About hdf5, pymc3 requires it and importing causes a kernel restart with the following error messages, curiously if I import h5py first this can be avoided.

Best wishes,

Gergely

import pymc3 as pm

Warning! ***HDF5 library version mismatched error*** The HDF5 header files used to compile this application do not match the version used by the HDF5 library to which this application is

...

...

Data corruption or segmentation faults may occur if the application continues. This can happen when an application was compiled by one version of HDF5 but linked with a different version of static or shared HDF5 library. You should recompile the application or check your shared library related settings such as 'LD_LIBRARY_PATH'. You can, at your own risk, disable this warning by setting the environment variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'. Setting it to 2 or higher will suppress the warning messages totally. Headers are 1.10.4, library is 1.10.5 SUMMARY OF THE HDF5 CONFIGURATION =================================

General Information: ------------------- HDF5 Version: 1.10.5 Configured on: Tue Oct 22 12:02:13 UTC 2019 Configured by: conda@16247e67ecd5 Host system: x86_64-conda_cos6-linux-gnu Uname information: Linux 16247e67ecd5 4.15.0-1059-azure #64-Ubuntu SMP Fri Sep 13 17:02:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Byte sex: little-endian Installation point: /home/gergely/anaconda3

Compiling Options: ------------------ Build Mode: production Debugging Symbols: no Asserts: no Profiling: no Optimization Level: high

Linking Options: ---------------- Libraries: static, shared Statically Linked Executables: LDFLAGS: -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/gergely/anaconda3/lib -Wl,-rpath-link,/home/gergely/anaconda3/lib -L/home/gergely/anaconda3/lib H5_LDFLAGS: AM_LDFLAGS: -L/home/gergely/anaconda3/lib Extra libraries: -lrt -lpthread -lz -ldl -lm Archiver:

/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar

...

AR_FLAGS: cr Ranlib:

/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib

...

Languages: ---------- C: yes C Compiler:

/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc

...

CPPFLAGS: -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/home/gergely/anaconda3/include H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200809L -DNDEBUG -UH5_DEBUG_API AM_CPPFLAGS: -I/home/gergely/anaconda3/include C Flags: -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/home/gergely/anaconda3/include

-fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5

...

-fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix H5 C Flags: -std=c99 -pedantic -Wall -Wextra -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual -Wconversion -Wdeclaration-after-statement -Wdisabled-optimization -Wfloat-equal -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum -Wundef -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings -finline-functions -s -Wno-inline -Wno-aggregate-return -Wno-missing-format-attribute -Wno-missing-noreturn -O AM C Flags: Shared C Library: yes Static C Library: yes

Fortran: yes Fortran Compiler:

/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran

...

Fortran Flags: H5 Fortran Flags: -pedantic -Wall -Wextra -Wunderflow -Wimplicit-interface -Wsurprising -Wno-c-binding-type -s -O2 AM Fortran Flags: Shared Fortran Library: yes Static Fortran Library: yes

C++: yes C++ Compiler:

/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++

...

C++ Flags: -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/home/gergely/anaconda3/include

-fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5

...

-fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix H5 C++ Flags: -pedantic -Wall -W -Wundef -Wshadow -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Wconversion -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor -Wctor-dtor-privacy -Wabi -finline-functions -s -O AM C++ Flags: Shared C++ Library: yes Static C++ Library: yes

Java: no

Features: --------- Parallel HDF5: no Parallel Filtered Dataset Writes: no Large Parallel I/O: no High-level library: yes Threadsafety: yes Default API mapping: v110 With deprecated public symbols: yes I/O filters (external): deflate(zlib) MPE: no Direct VFD: no dmalloc: no Packages w/ extra debug output: none API tracing: no Using memory checker: yes Memory allocation sanity checks: no Function stack tracing: no Strict file format checks: no Optimization instrumentation: no

On Tue, Dec 17, 2019 at 8:35 AM Billy Poon wrote:

...

Hi Gergely,

Let me build the test package tomorrow. All the gritty details for

building with conda is being finalized and the official documentation will be updated to describe the steps. It would be too confusing to keep changing the documentation as the process evolves. You do have the general

...

...

...

1) Installing dependencies. The cctbx_dependencies metapackage was

an initial approach for managing the CCTBX dependencies, but after contacting the conda-forge folks, they recommended using the --only-deps flag. So when the CCTBX conda package is available, you'll be able to get a set of dependencies with,

...

conda install -c conda-forge --only-deps cctbx

By default, the bootstrap.py file will automatically install a set

of dependencies in the "conda_base" directory (and a conda installation if one is not found). It just uses standard conda environment files located in

...

...

...

2) Building. The bootstrap.py file handles that with SCons.

3) Running. After building, there should be a setpaths.sh (and .csh)

file that adds build/bin to your path. The build/bin directory has our dispatchers, which are just scripts that set up the environment variables for you. This prevents other programs from loading our libraries, whose versions may conflict. You should see that there is a "python" dispatcher, which is a convenience for developers. Otherwise, you can use

...

...

...

The running part is where the conda package for CCTBX will be

different than this build. Since our Python files and extensions modules will be in the "site-packages" directory for Python, the PYTHONPATH variable will not be needed (and conda suggests that that variable not be set). The other CCTBX libraries will be in $CONDA_PREFIX/lib, so LD_LIBRARY_PATH is not needed. In an active environment, PATH will already be modified. And then our LIBTBX_BUILD directory can be set to sys.prefix.

...

So with the conda package, you would only need to activate your

conda environment and CCTBX should integrate with other conda packages. What is the conflict with hdf5? That's something that should be fixed. Thanks!

...

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Mon, Dec 16, 2019 at 2:53 AM Gergely Katona

wrote:

...

...

Dear Billy,

Thank you for this update and for your efforts! I found a solution

and

...

indeed most things already work in anaconda3. The steps I took (even if these instructions will have short expiry date):

Modified .condarc with: channels: - cctbx - conda-forge - defaults

run conda conda install cctbx_dependencies python=3.6

I expect this will work without python=3.6 in the near future.

Then compiling cctbx with anaconda3 python went without any problem when using these flags: python bootstrap.py --use-conda --python3 --nproc=12

Then I probably did the unorthodox thing and sourced these

...

...

...

...

setenv LIBTBX_BUILD /home/gergely/cctbx/build setenv PATH ${PATH}:/home/gergely/cctbx/build/bin setenv PYTHONPATH

/home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH

...

setenv LD_LIBRARY_PATH

/home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH

...

There is probably a better way to put this into conda environment. With these steps I could run one of my scripts depending on cctbx.

The

...

only problem I found is that hdf5 library had conflict with another package in conda, but I does not find this as a showstopper.

Best wishes,

Gergely

On Fri, Dec 13, 2019 at 7:23 PM Billy Poon wrote: > > Hi Gergely, > > It's still a work in progress. I'm sorting out some Windows issues right now. I can probably build a test package on a separate channel for people that want to test it (let's say next week?). I'll provide instructions, but basically, the test conda package will be in its own separate channel and the dependencies will be pulled from the conda-forge channel. I want most things to be working correctly on Python 2.7, 3.6, 3.7, and 3.8 on all 3 platforms. > > Thanks! > > -- > Billy K. Poon > Research Scientist, Molecular Biophysics and Integrated Bioimaging > Lawrence Berkeley National Laboratory > 1 Cyclotron Road, M/S 33R0345 > Berkeley, CA 94720 > Tel: (510) 486-5709 > Fax: (510) 486-5909 > Web: https://phenix-online.org > > > On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona wrote: >> >> Dear Billy, >> >> This sounds very promising and exciting. I am not sure if cctbx is >> already functional as a conda package in anaconda3 (Linux) or

...

...

...

...

>> still work in progress. My technical expertise does not allow me to >> tell the difference. What I tried: >> >> Fresh install of anaconda3. Adding - cctbx and - conda-forge to >> .condarc . Installing pyside2 with conda. Running conda install >> conda_dependencies . I get a lot package version conflict, and I >> cannot import cctbx or iotbx to anaconda python. Am I following

...

...

...

...

>> right instructions? Or it is too early to expect that cctbx works when >> installed through conda? >> >> Best wishes, >> >> Gergely >> >> On Wed, Nov 27, 2019 at 3:56 PM Billy Poon wrote: >> > >> > Hi all, >> > >> > For a brief update, I have submitted a recipe for cctbxlite to conda-forge (https://github.com/conda-forge/staged-recipes/pull/10021) and support for Python 3.7 and 3.8 is being added ( https://github.com/cctbx/cctbx_project/pull/409). With some fixes for Windows (https://github.com/cctbx/cctbx_project/pull/416), all platforms (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and 3.8. Some additional changes will be needed to get Windows to work with Python 3 and for tests to pass with Boost 1.70.0. That will enable the conda-forge recipe to build for all platforms and for all supported versions of Python. >> > >> > Currently, the conda-forge recipe will install into the conda

...

...

...

...

>> > >> > -- >> > Billy K. Poon >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging >> > Lawrence Berkeley National Laboratory >> > 1 Cyclotron Road, M/S 33R0345 >> > Berkeley, CA 94720 >> > Tel: (510) 486-5709 >> > Fax: (510) 486-5909 >> > Web: https://phenix-online.org >> > >> > >> > On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll wrote: >> >> >> >> Hi Luc, >> >> >> >> That sounds promising. From there, I’d need to work out how to make a fully-packaged installer (basically a modified wheel file) for

...

...

...

...

>> >> >> >> Thanks, >> >> >> >> Tristan >> >> >> >> >> >> >> >> Tristan Croll >> >> Research Fellow >> >> Cambridge Institute for Medical Research >> >> University of Cambridge CB2 0XY >> >> >> >> >> >> >> >> >> >> > On 25 Aug 2019, at 18:31, Luc Bourhis < [email protected]> wrote: >> >> > >> >> > Hi Tristan, >> >> > >> >> > cctbx could be built to use your ChimeraX python, now that cctbx is moving to Python 3. The option —with-python is there for that with

...

...

...

...

>> >> > >> >> > Best wishes, >> >> > >> >> > Luc >> >> > >> >> > >> >> >> On 23 Aug 2019, at 19:02, Tristan Croll wrote: >> >> >> >> >> >> To add my two cents on this: probably the second-most common question I've had about ISOLDE's implementation is, "why didn't you use CCTBX?". The honest answer to that is, "I didn't know how." >> >> >> >> >> >> Still don't, really - although the current developments are rather promising. The problem I've faced is that CCTBX was designed as its own self-contained Python (2.7, until very recently) environment, with its own interpreter and a lot of very specific environment setup. Meanwhile I'm developing ISOLDE in ChimeraX, which is *also* its own self-contained Python (3.7) environment. To plug one into the other in that form... well, I don't think I'm a good enough programmer to really know where to start. >> >> >> >> >> >> The move to Conda and a more modular CCTBX architecture should make a lot more possible in that direction. Pip would be even better for me personally (ChimeraX can install directly from the PyPI, but doesn't interact with Conda) - but I understand pretty well the substantial challenge that would amount to (not least being that the PyPI imposes a

...

...

...

...

>> >> >> >> >> >> Best regards, >> >> >> >> >> >> Tristan >> >> >> >> >> >>> On 2019-08-23 09:28, Luc Bourhis wrote: >> >> >>> Hi Graeme, >> >> >>> Yes, I know. But “black" is a program doing a very

...

...

...

...

>> >> >>> (code formatting from the top of my head). Requiring to use a wrapper >> >> >>> for python itself is another level. But ok, I think I am mellowing to >> >> >>> the idea after all! Talking with people around me, and extrapolating, >> >> >>> I would bet that, right now, a great majority of people interested by >> >> >>> cctbx in pip have already used the cctbx, so they know about the >> >> >>> Python wrapper, and they would not be too sanguine about

...

...

...

...

>> >> >>> concern is for the future, when pip will be the first time some people >> >> >>> use cctbx. Big fat warning notices on PyPI page and a better error >> >> >>> message when cctbx fails because LIBTBX_BUILD is not set would be >> >> >>> needed but that could be all right. >> >> >>> If we do a pip installer, we should aim at a minimal install: cctbx, >> >> >>> iotbx and their dependencies, and that’s it. >> >> >>> Best wishes, >> >> >>> Luc >> >> >>>> On 23 Aug 2019, at 07:17, [email protected] < [email protected]> wrote: >> >> >>>> Without discussing the merits of this or whether we _choose_ to make the move to supporting PIP, I am certain it would be _possible_ - many other packages make dispatcher scripts when you pip install them e.g. >> >> >>>> Silver-Surfer rescale_f2 :) $ which black; cat $(which black) >> >> >>>> /Library/Frameworks/Python.framework/Versions/3.6/bin/black >> >> >>>> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 >> >> >>>> # -*- coding: utf-8 -*- >> >> >>>> import re >> >> >>>> import sys >> >> >>>> from black import main >> >> >>>> if __name__ == '__main__': >> >> >>>> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) >> >> >>>> sys.exit(main()) >> >> >>>> So we _could_ work around the absence of LIBTBX_BUILD etc. in the system. Whether or not we elect to do the work is a different question, and it seems clear that here are very mixed opinions on this. >> >> >>>> Best wishes Graeme >> >> >>>> On 23 Aug 2019, at 01:21, Luc Bourhis < [email protected]mailto:[email protected]> wrote: >> >> >>>> Hi, >> >> >>>> Even if we managed to ship our the boost dynamic

...

...

...

...

>> >> >>>> Best wishes, >> >> >>>> Luc >> >> >>>> On 16 Aug 2019, at 22:47, Luc Bourhis < [email protected]mailto:[email protected]> wrote: >> >> >>>> Hi, >> >> >>>> I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond

...

...

...

...

>> >> >>>> Best wishes, >> >> >>>> Luc >> >> >>>> On 16 Aug 2019, at 21:45, Aaron Brewster < [email protected]mailto:[email protected]> wrote: >> >> >>>> Hi, to avoid clouding Dorothee's documentation email

...

...

...

...

>> >> >>>> All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip

...

...

...

...

>> >> >>>> There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6). >> >> >>>> If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py

...

...

...

...

>> >> >>>> Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does

...

...

...

...

>> >> >>>> Finally, there's the option of providing a source pip

...

...

...

...

>> >> >>>> Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort. >> >> >>>> Thanks, >> >> >>>> -Aaron >> >> >>>> _______________________________________________ >> >> >>>> cctbxbb mailing list >> >> >>>> [email protected] >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >>>> _______________________________________________ >> >> >>>> cctbxbb mailing list >> >> >>>> [email protected] >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >>>> _______________________________________________ >> >> >>>> cctbxbb mailing list >> >> >>>> [email protected] >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >>>> -- >> >> >>>> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of

...

...

...

...

>> >> >>>> Any opinions expressed within this e-mail are those of

...

...

...

...

>> >> >>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept

wrote: linked. process, though, which is summarized as follows. libtbx/auto_build/conda_envs, so you do not need to install cctbx_dependencies as a separate step. The environment files avoid channel issues by explicitly defining the channel to pull the packages from and the cctbx channel just stores copies of packages from conda-forge. There were issues earlier where the conda-forge packages would sometimes be moved to a different label. The --use-conda flag also accepts a path to $CONDA_PREFIX if you want to use a specific environment for testing. libtbx.python, which will be able to import CCTBX modules. directories: this is the python and cctbx imports can be done without sourcing the environment scripts. It looks like a lot of the environment variables being set in the dispatchers can be removed since the Python files and C++ extensions are in the right places. I'll update the libtbx_env file so that commands that load the environment can work correctly. the ChimeraX ToolShed - the aim is for the end user to not have to worry about any of this. That adds a couple of complications - e.g. $LIBTBX_BUILD would need to be set dynamically before first import - but doesn’t seem insurmountable. the bootstrap script. The specific environment setup boil down to setting two environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on Linux, PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a framework such as ChimeraX, that should not be difficult to ensure those two variables are set. limit - around 100MB from memory? - on the size of an individual package). particular task that. My libraries with pip, it would still not be pip-like, as we would still need our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work with the standard python exe. LD_LIBRARY_PATH, we could get around that by changing the way we compile, using -Wl,-R, which is the runtime equivalent of build time -L. That’s a significant change that would need to be tested. But there is no way around setting LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps there is a way to hack libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip installs? pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.orghttp://anaconda.org/, and then to advertise it hard to every potential user out there. thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip ( https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-...). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not. package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard ( https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support. package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4. pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :) package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users. the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. the individual and not necessarily of Diamond Light Source Ltd. liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.

...

...

...

...

>> >> >>>> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom >> >> >>>> _______________________________________________ >> >> >>>> cctbxbb mailing list >> >> >>>> [email protected] >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >>> _______________________________________________ >> >> >>> cctbxbb mailing list >> >> >>> [email protected] >> >> >>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> cctbxbb mailing list >> >> >> [email protected] >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> > >> >> > >> >> > _______________________________________________ >> >> > cctbxbb mailing list >> >> > [email protected] >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> _______________________________________________ >> >> cctbxbb mailing list >> >> [email protected] >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> > >> > _______________________________________________ >> > cctbxbb mailing list >> > [email protected] >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> -- >> Gergely Katona, PhD >> Associate Professor >> Department of Chemistry and Molecular Biology, University of Gothenburg >> Box 462, 40530 Göteborg, Sweden >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 >> Web: http://katonalab.eu, Email: [email protected] >> >> _______________________________________________ >> cctbxbb mailing list >> [email protected] >> http://phenix-online.org/mailman/listinfo/cctbxbb > > _______________________________________________ > cctbxbb mailing list > [email protected] > http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, PhD Associate Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, PhD Associate Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, PhD Associate Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

I found some other errors when using the test environment. --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) <ipython-input-1-fe308b345b3f> in <module> 10 import scipy as sp 11 from scipy import linalg ---> 12 import iotbx.pdb 13 import cctbx 14 import pandas as pd ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/__init__.py in <module> 8 9 import iotbx.pdb.records ---> 10 import iotbx.pdb.hierarchy 11 from scitbx import matrix 12 ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/hierarchy.py in <module> 14 from six.moves import cStringIO as StringIO 15 from iotbx.pdb import hy36encode, hy36decode ---> 16 import iotbx.cif.model 17 from cctbx import crystal 18 from libtbx import group_args ~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/cif/__init__.py in <module> 19 20 from cctbx.array_family import flex ---> 21 from cctbx import miller 22 from iotbx.cif import model, builders, geometry 23 from libtbx.containers import OrderedDict ~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/miller/__init__.py in <module> 11 12 from cctbx import crystal ---> 13 from cctbx import maptbx 14 from cctbx import sgtbx 15 from cctbx.sgtbx import lattice_symmetry ~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/maptbx/__init__.py in <module> 15 from libtbx import adopt_init_args 16 from libtbx.utils import Sorry ---> 17 import libtbx.load_env 18 import math 19 import sys, os ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/load_env.py in <module> 3 import libtbx.env_config 4 import os ----> 5 libtbx.env = libtbx.env_config.unpickle() 6 libtbx.env.set_os_environ_all_dist() 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME") ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/env_config.py in unpickle() 2736 env = pickle.load(libtbx_env) 2737 if (env.python_version_major_minor != sys.version_info[:2]): -> 2738 env.raise_python_version_incompatible() 2739 if (op.realpath(build_path) != op.realpath(abs(env.build_path))): 2740 env.build_path.reset(build_path) ~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/env_config.py in raise_python_version_incompatible(self, prev_pvmm) 469 if (prev_pvmm is None): 470 prev_pvmm = "%d.%d" % self.python_version_major_minor --> 471 raise RuntimeError("Python version incompatible with this build:\n" 472 + " Build directory: %s\n" % show_string(abs(self.build_path)) 473 + " Python version used initially: %s\n" % prev_pvmm RuntimeError: Python version incompatible with this build: Build directory: "/home/gergely/anaconda3" Python version used initially: 3.6 Python version in use now: 3.8 Interestingly in the base environment all import worked, but I got an error at: ArgumentError Traceback (most recent call last) <ipython-input-4-5bc71c36e121> in <module> 33 atm_odd.set_b(0) 34 ---> 35 odd.write_pdb_file("bdiff_in_even_000.pdb") #skriver ny pdb med dessa inkluderade 36 dfWBvals=pd.DataFrame(rows_list) ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in write_pdb_file(self, file_name, open_append, crystal_symmetry, cryst1_z, write_scale_records, append_end, atom_hetatm, sigatm, anisou, siguij) 930 siguij=True): 931 if (crystal_symmetry is Auto): --> 932 crystal_symmetry = self.crystal_symmetry() 933 if (cryst1_z is Auto): 934 cryst1_z = self.extract_cryst1_z_columns() ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in crystal_symmetry(self, crystal_symmetry, weak_symmetry) 1218 crystal_symmetry=None, 1219 weak_symmetry=False): -> 1220 self_symmetry = self.crystal_symmetry_from_cryst1() 1221 if (self_symmetry is None): 1222 self_symmetry = self._crystal_symmetry_from_cns_remark_sg() ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in crystal_symmetry_from_cryst1(self) 1195 for line in self.crystallographic_section(): 1196 if (line.startswith("CRYST1")): -> 1197 return cryst1_interpretation.crystal_symmetry(cryst1_record=line) 1198 return None 1199 ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/cryst1_interpretation.py in crystal_symmetry(cryst1_record) 136 space_group_info=None) 137 space_group_info = categorize(cryst1_record.sgroup).space_group_info( --> 138 unit_cell=u) 139 return crystal.symmetry(unit_cell=u, space_group_info=space_group_info) ~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/cryst1_interpretation.py in space_group_info(self, unit_cell) 85 if (self.symbol is None): return None 86 if (self.category is None): ---> 87 try: return sgtbx.space_group_info(self.symbol) 88 except RuntimeError: return None 89 if (isinstance(unit_cell, uctbx.ext.unit_cell)): ~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/__init__.py in __init__(self, symbol, table_id, group, number, space_group_t_den) 100 assert group is None 101 if (table_id is None): --> 102 symbols = space_group_symbols(symbol) 103 else: 104 if (isinstance(symbol, int)): symbol = str(symbol) ArgumentError: Python argument types in space_group_symbols.__init__(space_group_symbols, str) did not match C++ signature: __init__(_object*, int space_group_number) __init__(_object*, int space_group_number, std::__cxx11::basic_string extension='') __init__(_object*, int space_group_number, std::__cxx11::basic_string extension='', std::__cxx11::basic_string table_id='') __init__(_object*, std::__cxx11::basic_string symbol) __init__(_object*, std::__cxx11::basic_string symbol, std::__cxx11::basic_string table_id='') Gergely On Fri, Jan 3, 2020 at 1:44 PM Gergely Katona wrote:

Dear Billy,

I did some tests installing in my base conda was not troublefree in python 3.8. Running iotbx.fetch_pdb --all --mtz 1kp8 resulted in boost related error when converting to mtz. Installing into a test environment went fine with iotbx.fetch_pdb --all --mtz 1kp8 running successfully. I could add pymc3, jupyter and seaborn and iotbx still did not break. My LIBTBX_BUILD is not set however in the test environment, which breaks miller when I was trying to import it. Manually setting the LIBTBX_BUILD to CONDA_PREFIX (setenv LIBTBX_BUILD $CONDA_PREFIX) fixes this and one of my script works fine! I will come back if I find some problems with my other scripts, but so far this looks promising!

Happy new year!

Gergely

On Fri, Dec 20, 2019 at 8:16 PM Billy Poon wrote:

...

Hi everyone,

I updated the conda packages on the cctbx-dev channel and there are also now packages for macOS. The old packages have been removed so you can create a new environment with

conda create -n test -c cctbx-dev cctbx python=2.7 conda activate test

where "test" is the environment name and 2.7 is the python version. For the python version, you can also pick 3.6, 3.7, or 3.8. I think the latest conda will install 3.8 as the default python if the python argument is not provided. Or install into an existing environment (with python already installed) with the same command as before,

conda install -c cctbx-dev cctbx

Again, your .condarc file should have conda-forge as the first channel so that dependencies can be pulled correctly.

The dispatchers should work now and $LIBTBX_BUILD will be set on activation of an environment (and unset on environment deactivation) or after installing into an existing environment, so you can do more interesting things like

iotbx.fetch_pdb --all --mtz 1kp8

However, this package does not install databases, so

molprobity.molprobity 1kp8.pdb 1kp8.mtz

will fail because the database is not available (also probe and reduce are not built).

Some known issues in no particular order,

1) You can run "libtbx.run_tests_parallel module=cctbx nproc=Auto" to run tests, but not all the test files were copied, so I got 30 failures here. The final package will probably not have the test files. 2) The other python dispatchers, like "cctbx.python", were not updated, so those will fail. But you can just run "python" in the active environment. 3) Parts of mmtbx still need to be updated for Python 3. 4) The LIBTBX_BUILD variable should be set for sh and csh shells. Let me know if it's not working for your shell (i.e. "echo $LIBTBX_BUILD" should show the same thing as $CONDA_PREFIX). 5) One future step is to enable building other CCTBX-like modules with this conda package. This will require some bookkeeping updates so that modules in the conda environment and the modules being built are tracked properly. Also, making sure that all the necessary headers exist in $PREFIX/include.

Gergely, let me know if your scripts are working.

Happy holidays!

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Wed, Dec 18, 2019 at 11:25 AM Billy Poon wrote:

...

Hi Gergely,

That's the other information that hasn't been updated yet in the conda package. :)

For LIBTBX_BUILD, I can set up environment variables sort of how the conda gcc package sets up environment variables. I have to double check to see if they're set after installation of the package and after activation of an environment. The variable will just be a copy of $CONDA_PREFIX.

The libtbx_env file is a file that exists in $LIBTBX_BUILD that basically stores configuration information. I'll have to modify the contents to update the locations of the different modules since the conda package will not have a "modules" directory.

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Wed, Dec 18, 2019 at 2:08 AM Gergely Katona wrote:

...

Hi Billy,

Conda install went fine with your instructions after rearranging the channels by putting cctbx last. I removed all environmental variables and previous build of cctbx.

import sys print (sys.path, sys.prefix)

['/home/gergely/anaconda3/lib/python36.zip', '/home/gergely/anaconda3/lib/python3.6', '/home/gergely/anaconda3/lib/python3.6/lib-dynload', '', '/home/gergely/anaconda3/lib/python3.6/site-packages', '/home/gergely/anaconda3/lib/python3.6/site-packages/IPython/extensions', '/home/gergely/.ipython'] /home/gergely/anaconda3

Many imports went fine including pymc3, but I encountered problems with these three:

from cctbx import miller import iotbx.pdb from iotbx import reflection_file_reader, mtz

I also got type error when handling space groups.

--------------------------------------------------------------------------- KeyError Traceback (most recent call last) <ipython-input-5-cedbd4f84f3d> in <module> 25 from cctbx import uctbx 26 from cctbx import sgtbx ---> 27 from cctbx import miller 28 #from iotbx import reflection_file_reader, mtz 29 sns.set_context("poster")

~/anaconda3/lib/python3.6/site-packages/cctbx/miller/__init__.py in <module> 11 12 from cctbx import crystal ---> 13 from cctbx import maptbx 14 from cctbx import sgtbx 15 from cctbx.sgtbx import lattice_symmetry

~/anaconda3/lib/python3.6/site-packages/cctbx/maptbx/__init__.py in <module> 15 from libtbx import adopt_init_args 16 from libtbx.utils import Sorry ---> 17 import libtbx.load_env 18 import math 19 import sys, os

~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module> 3 import libtbx.env_config 4 import os ----> 5 libtbx.env = libtbx.env_config.unpickle() 6 libtbx.env.set_os_environ_all_dist() 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME")

~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle() 2603 2604 def unpickle(): -> 2605 build_path = os.environ["LIBTBX_BUILD"] 2606 set_preferred_sys_prefix_and_sys_executable(build_path=build_path) 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb")

~/anaconda3/lib/python3.6/os.py in __getitem__(self, key) 667 except KeyError: 668 # raise KeyError with the original key value --> 669 raise KeyError(key) from None 670 return self.decodevalue(value) 671

KeyError: 'LIBTBX_BUILD'

--------------------------------------------------------------------------- ArgumentError Traceback (most recent call last) <ipython-input-7-2acd3a9ce26a> in <module> 26 return ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar 27 ---> 28 ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar=initializecrystal()

<ipython-input-7-2acd3a9ce26a> in initializecrystal() 11 uc = uctbx.unit_cell(unit_cell) 12 wavelength = 1.54980 ---> 13 xtal_symm = crystal.symmetry(unit_cell=unit_cell, space_group_symbol="P 43 21 2") 14 15 ms = miller.build_set(crystal_symmetry=xtal_symm,anomalous_flag=True, d_min=1.61)

~/anaconda3/lib/python3.6/site-packages/cctbx/crystal/__init__.py in __init__(self, unit_cell, space_group_symbol, space_group_info, space_group, correct_rhombohedral_setting_if_necessary, assert_is_compatible_unit_cell, raise_sorry_if_incompatible_unit_cell, force_compatible_unit_cell) 74 if (space_group_symbol is not None): 75 self._space_group_info = sgtbx.space_group_info( ---> 76 symbol=space_group_symbol) 77 elif (space_group is not None): 78 if (isinstance(space_group, sgtbx.space_group)):

~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/__init__.py in __init__(self, symbol, table_id, group, number, space_group_t_den) 100 assert group is None 101 if (table_id is None): --> 102 symbols = space_group_symbols(symbol) 103 else: 104 if (isinstance(symbol, int)): symbol = str(symbol)

ArgumentError: Python argument types in space_group_symbols.__init__(space_group_symbols, str) did not match C++ signature: __init__(_object*, int space_group_number) __init__(_object*, int space_group_number, std::__cxx11::basic_string extension='') __init__(_object*, int space_group_number, std::__cxx11::basic_string extension='', std::__cxx11::basic_string table_id='') __init__(_object*, std::__cxx11::basic_string symbol) __init__(_object*, std::__cxx11::basic_string symbol, std::__cxx11::basic_string table_id='')

When I tried to set LIBTBX_BUILD to /home/gergely/anaconda3 (this is sys.prefix on my system), I got other problems when importing. Can LIBTBX_BUILD be set in the conda package?

--------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) <ipython-input-1-7e2b144826cc> in <module> 26 from cctbx import sgtbx 27 #from cctbx import miller ---> 28 from iotbx import reflection_file_reader, mtz 29 sns.set_context("poster") 30 plt.rcParams.update({'figure.autolayout': True})

~/anaconda3/lib/python3.6/site-packages/iotbx/reflection_file_reader.py in <module> 53 54 from __future__ import absolute_import, division, print_function ---> 55 from iotbx import mtz 56 from iotbx.scalepack import merge as scalepack_merge 57 from iotbx.scalepack import no_merge_original_index as scalepack_no_merge

~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/__init__.py in <module> 9 import iotbx_mtz_ext as ext 10 ---> 11 from iotbx.mtz import extract_from_symmetry_lib 12 from cctbx import xray 13 import cctbx.xray.observation_types

~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/extract_from_symmetry_lib.py in <module> 1 from __future__ import absolute_import, division, print_function 2 from cctbx import sgtbx ----> 3 import libtbx.load_env 4 import os.path as op 5 from six.moves import range

~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module> 3 import libtbx.env_config 4 import os ----> 5 libtbx.env = libtbx.env_config.unpickle() 6 libtbx.env.set_os_environ_all_dist() 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME")

~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle() 2605 build_path = os.environ["LIBTBX_BUILD"] 2606 set_preferred_sys_prefix_and_sys_executable(build_path=build_path) -> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb") 2608 env = pickle.load(libtbx_env) 2609 if (env.python_version_major_minor != sys.version_info[:2]):

FileNotFoundError: [Errno 2] No such file or directory: '/home/gergely/anaconda3/libtbx_env'

On Wed, Dec 18, 2019 at 9:10 AM Billy Poon wrote:

...

Hi Gergely,

I've uploaded linux packages to a new channel, cctbx-dev, and you can install it with

conda install -c cctbx-dev cctbx

in your current environment. You should set your ~/.condarc file to pull the other dependencies from the conda-forge channel first, so put conda-forge above cctbx. Mine looks like

channels: - conda-forge - defaults - cctbx

Do you need dxtbx for your scripts? This package does not build that part. I think the plan is to build a separate conda package for dxtbx so that it can be updated more frequently. I can rebuild the packages to include it for testing, but the one being submitted to conda-forge will not have it.

Also, your error message is probably due to version of HDF5 that the development build installs. The bootstrap.py script will install 1.10.4, but your other dependency is looking for 1.10.5. Installing this cctbx conda package should install 1.10.5, which should fix the issue. Also, I'm updating those environments and 1.10.5 will be the new default version.

Lastly, the dispatchers will not work in these packages because the old paths during the build process are still in them and many of them expect some additional information that has not been updated in the packages yet. I'm in the process of doing that and will update the cctbx-dev channel when that's done. However, by starting python, you can import cctbx modules. So you can run commands like

from scitbx.array_family import flex a = flex.random_double(1000000) b = flex.min_max_mean_double(a) b.min b.max b.mean

Thanks!

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Tue, Dec 17, 2019 at 5:39 AM Gergely Katona wrote:

...

Dear Billy,

Thank you for the detailed explanation, I look forward to do further testing! I aim to pool all modules under the same environment, if there are conflicts then I just try to reshuffle the order of imports. So far this did not cause problems for me even when I was using system python, but of course it is not the most prudent thing to do. With anaconda everything is much more standardized and isolated already and without being superuser I can have my familiar environment at any synchrotron based cluster. It is great that cctbx will be an integral part of this ecosystem and this was also the last thing holding me back from adopting python3. About hdf5, pymc3 requires it and importing causes a kernel restart with the following error messages, curiously if I import h5py first this can be avoided.

Best wishes,

Gergely

import pymc3 as pm

Warning! ***HDF5 library version mismatched error*** The HDF5 header files used to compile this application do not match the version used by the HDF5 library to which this application is linked. Data corruption or segmentation faults may occur if the application continues. This can happen when an application was compiled by one version of HDF5 but linked with a different version of static or shared HDF5 library. You should recompile the application or check your shared library related settings such as 'LD_LIBRARY_PATH'. You can, at your own risk, disable this warning by setting the environment variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'. Setting it to 2 or higher will suppress the warning messages totally. Headers are 1.10.4, library is 1.10.5 SUMMARY OF THE HDF5 CONFIGURATION =================================

General Information: ------------------- HDF5 Version: 1.10.5 Configured on: Tue Oct 22 12:02:13 UTC 2019 Configured by: conda@16247e67ecd5 Host system: x86_64-conda_cos6-linux-gnu Uname information: Linux 16247e67ecd5 4.15.0-1059-azure #64-Ubuntu SMP Fri Sep 13 17:02:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Byte sex: little-endian Installation point: /home/gergely/anaconda3

Compiling Options: ------------------ Build Mode: production Debugging Symbols: no Asserts: no Profiling: no Optimization Level: high

Linking Options: ---------------- Libraries: static, shared Statically Linked Executables: LDFLAGS: -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/gergely/anaconda3/lib -Wl,-rpath-link,/home/gergely/anaconda3/lib -L/home/gergely/anaconda3/lib H5_LDFLAGS: AM_LDFLAGS: -L/home/gergely/anaconda3/lib Extra libraries: -lrt -lpthread -lz -ldl -lm Archiver: /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar AR_FLAGS: cr Ranlib: /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib

Languages: ---------- C: yes C Compiler: /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc CPPFLAGS: -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -I/home/gergely/anaconda3/include H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200809L -DNDEBUG -UH5_DEBUG_API AM_CPPFLAGS: -I/home/gergely/anaconda3/include C Flags: -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/home/gergely/anaconda3/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5 -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix H5 C Flags: -std=c99 -pedantic -Wall -Wextra -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual -Wconversion -Wdeclaration-after-statement -Wdisabled-optimization -Wfloat-equal -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum -Wundef -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings -finline-functions -s -Wno-inline -Wno-aggregate-return -Wno-missing-format-attribute -Wno-missing-noreturn -O AM C Flags: Shared C Library: yes Static C Library: yes

Fortran: yes Fortran Compiler: /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran Fortran Flags: H5 Fortran Flags: -pedantic -Wall -Wextra -Wunderflow -Wimplicit-interface -Wsurprising -Wno-c-binding-type -s -O2 AM Fortran Flags: Shared Fortran Library: yes Static Fortran Library: yes

C++: yes C++ Compiler: /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++ C++ Flags: -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -I/home/gergely/anaconda3/include -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5 -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix H5 C++ Flags: -pedantic -Wall -W -Wundef -Wshadow -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Wconversion -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor -Wctor-dtor-privacy -Wabi -finline-functions -s -O AM C++ Flags: Shared C++ Library: yes Static C++ Library: yes

Java: no

Features: --------- Parallel HDF5: no Parallel Filtered Dataset Writes: no Large Parallel I/O: no High-level library: yes Threadsafety: yes Default API mapping: v110 With deprecated public symbols: yes I/O filters (external): deflate(zlib) MPE: no Direct VFD: no dmalloc: no Packages w/ extra debug output: none API tracing: no Using memory checker: yes Memory allocation sanity checks: no Function stack tracing: no Strict file format checks: no Optimization instrumentation: no

On Tue, Dec 17, 2019 at 8:35 AM Billy Poon wrote: > > Hi Gergely, > > Let me build the test package tomorrow. All the gritty details for building with conda is being finalized and the official documentation will be updated to describe the steps. It would be too confusing to keep changing the documentation as the process evolves. You do have the general process, though, which is summarized as follows. > > 1) Installing dependencies. The cctbx_dependencies metapackage was an initial approach for managing the CCTBX dependencies, but after contacting the conda-forge folks, they recommended using the --only-deps flag. So when the CCTBX conda package is available, you'll be able to get a set of dependencies with, > > conda install -c conda-forge --only-deps cctbx > > By default, the bootstrap.py file will automatically install a set of dependencies in the "conda_base" directory (and a conda installation if one is not found). It just uses standard conda environment files located in libtbx/auto_build/conda_envs, so you do not need to install cctbx_dependencies as a separate step. The environment files avoid channel issues by explicitly defining the channel to pull the packages from and the cctbx channel just stores copies of packages from conda-forge. There were issues earlier where the conda-forge packages would sometimes be moved to a different label. The --use-conda flag also accepts a path to $CONDA_PREFIX if you want to use a specific environment for testing. > > 2) Building. The bootstrap.py file handles that with SCons. > > 3) Running. After building, there should be a setpaths.sh (and .csh) file that adds build/bin to your path. The build/bin directory has our dispatchers, which are just scripts that set up the environment variables for you. This prevents other programs from loading our libraries, whose versions may conflict. You should see that there is a "python" dispatcher, which is a convenience for developers. Otherwise, you can use libtbx.python, which will be able to import CCTBX modules. > > The running part is where the conda package for CCTBX will be different than this build. Since our Python files and extensions modules will be in the "site-packages" directory for Python, the PYTHONPATH variable will not be needed (and conda suggests that that variable not be set). The other CCTBX libraries will be in $CONDA_PREFIX/lib, so LD_LIBRARY_PATH is not needed. In an active environment, PATH will already be modified. And then our LIBTBX_BUILD directory can be set to sys.prefix. > > So with the conda package, you would only need to activate your conda environment and CCTBX should integrate with other conda packages. What is the conflict with hdf5? That's something that should be fixed. Thanks! > > -- > Billy K. Poon > Research Scientist, Molecular Biophysics and Integrated Bioimaging > Lawrence Berkeley National Laboratory > 1 Cyclotron Road, M/S 33R0345 > Berkeley, CA 94720 > Tel: (510) 486-5709 > Fax: (510) 486-5909 > Web: https://phenix-online.org > > > On Mon, Dec 16, 2019 at 2:53 AM Gergely Katona wrote: >> >> Dear Billy, >> >> Thank you for this update and for your efforts! I found a solution and >> indeed most things already work in anaconda3. The steps I took (even >> if these instructions will have short expiry date): >> >> Modified .condarc with: >> channels: >> - cctbx >> - conda-forge >> - defaults >> >> run >> conda conda install cctbx_dependencies python=3.6 >> >> I expect this will work without python=3.6 in the near future. >> >> Then compiling cctbx with anaconda3 python went without any problem >> when using these flags: >> python bootstrap.py --use-conda --python3 --nproc=12 >> >> Then I probably did the unorthodox thing and sourced these directories: >> >> setenv LIBTBX_BUILD /home/gergely/cctbx/build >> setenv PATH ${PATH}:/home/gergely/cctbx/build/bin >> setenv PYTHONPATH >> /home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH >> setenv LD_LIBRARY_PATH >> /home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH >> >> There is probably a better way to put this into conda environment. >> With these steps I could run one of my scripts depending on cctbx. The >> only problem I found is that hdf5 library had conflict with another >> package in conda, but I does not find this as a showstopper. >> >> Best wishes, >> >> Gergely >> >> >> >> >> >> >> On Fri, Dec 13, 2019 at 7:23 PM Billy Poon wrote: >> > >> > Hi Gergely, >> > >> > It's still a work in progress. I'm sorting out some Windows issues right now. I can probably build a test package on a separate channel for people that want to test it (let's say next week?). I'll provide instructions, but basically, the test conda package will be in its own separate channel and the dependencies will be pulled from the conda-forge channel. I want most things to be working correctly on Python 2.7, 3.6, 3.7, and 3.8 on all 3 platforms. >> > >> > Thanks! >> > >> > -- >> > Billy K. Poon >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging >> > Lawrence Berkeley National Laboratory >> > 1 Cyclotron Road, M/S 33R0345 >> > Berkeley, CA 94720 >> > Tel: (510) 486-5709 >> > Fax: (510) 486-5909 >> > Web: https://phenix-online.org >> > >> > >> > On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona wrote: >> >> >> >> Dear Billy, >> >> >> >> This sounds very promising and exciting. I am not sure if cctbx is >> >> already functional as a conda package in anaconda3 (Linux) or this is >> >> still work in progress. My technical expertise does not allow me to >> >> tell the difference. What I tried: >> >> >> >> Fresh install of anaconda3. Adding - cctbx and - conda-forge to >> >> .condarc . Installing pyside2 with conda. Running conda install >> >> conda_dependencies . I get a lot package version conflict, and I >> >> cannot import cctbx or iotbx to anaconda python. Am I following the >> >> right instructions? Or it is too early to expect that cctbx works when >> >> installed through conda? >> >> >> >> Best wishes, >> >> >> >> Gergely >> >> >> >> On Wed, Nov 27, 2019 at 3:56 PM Billy Poon wrote: >> >> > >> >> > Hi all, >> >> > >> >> > For a brief update, I have submitted a recipe for cctbxlite to conda-forge (https://github.com/conda-forge/staged-recipes/pull/10021) and support for Python 3.7 and 3.8 is being added (https://github.com/cctbx/cctbx_project/pull/409). With some fixes for Windows (https://github.com/cctbx/cctbx_project/pull/416), all platforms (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and 3.8. Some additional changes will be needed to get Windows to work with Python 3 and for tests to pass with Boost 1.70.0. That will enable the conda-forge recipe to build for all platforms and for all supported versions of Python. >> >> > >> >> > Currently, the conda-forge recipe will install into the conda python and cctbx imports can be done without sourcing the environment scripts. It looks like a lot of the environment variables being set in the dispatchers can be removed since the Python files and C++ extensions are in the right places. I'll update the libtbx_env file so that commands that load the environment can work correctly. >> >> > >> >> > -- >> >> > Billy K. Poon >> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging >> >> > Lawrence Berkeley National Laboratory >> >> > 1 Cyclotron Road, M/S 33R0345 >> >> > Berkeley, CA 94720 >> >> > Tel: (510) 486-5709 >> >> > Fax: (510) 486-5909 >> >> > Web: https://phenix-online.org >> >> > >> >> > >> >> > On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll wrote: >> >> >> >> >> >> Hi Luc, >> >> >> >> >> >> That sounds promising. From there, I’d need to work out how to make a fully-packaged installer (basically a modified wheel file) for the ChimeraX ToolShed - the aim is for the end user to not have to worry about any of this. That adds a couple of complications - e.g. $LIBTBX_BUILD would need to be set dynamically before first import - but doesn’t seem insurmountable. >> >> >> >> >> >> Thanks, >> >> >> >> >> >> Tristan >> >> >> >> >> >> >> >> >> >> >> >> Tristan Croll >> >> >> Research Fellow >> >> >> Cambridge Institute for Medical Research >> >> >> University of Cambridge CB2 0XY >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > On 25 Aug 2019, at 18:31, Luc Bourhis wrote: >> >> >> > >> >> >> > Hi Tristan, >> >> >> > >> >> >> > cctbx could be built to use your ChimeraX python, now that cctbx is moving to Python 3. The option —with-python is there for that with the bootstrap script. The specific environment setup boil down to setting two environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on Linux, PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a framework such as ChimeraX, that should not be difficult to ensure those two variables are set. >> >> >> > >> >> >> > Best wishes, >> >> >> > >> >> >> > Luc >> >> >> > >> >> >> > >> >> >> >> On 23 Aug 2019, at 19:02, Tristan Croll wrote: >> >> >> >> >> >> >> >> To add my two cents on this: probably the second-most common question I've had about ISOLDE's implementation is, "why didn't you use CCTBX?". The honest answer to that is, "I didn't know how." >> >> >> >> >> >> >> >> Still don't, really - although the current developments are rather promising. The problem I've faced is that CCTBX was designed as its own self-contained Python (2.7, until very recently) environment, with its own interpreter and a lot of very specific environment setup. Meanwhile I'm developing ISOLDE in ChimeraX, which is *also* its own self-contained Python (3.7) environment. To plug one into the other in that form... well, I don't think I'm a good enough programmer to really know where to start. >> >> >> >> >> >> >> >> The move to Conda and a more modular CCTBX architecture should make a lot more possible in that direction. Pip would be even better for me personally (ChimeraX can install directly from the PyPI, but doesn't interact with Conda) - but I understand pretty well the substantial challenge that would amount to (not least being that the PyPI imposes a limit - around 100MB from memory? - on the size of an individual package). >> >> >> >> >> >> >> >> Best regards, >> >> >> >> >> >> >> >> Tristan >> >> >> >> >> >> >> >>> On 2019-08-23 09:28, Luc Bourhis wrote: >> >> >> >>> Hi Graeme, >> >> >> >>> Yes, I know. But “black" is a program doing a very particular task >> >> >> >>> (code formatting from the top of my head). Requiring to use a wrapper >> >> >> >>> for python itself is another level. But ok, I think I am mellowing to >> >> >> >>> the idea after all! Talking with people around me, and extrapolating, >> >> >> >>> I would bet that, right now, a great majority of people interested by >> >> >> >>> cctbx in pip have already used the cctbx, so they know about the >> >> >> >>> Python wrapper, and they would not be too sanguine about that. My >> >> >> >>> concern is for the future, when pip will be the first time some people >> >> >> >>> use cctbx. Big fat warning notices on PyPI page and a better error >> >> >> >>> message when cctbx fails because LIBTBX_BUILD is not set would be >> >> >> >>> needed but that could be all right. >> >> >> >>> If we do a pip installer, we should aim at a minimal install: cctbx, >> >> >> >>> iotbx and their dependencies, and that’s it. >> >> >> >>> Best wishes, >> >> >> >>> Luc >> >> >> >>>> On 23 Aug 2019, at 07:17, [email protected] wrote: >> >> >> >>>> Without discussing the merits of this or whether we _choose_ to make the move to supporting PIP, I am certain it would be _possible_ - many other packages make dispatcher scripts when you pip install them e.g. >> >> >> >>>> Silver-Surfer rescale_f2 :) $ which black; cat $(which black) >> >> >> >>>> /Library/Frameworks/Python.framework/Versions/3.6/bin/black >> >> >> >>>> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 >> >> >> >>>> # -*- coding: utf-8 -*- >> >> >> >>>> import re >> >> >> >>>> import sys >> >> >> >>>> from black import main >> >> >> >>>> if __name__ == '__main__': >> >> >> >>>> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) >> >> >> >>>> sys.exit(main()) >> >> >> >>>> So we _could_ work around the absence of LIBTBX_BUILD etc. in the system. Whether or not we elect to do the work is a different question, and it seems clear that here are very mixed opinions on this. >> >> >> >>>> Best wishes Graeme >> >> >> >>>> On 23 Aug 2019, at 01:21, Luc Bourhis mailto:[email protected]> wrote: >> >> >> >>>> Hi, >> >> >> >>>> Even if we managed to ship our the boost dynamic libraries with pip, it would still not be pip-like, as we would still need our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work with the standard python exe. LD_LIBRARY_PATH, we could get around that by changing the way we compile, using -Wl,-R, which is the runtime equivalent of build time -L. That’s a significant change that would need to be tested. But there is no way around setting LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps there is a way to hack libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip installs? >> >> >> >>>> Best wishes, >> >> >> >>>> Luc >> >> >> >>>> On 16 Aug 2019, at 22:47, Luc Bourhis mailto:[email protected]> wrote: >> >> >> >>>> Hi, >> >> >> >>>> I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.orghttp://anaconda.org/, and then to advertise it hard to every potential user out there. >> >> >> >>>> Best wishes, >> >> >> >>>> Luc >> >> >> >>>> On 16 Aug 2019, at 21:45, Aaron Brewster mailto:[email protected]> wrote: >> >> >> >>>> Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-...). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not. >> >> >> >>>> All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support. >> >> >> >>>> There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6). >> >> >> >>>> If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4. >> >> >> >>>> Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :) >> >> >> >>>> Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users. >> >> >> >>>> Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort. >> >> >> >>>> Thanks, >> >> >> >>>> -Aaron >> >> >> >>>> _______________________________________________ >> >> >> >>>> cctbxbb mailing list >> >> >> >>>> [email protected]mailto:[email protected] >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >>>> _______________________________________________ >> >> >> >>>> cctbxbb mailing list >> >> >> >>>> [email protected]mailto:[email protected] >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >>>> _______________________________________________ >> >> >> >>>> cctbxbb mailing list >> >> >> >>>> [email protected]mailto:[email protected] >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >>>> -- >> >> >> >>>> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. >> >> >> >>>> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. >> >> >> >>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. >> >> >> >>>> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom >> >> >> >>>> _______________________________________________ >> >> >> >>>> cctbxbb mailing list >> >> >> >>>> [email protected] >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >>> _______________________________________________ >> >> >> >>> cctbxbb mailing list >> >> >> >>> [email protected] >> >> >> >>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> >> cctbxbb mailing list >> >> >> >> [email protected] >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> > >> >> >> > >> >> >> > _______________________________________________ >> >> >> > cctbxbb mailing list >> >> >> > [email protected] >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> cctbxbb mailing list >> >> >> [email protected] >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> > >> >> > _______________________________________________ >> >> > cctbxbb mailing list >> >> > [email protected] >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> >> >> -- >> >> Gergely Katona, PhD >> >> Associate Professor >> >> Department of Chemistry and Molecular Biology, University of Gothenburg >> >> Box 462, 40530 Göteborg, Sweden >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 >> >> Web: http://katonalab.eu, Email: [email protected] >> >> >> >> _______________________________________________ >> >> cctbxbb mailing list >> >> [email protected] >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> > >> > _______________________________________________ >> > cctbxbb mailing list >> > [email protected] >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> -- >> Gergely Katona, PhD >> Associate Professor >> Department of Chemistry and Molecular Biology, University of Gothenburg >> Box 462, 40530 Göteborg, Sweden >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 >> Web: http://katonalab.eu, Email: [email protected] >> >> _______________________________________________ >> cctbxbb mailing list >> [email protected] >> http://phenix-online.org/mailman/listinfo/cctbxbb > > _______________________________________________ > cctbxbb mailing list > [email protected] > http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, PhD Associate Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, PhD Associate Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

-- Gergely Katona, Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

Hi Billy, Here come the package lists from conda. I will do more testing with the cleaned condas installation. Indeed, the boost packages came from the conda-forge channel but for some odd reason in the test environment not in the base. Best wishes, Gergely On Fri, Jan 3, 2020 at 9:05 PM Billy Poon wrote:

Hi Gergely,

Happy New Year!

Can you provide the output to "conda list" for each environment?

I suspect that the old version of cctbx might still be around. Is there a file named PYTHON_VERSION_MAJOR_MINOR in "/home/gergely/anaconda3/lib"? If so, delete that file. To make sure you are downloading a new copy of the package, clear the cache where conda keeps copies of all the packages. To do that, you can run "conda clean -y --all". You can remove the "-y" flag if you want to see what will be deleted. The hash for the new linux package should be h8b68381_0.

The last error looks like an incompatible version of boost or gcc library was installed. The output from "conda list" should show the boost and boost-cpp packages coming from conda-forge. I would avoid installing into the root or base environment because that usually pulls from the default channels. All the dependencies should be pulled from the conda-forge channel.

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Fri, Jan 3, 2020 at 2:20 PM Gergely Katona wrote:

...

I found some other errors when using the test environment.

--------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) <ipython-input-1-fe308b345b3f> in <module> 10 import scipy as sp 11 from scipy import linalg ---> 12 import iotbx.pdb 13 import cctbx 14 import pandas as pd

~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/__init__.py in <module> 8 9 import iotbx.pdb.records ---> 10 import iotbx.pdb.hierarchy 11 from scitbx import matrix 12

~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/hierarchy.py in <module> 14 from six.moves import cStringIO as StringIO 15 from iotbx.pdb import hy36encode, hy36decode ---> 16 import iotbx.cif.model 17 from cctbx import crystal 18 from libtbx import group_args

~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/cif/__init__.py in <module> 19 20 from cctbx.array_family import flex ---> 21 from cctbx import miller 22 from iotbx.cif import model, builders, geometry 23 from libtbx.containers import OrderedDict

~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/miller/__init__.py in <module> 11 12 from cctbx import crystal ---> 13 from cctbx import maptbx 14 from cctbx import sgtbx 15 from cctbx.sgtbx import lattice_symmetry

~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/maptbx/__init__.py in <module> 15 from libtbx import adopt_init_args 16 from libtbx.utils import Sorry ---> 17 import libtbx.load_env 18 import math 19 import sys, os

~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/load_env.py in <module> 3 import libtbx.env_config 4 import os ----> 5 libtbx.env = libtbx.env_config.unpickle() 6 libtbx.env.set_os_environ_all_dist() 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME")

~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/env_config.py in unpickle() 2736 env = pickle.load(libtbx_env) 2737 if (env.python_version_major_minor != sys.version_info[:2]): -> 2738 env.raise_python_version_incompatible() 2739 if (op.realpath(build_path) != op.realpath(abs(env.build_path))): 2740 env.build_path.reset(build_path)

~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/env_config.py in raise_python_version_incompatible(self, prev_pvmm) 469 if (prev_pvmm is None): 470 prev_pvmm = "%d.%d" % self.python_version_major_minor --> 471 raise RuntimeError("Python version incompatible with this build:\n" 472 + " Build directory: %s\n" % show_string(abs(self.build_path)) 473 + " Python version used initially: %s\n" % prev_pvmm

RuntimeError: Python version incompatible with this build: Build directory: "/home/gergely/anaconda3" Python version used initially: 3.6 Python version in use now: 3.8

Interestingly in the base environment all import worked, but I got an error at:

ArgumentError Traceback (most recent call last) <ipython-input-4-5bc71c36e121> in <module> 33 atm_odd.set_b(0) 34 ---> 35 odd.write_pdb_file("bdiff_in_even_000.pdb") #skriver ny pdb med dessa inkluderade 36 dfWBvals=pd.DataFrame(rows_list)

~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in write_pdb_file(self, file_name, open_append, crystal_symmetry, cryst1_z, write_scale_records, append_end, atom_hetatm, sigatm, anisou, siguij) 930 siguij=True): 931 if (crystal_symmetry is Auto): --> 932 crystal_symmetry = self.crystal_symmetry() 933 if (cryst1_z is Auto): 934 cryst1_z = self.extract_cryst1_z_columns()

~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in crystal_symmetry(self, crystal_symmetry, weak_symmetry) 1218 crystal_symmetry=None, 1219 weak_symmetry=False): -> 1220 self_symmetry = self.crystal_symmetry_from_cryst1() 1221 if (self_symmetry is None): 1222 self_symmetry = self._crystal_symmetry_from_cns_remark_sg()

~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in crystal_symmetry_from_cryst1(self) 1195 for line in self.crystallographic_section(): 1196 if (line.startswith("CRYST1")): -> 1197 return cryst1_interpretation.crystal_symmetry(cryst1_record=line) 1198 return None 1199

~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/cryst1_interpretation.py in crystal_symmetry(cryst1_record) 136 space_group_info=None) 137 space_group_info = categorize(cryst1_record.sgroup).space_group_info( --> 138 unit_cell=u) 139 return crystal.symmetry(unit_cell=u, space_group_info=space_group_info)

~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/cryst1_interpretation.py in space_group_info(self, unit_cell) 85 if (self.symbol is None): return None 86 if (self.category is None): ---> 87 try: return sgtbx.space_group_info(self.symbol) 88 except RuntimeError: return None 89 if (isinstance(unit_cell, uctbx.ext.unit_cell)):

~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/__init__.py in __init__(self, symbol, table_id, group, number, space_group_t_den) 100 assert group is None 101 if (table_id is None): --> 102 symbols = space_group_symbols(symbol) 103 else: 104 if (isinstance(symbol, int)): symbol = str(symbol)

ArgumentError: Python argument types in space_group_symbols.__init__(space_group_symbols, str) did not match C++ signature: __init__(_object*, int space_group_number) __init__(_object*, int space_group_number, std::__cxx11::basic_string extension='') __init__(_object*, int space_group_number, std::__cxx11::basic_string extension='', std::__cxx11::basic_string table_id='') __init__(_object*, std::__cxx11::basic_string symbol) __init__(_object*, std::__cxx11::basic_string symbol, std::__cxx11::basic_string table_id='')

Gergely

On Fri, Jan 3, 2020 at 1:44 PM Gergely Katona wrote:

...

Dear Billy,

I did some tests installing in my base conda was not troublefree in python 3.8. Running iotbx.fetch_pdb --all --mtz 1kp8 resulted in boost related error when converting to mtz. Installing into a test environment went fine with iotbx.fetch_pdb --all --mtz 1kp8 running successfully. I could add pymc3, jupyter and seaborn and iotbx still did not break. My LIBTBX_BUILD is not set however in the test environment, which breaks miller when I was trying to import it. Manually setting the LIBTBX_BUILD to CONDA_PREFIX (setenv LIBTBX_BUILD $CONDA_PREFIX) fixes this and one of my script works fine! I will come back if I find some problems with my other scripts, but so far this looks promising!

Happy new year!

Gergely

On Fri, Dec 20, 2019 at 8:16 PM Billy Poon wrote:

...

Hi everyone,

I updated the conda packages on the cctbx-dev channel and there are also now packages for macOS. The old packages have been removed so you can create a new environment with

conda create -n test -c cctbx-dev cctbx python=2.7 conda activate test

where "test" is the environment name and 2.7 is the python version. For the python version, you can also pick 3.6, 3.7, or 3.8. I think the latest conda will install 3.8 as the default python if the python argument is not provided. Or install into an existing environment (with python already installed) with the same command as before,

conda install -c cctbx-dev cctbx

Again, your .condarc file should have conda-forge as the first channel so that dependencies can be pulled correctly.

The dispatchers should work now and $LIBTBX_BUILD will be set on activation of an environment (and unset on environment deactivation) or after installing into an existing environment, so you can do more interesting things like

iotbx.fetch_pdb --all --mtz 1kp8

However, this package does not install databases, so

molprobity.molprobity 1kp8.pdb 1kp8.mtz

will fail because the database is not available (also probe and reduce are not built).

Some known issues in no particular order,

1) You can run "libtbx.run_tests_parallel module=cctbx nproc=Auto" to run tests, but not all the test files were copied, so I got 30 failures here. The final package will probably not have the test files. 2) The other python dispatchers, like "cctbx.python", were not updated, so those will fail. But you can just run "python" in the active environment. 3) Parts of mmtbx still need to be updated for Python 3. 4) The LIBTBX_BUILD variable should be set for sh and csh shells. Let me know if it's not working for your shell (i.e. "echo $LIBTBX_BUILD" should show the same thing as $CONDA_PREFIX). 5) One future step is to enable building other CCTBX-like modules with this conda package. This will require some bookkeeping updates so that modules in the conda environment and the modules being built are tracked properly. Also, making sure that all the necessary headers exist in $PREFIX/include.

Gergely, let me know if your scripts are working.

Happy holidays!

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Wed, Dec 18, 2019 at 11:25 AM Billy Poon wrote:

...

Hi Gergely,

That's the other information that hasn't been updated yet in the conda package. :)

For LIBTBX_BUILD, I can set up environment variables sort of how the conda gcc package sets up environment variables. I have to double check to see if they're set after installation of the package and after activation of an environment. The variable will just be a copy of $CONDA_PREFIX.

The libtbx_env file is a file that exists in $LIBTBX_BUILD that basically stores configuration information. I'll have to modify the contents to update the locations of the different modules since the conda package will not have a "modules" directory.

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Wed, Dec 18, 2019 at 2:08 AM Gergely Katona wrote:

...

Hi Billy,

Conda install went fine with your instructions after rearranging the channels by putting cctbx last. I removed all environmental variables and previous build of cctbx.

import sys print (sys.path, sys.prefix)

['/home/gergely/anaconda3/lib/python36.zip', '/home/gergely/anaconda3/lib/python3.6', '/home/gergely/anaconda3/lib/python3.6/lib-dynload', '', '/home/gergely/anaconda3/lib/python3.6/site-packages', '/home/gergely/anaconda3/lib/python3.6/site-packages/IPython/extensions', '/home/gergely/.ipython'] /home/gergely/anaconda3

Many imports went fine including pymc3, but I encountered problems with these three:

from cctbx import miller import iotbx.pdb from iotbx import reflection_file_reader, mtz

I also got type error when handling space groups.

--------------------------------------------------------------------------- KeyError Traceback (most recent call last) <ipython-input-5-cedbd4f84f3d> in <module> 25 from cctbx import uctbx 26 from cctbx import sgtbx ---> 27 from cctbx import miller 28 #from iotbx import reflection_file_reader, mtz 29 sns.set_context("poster")

~/anaconda3/lib/python3.6/site-packages/cctbx/miller/__init__.py in <module> 11 12 from cctbx import crystal ---> 13 from cctbx import maptbx 14 from cctbx import sgtbx 15 from cctbx.sgtbx import lattice_symmetry

~/anaconda3/lib/python3.6/site-packages/cctbx/maptbx/__init__.py in <module> 15 from libtbx import adopt_init_args 16 from libtbx.utils import Sorry ---> 17 import libtbx.load_env 18 import math 19 import sys, os

~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module> 3 import libtbx.env_config 4 import os ----> 5 libtbx.env = libtbx.env_config.unpickle() 6 libtbx.env.set_os_environ_all_dist() 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME")

~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle() 2603 2604 def unpickle(): -> 2605 build_path = os.environ["LIBTBX_BUILD"] 2606 set_preferred_sys_prefix_and_sys_executable(build_path=build_path) 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb")

~/anaconda3/lib/python3.6/os.py in __getitem__(self, key) 667 except KeyError: 668 # raise KeyError with the original key value --> 669 raise KeyError(key) from None 670 return self.decodevalue(value) 671

KeyError: 'LIBTBX_BUILD'

--------------------------------------------------------------------------- ArgumentError Traceback (most recent call last) <ipython-input-7-2acd3a9ce26a> in <module> 26 return ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar 27 ---> 28 ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar=initializecrystal()

<ipython-input-7-2acd3a9ce26a> in initializecrystal() 11 uc = uctbx.unit_cell(unit_cell) 12 wavelength = 1.54980 ---> 13 xtal_symm = crystal.symmetry(unit_cell=unit_cell, space_group_symbol="P 43 21 2") 14 15 ms = miller.build_set(crystal_symmetry=xtal_symm,anomalous_flag=True, d_min=1.61)

~/anaconda3/lib/python3.6/site-packages/cctbx/crystal/__init__.py in __init__(self, unit_cell, space_group_symbol, space_group_info, space_group, correct_rhombohedral_setting_if_necessary, assert_is_compatible_unit_cell, raise_sorry_if_incompatible_unit_cell, force_compatible_unit_cell) 74 if (space_group_symbol is not None): 75 self._space_group_info = sgtbx.space_group_info( ---> 76 symbol=space_group_symbol) 77 elif (space_group is not None): 78 if (isinstance(space_group, sgtbx.space_group)):

~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/__init__.py in __init__(self, symbol, table_id, group, number, space_group_t_den) 100 assert group is None 101 if (table_id is None): --> 102 symbols = space_group_symbols(symbol) 103 else: 104 if (isinstance(symbol, int)): symbol = str(symbol)

ArgumentError: Python argument types in space_group_symbols.__init__(space_group_symbols, str) did not match C++ signature: __init__(_object*, int space_group_number) __init__(_object*, int space_group_number, std::__cxx11::basic_string extension='') __init__(_object*, int space_group_number, std::__cxx11::basic_string extension='', std::__cxx11::basic_string table_id='') __init__(_object*, std::__cxx11::basic_string symbol) __init__(_object*, std::__cxx11::basic_string symbol, std::__cxx11::basic_string table_id='')

When I tried to set LIBTBX_BUILD to /home/gergely/anaconda3 (this is sys.prefix on my system), I got other problems when importing. Can LIBTBX_BUILD be set in the conda package?

--------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) <ipython-input-1-7e2b144826cc> in <module> 26 from cctbx import sgtbx 27 #from cctbx import miller ---> 28 from iotbx import reflection_file_reader, mtz 29 sns.set_context("poster") 30 plt.rcParams.update({'figure.autolayout': True})

~/anaconda3/lib/python3.6/site-packages/iotbx/reflection_file_reader.py in <module> 53 54 from __future__ import absolute_import, division, print_function ---> 55 from iotbx import mtz 56 from iotbx.scalepack import merge as scalepack_merge 57 from iotbx.scalepack import no_merge_original_index as scalepack_no_merge

~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/__init__.py in <module> 9 import iotbx_mtz_ext as ext 10 ---> 11 from iotbx.mtz import extract_from_symmetry_lib 12 from cctbx import xray 13 import cctbx.xray.observation_types

~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/extract_from_symmetry_lib.py in <module> 1 from __future__ import absolute_import, division, print_function 2 from cctbx import sgtbx ----> 3 import libtbx.load_env 4 import os.path as op 5 from six.moves import range

~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module> 3 import libtbx.env_config 4 import os ----> 5 libtbx.env = libtbx.env_config.unpickle() 6 libtbx.env.set_os_environ_all_dist() 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME")

~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle() 2605 build_path = os.environ["LIBTBX_BUILD"] 2606 set_preferred_sys_prefix_and_sys_executable(build_path=build_path) -> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb") 2608 env = pickle.load(libtbx_env) 2609 if (env.python_version_major_minor != sys.version_info[:2]):

FileNotFoundError: [Errno 2] No such file or directory: '/home/gergely/anaconda3/libtbx_env'

On Wed, Dec 18, 2019 at 9:10 AM Billy Poon wrote: > > Hi Gergely, > > I've uploaded linux packages to a new channel, cctbx-dev, and you can install it with > > conda install -c cctbx-dev cctbx > > in your current environment. You should set your ~/.condarc file to pull the other dependencies from the conda-forge channel first, so put conda-forge above cctbx. Mine looks like > > channels: > - conda-forge > - defaults > - cctbx > > Do you need dxtbx for your scripts? This package does not build that part. I think the plan is to build a separate conda package for dxtbx so that it can be updated more frequently. I can rebuild the packages to include it for testing, but the one being submitted to conda-forge will not have it. > > Also, your error message is probably due to version of HDF5 that the development build installs. The bootstrap.py script will install 1.10.4, but your other dependency is looking for 1.10.5. Installing this cctbx conda package should install 1.10.5, which should fix the issue. Also, I'm updating those environments and 1.10.5 will be the new default version. > > Lastly, the dispatchers will not work in these packages because the old paths during the build process are still in them and many of them expect some additional information that has not been updated in the packages yet. I'm in the process of doing that and will update the cctbx-dev channel when that's done. However, by starting python, you can import cctbx modules. So you can run commands like > > from scitbx.array_family import flex > a = flex.random_double(1000000) > b = flex.min_max_mean_double(a) > b.min > b.max > b.mean > > Thanks! > > -- > Billy K. Poon > Research Scientist, Molecular Biophysics and Integrated Bioimaging > Lawrence Berkeley National Laboratory > 1 Cyclotron Road, M/S 33R0345 > Berkeley, CA 94720 > Tel: (510) 486-5709 > Fax: (510) 486-5909 > Web: https://phenix-online.org > > > On Tue, Dec 17, 2019 at 5:39 AM Gergely Katona wrote: >> >> Dear Billy, >> >> Thank you for the detailed explanation, I look forward to do further >> testing! I aim to pool all modules under the same environment, if >> there are conflicts then I just try to reshuffle the order of imports. >> So far this did not cause problems for me even when I was using system >> python, but of course it is not the most prudent thing to do. With >> anaconda everything is much more standardized and isolated already and >> without being superuser I can have my familiar environment at any >> synchrotron based cluster. It is great that cctbx will be an integral >> part of this ecosystem and this was also the last thing holding me >> back from adopting python3. About hdf5, pymc3 requires it and >> importing causes a kernel restart with the following error messages, >> curiously if I import h5py first this can be avoided. >> >> Best wishes, >> >> Gergely >> >> import pymc3 as pm >> >> Warning! ***HDF5 library version mismatched error*** >> The HDF5 header files used to compile this application do not match >> the version used by the HDF5 library to which this application is linked. >> Data corruption or segmentation faults may occur if the application continues. >> This can happen when an application was compiled by one version of HDF5 but >> linked with a different version of static or shared HDF5 library. >> You should recompile the application or check your shared library related >> settings such as 'LD_LIBRARY_PATH'. >> You can, at your own risk, disable this warning by setting the environment >> variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'. >> Setting it to 2 or higher will suppress the warning messages totally. >> Headers are 1.10.4, library is 1.10.5 >> SUMMARY OF THE HDF5 CONFIGURATION >> ================================= >> >> General Information: >> ------------------- >> HDF5 Version: 1.10.5 >> Configured on: Tue Oct 22 12:02:13 UTC 2019 >> Configured by: conda@16247e67ecd5 >> Host system: x86_64-conda_cos6-linux-gnu >> Uname information: Linux 16247e67ecd5 4.15.0-1059-azure >> #64-Ubuntu SMP Fri Sep 13 17:02:44 UTC 2019 x86_64 x86_64 x86_64 >> GNU/Linux >> Byte sex: little-endian >> Installation point: /home/gergely/anaconda3 >> >> Compiling Options: >> ------------------ >> Build Mode: production >> Debugging Symbols: no >> Asserts: no >> Profiling: no >> Optimization Level: high >> >> Linking Options: >> ---------------- >> Libraries: static, shared >> Statically Linked Executables: >> LDFLAGS: -Wl,-O2 -Wl,--sort-common >> -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags >> -Wl,--gc-sections -Wl,-rpath,/home/gergely/anaconda3/lib >> -Wl,-rpath-link,/home/gergely/anaconda3/lib >> -L/home/gergely/anaconda3/lib >> H5_LDFLAGS: >> AM_LDFLAGS: -L/home/gergely/anaconda3/lib >> Extra libraries: -lrt -lpthread -lz -ldl -lm >> Archiver: >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar >> AR_FLAGS: cr >> Ranlib: >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib >> >> Languages: >> ---------- >> C: yes >> C Compiler: >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc >> CPPFLAGS: -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 >> -I/home/gergely/anaconda3/include >> H5_CPPFLAGS: -D_GNU_SOURCE >> -D_POSIX_C_SOURCE=200809L -DNDEBUG -UH5_DEBUG_API >> AM_CPPFLAGS: -I/home/gergely/anaconda3/include >> C Flags: -march=nocona -mtune=haswell >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include >> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5 >> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix >> H5 C Flags: -std=c99 -pedantic -Wall -Wextra >> -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual -Wconversion >> -Wdeclaration-after-statement -Wdisabled-optimization -Wfloat-equal >> -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations >> -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs >> -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls >> -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum -Wundef >> -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings >> -finline-functions -s -Wno-inline -Wno-aggregate-return >> -Wno-missing-format-attribute -Wno-missing-noreturn -O >> AM C Flags: >> Shared C Library: yes >> Static C Library: yes >> >> >> Fortran: yes >> Fortran Compiler: >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran >> Fortran Flags: >> H5 Fortran Flags: -pedantic -Wall -Wextra -Wunderflow >> -Wimplicit-interface -Wsurprising -Wno-c-binding-type -s -O2 >> AM Fortran Flags: >> Shared Fortran Library: yes >> Static Fortran Library: yes >> >> C++: yes >> C++ Compiler: >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++ >> C++ Flags: -fvisibility-inlines-hidden >> -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include >> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5 >> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix >> H5 C++ Flags: -pedantic -Wall -W -Wundef -Wshadow >> -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Wconversion >> -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual >> -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor >> -Wctor-dtor-privacy -Wabi -finline-functions -s -O >> AM C++ Flags: >> Shared C++ Library: yes >> Static C++ Library: yes >> >> Java: no >> >> >> Features: >> --------- >> Parallel HDF5: no >> Parallel Filtered Dataset Writes: no >> Large Parallel I/O: no >> High-level library: yes >> Threadsafety: yes >> Default API mapping: v110 >> With deprecated public symbols: yes >> I/O filters (external): deflate(zlib) >> MPE: no >> Direct VFD: no >> dmalloc: no >> Packages w/ extra debug output: none >> API tracing: no >> Using memory checker: yes >> Memory allocation sanity checks: no >> Function stack tracing: no >> Strict file format checks: no >> Optimization instrumentation: no >> >> On Tue, Dec 17, 2019 at 8:35 AM Billy Poon wrote: >> > >> > Hi Gergely, >> > >> > Let me build the test package tomorrow. All the gritty details for building with conda is being finalized and the official documentation will be updated to describe the steps. It would be too confusing to keep changing the documentation as the process evolves. You do have the general process, though, which is summarized as follows. >> > >> > 1) Installing dependencies. The cctbx_dependencies metapackage was an initial approach for managing the CCTBX dependencies, but after contacting the conda-forge folks, they recommended using the --only-deps flag. So when the CCTBX conda package is available, you'll be able to get a set of dependencies with, >> > >> > conda install -c conda-forge --only-deps cctbx >> > >> > By default, the bootstrap.py file will automatically install a set of dependencies in the "conda_base" directory (and a conda installation if one is not found). It just uses standard conda environment files located in libtbx/auto_build/conda_envs, so you do not need to install cctbx_dependencies as a separate step. The environment files avoid channel issues by explicitly defining the channel to pull the packages from and the cctbx channel just stores copies of packages from conda-forge. There were issues earlier where the conda-forge packages would sometimes be moved to a different label. The --use-conda flag also accepts a path to $CONDA_PREFIX if you want to use a specific environment for testing. >> > >> > 2) Building. The bootstrap.py file handles that with SCons. >> > >> > 3) Running. After building, there should be a setpaths.sh (and .csh) file that adds build/bin to your path. The build/bin directory has our dispatchers, which are just scripts that set up the environment variables for you. This prevents other programs from loading our libraries, whose versions may conflict. You should see that there is a "python" dispatcher, which is a convenience for developers. Otherwise, you can use libtbx.python, which will be able to import CCTBX modules. >> > >> > The running part is where the conda package for CCTBX will be different than this build. Since our Python files and extensions modules will be in the "site-packages" directory for Python, the PYTHONPATH variable will not be needed (and conda suggests that that variable not be set). The other CCTBX libraries will be in $CONDA_PREFIX/lib, so LD_LIBRARY_PATH is not needed. In an active environment, PATH will already be modified. And then our LIBTBX_BUILD directory can be set to sys.prefix. >> > >> > So with the conda package, you would only need to activate your conda environment and CCTBX should integrate with other conda packages. What is the conflict with hdf5? That's something that should be fixed. Thanks! >> > >> > -- >> > Billy K. Poon >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging >> > Lawrence Berkeley National Laboratory >> > 1 Cyclotron Road, M/S 33R0345 >> > Berkeley, CA 94720 >> > Tel: (510) 486-5709 >> > Fax: (510) 486-5909 >> > Web: https://phenix-online.org >> > >> > >> > On Mon, Dec 16, 2019 at 2:53 AM Gergely Katona wrote: >> >> >> >> Dear Billy, >> >> >> >> Thank you for this update and for your efforts! I found a solution and >> >> indeed most things already work in anaconda3. The steps I took (even >> >> if these instructions will have short expiry date): >> >> >> >> Modified .condarc with: >> >> channels: >> >> - cctbx >> >> - conda-forge >> >> - defaults >> >> >> >> run >> >> conda conda install cctbx_dependencies python=3.6 >> >> >> >> I expect this will work without python=3.6 in the near future. >> >> >> >> Then compiling cctbx with anaconda3 python went without any problem >> >> when using these flags: >> >> python bootstrap.py --use-conda --python3 --nproc=12 >> >> >> >> Then I probably did the unorthodox thing and sourced these directories: >> >> >> >> setenv LIBTBX_BUILD /home/gergely/cctbx/build >> >> setenv PATH ${PATH}:/home/gergely/cctbx/build/bin >> >> setenv PYTHONPATH >> >> /home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH >> >> setenv LD_LIBRARY_PATH >> >> /home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH >> >> >> >> There is probably a better way to put this into conda environment. >> >> With these steps I could run one of my scripts depending on cctbx. The >> >> only problem I found is that hdf5 library had conflict with another >> >> package in conda, but I does not find this as a showstopper. >> >> >> >> Best wishes, >> >> >> >> Gergely >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Fri, Dec 13, 2019 at 7:23 PM Billy Poon wrote: >> >> > >> >> > Hi Gergely, >> >> > >> >> > It's still a work in progress. I'm sorting out some Windows issues right now. I can probably build a test package on a separate channel for people that want to test it (let's say next week?). I'll provide instructions, but basically, the test conda package will be in its own separate channel and the dependencies will be pulled from the conda-forge channel. I want most things to be working correctly on Python 2.7, 3.6, 3.7, and 3.8 on all 3 platforms. >> >> > >> >> > Thanks! >> >> > >> >> > -- >> >> > Billy K. Poon >> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging >> >> > Lawrence Berkeley National Laboratory >> >> > 1 Cyclotron Road, M/S 33R0345 >> >> > Berkeley, CA 94720 >> >> > Tel: (510) 486-5709 >> >> > Fax: (510) 486-5909 >> >> > Web: https://phenix-online.org >> >> > >> >> > >> >> > On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona wrote: >> >> >> >> >> >> Dear Billy, >> >> >> >> >> >> This sounds very promising and exciting. I am not sure if cctbx is >> >> >> already functional as a conda package in anaconda3 (Linux) or this is >> >> >> still work in progress. My technical expertise does not allow me to >> >> >> tell the difference. What I tried: >> >> >> >> >> >> Fresh install of anaconda3. Adding - cctbx and - conda-forge to >> >> >> .condarc . Installing pyside2 with conda. Running conda install >> >> >> conda_dependencies . I get a lot package version conflict, and I >> >> >> cannot import cctbx or iotbx to anaconda python. Am I following the >> >> >> right instructions? Or it is too early to expect that cctbx works when >> >> >> installed through conda? >> >> >> >> >> >> Best wishes, >> >> >> >> >> >> Gergely >> >> >> >> >> >> On Wed, Nov 27, 2019 at 3:56 PM Billy Poon wrote: >> >> >> > >> >> >> > Hi all, >> >> >> > >> >> >> > For a brief update, I have submitted a recipe for cctbxlite to conda-forge (https://github.com/conda-forge/staged-recipes/pull/10021) and support for Python 3.7 and 3.8 is being added (https://github.com/cctbx/cctbx_project/pull/409). With some fixes for Windows (https://github.com/cctbx/cctbx_project/pull/416), all platforms (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and 3.8. Some additional changes will be needed to get Windows to work with Python 3 and for tests to pass with Boost 1.70.0. That will enable the conda-forge recipe to build for all platforms and for all supported versions of Python. >> >> >> > >> >> >> > Currently, the conda-forge recipe will install into the conda python and cctbx imports can be done without sourcing the environment scripts. It looks like a lot of the environment variables being set in the dispatchers can be removed since the Python files and C++ extensions are in the right places. I'll update the libtbx_env file so that commands that load the environment can work correctly. >> >> >> > >> >> >> > -- >> >> >> > Billy K. Poon >> >> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging >> >> >> > Lawrence Berkeley National Laboratory >> >> >> > 1 Cyclotron Road, M/S 33R0345 >> >> >> > Berkeley, CA 94720 >> >> >> > Tel: (510) 486-5709 >> >> >> > Fax: (510) 486-5909 >> >> >> > Web: https://phenix-online.org >> >> >> > >> >> >> > >> >> >> > On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll wrote: >> >> >> >> >> >> >> >> Hi Luc, >> >> >> >> >> >> >> >> That sounds promising. From there, I’d need to work out how to make a fully-packaged installer (basically a modified wheel file) for the ChimeraX ToolShed - the aim is for the end user to not have to worry about any of this. That adds a couple of complications - e.g. $LIBTBX_BUILD would need to be set dynamically before first import - but doesn’t seem insurmountable. >> >> >> >> >> >> >> >> Thanks, >> >> >> >> >> >> >> >> Tristan >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Tristan Croll >> >> >> >> Research Fellow >> >> >> >> Cambridge Institute for Medical Research >> >> >> >> University of Cambridge CB2 0XY >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > On 25 Aug 2019, at 18:31, Luc Bourhis wrote: >> >> >> >> > >> >> >> >> > Hi Tristan, >> >> >> >> > >> >> >> >> > cctbx could be built to use your ChimeraX python, now that cctbx is moving to Python 3. The option —with-python is there for that with the bootstrap script. The specific environment setup boil down to setting two environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on Linux, PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a framework such as ChimeraX, that should not be difficult to ensure those two variables are set. >> >> >> >> > >> >> >> >> > Best wishes, >> >> >> >> > >> >> >> >> > Luc >> >> >> >> > >> >> >> >> > >> >> >> >> >> On 23 Aug 2019, at 19:02, Tristan Croll wrote: >> >> >> >> >> >> >> >> >> >> To add my two cents on this: probably the second-most common question I've had about ISOLDE's implementation is, "why didn't you use CCTBX?". The honest answer to that is, "I didn't know how." >> >> >> >> >> >> >> >> >> >> Still don't, really - although the current developments are rather promising. The problem I've faced is that CCTBX was designed as its own self-contained Python (2.7, until very recently) environment, with its own interpreter and a lot of very specific environment setup. Meanwhile I'm developing ISOLDE in ChimeraX, which is *also* its own self-contained Python (3.7) environment. To plug one into the other in that form... well, I don't think I'm a good enough programmer to really know where to start. >> >> >> >> >> >> >> >> >> >> The move to Conda and a more modular CCTBX architecture should make a lot more possible in that direction. Pip would be even better for me personally (ChimeraX can install directly from the PyPI, but doesn't interact with Conda) - but I understand pretty well the substantial challenge that would amount to (not least being that the PyPI imposes a limit - around 100MB from memory? - on the size of an individual package). >> >> >> >> >> >> >> >> >> >> Best regards, >> >> >> >> >> >> >> >> >> >> Tristan >> >> >> >> >> >> >> >> >> >>> On 2019-08-23 09:28, Luc Bourhis wrote: >> >> >> >> >>> Hi Graeme, >> >> >> >> >>> Yes, I know. But “black" is a program doing a very particular task >> >> >> >> >>> (code formatting from the top of my head). Requiring to use a wrapper >> >> >> >> >>> for python itself is another level. But ok, I think I am mellowing to >> >> >> >> >>> the idea after all! Talking with people around me, and extrapolating, >> >> >> >> >>> I would bet that, right now, a great majority of people interested by >> >> >> >> >>> cctbx in pip have already used the cctbx, so they know about the >> >> >> >> >>> Python wrapper, and they would not be too sanguine about that. My >> >> >> >> >>> concern is for the future, when pip will be the first time some people >> >> >> >> >>> use cctbx. Big fat warning notices on PyPI page and a better error >> >> >> >> >>> message when cctbx fails because LIBTBX_BUILD is not set would be >> >> >> >> >>> needed but that could be all right. >> >> >> >> >>> If we do a pip installer, we should aim at a minimal install: cctbx, >> >> >> >> >>> iotbx and their dependencies, and that’s it. >> >> >> >> >>> Best wishes, >> >> >> >> >>> Luc >> >> >> >> >>>> On 23 Aug 2019, at 07:17, [email protected] wrote: >> >> >> >> >>>> Without discussing the merits of this or whether we _choose_ to make the move to supporting PIP, I am certain it would be _possible_ - many other packages make dispatcher scripts when you pip install them e.g. >> >> >> >> >>>> Silver-Surfer rescale_f2 :) $ which black; cat $(which black) >> >> >> >> >>>> /Library/Frameworks/Python.framework/Versions/3.6/bin/black >> >> >> >> >>>> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 >> >> >> >> >>>> # -*- coding: utf-8 -*- >> >> >> >> >>>> import re >> >> >> >> >>>> import sys >> >> >> >> >>>> from black import main >> >> >> >> >>>> if __name__ == '__main__': >> >> >> >> >>>> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) >> >> >> >> >>>> sys.exit(main()) >> >> >> >> >>>> So we _could_ work around the absence of LIBTBX_BUILD etc. in the system. Whether or not we elect to do the work is a different question, and it seems clear that here are very mixed opinions on this. >> >> >> >> >>>> Best wishes Graeme >> >> >> >> >>>> On 23 Aug 2019, at 01:21, Luc Bourhis mailto:[email protected]> wrote: >> >> >> >> >>>> Hi, >> >> >> >> >>>> Even if we managed to ship our the boost dynamic libraries with pip, it would still not be pip-like, as we would still need our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work with the standard python exe. LD_LIBRARY_PATH, we could get around that by changing the way we compile, using -Wl,-R, which is the runtime equivalent of build time -L. That’s a significant change that would need to be tested. But there is no way around setting LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps there is a way to hack libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip installs? >> >> >> >> >>>> Best wishes, >> >> >> >> >>>> Luc >> >> >> >> >>>> On 16 Aug 2019, at 22:47, Luc Bourhis mailto:[email protected]> wrote: >> >> >> >> >>>> Hi, >> >> >> >> >>>> I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.orghttp://anaconda.org/, and then to advertise it hard to every potential user out there. >> >> >> >> >>>> Best wishes, >> >> >> >> >>>> Luc >> >> >> >> >>>> On 16 Aug 2019, at 21:45, Aaron Brewster mailto:[email protected]> wrote: >> >> >> >> >>>> Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip (https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-...). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not. >> >> >> >> >>>> All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard (https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support. >> >> >> >> >>>> There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6). >> >> >> >> >>>> If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4. >> >> >> >> >>>> Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :) >> >> >> >> >>>> Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users. >> >> >> >> >>>> Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort. >> >> >> >> >>>> Thanks, >> >> >> >> >>>> -Aaron >> >> >> >> >>>> _______________________________________________ >> >> >> >> >>>> cctbxbb mailing list >> >> >> >> >>>> [email protected]mailto:[email protected] >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >>>> _______________________________________________ >> >> >> >> >>>> cctbxbb mailing list >> >> >> >> >>>> [email protected]mailto:[email protected] >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >>>> _______________________________________________ >> >> >> >> >>>> cctbxbb mailing list >> >> >> >> >>>> [email protected]mailto:[email protected] >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >>>> -- >> >> >> >> >>>> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. >> >> >> >> >>>> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. >> >> >> >> >>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. >> >> >> >> >>>> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom >> >> >> >> >>>> _______________________________________________ >> >> >> >> >>>> cctbxbb mailing list >> >> >> >> >>>> [email protected] >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >>> _______________________________________________ >> >> >> >> >>> cctbxbb mailing list >> >> >> >> >>> [email protected] >> >> >> >> >>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> >> >> cctbxbb mailing list >> >> >> >> >> [email protected] >> >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> > >> >> >> >> > >> >> >> >> > _______________________________________________ >> >> >> >> > cctbxbb mailing list >> >> >> >> > [email protected] >> >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> >> cctbxbb mailing list >> >> >> >> [email protected] >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> > >> >> >> > _______________________________________________ >> >> >> > cctbxbb mailing list >> >> >> > [email protected] >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Gergely Katona, PhD >> >> >> Associate Professor >> >> >> Department of Chemistry and Molecular Biology, University of Gothenburg >> >> >> Box 462, 40530 Göteborg, Sweden >> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 >> >> >> Web: http://katonalab.eu, Email: [email protected] >> >> >> >> >> >> _______________________________________________ >> >> >> cctbxbb mailing list >> >> >> [email protected] >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> > >> >> > _______________________________________________ >> >> > cctbxbb mailing list >> >> > [email protected] >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> >> >> -- >> >> Gergely Katona, PhD >> >> Associate Professor >> >> Department of Chemistry and Molecular Biology, University of Gothenburg >> >> Box 462, 40530 Göteborg, Sweden >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 >> >> Web: http://katonalab.eu, Email: [email protected] >> >> >> >> _______________________________________________ >> >> cctbxbb mailing list >> >> [email protected] >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> > >> > _______________________________________________ >> > cctbxbb mailing list >> > [email protected] >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> -- >> Gergely Katona, PhD >> Associate Professor >> Department of Chemistry and Molecular Biology, University of Gothenburg >> Box 462, 40530 Göteborg, Sweden >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 >> Web: http://katonalab.eu, Email: [email protected] >> >> _______________________________________________ >> cctbxbb mailing list >> [email protected] >> http://phenix-online.org/mailman/listinfo/cctbxbb > > _______________________________________________ > cctbxbb mailing list > [email protected] > http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, PhD Associate Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

-- Gergely Katona, Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

Hi Gergely, So for your base environment, the std::__cxx11::basic_string error is caused by the boost package coming from the cctbx channel. I built that copy of boost to support older compilers (developer environments may use GCC versions that are different than the one used for building conda-forge packages), but I'll just move it to another channel so it's less confusing. I'm surprised that the dependency resolution step kept that package instead of replacing it with the version from conda-forge. What shell are you using? The LIBTBX_BUILD environment variable should be set for sh and csh type shells. What files exist in $CONDA_PREFIX/etc/conda/activate.d ? Thanks for testing! -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org On Tue, Jan 7, 2020 at 12:54 AM Gergely Katona wrote:

Ah, an there is no sign of the mentioned file in PYTHON_VERSION_MAJOR_MINOR in /home/gergely/anaconda3/lib before I forget.

Gergely

On Tue, Jan 7, 2020 at 9:37 AM Gergely Katona wrote:

...

Hi Billy,

Here come the package lists from conda. I will do more testing with the cleaned condas installation. Indeed, the boost packages came from the conda-forge channel but for some odd reason in the test environment not in the base.

Best wishes,

Gergely

On Fri, Jan 3, 2020 at 9:05 PM Billy Poon wrote:

...

Hi Gergely,

Happy New Year!

Can you provide the output to "conda list" for each environment?

I suspect that the old version of cctbx might still be around. Is

...

...

The last error looks like an incompatible version of boost or gcc

there a file named PYTHON_VERSION_MAJOR_MINOR in "/home/gergely/anaconda3/lib"? If so, delete that file. To make sure you are downloading a new copy of the package, clear the cache where conda keeps copies of all the packages. To do that, you can run "conda clean -y --all". You can remove the "-y" flag if you want to see what will be deleted. The hash for the new linux package should be h8b68381_0. library was installed. The output from "conda list" should show the boost and boost-cpp packages coming from conda-forge. I would avoid installing into the root or base environment because that usually pulls from the default channels. All the dependencies should be pulled from the conda-forge channel.

...

...

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Fri, Jan 3, 2020 at 2:20 PM Gergely Katona

wrote:

...

...

I found some other errors when using the test environment.

---

...

...

...

RuntimeError Traceback (most recent call last) <ipython-input-1-fe308b345b3f> in <module> 10 import scipy as sp 11 from scipy import linalg ---> 12 import iotbx.pdb 13 import cctbx 14 import pandas as pd

~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/__init__.py

...

in <module> 8 9 import iotbx.pdb.records ---> 10 import iotbx.pdb.hierarchy 11 from scitbx import matrix 12

~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/hierarchy.py

...

in <module> 14 from six.moves import cStringIO as StringIO 15 from iotbx.pdb import hy36encode, hy36decode ---> 16 import iotbx.cif.model 17 from cctbx import crystal 18 from libtbx import group_args

~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/cif/__init__.py

...

in <module> 19 20 from cctbx.array_family import flex ---> 21 from cctbx import miller 22 from iotbx.cif import model, builders, geometry 23 from libtbx.containers import OrderedDict

~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/miller/__init__.py

...

in <module> 11 12 from cctbx import crystal ---> 13 from cctbx import maptbx 14 from cctbx import sgtbx 15 from cctbx.sgtbx import lattice_symmetry

~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/maptbx/__init__.py

...

in <module> 15 from libtbx import adopt_init_args 16 from libtbx.utils import Sorry ---> 17 import libtbx.load_env 18 import math 19 import sys, os

~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/load_env.py in <module> 3 import libtbx.env_config 4 import os ----> 5 libtbx.env = libtbx.env_config.unpickle() 6 libtbx.env.set_os_environ_all_dist() 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME")

~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/env_config.py in unpickle() 2736 env = pickle.load(libtbx_env) 2737 if (env.python_version_major_minor != sys.version_info[:2]): -> 2738 env.raise_python_version_incompatible() 2739 if (op.realpath(build_path) != op.realpath(abs(env.build_path))): 2740 env.build_path.reset(build_path)

~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/env_config.py in raise_python_version_incompatible(self, prev_pvmm) 469 if (prev_pvmm is None): 470 prev_pvmm = "%d.%d" % self.python_version_major_minor --> 471 raise RuntimeError("Python version incompatible with this build:\n" 472 + " Build directory: %s\n" % show_string(abs(self.build_path)) 473 + " Python version used initially: %s\n" % prev_pvmm

RuntimeError: Python version incompatible with this build: Build directory: "/home/gergely/anaconda3" Python version used initially: 3.6 Python version in use now: 3.8

Interestingly in the base environment all import worked, but I got an error at:

ArgumentError Traceback (most recent call last) <ipython-input-4-5bc71c36e121> in <module> 33 atm_odd.set_b(0) 34 ---> 35 odd.write_pdb_file("bdiff_in_even_000.pdb") #skriver ny pdb med dessa inkluderade 36 dfWBvals=pd.DataFrame(rows_list)

~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in write_pdb_file(self, file_name, open_append, crystal_symmetry, cryst1_z, write_scale_records, append_end, atom_hetatm, sigatm, anisou, siguij) 930 siguij=True): 931 if (crystal_symmetry is Auto): --> 932 crystal_symmetry = self.crystal_symmetry() 933 if (cryst1_z is Auto): 934 cryst1_z = self.extract_cryst1_z_columns()

~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in crystal_symmetry(self, crystal_symmetry, weak_symmetry) 1218 crystal_symmetry=None, 1219 weak_symmetry=False): -> 1220 self_symmetry = self.crystal_symmetry_from_cryst1() 1221 if (self_symmetry is None): 1222 self_symmetry = self._crystal_symmetry_from_cns_remark_sg()

~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in crystal_symmetry_from_cryst1(self) 1195 for line in self.crystallographic_section(): 1196 if (line.startswith("CRYST1")): -> 1197 return cryst1_interpretation.crystal_symmetry(cryst1_record=line) 1198 return None 1199

~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/cryst1_interpretation.py

...

in crystal_symmetry(cryst1_record) 136 space_group_info=None) 137 space_group_info = categorize(cryst1_record.sgroup).space_group_info( --> 138 unit_cell=u) 139 return crystal.symmetry(unit_cell=u, space_group_info=space_group_info)

...

in space_group_info(self, unit_cell) 85 if (self.symbol is None): return None 86 if (self.category is None): ---> 87 try: return sgtbx.space_group_info(self.symbol) 88 except RuntimeError: return None 89 if (isinstance(unit_cell, uctbx.ext.unit_cell)):

~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/__init__.py in __init__(self, symbol, table_id, group, number, space_group_t_den) 100 assert group is None 101 if (table_id is None): --> 102 symbols = space_group_symbols(symbol) 103 else: 104 if (isinstance(symbol, int)): symbol = str(symbol)

ArgumentError: Python argument types in space_group_symbols.__init__(space_group_symbols, str) did not match C++ signature: __init__(_object*, int space_group_number) __init__(_object*, int space_group_number, std::__cxx11::basic_string extension='') __init__(_object*, int space_group_number, std::__cxx11::basic_string extension='', std::__cxx11::basic_string table_id='') __init__(_object*, std::__cxx11::basic_string symbol) __init__(_object*, std::__cxx11::basic_string symbol, std::__cxx11::basic_string table_id='')

Gergely

On Fri, Jan 3, 2020 at 1:44 PM Gergely Katona wrote:

...

Dear Billy,

I did some tests installing in my base conda was not troublefree in python 3.8. Running iotbx.fetch_pdb --all --mtz 1kp8 resulted in

boost

...

related error when converting to mtz. Installing into a test environment went fine with iotbx.fetch_pdb --all --mtz 1kp8 running successfully. I could add pymc3, jupyter and seaborn and iotbx still did not break. My LIBTBX_BUILD is not set however in the test environment, which breaks miller when I was trying to import it. Manually setting the LIBTBX_BUILD to CONDA_PREFIX (setenv LIBTBX_BUILD $CONDA_PREFIX) fixes this and one of my script works fine! I will come back if I find some problems with my other scripts, but so far this looks promising!

Happy new year!

Gergely

On Fri, Dec 20, 2019 at 8:16 PM Billy Poon wrote:

...

Hi everyone,

I updated the conda packages on the cctbx-dev channel and there

are also now packages for macOS. The old packages have been removed so you can create a new environment with

...

conda create -n test -c cctbx-dev cctbx python=2.7 conda activate test

where "test" is the environment name and 2.7 is the python

version. For the python version, you can also pick 3.6, 3.7, or 3.8. I

...

...

...

conda install -c cctbx-dev cctbx

Again, your .condarc file should have conda-forge as the first

channel so that dependencies can be pulled correctly.

...

The dispatchers should work now and $LIBTBX_BUILD will be set on

activation of an environment (and unset on environment deactivation) or after installing into an existing environment, so you can do more interesting things like

...

iotbx.fetch_pdb --all --mtz 1kp8

However, this package does not install databases, so

molprobity.molprobity 1kp8.pdb 1kp8.mtz

will fail because the database is not available (also probe and

reduce are not built).

...

Some known issues in no particular order,

1) You can run "libtbx.run_tests_parallel module=cctbx

nproc=Auto" to run tests, but not all the test files were copied, so I got 30 failures here. The final package will probably not have the test files.

...

2) The other python dispatchers, like "cctbx.python", were not updated, so those will fail. But you can just run "python" in the active environment. 3) Parts of mmtbx still need to be updated for Python 3. 4) The LIBTBX_BUILD variable should be set for sh and csh shells. Let me know if it's not working for your shell (i.e. "echo $LIBTBX_BUILD" should show the same thing as $CONDA_PREFIX). 5) One future step is to enable building other CCTBX-like modules with this conda package. This will require some bookkeeping updates so that modules in the conda environment and the modules being built are tracked

...

...

...

Gergely, let me know if your scripts are working.

Happy holidays!

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Wed, Dec 18, 2019 at 11:25 AM Billy Poon

wrote:

...

> > Hi Gergely, > > That's the other information that hasn't been updated yet in the conda package. :) > > For LIBTBX_BUILD, I can set up environment variables sort of how

...

...

...

> > The libtbx_env file is a file that exists in $LIBTBX_BUILD that basically stores configuration information. I'll have to modify the contents to update the locations of the different modules since the conda

...

...

...

> > -- > Billy K. Poon > Research Scientist, Molecular Biophysics and Integrated Bioimaging > Lawrence Berkeley National Laboratory > 1 Cyclotron Road, M/S 33R0345 > Berkeley, CA 94720 > Tel: (510) 486-5709 > Fax: (510) 486-5909 > Web: https://phenix-online.org > > > On Wed, Dec 18, 2019 at 2:08 AM Gergely Katona < [email protected]> wrote: >> >> Hi Billy, >> >> Conda install went fine with your instructions after rearranging the >> channels by putting cctbx last. I removed all environmental variables >> and previous build of cctbx. >> >> import sys >> print (sys.path, sys.prefix) >> >> ['/home/gergely/anaconda3/lib/python36.zip', >> '/home/gergely/anaconda3/lib/python3.6', >> '/home/gergely/anaconda3/lib/python3.6/lib-dynload', '', >> '/home/gergely/anaconda3/lib/python3.6/site-packages', >> '/home/gergely/anaconda3/lib/python3.6/site-packages/IPython/extensions', >> '/home/gergely/.ipython'] /home/gergely/anaconda3 >> >> Many imports went fine including pymc3, but I encountered

~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/cryst1_interpretation.py think the latest conda will install 3.8 as the default python if the python argument is not provided. Or install into an existing environment (with python already installed) with the same command as before, properly. Also, making sure that all the necessary headers exist in $PREFIX/include. the conda gcc package sets up environment variables. I have to double check to see if they're set after installation of the package and after activation of an environment. The variable will just be a copy of $CONDA_PREFIX. package will not have a "modules" directory. problems

...

...

...

>> with these three: >> >> from cctbx import miller >> import iotbx.pdb >> from iotbx import reflection_file_reader, mtz >> >> >> I also got type error when handling space groups. >> >> >> >>

---

...

...

...

...

...

>> KeyError Traceback (most recent call last) >> <ipython-input-5-cedbd4f84f3d> in <module> >> 25 from cctbx import uctbx >> 26 from cctbx import sgtbx >> ---> 27 from cctbx import miller >> 28 #from iotbx import reflection_file_reader, mtz >> 29 sns.set_context("poster") >> >> ~/anaconda3/lib/python3.6/site-packages/cctbx/miller/__init__.py in <module> >> 11 >> 12 from cctbx import crystal >> ---> 13 from cctbx import maptbx >> 14 from cctbx import sgtbx >> 15 from cctbx.sgtbx import lattice_symmetry >> >> ~/anaconda3/lib/python3.6/site-packages/cctbx/maptbx/__init__.py in <module> >> 15 from libtbx import adopt_init_args >> 16 from libtbx.utils import Sorry >> ---> 17 import libtbx.load_env >> 18 import math >> 19 import sys, os >> >> ~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module> >> 3 import libtbx.env_config >> 4 import os >> ----> 5 libtbx.env = libtbx.env_config.unpickle() >> 6 libtbx.env.set_os_environ_all_dist() >> 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME") >> >> ~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle() >> 2603 >> 2604 def unpickle(): >> -> 2605 build_path = os.environ["LIBTBX_BUILD"] >> 2606 set_preferred_sys_prefix_and_sys_executable(build_path=build_path) >> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb") >> >> ~/anaconda3/lib/python3.6/os.py in __getitem__(self, key) >> 667 except KeyError: >> 668 # raise KeyError with the original key value >> --> 669 raise KeyError(key) from None >> 670 return self.decodevalue(value) >> 671 >> >> KeyError: 'LIBTBX_BUILD' >> >> >> >>

---

...

...

...

...

...

>> ArgumentError Traceback (most recent call last) >> <ipython-input-7-2acd3a9ce26a> in <module> >> 26 return ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar >> 27 >> ---> 28 ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar=initializecrystal() >> >> <ipython-input-7-2acd3a9ce26a> in initializecrystal() >> 11 uc = uctbx.unit_cell(unit_cell) >> 12 wavelength = 1.54980 >> ---> 13 xtal_symm = crystal.symmetry(unit_cell=unit_cell, >> space_group_symbol="P 43 21 2") >> 14 >> 15 ms = >> miller.build_set(crystal_symmetry=xtal_symm,anomalous_flag=True, >> d_min=1.61) >> >> ~/anaconda3/lib/python3.6/site-packages/cctbx/crystal/__init__.py in >> __init__(self, unit_cell, space_group_symbol, space_group_info, >> space_group, correct_rhombohedral_setting_if_necessary, >> assert_is_compatible_unit_cell, raise_sorry_if_incompatible_unit_cell, >> force_compatible_unit_cell) >> 74 if (space_group_symbol is not None): >> 75 self._space_group_info = sgtbx.space_group_info( >> ---> 76 symbol=space_group_symbol) >> 77 elif (space_group is not None): >> 78 if (isinstance(space_group, sgtbx.space_group)): >> >> ~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/__init__.py in >> __init__(self, symbol, table_id, group, number, space_group_t_den) >> 100 assert group is None >> 101 if (table_id is None): >> --> 102 symbols = space_group_symbols(symbol) >> 103 else: >> 104 if (isinstance(symbol, int)): symbol = str(symbol) >> >> ArgumentError: Python argument types in >> space_group_symbols.__init__(space_group_symbols, str) >> did not match C++ signature: >> __init__(_object*, int space_group_number) >> __init__(_object*, int space_group_number, >> std::__cxx11::basic_string> std::allocator<char> > extension='') >> __init__(_object*, int space_group_number, >> std::__cxx11::basic_string> std::allocator<char> > extension='', std::__cxx11::basic_string> std::char_traits<char>, std::allocator<char> > table_id='') >> __init__(_object*, std::__cxx11::basic_string> std::char_traits<char>, std::allocator<char> > symbol) >> __init__(_object*, std::__cxx11::basic_string> std::char_traits<char>, std::allocator<char> > symbol, >> std::__cxx11::basic_string> std::allocator<char> > table_id='') >> >> >> When I tried to set LIBTBX_BUILD to /home/gergely/anaconda3 (this is >> sys.prefix on my system), I got other problems when importing. Can >> LIBTBX_BUILD be set in the conda package? >> >> >> >>

---

...

...

...

...

...

>> FileNotFoundError Traceback (most recent call last) >> <ipython-input-1-7e2b144826cc> in <module> >> 26 from cctbx import sgtbx >> 27 #from cctbx import miller >> ---> 28 from iotbx import reflection_file_reader, mtz >> 29 sns.set_context("poster") >> 30 plt.rcParams.update({'figure.autolayout': True}) >> >> ~/anaconda3/lib/python3.6/site-packages/iotbx/reflection_file_reader.py >> in <module> >> 53 >> 54 from __future__ import absolute_import, division, print_function >> ---> 55 from iotbx import mtz >> 56 from iotbx.scalepack import merge as scalepack_merge >> 57 from iotbx.scalepack import no_merge_original_index as >> scalepack_no_merge >> >> ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/__init__.py in <module> >> 9 import iotbx_mtz_ext as ext >> 10 >> ---> 11 from iotbx.mtz import extract_from_symmetry_lib >> 12 from cctbx import xray >> 13 import cctbx.xray.observation_types >> >> ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/extract_from_symmetry_lib.py >> in <module> >> 1 from __future__ import absolute_import, division, print_function >> 2 from cctbx import sgtbx >> ----> 3 import libtbx.load_env >> 4 import os.path as op >> 5 from six.moves import range >> >> ~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module> >> 3 import libtbx.env_config >> 4 import os >> ----> 5 libtbx.env = libtbx.env_config.unpickle() >> 6 libtbx.env.set_os_environ_all_dist() >> 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME") >> >> ~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle() >> 2605 build_path = os.environ["LIBTBX_BUILD"] >> 2606 set_preferred_sys_prefix_and_sys_executable(build_path=build_path) >> -> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb") >> 2608 env = pickle.load(libtbx_env) >> 2609 if (env.python_version_major_minor != sys.version_info[:2]): >> >> FileNotFoundError: [Errno 2] No such file or directory: >> '/home/gergely/anaconda3/libtbx_env' >> >> On Wed, Dec 18, 2019 at 9:10 AM Billy Poon wrote: >> > >> > Hi Gergely, >> > >> > I've uploaded linux packages to a new channel, cctbx-dev, and you can install it with >> > >> > conda install -c cctbx-dev cctbx >> > >> > in your current environment. You should set your ~/.condarc file to pull the other dependencies from the conda-forge channel first, so put conda-forge above cctbx. Mine looks like >> > >> > channels: >> > - conda-forge >> > - defaults >> > - cctbx >> > >> > Do you need dxtbx for your scripts? This package does not build that part. I think the plan is to build a separate conda package for dxtbx so that it can be updated more frequently. I can rebuild the packages to include it for testing, but the one being submitted to conda-forge will not have it. >> > >> > Also, your error message is probably due to version of HDF5 that the development build installs. The bootstrap.py script will install 1.10.4, but your other dependency is looking for 1.10.5. Installing this cctbx conda package should install 1.10.5, which should fix the issue. Also, I'm updating those environments and 1.10.5 will be the new default version. >> > >> > Lastly, the dispatchers will not work in these packages because the old paths during the build process are still in them and many of them expect some additional information that has not been updated in the packages yet. I'm in the process of doing that and will update the cctbx-dev channel when that's done. However, by starting python, you can import cctbx modules. So you can run commands like >> > >> > from scitbx.array_family import flex >> > a = flex.random_double(1000000) >> > b = flex.min_max_mean_double(a) >> > b.min >> > b.max >> > b.mean >> > >> > Thanks! >> > >> > -- >> > Billy K. Poon >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging >> > Lawrence Berkeley National Laboratory >> > 1 Cyclotron Road, M/S 33R0345 >> > Berkeley, CA 94720 >> > Tel: (510) 486-5709 >> > Fax: (510) 486-5909 >> > Web: https://phenix-online.org >> > >> > >> > On Tue, Dec 17, 2019 at 5:39 AM Gergely Katona < [email protected]> wrote: >> >> >> >> Dear Billy, >> >> >> >> Thank you for the detailed explanation, I look forward to do further >> >> testing! I aim to pool all modules under the same environment, if >> >> there are conflicts then I just try to reshuffle the order of imports. >> >> So far this did not cause problems for me even when I was using system >> >> python, but of course it is not the most prudent thing to do. With >> >> anaconda everything is much more standardized and isolated already and >> >> without being superuser I can have my familiar environment at any >> >> synchrotron based cluster. It is great that cctbx will be an integral >> >> part of this ecosystem and this was also the last thing holding me >> >> back from adopting python3. About hdf5, pymc3 requires it and >> >> importing causes a kernel restart with the following error messages, >> >> curiously if I import h5py first this can be avoided. >> >> >> >> Best wishes, >> >> >> >> Gergely >> >> >> >> import pymc3 as pm >> >> >> >> Warning! ***HDF5 library version mismatched error*** >> >> The HDF5 header files used to compile this application do not match >> >> the version used by the HDF5 library to which this application is linked. >> >> Data corruption or segmentation faults may occur if the application continues. >> >> This can happen when an application was compiled by one version of HDF5 but >> >> linked with a different version of static or shared HDF5 library. >> >> You should recompile the application or check your shared library related >> >> settings such as 'LD_LIBRARY_PATH'. >> >> You can, at your own risk, disable this warning by setting the environment >> >> variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'. >> >> Setting it to 2 or higher will suppress the warning messages totally. >> >> Headers are 1.10.4, library is 1.10.5 >> >> SUMMARY OF THE HDF5 CONFIGURATION >> >> ================================= >> >> >> >> General Information: >> >> ------------------- >> >> HDF5 Version: 1.10.5 >> >> Configured on: Tue Oct 22 12:02:13 UTC 2019 >> >> Configured by: conda@16247e67ecd5 >> >> Host system: x86_64-conda_cos6-linux-gnu >> >> Uname information: Linux 16247e67ecd5 4.15.0-1059-azure >> >> #64-Ubuntu SMP Fri Sep 13 17:02:44 UTC 2019 x86_64 x86_64 x86_64 >> >> GNU/Linux >> >> Byte sex: little-endian >> >> Installation point: /home/gergely/anaconda3 >> >> >> >> Compiling Options: >> >> ------------------ >> >> Build Mode: production >> >> Debugging Symbols: no >> >> Asserts: no >> >> Profiling: no >> >> Optimization Level: high >> >> >> >> Linking Options: >> >> ---------------- >> >> Libraries: static, shared >> >> Statically Linked Executables: >> >> LDFLAGS: -Wl,-O2 -Wl,--sort-common >> >> -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags >> >> -Wl,--gc-sections -Wl,-rpath,/home/gergely/anaconda3/lib >> >> -Wl,-rpath-link,/home/gergely/anaconda3/lib >> >> -L/home/gergely/anaconda3/lib >> >> H5_LDFLAGS: >> >> AM_LDFLAGS: -L/home/gergely/anaconda3/lib >> >> Extra libraries: -lrt -lpthread -lz -ldl -lm >> >> Archiver: >> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar >> >> AR_FLAGS: cr >> >> Ranlib: >> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib >> >> >> >> Languages: >> >> ---------- >> >> C: yes >> >> C Compiler: >> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc >> >> CPPFLAGS: -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 >> >> -I/home/gergely/anaconda3/include >> >> H5_CPPFLAGS: -D_GNU_SOURCE >> >> -D_POSIX_C_SOURCE=200809L -DNDEBUG -UH5_DEBUG_API >> >> AM_CPPFLAGS: -I/home/gergely/anaconda3/include >> >> C Flags: -march=nocona -mtune=haswell >> >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 >> >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include >> >> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5 >> >> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix >> >> H5 C Flags: -std=c99 -pedantic -Wall -Wextra >> >> -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual -Wconversion >> >> -Wdeclaration-after-statement -Wdisabled-optimization -Wfloat-equal >> >> -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations >> >> -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs >> >> -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls >> >> -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum -Wundef >> >> -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings >> >> -finline-functions -s -Wno-inline -Wno-aggregate-return >> >> -Wno-missing-format-attribute -Wno-missing-noreturn -O >> >> AM C Flags: >> >> Shared C Library: yes >> >> Static C Library: yes >> >> >> >> >> >> Fortran: yes >> >> Fortran Compiler: >> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran >> >> Fortran Flags: >> >> H5 Fortran Flags: -pedantic -Wall -Wextra -Wunderflow >> >> -Wimplicit-interface -Wsurprising -Wno-c-binding-type -s -O2 >> >> AM Fortran Flags: >> >> Shared Fortran Library: yes >> >> Static Fortran Library: yes >> >> >> >> C++: yes >> >> C++ Compiler: >> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++ >> >> C++ Flags: -fvisibility-inlines-hidden >> >> -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell >> >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 >> >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include >> >> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5 >> >> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix >> >> H5 C++ Flags: -pedantic -Wall -W -Wundef -Wshadow >> >> -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Wconversion >> >> -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual >> >> -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor >> >> -Wctor-dtor-privacy -Wabi -finline-functions -s -O >> >> AM C++ Flags: >> >> Shared C++ Library: yes >> >> Static C++ Library: yes >> >> >> >> Java: no >> >> >> >> >> >> Features: >> >> --------- >> >> Parallel HDF5: no >> >> Parallel Filtered Dataset Writes: no >> >> Large Parallel I/O: no >> >> High-level library: yes >> >> Threadsafety: yes >> >> Default API mapping: v110 >> >> With deprecated public symbols: yes >> >> I/O filters (external): deflate(zlib) >> >> MPE: no >> >> Direct VFD: no >> >> dmalloc: no >> >> Packages w/ extra debug output: none >> >> API tracing: no >> >> Using memory checker: yes >> >> Memory allocation sanity checks: no >> >> Function stack tracing: no >> >> Strict file format checks: no >> >> Optimization instrumentation: no >> >> >> >> On Tue, Dec 17, 2019 at 8:35 AM Billy Poon wrote: >> >> > >> >> > Hi Gergely, >> >> > >> >> > Let me build the test package tomorrow. All the gritty details for building with conda is being finalized and the official documentation will be updated to describe the steps. It would be too confusing to keep changing the documentation as the process evolves. You do have the general process, though, which is summarized as follows. >> >> > >> >> > 1) Installing dependencies. The cctbx_dependencies metapackage was an initial approach for managing the CCTBX dependencies, but after contacting the conda-forge folks, they recommended using the --only-deps flag. So when the CCTBX conda package is available, you'll be able to get a set of dependencies with, >> >> > >> >> > conda install -c conda-forge --only-deps cctbx >> >> > >> >> > By default, the bootstrap.py file will automatically install a set of dependencies in the "conda_base" directory (and a conda installation if one is not found). It just uses standard conda environment files located in libtbx/auto_build/conda_envs, so you do not need to install cctbx_dependencies as a separate step. The environment files avoid channel issues by explicitly defining the channel to pull the packages from and the cctbx channel just stores copies of packages from conda-forge. There were issues earlier where the conda-forge packages would sometimes be moved to a different label. The --use-conda flag also accepts a path to $CONDA_PREFIX if you want to use a specific environment for testing. >> >> > >> >> > 2) Building. The bootstrap.py file handles that with SCons. >> >> > >> >> > 3) Running. After building, there should be a setpaths.sh (and .csh) file that adds build/bin to your path. The build/bin directory has our dispatchers, which are just scripts that set up the environment variables for you. This prevents other programs from loading our libraries, whose versions may conflict. You should see that there is a "python" dispatcher, which is a convenience for developers. Otherwise, you can use libtbx.python, which will be able to import CCTBX modules. >> >> > >> >> > The running part is where the conda package for CCTBX will be different than this build. Since our Python files and extensions modules will be in the "site-packages" directory for Python, the PYTHONPATH variable will not be needed (and conda suggests that that variable not be set). The other CCTBX libraries will be in $CONDA_PREFIX/lib, so LD_LIBRARY_PATH is not needed. In an active environment, PATH will already be modified. And then our LIBTBX_BUILD directory can be set to sys.prefix. >> >> > >> >> > So with the conda package, you would only need to activate your conda environment and CCTBX should integrate with other conda packages. What is the conflict with hdf5? That's something that should be fixed. Thanks! >> >> > >> >> > -- >> >> > Billy K. Poon >> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging >> >> > Lawrence Berkeley National Laboratory >> >> > 1 Cyclotron Road, M/S 33R0345 >> >> > Berkeley, CA 94720 >> >> > Tel: (510) 486-5709 >> >> > Fax: (510) 486-5909 >> >> > Web: https://phenix-online.org >> >> > >> >> > >> >> > On Mon, Dec 16, 2019 at 2:53 AM Gergely Katona < [email protected]> wrote: >> >> >> >> >> >> Dear Billy, >> >> >> >> >> >> Thank you for this update and for your efforts! I found a solution and >> >> >> indeed most things already work in anaconda3. The steps I took (even >> >> >> if these instructions will have short expiry date): >> >> >> >> >> >> Modified .condarc with: >> >> >> channels: >> >> >> - cctbx >> >> >> - conda-forge >> >> >> - defaults >> >> >> >> >> >> run >> >> >> conda conda install cctbx_dependencies python=3.6 >> >> >> >> >> >> I expect this will work without python=3.6 in the near future. >> >> >> >> >> >> Then compiling cctbx with anaconda3 python went without any problem >> >> >> when using these flags: >> >> >> python bootstrap.py --use-conda --python3 --nproc=12 >> >> >> >> >> >> Then I probably did the unorthodox thing and sourced these directories: >> >> >> >> >> >> setenv LIBTBX_BUILD /home/gergely/cctbx/build >> >> >> setenv PATH ${PATH}:/home/gergely/cctbx/build/bin >> >> >> setenv PYTHONPATH >> >> >> /home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH >> >> >> setenv LD_LIBRARY_PATH >> >> >> /home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH >> >> >> >> >> >> There is probably a better way to put this into conda environment. >> >> >> With these steps I could run one of my scripts depending on cctbx. The >> >> >> only problem I found is that hdf5 library had conflict with another >> >> >> package in conda, but I does not find this as a showstopper. >> >> >> >> >> >> Best wishes, >> >> >> >> >> >> Gergely >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Fri, Dec 13, 2019 at 7:23 PM Billy Poon < [email protected]> wrote: >> >> >> > >> >> >> > Hi Gergely, >> >> >> > >> >> >> > It's still a work in progress. I'm sorting out some Windows issues right now. I can probably build a test package on a separate channel for people that want to test it (let's say next week?). I'll provide instructions, but basically, the test conda package will be in its own separate channel and the dependencies will be pulled from the conda-forge channel. I want most things to be working correctly on Python 2.7, 3.6, 3.7, and 3.8 on all 3 platforms. >> >> >> > >> >> >> > Thanks! >> >> >> > >> >> >> > -- >> >> >> > Billy K. Poon >> >> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging >> >> >> > Lawrence Berkeley National Laboratory >> >> >> > 1 Cyclotron Road, M/S 33R0345 >> >> >> > Berkeley, CA 94720 >> >> >> > Tel: (510) 486-5709 >> >> >> > Fax: (510) 486-5909 >> >> >> > Web: https://phenix-online.org >> >> >> > >> >> >> > >> >> >> > On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona < [email protected]> wrote: >> >> >> >> >> >> >> >> Dear Billy, >> >> >> >> >> >> >> >> This sounds very promising and exciting. I am not sure if cctbx is >> >> >> >> already functional as a conda package in anaconda3 (Linux) or this is >> >> >> >> still work in progress. My technical expertise does not allow me to >> >> >> >> tell the difference. What I tried: >> >> >> >> >> >> >> >> Fresh install of anaconda3. Adding - cctbx and - conda-forge to >> >> >> >> .condarc . Installing pyside2 with conda. Running conda install >> >> >> >> conda_dependencies . I get a lot package version conflict, and I >> >> >> >> cannot import cctbx or iotbx to anaconda python. Am I following the >> >> >> >> right instructions? Or it is too early to expect that cctbx works when >> >> >> >> installed through conda? >> >> >> >> >> >> >> >> Best wishes, >> >> >> >> >> >> >> >> Gergely >> >> >> >> >> >> >> >> On Wed, Nov 27, 2019 at 3:56 PM Billy Poon < [email protected]> wrote: >> >> >> >> > >> >> >> >> > Hi all, >> >> >> >> > >> >> >> >> > For a brief update, I have submitted a recipe for cctbxlite to conda-forge ( https://github.com/conda-forge/staged-recipes/pull/10021) and support for Python 3.7 and 3.8 is being added ( https://github.com/cctbx/cctbx_project/pull/409). With some fixes for Windows (https://github.com/cctbx/cctbx_project/pull/416), all platforms (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and 3.8. Some additional changes will be needed to get Windows to work with Python 3 and for tests to pass with Boost 1.70.0. That will enable the conda-forge recipe to build for all platforms and for all supported versions of Python. >> >> >> >> > >> >> >> >> > Currently, the conda-forge recipe will install into the conda python and cctbx imports can be done without sourcing the environment scripts. It looks like a lot of the environment variables being set in the dispatchers can be removed since the Python files and C++ extensions are in the right places. I'll update the libtbx_env file so that commands that load the environment can work correctly. >> >> >> >> > >> >> >> >> > -- >> >> >> >> > Billy K. Poon >> >> >> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging >> >> >> >> > Lawrence Berkeley National Laboratory >> >> >> >> > 1 Cyclotron Road, M/S 33R0345 >> >> >> >> > Berkeley, CA 94720 >> >> >> >> > Tel: (510) 486-5709 >> >> >> >> > Fax: (510) 486-5909 >> >> >> >> > Web: https://phenix-online.org >> >> >> >> > >> >> >> >> > >> >> >> >> > On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll < [email protected]> wrote: >> >> >> >> >> >> >> >> >> >> Hi Luc, >> >> >> >> >> >> >> >> >> >> That sounds promising. From there, I’d need to work out how to make a fully-packaged installer (basically a modified wheel file) for the ChimeraX ToolShed - the aim is for the end user to not have to worry about any of this. That adds a couple of complications - e.g. $LIBTBX_BUILD would need to be set dynamically before first import - but doesn’t seem insurmountable. >> >> >> >> >> >> >> >> >> >> Thanks, >> >> >> >> >> >> >> >> >> >> Tristan >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Tristan Croll >> >> >> >> >> Research Fellow >> >> >> >> >> Cambridge Institute for Medical Research >> >> >> >> >> University of Cambridge CB2 0XY >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> > On 25 Aug 2019, at 18:31, Luc Bourhis < [email protected]> wrote: >> >> >> >> >> > >> >> >> >> >> > Hi Tristan, >> >> >> >> >> > >> >> >> >> >> > cctbx could be built to use your ChimeraX python, now that cctbx is moving to Python 3. The option —with-python is there for that with the bootstrap script. The specific environment setup boil down to setting two environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on Linux, PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a framework such as ChimeraX, that should not be difficult to ensure those two variables are set. >> >> >> >> >> > >> >> >> >> >> > Best wishes, >> >> >> >> >> > >> >> >> >> >> > Luc >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> >> On 23 Aug 2019, at 19:02, Tristan Croll < [email protected]> wrote: >> >> >> >> >> >> >> >> >> >> >> >> To add my two cents on this: probably the second-most common question I've had about ISOLDE's implementation is, "why didn't you use CCTBX?". The honest answer to that is, "I didn't know how." >> >> >> >> >> >> >> >> >> >> >> >> Still don't, really - although the current developments are rather promising. The problem I've faced is that CCTBX was designed as its own self-contained Python (2.7, until very recently) environment, with its own interpreter and a lot of very specific environment setup. Meanwhile I'm developing ISOLDE in ChimeraX, which is *also* its own self-contained Python (3.7) environment. To plug one into the other in that form... well, I don't think I'm a good enough programmer to really know where to start. >> >> >> >> >> >> >> >> >> >> >> >> The move to Conda and a more modular CCTBX architecture should make a lot more possible in that direction. Pip would be even better for me personally (ChimeraX can install directly from the PyPI, but doesn't interact with Conda) - but I understand pretty well the substantial challenge that would amount to (not least being that the PyPI imposes a limit - around 100MB from memory? - on the size of an individual package). >> >> >> >> >> >> >> >> >> >> >> >> Best regards, >> >> >> >> >> >> >> >> >> >> >> >> Tristan >> >> >> >> >> >> >> >> >> >> >> >>> On 2019-08-23 09:28, Luc Bourhis wrote: >> >> >> >> >> >>> Hi Graeme, >> >> >> >> >> >>> Yes, I know. But “black" is a program doing a very particular task >> >> >> >> >> >>> (code formatting from the top of my head). Requiring to use a wrapper >> >> >> >> >> >>> for python itself is another level. But ok, I think I am mellowing to >> >> >> >> >> >>> the idea after all! Talking with people around me, and extrapolating, >> >> >> >> >> >>> I would bet that, right now, a great majority of people interested by >> >> >> >> >> >>> cctbx in pip have already used the cctbx, so they know about the >> >> >> >> >> >>> Python wrapper, and they would not be too sanguine about that. My >> >> >> >> >> >>> concern is for the future, when pip will be the first time some people >> >> >> >> >> >>> use cctbx. Big fat warning notices on PyPI page and a better error >> >> >> >> >> >>> message when cctbx fails because LIBTBX_BUILD is not set would be >> >> >> >> >> >>> needed but that could be all right. >> >> >> >> >> >>> If we do a pip installer, we should aim at a minimal install: cctbx, >> >> >> >> >> >>> iotbx and their dependencies, and that’s it. >> >> >> >> >> >>> Best wishes, >> >> >> >> >> >>> Luc >> >> >> >> >> >>>> On 23 Aug 2019, at 07:17, [email protected] wrote: >> >> >> >> >> >>>> Without discussing the merits of this or whether we _choose_ to make the move to supporting PIP, I am certain it would be _possible_ - many other packages make dispatcher scripts when you pip install them e.g. >> >> >> >> >> >>>> Silver-Surfer rescale_f2 :) $ which black; cat $(which black) >> >> >> >> >> >>>> /Library/Frameworks/Python.framework/Versions/3.6/bin/black >> >> >> >> >> >>>> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 >> >> >> >> >> >>>> # -*- coding: utf-8 -*- >> >> >> >> >> >>>> import re >> >> >> >> >> >>>> import sys >> >> >> >> >> >>>> from black import main >> >> >> >> >> >>>> if __name__ == '__main__': >> >> >> >> >> >>>> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) >> >> >> >> >> >>>> sys.exit(main()) >> >> >> >> >> >>>> So we _could_ work around the absence of LIBTBX_BUILD etc. in the system. Whether or not we elect to do the work is a different question, and it seems clear that here are very mixed opinions on this. >> >> >> >> >> >>>> Best wishes Graeme >> >> >> >> >> >>>> On 23 Aug 2019, at 01:21, Luc Bourhis < [email protected]mailto:[email protected]> wrote: >> >> >> >> >> >>>> Hi, >> >> >> >> >> >>>> Even if we managed to ship our the boost dynamic libraries with pip, it would still not be pip-like, as we would still need our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work with the standard python exe. LD_LIBRARY_PATH, we could get around that by changing the way we compile, using -Wl,-R, which is the runtime equivalent of build time -L. That’s a significant change that would need to be tested. But there is no way around setting LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps there is a way to hack libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip installs? >> >> >> >> >> >>>> Best wishes, >> >> >> >> >> >>>> Luc >> >> >> >> >> >>>> On 16 Aug 2019, at 22:47, Luc Bourhis < [email protected]mailto:[email protected]> wrote: >> >> >> >> >> >>>> Hi, >> >> >> >> >> >>>> I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.orghttp://anaconda.org/, and then to advertise it hard to every potential user out there. >> >> >> >> >> >>>> Best wishes, >> >> >> >> >> >>>> Luc >> >> >> >> >> >>>> On 16 Aug 2019, at 21:45, Aaron Brewster < [email protected]mailto:[email protected]> wrote: >> >> >> >> >> >>>> Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip ( https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-...). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not. >> >> >> >> >> >>>> All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard ( https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support. >> >> >> >> >> >>>> There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6). >> >> >> >> >> >>>> If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4. >> >> >> >> >> >>>> Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :) >> >> >> >> >> >>>> Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users. >> >> >> >> >> >>>> Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort. >> >> >> >> >> >>>> Thanks, >> >> >> >> >> >>>> -Aaron >> >> >> >> >> >>>> _______________________________________________ >> >> >> >> >> >>>> cctbxbb mailing list >> >> >> >> >> >>>> [email protected] >> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >>>> _______________________________________________ >> >> >> >> >> >>>> cctbxbb mailing list >> >> >> >> >> >>>> [email protected] >> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >>>> _______________________________________________ >> >> >> >> >> >>>> cctbxbb mailing list >> >> >> >> >> >>>> [email protected] >> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >>>> -- >> >> >> >> >> >>>> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. >> >> >> >> >> >>>> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. >> >> >> >> >> >>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. >> >> >> >> >> >>>> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom >> >> >> >> >> >>>> _______________________________________________ >> >> >> >> >> >>>> cctbxbb mailing list >> >> >> >> >> >>>> [email protected] >> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >>> _______________________________________________ >> >> >> >> >> >>> cctbxbb mailing list >> >> >> >> >> >>> [email protected] >> >> >> >> >> >>> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> >> >> >> cctbxbb mailing list >> >> >> >> >> >> [email protected] >> >> >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > _______________________________________________ >> >> >> >> >> > cctbxbb mailing list >> >> >> >> >> > [email protected] >> >> >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> >> >> cctbxbb mailing list >> >> >> >> >> [email protected] >> >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> > >> >> >> >> > _______________________________________________ >> >> >> >> > cctbxbb mailing list >> >> >> >> > [email protected] >> >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> Gergely Katona, PhD >> >> >> >> Associate Professor >> >> >> >> Department of Chemistry and Molecular Biology, University of Gothenburg >> >> >> >> Box 462, 40530 Göteborg, Sweden >> >> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 >> >> >> >> Web: http://katonalab.eu, Email: [email protected] >> >> >> >> >> >> >> >> _______________________________________________ >> >> >> >> cctbxbb mailing list >> >> >> >> [email protected] >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> > >> >> >> > _______________________________________________ >> >> >> > cctbxbb mailing list >> >> >> > [email protected] >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Gergely Katona, PhD >> >> >> Associate Professor >> >> >> Department of Chemistry and Molecular Biology, University of Gothenburg >> >> >> Box 462, 40530 Göteborg, Sweden >> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 >> >> >> Web: http://katonalab.eu, Email: [email protected] >> >> >> >> >> >> _______________________________________________ >> >> >> cctbxbb mailing list >> >> >> [email protected] >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> >> > >> >> > _______________________________________________ >> >> > cctbxbb mailing list >> >> > [email protected] >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> >> >> >> >> -- >> >> Gergely Katona, PhD >> >> Associate Professor >> >> Department of Chemistry and Molecular Biology, University of Gothenburg >> >> Box 462, 40530 Göteborg, Sweden >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 >> >> Web: http://katonalab.eu, Email: [email protected] >> >> >> >> _______________________________________________ >> >> cctbxbb mailing list >> >> [email protected] >> >> http://phenix-online.org/mailman/listinfo/cctbxbb >> > >> > _______________________________________________ >> > cctbxbb mailing list >> > [email protected] >> > http://phenix-online.org/mailman/listinfo/cctbxbb >> >> >> >> -- >> Gergely Katona, PhD >> Associate Professor >> Department of Chemistry and Molecular Biology, University of Gothenburg >> Box 462, 40530 Göteborg, Sweden >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 >> Web: http://katonalab.eu, Email: [email protected] >> >> _______________________________________________ >> cctbxbb mailing list >> [email protected] >> http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

-- Gergely Katona, Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

-- Gergely Katona, Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Hi Gergely, Oops, it's my mistake. I should use "set" and "unset" instead of "setenv" and "unsetenv". I'll build a new package probably next week. I want to clean up some more things for the conda packages. -- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org On Tue, Jan 14, 2020 at 3:45 AM Gergely Katona wrote:

Hi Billy,

I am using tcsh, the conda activate.d directory contains a sh and csh script for cctbx with a single line assigning the CONDA_PREFIX to the LIBTBX_BUILD environmental variable, it looks proper, I do not know why these are not executed/working. CONDA_PREFIX only get set when I use conda python, not when I open a terminal.

Best wishes,

Gergely

On Fri, Jan 10, 2020 at 5:40 PM Billy Poon wrote:

...

Hi Gergely,

So for your base environment, the std::__cxx11::basic_string error is

caused by the boost package coming from the cctbx channel. I built that copy of boost to support older compilers (developer environments may use GCC versions that are different than the one used for building conda-forge packages), but I'll just move it to another channel so it's less confusing. I'm surprised that the dependency resolution step kept that package instead of replacing it with the version from conda-forge.

...

What shell are you using? The LIBTBX_BUILD environment variable should

be set for sh and csh type shells. What files exist in $CONDA_PREFIX/etc/conda/activate.d ?

...

Thanks for testing!

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Tue, Jan 7, 2020 at 12:54 AM Gergely Katona

...

...

Ah, an there is no sign of the mentioned file in PYTHON_VERSION_MAJOR_MINOR in /home/gergely/anaconda3/lib before I forget.

Gergely

On Tue, Jan 7, 2020 at 9:37 AM Gergely Katona

wrote:

...

...

Hi Billy,

Here come the package lists from conda. I will do more testing with the cleaned condas installation. Indeed, the boost packages came from the conda-forge channel but for some odd reason in the test environment not in the base.

Best wishes,

Gergely

On Fri, Jan 3, 2020 at 9:05 PM Billy Poon wrote:

...

Hi Gergely,

Happy New Year!

Can you provide the output to "conda list" for each environment?

I suspect that the old version of cctbx might still be around. Is

...

...

...

...

The last error looks like an incompatible version of boost or gcc

wrote: there a file named PYTHON_VERSION_MAJOR_MINOR in "/home/gergely/anaconda3/lib"? If so, delete that file. To make sure you are downloading a new copy of the package, clear the cache where conda keeps copies of all the packages. To do that, you can run "conda clean -y --all". You can remove the "-y" flag if you want to see what will be deleted. The hash for the new linux package should be h8b68381_0. library was installed. The output from "conda list" should show the boost and boost-cpp packages coming from conda-forge. I would avoid installing into the root or base environment because that usually pulls from the default channels. All the dependencies should be pulled from the conda-forge channel.

...

...

...

...

-- Billy K. Poon Research Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org

On Fri, Jan 3, 2020 at 2:20 PM Gergely Katona

wrote:

...

...

I found some other errors when using the test environment.

---

...

...

...

...

...

RuntimeError Traceback (most recent call last) <ipython-input-1-fe308b345b3f> in <module> 10 import scipy as sp 11 from scipy import linalg ---> 12 import iotbx.pdb 13 import cctbx 14 import pandas as pd

~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/__init__.py

...

in <module> 8 9 import iotbx.pdb.records ---> 10 import iotbx.pdb.hierarchy 11 from scitbx import matrix 12

~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/pdb/hierarchy.py

...

in <module> 14 from six.moves import cStringIO as StringIO 15 from iotbx.pdb import hy36encode, hy36decode ---> 16 import iotbx.cif.model 17 from cctbx import crystal 18 from libtbx import group_args

~/anaconda3/envs/test/lib/python3.8/site-packages/iotbx/cif/__init__.py

...

in <module> 19 20 from cctbx.array_family import flex ---> 21 from cctbx import miller 22 from iotbx.cif import model, builders, geometry 23 from libtbx.containers import OrderedDict

~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/miller/__init__.py

...

in <module> 11 12 from cctbx import crystal ---> 13 from cctbx import maptbx 14 from cctbx import sgtbx 15 from cctbx.sgtbx import lattice_symmetry

~/anaconda3/envs/test/lib/python3.8/site-packages/cctbx/maptbx/__init__.py

...

in <module> 15 from libtbx import adopt_init_args 16 from libtbx.utils import Sorry ---> 17 import libtbx.load_env 18 import math 19 import sys, os

~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/load_env.py in <module>

...

3 import libtbx.env_config 4 import os ----> 5 libtbx.env = libtbx.env_config.unpickle() 6 libtbx.env.set_os_environ_all_dist() 7 libtbx.env.dispatcher_name =

os.environ.get("LIBTBX_DISPATCHER_NAME")

...

~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/env_config.py

...

in unpickle() 2736 env = pickle.load(libtbx_env) 2737 if (env.python_version_major_minor != sys.version_info[:2]): -> 2738 env.raise_python_version_incompatible() 2739 if (op.realpath(build_path) != op.realpath(abs(env.build_path))): 2740 env.build_path.reset(build_path)

...

in raise_python_version_incompatible(self, prev_pvmm) 469 if (prev_pvmm is None): 470 prev_pvmm = "%d.%d" % self.python_version_major_minor --> 471 raise RuntimeError("Python version incompatible with

~/anaconda3/envs/test/lib/python3.8/site-packages/libtbx/env_config.py this build:\n"

...

472 + " Build directory: %s\n" %

show_string(abs(self.build_path))

...

473 + " Python version used initially: %s\n" % prev_pvmm

RuntimeError: Python version incompatible with this build: Build directory: "/home/gergely/anaconda3" Python version used initially: 3.6 Python version in use now: 3.8

Interestingly in the base environment all import worked, but I got

an error at:

...

ArgumentError Traceback (most recent

call last)

...

<ipython-input-4-5bc71c36e121> in <module> 33 atm_odd.set_b(0) 34 ---> 35 odd.write_pdb_file("bdiff_in_even_000.pdb") #skriver ny pdb med dessa inkluderade 36 dfWBvals=pd.DataFrame(rows_list)

~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in write_pdb_file(self, file_name, open_append, crystal_symmetry, cryst1_z, write_scale_records, append_end, atom_hetatm, sigatm, anisou, siguij) 930 siguij=True): 931 if (crystal_symmetry is Auto): --> 932 crystal_symmetry = self.crystal_symmetry() 933 if (cryst1_z is Auto): 934 cryst1_z = self.extract_cryst1_z_columns()

~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in crystal_symmetry(self, crystal_symmetry, weak_symmetry) 1218 crystal_symmetry=None, 1219 weak_symmetry=False): -> 1220 self_symmetry = self.crystal_symmetry_from_cryst1() 1221 if (self_symmetry is None): 1222 self_symmetry = self._crystal_symmetry_from_cns_remark_sg()

~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/__init__.py in crystal_symmetry_from_cryst1(self) 1195 for line in self.crystallographic_section(): 1196 if (line.startswith("CRYST1")): -> 1197 return cryst1_interpretation.crystal_symmetry(cryst1_record=line) 1198 return None 1199

~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/cryst1_interpretation.py

...

in crystal_symmetry(cryst1_record) 136 space_group_info=None) 137 space_group_info = categorize(cryst1_record.sgroup).space_group_info( --> 138 unit_cell=u) 139 return crystal.symmetry(unit_cell=u, space_group_info=space_group_info)

...

in space_group_info(self, unit_cell) 85 if (self.symbol is None): return None 86 if (self.category is None): ---> 87 try: return sgtbx.space_group_info(self.symbol) 88 except RuntimeError: return None 89 if (isinstance(unit_cell, uctbx.ext.unit_cell)):

~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/__init__.py in __init__(self, symbol, table_id, group, number, space_group_t_den) 100 assert group is None 101 if (table_id is None): --> 102 symbols = space_group_symbols(symbol) 103 else: 104 if (isinstance(symbol, int)): symbol = str(symbol)

ArgumentError: Python argument types in space_group_symbols.__init__(space_group_symbols, str) did not match C++ signature: __init__(_object*, int space_group_number) __init__(_object*, int space_group_number, std::__cxx11::basic_string extension='') __init__(_object*, int space_group_number, std::__cxx11::basic_string extension='', std::__cxx11::basic_string table_id='') __init__(_object*, std::__cxx11::basic_string symbol) __init__(_object*, std::__cxx11::basic_string symbol, std::__cxx11::basic_string table_id='')

Gergely

On Fri, Jan 3, 2020 at 1:44 PM Gergely Katona wrote: > > Dear Billy, > > I did some tests installing in my base conda was not troublefree in > python 3.8. Running iotbx.fetch_pdb --all --mtz 1kp8 resulted in boost > related error when converting to mtz. Installing into a test > environment went fine with iotbx.fetch_pdb --all --mtz 1kp8 running > successfully. I could add pymc3, jupyter and seaborn and iotbx still > did not break. My LIBTBX_BUILD is not set however in the test > environment, which breaks miller when I was trying to import it. > Manually setting the LIBTBX_BUILD to CONDA_PREFIX (setenv LIBTBX_BUILD > $CONDA_PREFIX) fixes this and one of my script works fine! I will come > back if I find some problems with my other scripts, but so far

...

> looks promising! > > > Happy new year! > > Gergely > > > > > > On Fri, Dec 20, 2019 at 8:16 PM Billy Poon wrote: > > > > Hi everyone, > > > > I updated the conda packages on the cctbx-dev channel and

...

> > > > conda create -n test -c cctbx-dev cctbx python=2.7 > > conda activate test > > > > where "test" is the environment name and 2.7 is the python version. For the python version, you can also pick 3.6, 3.7, or 3.8. I

...

> > > > conda install -c cctbx-dev cctbx > > > > Again, your .condarc file should have conda-forge as the first channel so that dependencies can be pulled correctly. > > > > The dispatchers should work now and $LIBTBX_BUILD will be set on activation of an environment (and unset on environment deactivation) or after installing into an existing environment, so you can do more interesting things like > > > > iotbx.fetch_pdb --all --mtz 1kp8 > > > > However, this package does not install databases, so > > > > molprobity.molprobity 1kp8.pdb 1kp8.mtz > > > > will fail because the database is not available (also probe and reduce are not built). > > > > Some known issues in no particular order, > > > > 1) You can run "libtbx.run_tests_parallel module=cctbx nproc=Auto" to run tests, but not all the test files were copied, so I got 30 failures here. The final package will probably not have the test files. > > 2) The other python dispatchers, like "cctbx.python", were not updated, so those will fail. But you can just run "python" in the active environment. > > 3) Parts of mmtbx still need to be updated for Python 3. > > 4) The LIBTBX_BUILD variable should be set for sh and csh shells. Let me know if it's not working for your shell (i.e. "echo $LIBTBX_BUILD" should show the same thing as $CONDA_PREFIX). > > 5) One future step is to enable building other CCTBX-like modules with this conda package. This will require some bookkeeping updates so that modules in the conda environment and the modules being built are

...

> > > > Gergely, let me know if your scripts are working. > > > > Happy holidays! > > > > -- > > Billy K. Poon > > Research Scientist, Molecular Biophysics and Integrated Bioimaging > > Lawrence Berkeley National Laboratory > > 1 Cyclotron Road, M/S 33R0345 > > Berkeley, CA 94720 > > Tel: (510) 486-5709 > > Fax: (510) 486-5909 > > Web: https://phenix-online.org > > > > > > On Wed, Dec 18, 2019 at 11:25 AM Billy Poon wrote: > >> > >> Hi Gergely, > >> > >> That's the other information that hasn't been updated yet in

...

> >> > >> For LIBTBX_BUILD, I can set up environment variables sort of how the conda gcc package sets up environment variables. I have to double check to see if they're set after installation of the package and after activation of an environment. The variable will just be a copy of $CONDA_PREFIX. > >> > >> The libtbx_env file is a file that exists in $LIBTBX_BUILD

...

> >> > >> -- > >> Billy K. Poon > >> Research Scientist, Molecular Biophysics and Integrated Bioimaging > >> Lawrence Berkeley National Laboratory > >> 1 Cyclotron Road, M/S 33R0345 > >> Berkeley, CA 94720 > >> Tel: (510) 486-5709 > >> Fax: (510) 486-5909 > >> Web: https://phenix-online.org > >> > >> > >> On Wed, Dec 18, 2019 at 2:08 AM Gergely Katona < [email protected]> wrote: > >>> > >>> Hi Billy, > >>> > >>> Conda install went fine with your instructions after rearranging the > >>> channels by putting cctbx last. I removed all environmental variables > >>> and previous build of cctbx. > >>> > >>> import sys > >>> print (sys.path, sys.prefix) > >>> > >>> ['/home/gergely/anaconda3/lib/python36.zip', > >>> '/home/gergely/anaconda3/lib/python3.6', > >>> '/home/gergely/anaconda3/lib/python3.6/lib-dynload', '', > >>> '/home/gergely/anaconda3/lib/python3.6/site-packages', > >>> '/home/gergely/anaconda3/lib/python3.6/site-packages/IPython/extensions', > >>> '/home/gergely/.ipython'] /home/gergely/anaconda3 > >>> > >>> Many imports went fine including pymc3, but I encountered

~/anaconda3/lib/python3.6/site-packages/iotbx/pdb/cryst1_interpretation.py this there are also now packages for macOS. The old packages have been removed so you can create a new environment with think the latest conda will install 3.8 as the default python if the python argument is not provided. Or install into an existing environment (with python already installed) with the same command as before, tracked properly. Also, making sure that all the necessary headers exist in $PREFIX/include. the conda package. :) that basically stores configuration information. I'll have to modify the contents to update the locations of the different modules since the conda package will not have a "modules" directory. problems

...

> >>> with these three: > >>> > >>> from cctbx import miller > >>> import iotbx.pdb > >>> from iotbx import reflection_file_reader, mtz > >>> > >>> > >>> I also got type error when handling space groups. > >>> > >>> > >>> > >>>

---

...

...

...

...

...

> >>> KeyError Traceback (most recent call last) > >>> <ipython-input-5-cedbd4f84f3d> in <module> > >>> 25 from cctbx import uctbx > >>> 26 from cctbx import sgtbx > >>> ---> 27 from cctbx import miller > >>> 28 #from iotbx import reflection_file_reader, mtz > >>> 29 sns.set_context("poster") > >>> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/miller/__init__.py in <module> > >>> 11 > >>> 12 from cctbx import crystal > >>> ---> 13 from cctbx import maptbx > >>> 14 from cctbx import sgtbx > >>> 15 from cctbx.sgtbx import lattice_symmetry > >>> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/maptbx/__init__.py in <module> > >>> 15 from libtbx import adopt_init_args > >>> 16 from libtbx.utils import Sorry > >>> ---> 17 import libtbx.load_env > >>> 18 import math > >>> 19 import sys, os > >>> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module> > >>> 3 import libtbx.env_config > >>> 4 import os > >>> ----> 5 libtbx.env = libtbx.env_config.unpickle() > >>> 6 libtbx.env.set_os_environ_all_dist() > >>> 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME") > >>> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle() > >>> 2603 > >>> 2604 def unpickle(): > >>> -> 2605 build_path = os.environ["LIBTBX_BUILD"] > >>> 2606 set_preferred_sys_prefix_and_sys_executable(build_path=build_path) > >>> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb") > >>> > >>> ~/anaconda3/lib/python3.6/os.py in __getitem__(self, key) > >>> 667 except KeyError: > >>> 668 # raise KeyError with the original key value > >>> --> 669 raise KeyError(key) from None > >>> 670 return self.decodevalue(value) > >>> 671 > >>> > >>> KeyError: 'LIBTBX_BUILD' > >>> > >>> > >>> > >>>

---

...

...

...

...

...

> >>> ArgumentError Traceback (most recent call last) > >>> <ipython-input-7-2acd3a9ce26a> in <module> > >>> 26 return ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar > >>> 27 > >>> ---> 28 ms,msnam,mscent,msacent,msnamacent,msnamcent,msnamacent_dstar=initializecrystal() > >>> > >>> <ipython-input-7-2acd3a9ce26a> in initializecrystal() > >>> 11 uc = uctbx.unit_cell(unit_cell) > >>> 12 wavelength = 1.54980 > >>> ---> 13 xtal_symm = crystal.symmetry(unit_cell=unit_cell, > >>> space_group_symbol="P 43 21 2") > >>> 14 > >>> 15 ms = > >>> miller.build_set(crystal_symmetry=xtal_symm,anomalous_flag=True, > >>> d_min=1.61) > >>> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/crystal/__init__.py in > >>> __init__(self, unit_cell, space_group_symbol, space_group_info, > >>> space_group, correct_rhombohedral_setting_if_necessary, > >>> assert_is_compatible_unit_cell, raise_sorry_if_incompatible_unit_cell, > >>> force_compatible_unit_cell) > >>> 74 if (space_group_symbol is not None): > >>> 75 self._space_group_info = sgtbx.space_group_info( > >>> ---> 76 symbol=space_group_symbol) > >>> 77 elif (space_group is not None): > >>> 78 if (isinstance(space_group, sgtbx.space_group)): > >>> > >>> ~/anaconda3/lib/python3.6/site-packages/cctbx/sgtbx/__init__.py in > >>> __init__(self, symbol, table_id, group, number, space_group_t_den) > >>> 100 assert group is None > >>> 101 if (table_id is None): > >>> --> 102 symbols = space_group_symbols(symbol) > >>> 103 else: > >>> 104 if (isinstance(symbol, int)): symbol = str(symbol) > >>> > >>> ArgumentError: Python argument types in > >>> space_group_symbols.__init__(space_group_symbols, str) > >>> did not match C++ signature: > >>> __init__(_object*, int space_group_number) > >>> __init__(_object*, int space_group_number, > >>> std::__cxx11::basic_string >>> std::allocator<char> > extension='') > >>> __init__(_object*, int space_group_number, > >>> std::__cxx11::basic_string >>> std::allocator<char> > extension='', std::__cxx11::basic_string >>> std::char_traits<char>, std::allocator<char> > table_id='') > >>> __init__(_object*, std::__cxx11::basic_string >>> std::char_traits<char>, std::allocator<char> > symbol) > >>> __init__(_object*, std::__cxx11::basic_string >>> std::char_traits<char>, std::allocator<char> > symbol, > >>> std::__cxx11::basic_string >>> std::allocator<char> > table_id='') > >>> > >>> > >>> When I tried to set LIBTBX_BUILD to /home/gergely/anaconda3 (this is > >>> sys.prefix on my system), I got other problems when importing. Can > >>> LIBTBX_BUILD be set in the conda package? > >>> > >>> > >>> > >>>

---

...

...

...

...

...

> >>> FileNotFoundError Traceback (most recent call last) > >>> <ipython-input-1-7e2b144826cc> in <module> > >>> 26 from cctbx import sgtbx > >>> 27 #from cctbx import miller > >>> ---> 28 from iotbx import reflection_file_reader, mtz > >>> 29 sns.set_context("poster") > >>> 30 plt.rcParams.update({'figure.autolayout': True}) > >>> > >>> ~/anaconda3/lib/python3.6/site-packages/iotbx/reflection_file_reader.py > >>> in <module> > >>> 53 > >>> 54 from __future__ import absolute_import, division, print_function > >>> ---> 55 from iotbx import mtz > >>> 56 from iotbx.scalepack import merge as scalepack_merge > >>> 57 from iotbx.scalepack import no_merge_original_index as > >>> scalepack_no_merge > >>> > >>> ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/__init__.py in <module> > >>> 9 import iotbx_mtz_ext as ext > >>> 10 > >>> ---> 11 from iotbx.mtz import extract_from_symmetry_lib > >>> 12 from cctbx import xray > >>> 13 import cctbx.xray.observation_types > >>> > >>> ~/anaconda3/lib/python3.6/site-packages/iotbx/mtz/extract_from_symmetry_lib.py > >>> in <module> > >>> 1 from __future__ import absolute_import, division, print_function > >>> 2 from cctbx import sgtbx > >>> ----> 3 import libtbx.load_env > >>> 4 import os.path as op > >>> 5 from six.moves import range > >>> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/load_env.py in <module> > >>> 3 import libtbx.env_config > >>> 4 import os > >>> ----> 5 libtbx.env = libtbx.env_config.unpickle() > >>> 6 libtbx.env.set_os_environ_all_dist() > >>> 7 libtbx.env.dispatcher_name = os.environ.get("LIBTBX_DISPATCHER_NAME") > >>> > >>> ~/anaconda3/lib/python3.6/site-packages/libtbx/env_config.py in unpickle() > >>> 2605 build_path = os.environ["LIBTBX_BUILD"] > >>> 2606 set_preferred_sys_prefix_and_sys_executable(build_path=build_path) > >>> -> 2607 libtbx_env = open(op.join(build_path, "libtbx_env"), "rb") > >>> 2608 env = pickle.load(libtbx_env) > >>> 2609 if (env.python_version_major_minor != sys.version_info[:2]): > >>> > >>> FileNotFoundError: [Errno 2] No such file or directory: > >>> '/home/gergely/anaconda3/libtbx_env' > >>> > >>> On Wed, Dec 18, 2019 at 9:10 AM Billy Poon wrote: > >>> > > >>> > Hi Gergely, > >>> > > >>> > I've uploaded linux packages to a new channel, cctbx-dev, and you can install it with > >>> > > >>> > conda install -c cctbx-dev cctbx > >>> > > >>> > in your current environment. You should set your ~/.condarc file to pull the other dependencies from the conda-forge channel first, so put conda-forge above cctbx. Mine looks like > >>> > > >>> > channels: > >>> > - conda-forge > >>> > - defaults > >>> > - cctbx > >>> > > >>> > Do you need dxtbx for your scripts? This package does not build that part. I think the plan is to build a separate conda package for dxtbx so that it can be updated more frequently. I can rebuild the packages to include it for testing, but the one being submitted to conda-forge will not have it. > >>> > > >>> > Also, your error message is probably due to version of HDF5 that the development build installs. The bootstrap.py script will install 1.10.4, but your other dependency is looking for 1.10.5. Installing this cctbx conda package should install 1.10.5, which should fix the issue. Also, I'm updating those environments and 1.10.5 will be the new default version. > >>> > > >>> > Lastly, the dispatchers will not work in these packages because the old paths during the build process are still in them and many of them expect some additional information that has not been updated in the packages yet. I'm in the process of doing that and will update the cctbx-dev channel when that's done. However, by starting python, you can import cctbx modules. So you can run commands like > >>> > > >>> > from scitbx.array_family import flex > >>> > a = flex.random_double(1000000) > >>> > b = flex.min_max_mean_double(a) > >>> > b.min > >>> > b.max > >>> > b.mean > >>> > > >>> > Thanks! > >>> > > >>> > -- > >>> > Billy K. Poon > >>> > Research Scientist, Molecular Biophysics and Integrated Bioimaging > >>> > Lawrence Berkeley National Laboratory > >>> > 1 Cyclotron Road, M/S 33R0345 > >>> > Berkeley, CA 94720 > >>> > Tel: (510) 486-5709 > >>> > Fax: (510) 486-5909 > >>> > Web: https://phenix-online.org > >>> > > >>> > > >>> > On Tue, Dec 17, 2019 at 5:39 AM Gergely Katona < [email protected]> wrote: > >>> >> > >>> >> Dear Billy, > >>> >> > >>> >> Thank you for the detailed explanation, I look forward to do further > >>> >> testing! I aim to pool all modules under the same environment, if > >>> >> there are conflicts then I just try to reshuffle the order of imports. > >>> >> So far this did not cause problems for me even when I was using system > >>> >> python, but of course it is not the most prudent thing to do. With > >>> >> anaconda everything is much more standardized and isolated already and > >>> >> without being superuser I can have my familiar environment at any > >>> >> synchrotron based cluster. It is great that cctbx will be an integral > >>> >> part of this ecosystem and this was also the last thing holding me > >>> >> back from adopting python3. About hdf5, pymc3 requires it and > >>> >> importing causes a kernel restart with the following error messages, > >>> >> curiously if I import h5py first this can be avoided. > >>> >> > >>> >> Best wishes, > >>> >> > >>> >> Gergely > >>> >> > >>> >> import pymc3 as pm > >>> >> > >>> >> Warning! ***HDF5 library version mismatched error*** > >>> >> The HDF5 header files used to compile this application do not match > >>> >> the version used by the HDF5 library to which this application is linked. > >>> >> Data corruption or segmentation faults may occur if the application continues. > >>> >> This can happen when an application was compiled by one version of HDF5 but > >>> >> linked with a different version of static or shared HDF5 library. > >>> >> You should recompile the application or check your shared library related > >>> >> settings such as 'LD_LIBRARY_PATH'. > >>> >> You can, at your own risk, disable this warning by setting the environment > >>> >> variable 'HDF5_DISABLE_VERSION_CHECK' to a value of '1'. > >>> >> Setting it to 2 or higher will suppress the warning messages totally. > >>> >> Headers are 1.10.4, library is 1.10.5 > >>> >> SUMMARY OF THE HDF5 CONFIGURATION > >>> >> ================================= > >>> >> > >>> >> General Information: > >>> >> ------------------- > >>> >> HDF5 Version: 1.10.5 > >>> >> Configured on: Tue Oct 22 12:02:13 UTC 2019 > >>> >> Configured by: conda@16247e67ecd5 > >>> >> Host system: x86_64-conda_cos6-linux-gnu > >>> >> Uname information: Linux 16247e67ecd5 4.15.0-1059-azure > >>> >> #64-Ubuntu SMP Fri Sep 13 17:02:44 UTC 2019 x86_64 x86_64 x86_64 > >>> >> GNU/Linux > >>> >> Byte sex: little-endian > >>> >> Installation point: /home/gergely/anaconda3 > >>> >> > >>> >> Compiling Options: > >>> >> ------------------ > >>> >> Build Mode: production > >>> >> Debugging Symbols: no > >>> >> Asserts: no > >>> >> Profiling: no > >>> >> Optimization Level: high > >>> >> > >>> >> Linking Options: > >>> >> ---------------- > >>> >> Libraries: static, shared > >>> >> Statically Linked Executables: > >>> >> LDFLAGS: -Wl,-O2 -Wl,--sort-common > >>> >> -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags > >>> >> -Wl,--gc-sections -Wl,-rpath,/home/gergely/anaconda3/lib > >>> >> -Wl,-rpath-link,/home/gergely/anaconda3/lib > >>> >> -L/home/gergely/anaconda3/lib > >>> >> H5_LDFLAGS: > >>> >> AM_LDFLAGS: -L/home/gergely/anaconda3/lib > >>> >> Extra libraries: -lrt -lpthread -lz -ldl -lm > >>> >> Archiver: > >>> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar > >>> >> AR_FLAGS: cr > >>> >> Ranlib: > >>> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib > >>> >> > >>> >> Languages: > >>> >> ---------- > >>> >> C: yes > >>> >> C Compiler: > >>> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc > >>> >> CPPFLAGS: -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 > >>> >> -I/home/gergely/anaconda3/include > >>> >> H5_CPPFLAGS: -D_GNU_SOURCE > >>> >> -D_POSIX_C_SOURCE=200809L -DNDEBUG -UH5_DEBUG_API > >>> >> AM_CPPFLAGS: -I/home/gergely/anaconda3/include > >>> >> C Flags: -march=nocona -mtune=haswell > >>> >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 > >>> >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include > >>> >> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5 > >>> >> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix > >>> >> H5 C Flags: -std=c99 -pedantic -Wall -Wextra > >>> >> -Wbad-function-cast -Wc++-compat -Wcast-align -Wcast-qual -Wconversion > >>> >> -Wdeclaration-after-statement -Wdisabled-optimization -Wfloat-equal > >>> >> -Wformat=2 -Winit-self -Winvalid-pch -Wmissing-declarations > >>> >> -Wmissing-include-dirs -Wmissing-prototypes -Wnested-externs > >>> >> -Wold-style-definition -Wpacked -Wpointer-arith -Wredundant-decls > >>> >> -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum -Wundef > >>> >> -Wunused-macros -Wunsafe-loop-optimizations -Wwrite-strings > >>> >> -finline-functions -s -Wno-inline -Wno-aggregate-return > >>> >> -Wno-missing-format-attribute -Wno-missing-noreturn -O > >>> >> AM C Flags: > >>> >> Shared C Library: yes > >>> >> Static C Library: yes > >>> >> > >>> >> > >>> >> Fortran: yes > >>> >> Fortran Compiler: > >>> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-gfortran > >>> >> Fortran Flags: > >>> >> H5 Fortran Flags: -pedantic -Wall -Wextra -Wunderflow > >>> >> -Wimplicit-interface -Wsurprising -Wno-c-binding-type -s -O2 > >>> >> AM Fortran Flags: > >>> >> Shared Fortran Library: yes > >>> >> Static Fortran Library: yes > >>> >> > >>> >> C++: yes > >>> >> C++ Compiler: > >>> >> /home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++ > >>> >> C++ Flags: -fvisibility-inlines-hidden > >>> >> -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell > >>> >> -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 > >>> >> -ffunction-sections -pipe -I/home/gergely/anaconda3/include > >>> >> -fdebug-prefix-map=/home/conda/feedstock_root/build_artifacts/hdf5_split_1571745596770/work=/usr/local/src/conda/hdf5_split-1.10.5 > >>> >> -fdebug-prefix-map=/home/gergely/anaconda3=/usr/local/src/conda-prefix > >>> >> H5 C++ Flags: -pedantic -Wall -W -Wundef -Wshadow > >>> >> -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Wconversion > >>> >> -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual > >>> >> -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor > >>> >> -Wctor-dtor-privacy -Wabi -finline-functions -s -O > >>> >> AM C++ Flags: > >>> >> Shared C++ Library: yes > >>> >> Static C++ Library: yes > >>> >> > >>> >> Java: no > >>> >> > >>> >> > >>> >> Features: > >>> >> --------- > >>> >> Parallel HDF5: no > >>> >> Parallel Filtered Dataset Writes: no > >>> >> Large Parallel I/O: no > >>> >> High-level library: yes > >>> >> Threadsafety: yes > >>> >> Default API mapping: v110 > >>> >> With deprecated public symbols: yes > >>> >> I/O filters (external): deflate(zlib) > >>> >> MPE: no > >>> >> Direct VFD: no > >>> >> dmalloc: no > >>> >> Packages w/ extra debug output: none > >>> >> API tracing: no > >>> >> Using memory checker: yes > >>> >> Memory allocation sanity checks: no > >>> >> Function stack tracing: no > >>> >> Strict file format checks: no > >>> >> Optimization instrumentation: no > >>> >> > >>> >> On Tue, Dec 17, 2019 at 8:35 AM Billy Poon < [email protected]> wrote: > >>> >> > > >>> >> > Hi Gergely, > >>> >> > > >>> >> > Let me build the test package tomorrow. All the gritty details for building with conda is being finalized and the official documentation will be updated to describe the steps. It would be too confusing to keep changing the documentation as the process evolves. You do have the general process, though, which is summarized as follows. > >>> >> > > >>> >> > 1) Installing dependencies. The cctbx_dependencies metapackage was an initial approach for managing the CCTBX dependencies, but after contacting the conda-forge folks, they recommended using the --only-deps flag. So when the CCTBX conda package is available, you'll be able to get a set of dependencies with, > >>> >> > > >>> >> > conda install -c conda-forge --only-deps cctbx > >>> >> > > >>> >> > By default, the bootstrap.py file will automatically install a set of dependencies in the "conda_base" directory (and a conda installation if one is not found). It just uses standard conda environment files located in libtbx/auto_build/conda_envs, so you do not need to install cctbx_dependencies as a separate step. The environment files avoid channel issues by explicitly defining the channel to pull the packages from and the cctbx channel just stores copies of packages from conda-forge. There were issues earlier where the conda-forge packages would sometimes be moved to a different label. The --use-conda flag also accepts a path to $CONDA_PREFIX if you want to use a specific environment for testing. > >>> >> > > >>> >> > 2) Building. The bootstrap.py file handles that with SCons. > >>> >> > > >>> >> > 3) Running. After building, there should be a setpaths.sh (and .csh) file that adds build/bin to your path. The build/bin directory has our dispatchers, which are just scripts that set up the environment variables for you. This prevents other programs from loading our libraries, whose versions may conflict. You should see that there is a "python" dispatcher, which is a convenience for developers. Otherwise, you can use libtbx.python, which will be able to import CCTBX modules. > >>> >> > > >>> >> > The running part is where the conda package for CCTBX will be different than this build. Since our Python files and extensions modules will be in the "site-packages" directory for Python, the PYTHONPATH variable will not be needed (and conda suggests that that variable not be set). The other CCTBX libraries will be in $CONDA_PREFIX/lib, so LD_LIBRARY_PATH is not needed. In an active environment, PATH will already be modified. And then our LIBTBX_BUILD directory can be set to sys.prefix. > >>> >> > > >>> >> > So with the conda package, you would only need to activate your conda environment and CCTBX should integrate with other conda packages. What is the conflict with hdf5? That's something that should be fixed. Thanks! > >>> >> > > >>> >> > -- > >>> >> > Billy K. Poon > >>> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging > >>> >> > Lawrence Berkeley National Laboratory > >>> >> > 1 Cyclotron Road, M/S 33R0345 > >>> >> > Berkeley, CA 94720 > >>> >> > Tel: (510) 486-5709 > >>> >> > Fax: (510) 486-5909 > >>> >> > Web: https://phenix-online.org > >>> >> > > >>> >> > > >>> >> > On Mon, Dec 16, 2019 at 2:53 AM Gergely Katona < [email protected]> wrote: > >>> >> >> > >>> >> >> Dear Billy, > >>> >> >> > >>> >> >> Thank you for this update and for your efforts! I found a solution and > >>> >> >> indeed most things already work in anaconda3. The steps I took (even > >>> >> >> if these instructions will have short expiry date): > >>> >> >> > >>> >> >> Modified .condarc with: > >>> >> >> channels: > >>> >> >> - cctbx > >>> >> >> - conda-forge > >>> >> >> - defaults > >>> >> >> > >>> >> >> run > >>> >> >> conda conda install cctbx_dependencies python=3.6 > >>> >> >> > >>> >> >> I expect this will work without python=3.6 in the near future. > >>> >> >> > >>> >> >> Then compiling cctbx with anaconda3 python went without any problem > >>> >> >> when using these flags: > >>> >> >> python bootstrap.py --use-conda --python3 --nproc=12 > >>> >> >> > >>> >> >> Then I probably did the unorthodox thing and sourced these directories: > >>> >> >> > >>> >> >> setenv LIBTBX_BUILD /home/gergely/cctbx/build > >>> >> >> setenv PATH ${PATH}:/home/gergely/cctbx/build/bin > >>> >> >> setenv PYTHONPATH > >>> >> >> /home/gergely/cctbx/modules/cctbx_project:/home/gergely/cctbx/modules:/home/gergely/cctbx/modules/cctbx_project/boost_adaptbx:/home/gergely/cctbx/build/lib:/home/gergely/cctbx/conda_base/lib/python3.6/site-packages:$PYTHONPATH > >>> >> >> setenv LD_LIBRARY_PATH > >>> >> >> /home/gergely/cctbx/conda_base/lib:/home/gergely/cctbx/build/lib:$LD_LIBRARY_PATH > >>> >> >> > >>> >> >> There is probably a better way to put this into conda environment. > >>> >> >> With these steps I could run one of my scripts depending on cctbx. The > >>> >> >> only problem I found is that hdf5 library had conflict with another > >>> >> >> package in conda, but I does not find this as a showstopper. > >>> >> >> > >>> >> >> Best wishes, > >>> >> >> > >>> >> >> Gergely > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> On Fri, Dec 13, 2019 at 7:23 PM Billy Poon < [email protected]> wrote: > >>> >> >> > > >>> >> >> > Hi Gergely, > >>> >> >> > > >>> >> >> > It's still a work in progress. I'm sorting out some Windows issues right now. I can probably build a test package on a separate channel for people that want to test it (let's say next week?). I'll provide instructions, but basically, the test conda package will be in its own separate channel and the dependencies will be pulled from the conda-forge channel. I want most things to be working correctly on Python 2.7, 3.6, 3.7, and 3.8 on all 3 platforms. > >>> >> >> > > >>> >> >> > Thanks! > >>> >> >> > > >>> >> >> > -- > >>> >> >> > Billy K. Poon > >>> >> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging > >>> >> >> > Lawrence Berkeley National Laboratory > >>> >> >> > 1 Cyclotron Road, M/S 33R0345 > >>> >> >> > Berkeley, CA 94720 > >>> >> >> > Tel: (510) 486-5709 > >>> >> >> > Fax: (510) 486-5909 > >>> >> >> > Web: https://phenix-online.org > >>> >> >> > > >>> >> >> > > >>> >> >> > On Fri, Dec 13, 2019 at 5:44 AM Gergely Katona < [email protected]> wrote: > >>> >> >> >> > >>> >> >> >> Dear Billy, > >>> >> >> >> > >>> >> >> >> This sounds very promising and exciting. I am not sure if cctbx is > >>> >> >> >> already functional as a conda package in anaconda3 (Linux) or this is > >>> >> >> >> still work in progress. My technical expertise does not allow me to > >>> >> >> >> tell the difference. What I tried: > >>> >> >> >> > >>> >> >> >> Fresh install of anaconda3. Adding - cctbx and - conda-forge to > >>> >> >> >> .condarc . Installing pyside2 with conda. Running conda install > >>> >> >> >> conda_dependencies . I get a lot package version conflict, and I > >>> >> >> >> cannot import cctbx or iotbx to anaconda python. Am I following the > >>> >> >> >> right instructions? Or it is too early to expect that cctbx works when > >>> >> >> >> installed through conda? > >>> >> >> >> > >>> >> >> >> Best wishes, > >>> >> >> >> > >>> >> >> >> Gergely > >>> >> >> >> > >>> >> >> >> On Wed, Nov 27, 2019 at 3:56 PM Billy Poon < [email protected]> wrote: > >>> >> >> >> > > >>> >> >> >> > Hi all, > >>> >> >> >> > > >>> >> >> >> > For a brief update, I have submitted a recipe for cctbxlite to conda-forge ( https://github.com/conda-forge/staged-recipes/pull/10021) and support for Python 3.7 and 3.8 is being added ( https://github.com/cctbx/cctbx_project/pull/409). With some fixes for Windows (https://github.com/cctbx/cctbx_project/pull/416), all platforms (macOS, linux, and Windows) can build for Python 2.7, 3.6, 3.7, and 3.8. Some additional changes will be needed to get Windows to work with Python 3 and for tests to pass with Boost 1.70.0. That will enable the conda-forge recipe to build for all platforms and for all supported versions of Python. > >>> >> >> >> > > >>> >> >> >> > Currently, the conda-forge recipe will install into the conda python and cctbx imports can be done without sourcing the environment scripts. It looks like a lot of the environment variables being set in the dispatchers can be removed since the Python files and C++ extensions are in the right places. I'll update the libtbx_env file so that commands that load the environment can work correctly. > >>> >> >> >> > > >>> >> >> >> > -- > >>> >> >> >> > Billy K. Poon > >>> >> >> >> > Research Scientist, Molecular Biophysics and Integrated Bioimaging > >>> >> >> >> > Lawrence Berkeley National Laboratory > >>> >> >> >> > 1 Cyclotron Road, M/S 33R0345 > >>> >> >> >> > Berkeley, CA 94720 > >>> >> >> >> > Tel: (510) 486-5709 > >>> >> >> >> > Fax: (510) 486-5909 > >>> >> >> >> > Web: https://phenix-online.org > >>> >> >> >> > > >>> >> >> >> > > >>> >> >> >> > On Sun, Aug 25, 2019 at 2:33 PM Tristan Croll < [email protected]> wrote: > >>> >> >> >> >> > >>> >> >> >> >> Hi Luc, > >>> >> >> >> >> > >>> >> >> >> >> That sounds promising. From there, I’d need to work out how to make a fully-packaged installer (basically a modified wheel file) for the ChimeraX ToolShed - the aim is for the end user to not have to worry about any of this. That adds a couple of complications - e.g. $LIBTBX_BUILD would need to be set dynamically before first import - but doesn’t seem insurmountable. > >>> >> >> >> >> > >>> >> >> >> >> Thanks, > >>> >> >> >> >> > >>> >> >> >> >> Tristan > >>> >> >> >> >> > >>> >> >> >> >> > >>> >> >> >> >> > >>> >> >> >> >> Tristan Croll > >>> >> >> >> >> Research Fellow > >>> >> >> >> >> Cambridge Institute for Medical Research > >>> >> >> >> >> University of Cambridge CB2 0XY > >>> >> >> >> >> > >>> >> >> >> >> > >>> >> >> >> >> > >>> >> >> >> >> > >>> >> >> >> >> > On 25 Aug 2019, at 18:31, Luc Bourhis < [email protected]> wrote: > >>> >> >> >> >> > > >>> >> >> >> >> > Hi Tristan, > >>> >> >> >> >> > > >>> >> >> >> >> > cctbx could be built to use your ChimeraX python, now that cctbx is moving to Python 3. The option —with-python is there for that with the bootstrap script. The specific environment setup boil down to setting two environment variable LIBTBX_BUILD and either LD_LIBRARY_PATH on Linux, PATH on Win32, or DYLIB_LIBRARY_PATH on MacOS. If you work within a framework such as ChimeraX, that should not be difficult to ensure those two variables are set. > >>> >> >> >> >> > > >>> >> >> >> >> > Best wishes, > >>> >> >> >> >> > > >>> >> >> >> >> > Luc > >>> >> >> >> >> > > >>> >> >> >> >> > > >>> >> >> >> >> >> On 23 Aug 2019, at 19:02, Tristan Croll < [email protected]> wrote: > >>> >> >> >> >> >> > >>> >> >> >> >> >> To add my two cents on this: probably the second-most common question I've had about ISOLDE's implementation is, "why didn't you use CCTBX?". The honest answer to that is, "I didn't know how." > >>> >> >> >> >> >> > >>> >> >> >> >> >> Still don't, really - although the current developments are rather promising. The problem I've faced is that CCTBX was designed as its own self-contained Python (2.7, until very recently) environment, with its own interpreter and a lot of very specific environment setup. Meanwhile I'm developing ISOLDE in ChimeraX, which is *also* its own self-contained Python (3.7) environment. To plug one into the other in that form... well, I don't think I'm a good enough programmer to really know where to start. > >>> >> >> >> >> >> > >>> >> >> >> >> >> The move to Conda and a more modular CCTBX architecture should make a lot more possible in that direction. Pip would be even better for me personally (ChimeraX can install directly from the PyPI, but doesn't interact with Conda) - but I understand pretty well the substantial challenge that would amount to (not least being that the PyPI imposes a limit - around 100MB from memory? - on the size of an individual package). > >>> >> >> >> >> >> > >>> >> >> >> >> >> Best regards, > >>> >> >> >> >> >> > >>> >> >> >> >> >> Tristan > >>> >> >> >> >> >> > >>> >> >> >> >> >>> On 2019-08-23 09:28, Luc Bourhis wrote: > >>> >> >> >> >> >>> Hi Graeme, > >>> >> >> >> >> >>> Yes, I know. But “black" is a program doing a very particular task > >>> >> >> >> >> >>> (code formatting from the top of my head). Requiring to use a wrapper > >>> >> >> >> >> >>> for python itself is another level. But ok, I think I am mellowing to > >>> >> >> >> >> >>> the idea after all! Talking with people around me, and extrapolating, > >>> >> >> >> >> >>> I would bet that, right now, a great majority of people interested by > >>> >> >> >> >> >>> cctbx in pip have already used the cctbx, so they know about the > >>> >> >> >> >> >>> Python wrapper, and they would not be too sanguine about that. My > >>> >> >> >> >> >>> concern is for the future, when pip will be the first time some people > >>> >> >> >> >> >>> use cctbx. Big fat warning notices on PyPI page and a better error > >>> >> >> >> >> >>> message when cctbx fails because LIBTBX_BUILD is not set would be > >>> >> >> >> >> >>> needed but that could be all right. > >>> >> >> >> >> >>> If we do a pip installer, we should aim at a minimal install: cctbx, > >>> >> >> >> >> >>> iotbx and their dependencies, and that’s it. > >>> >> >> >> >> >>> Best wishes, > >>> >> >> >> >> >>> Luc > >>> >> >> >> >> >>>> On 23 Aug 2019, at 07:17, [email protected] wrote: > >>> >> >> >> >> >>>> Without discussing the merits of this or whether we _choose_ to make the move to supporting PIP, I am certain it would be _possible_ - many other packages make dispatcher scripts when you pip install them e.g. > >>> >> >> >> >> >>>> Silver-Surfer rescale_f2 :) $ which black; cat $(which black) > >>> >> >> >> >> >>>> /Library/Frameworks/Python.framework/Versions/3.6/bin/black > >>> >> >> >> >> >>>> #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 > >>> >> >> >> >> >>>> # -*- coding: utf-8 -*- > >>> >> >> >> >> >>>> import re > >>> >> >> >> >> >>>> import sys > >>> >> >> >> >> >>>> from black import main > >>> >> >> >> >> >>>> if __name__ == '__main__': > >>> >> >> >> >> >>>> sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) > >>> >> >> >> >> >>>> sys.exit(main()) > >>> >> >> >> >> >>>> So we _could_ work around the absence of LIBTBX_BUILD etc. in the system. Whether or not we elect to do the work is a different question, and it seems clear that here are very mixed opinions on this. > >>> >> >> >> >> >>>> Best wishes Graeme > >>> >> >> >> >> >>>> On 23 Aug 2019, at 01:21, Luc Bourhis < [email protected]mailto:[email protected]> wrote: > >>> >> >> >> >> >>>> Hi, > >>> >> >> >> >> >>>> Even if we managed to ship our the boost dynamic libraries with pip, it would still not be pip-like, as we would still need our python wrappers to set LIBTBX_BUILD and LD_LIBRARY_PATH. Normal pip packages work with the standard python exe. LD_LIBRARY_PATH, we could get around that by changing the way we compile, using -Wl,-R, which is the runtime equivalent of build time -L. That’s a significant change that would need to be tested. But there is no way around setting LIBTBX_BUILD right now. Leaving that to the user is horrible. Perhaps there is a way to hack libtbx/env_config.py so that we can hardwire LIBTBX_BUILD in there when pip installs? > >>> >> >> >> >> >>>> Best wishes, > >>> >> >> >> >> >>>> Luc > >>> >> >> >> >> >>>> On 16 Aug 2019, at 22:47, Luc Bourhis < [email protected]mailto:[email protected]> wrote: > >>> >> >> >> >> >>>> Hi, > >>> >> >> >> >> >>>> I did look into that many years ago, and even toyed with building a pip installer. What stopped me is the exact conclusion you reached too: the user would not have the pip experience he expects. You are right that it is a lot of effort but is it worth it? Considering that remark, I don’t think so. Now, Conda was created specifically to go beyond pip pure-python-only support. Since cctbx has garnered support for Conda, the best avenue imho is to go the extra length to have a package on Anaconda.orghttp://anaconda.org/, and then to advertise it hard to every potential user out there. > >>> >> >> >> >> >>>> Best wishes, > >>> >> >> >> >> >>>> Luc > >>> >> >> >> >> >>>> On 16 Aug 2019, at 21:45, Aaron Brewster < [email protected]mailto:[email protected]> wrote: > >>> >> >> >> >> >>>> Hi, to avoid clouding Dorothee's documentation email thread, which I think is a highly useful enterprise, here's some thoughts about putting cctbx into pip. Pip doesn't install non-python dependencies well. I don't think boost is available as a package on pip (at least the package version we use). wxPython4 isn't portable through pip ( https://wiki.wxpython.org/How%20to%20install%20wxPython#Installing_wxPython-...). MPI libraries are system dependent. If cctbx were a pure python package, pip would be fine, but cctbx is not. > >>> >> >> >> >> >>>> All that said, we could build a manylinux1 version of cctbx and upload it to PyPi (I'm just learning about this). For a pip package to be portable (which is a requirement for cctbx), it needs to conform to PEP513, the manylinux1 standard ( https://www.python.org/dev/peps/pep-0513/). For example, numpy is built according to this standard (see https://pypi.org/project/numpy/#files, where you'll see the manylinux1 wheel). Note, the manylinux1 standard is built with Centos 5.11 which we no longer support. > >>> >> >> >> >> >>>> There is also a manylinux2010 standard, which is based on Centos 6 (https://www.python.org/dev/peps/pep-0571/). This is likely a more attainable target (note though by default C++11 is not supported on Centos 6). > >>> >> >> >> >> >>>> If we built a manylinuxX version of cctbx and uploaded it to PyPi, the user would need all the non-python dependencies. There's no way to specify these in pip. For example, cctbx requires boost 1.63 or better. The user will need to have it in a place their python can find it, or we could package it ourselves and supply it, similar to how the pip h5py package now comes with an hd5f library, or how the pip numpy package includes an openblas library. We'd have to do the same for any packages we depend on that aren't on pip using the manylinux standards, such as wxPython4. > >>> >> >> >> >> >>>> Further, we need to think about how dials and other cctbx-based packages interact. If pip install cctbx is set up, how does pip install dials work, such that any dials shared libraries can find the cctbx libraries? Can shared libraries from one pip package link against libraries in another pip package? Would each package need to supply its own boost? Possibly this is well understood in the pip field, but not by me :) > >>> >> >> >> >> >>>> Finally, there's the option of providing a source pip package. This would require the full compiler toolchain for any given platform (macOS, linux, windows). These are likely available for developers, but not for general users. > >>> >> >> >> >> >>>> Anyway, these are some of the obstacles. Not saying it isn't possible, it's just a lot of effort. > >>> >> >> >> >> >>>> Thanks, > >>> >> >> >> >> >>>> -Aaron > >>> >> >> >> >> >>>>

---

...

...

...

...

...

> >>> >> >> >> >> >>>> cctbxbb mailing list > >>> >> >> >> >> >>>> [email protected] > >>> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb > >>> >> >> >> >> >>>>

---

...

...

...

...

...

> >>> >> >> >> >> >>>> cctbxbb mailing list > >>> >> >> >> >> >>>> [email protected] > >>> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb > >>> >> >> >> >> >>>>

---

...

...

...

...

...

> >>> >> >> >> >> >>>> cctbxbb mailing list > >>> >> >> >> >> >>>> [email protected] > >>> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb > >>> >> >> >> >> >>>> -- > >>> >> >> >> >> >>>> This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. > >>> >> >> >> >> >>>> Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. > >>> >> >> >> >> >>>> Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. > >>> >> >> >> >> >>>> Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom > >>> >> >> >> >> >>>>

---

...

...

...

...

...

> >>> >> >> >> >> >>>> cctbxbb mailing list > >>> >> >> >> >> >>>> [email protected] > >>> >> >> >> >> >>>> http://phenix-online.org/mailman/listinfo/cctbxbb > >>> >> >> >> >> >>>

---

...

...

...

...

...

> >>> >> >> >> >> >>> cctbxbb mailing list > >>> >> >> >> >> >>> [email protected] > >>> >> >> >> >> >>> http://phenix-online.org/mailman/listinfo/cctbxbb > >>> >> >> >> >> >> > >>> >> >> >> >> >> > >>> >> >> >> >> >>

---

...

...

...

...

...

> >>> >> >> >> >> >> cctbxbb mailing list > >>> >> >> >> >> >> [email protected] > >>> >> >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb > >>> >> >> >> >> > > >>> >> >> >> >> > > >>> >> >> >> >> > _______________________________________________ > >>> >> >> >> >> > cctbxbb mailing list > >>> >> >> >> >> > [email protected] > >>> >> >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb > >>> >> >> >> >> > >>> >> >> >> >> > >>> >> >> >> >> _______________________________________________ > >>> >> >> >> >> cctbxbb mailing list > >>> >> >> >> >> [email protected] > >>> >> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb > >>> >> >> >> > > >>> >> >> >> > _______________________________________________ > >>> >> >> >> > cctbxbb mailing list > >>> >> >> >> > [email protected] > >>> >> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb > >>> >> >> >> > >>> >> >> >> > >>> >> >> >> > >>> >> >> >> -- > >>> >> >> >> Gergely Katona, PhD > >>> >> >> >> Associate Professor > >>> >> >> >> Department of Chemistry and Molecular Biology, University of Gothenburg > >>> >> >> >> Box 462, 40530 Göteborg, Sweden > >>> >> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 > >>> >> >> >> Web: http://katonalab.eu, Email: [email protected] > >>> >> >> >> > >>> >> >> >> _______________________________________________ > >>> >> >> >> cctbxbb mailing list > >>> >> >> >> [email protected] > >>> >> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb > >>> >> >> > > >>> >> >> > _______________________________________________ > >>> >> >> > cctbxbb mailing list > >>> >> >> > [email protected] > >>> >> >> > http://phenix-online.org/mailman/listinfo/cctbxbb > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> -- > >>> >> >> Gergely Katona, PhD > >>> >> >> Associate Professor > >>> >> >> Department of Chemistry and Molecular Biology, University of Gothenburg > >>> >> >> Box 462, 40530 Göteborg, Sweden > >>> >> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 > >>> >> >> Web: http://katonalab.eu, Email: [email protected] > >>> >> >> > >>> >> >> _______________________________________________ > >>> >> >> cctbxbb mailing list > >>> >> >> [email protected] > >>> >> >> http://phenix-online.org/mailman/listinfo/cctbxbb > >>> >> > > >>> >> > _______________________________________________ > >>> >> > cctbxbb mailing list > >>> >> > [email protected] > >>> >> > http://phenix-online.org/mailman/listinfo/cctbxbb > >>> >> > >>> >> > >>> >> > >>> >> -- > >>> >> Gergely Katona, PhD > >>> >> Associate Professor > >>> >> Department of Chemistry and Molecular Biology, University of Gothenburg > >>> >> Box 462, 40530 Göteborg, Sweden > >>> >> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 > >>> >> Web: http://katonalab.eu, Email: [email protected] > >>> >> > >>> >> _______________________________________________ > >>> >> cctbxbb mailing list > >>> >> [email protected] > >>> >> http://phenix-online.org/mailman/listinfo/cctbxbb > >>> > > >>> > _______________________________________________ > >>> > cctbxbb mailing list > >>> > [email protected] > >>> > http://phenix-online.org/mailman/listinfo/cctbxbb > >>> > >>> > >>> > >>> -- > >>> Gergely Katona, PhD > >>> Associate Professor > >>> Department of Chemistry and Molecular Biology, University of Gothenburg > >>> Box 462, 40530 Göteborg, Sweden > >>> Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 > >>> Web: http://katonalab.eu, Email: [email protected] > >>> > >>> _______________________________________________ > >>> cctbxbb mailing list > >>> [email protected] > >>> http://phenix-online.org/mailman/listinfo/cctbxbb > > > > _______________________________________________ > > cctbxbb mailing list > > [email protected] > > http://phenix-online.org/mailman/listinfo/cctbxbb > > > > -- > Gergely Katona, Professor > Department of Chemistry and Molecular Biology, University of Gothenburg > Box 462, 40530 Göteborg, Sweden > Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 > Web: http://katonalab.eu, Email: [email protected]

-- Gergely Katona, Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

-- Gergely Katona, Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Gergely Katona, Professor Department of Chemistry and Molecular Biology, University of Gothenburg Box 462, 40530 Göteborg, Sweden Tel: +46-31-786-3959 / M: +46-70-912-3309 / Fax: +46-31-786-3910 Web: http://katonalab.eu, Email: [email protected]

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Hi Lance, Thank you for passing along the questionnaire. The more feedback we get, the better. If you could make examples of using the CCTBX available, that would be great! One common issue found in questionnaire is indeed the lack of examples. Thanks, Dorothee On Fri, Aug 16, 2019 at 12:06 PM Lance Westerhoff wrote:

Hi Dorothee-

More documentation sounds great. We’ve used CCTBX extensively so I’ll see what our group has to say concerning your request and I’ll pass along your questionnaire.

As part of your efforts, would you like examples of the use of CCTBX in production? We’ve developed a number CCTBX-based tools (even a stripped down, light weight QM/MM refinement engine we use for testing) and we’d be happy to opensource some of this code in our github account if you’d like. It doesn’t hurt us and if it helps folks see how CCTBX can be used, all the better. If you’re interested, let me see about opening it up. Thanks and have a great weekend!

-Lance ____________________ Lance M. Westerhoff, Ph.D. President and General Manager QuantumBio Inc.

WWW: http://www.quantumbioinc.com Email: [email protected]

Phone: 814-235-6908 Fax: 814-235-6909

This message and any attachments are solely for the intended recipient and should be considered confidential. If you are not the intended recipient, please immediately and permanently delete.

On Aug 15, 2019, at 3:04 PM, Dorothee Liebschner wrote:

Hi everyone,

During this year's ECM computing school, it came up that the CCTBX does not have a lot of documentation. This seems to be a major reason why people don't use it, even if they would like to employ the libraries. We would like to find out which parts of the CCTBX are the most interesting for users. We certainly cannot provide extensive documentation for the entire toolbox, but we can see what we can do once we know the main areas of interest. So if you were in the situation of wanting to use the CCTBX but decided to stay away from it because the documentation was non-existent or obscure, please let us know by filling out the questionnaire below (it's anonymous):

questionnaire CCTBX documentation https://forms.gle/9RawJFfHyEvhCVVP7

Best wishes,

Dorothee

-- Project Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org _______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

-- Project Scientist, Molecular Biophysics and Integrated Bioimaging Lawrence Berkeley National Laboratory 1 Cyclotron Road, M/S 33R0345 Berkeley, CA 94720 Tel: (510) 486-5709 Fax: (510) 486-5909 Web: https://phenix-online.org