Hi Graeme,
Here is a summary of the discussions across various email threads and meetings.
The plan is to get conda ready in October/November. With the delay from the Phenix release, there might be a delay for this work, but I'll have to see.
This will be a drop-in replacement for the "base" directory containing the dependencies. The usage of conda (or the current
bootstrap.py/install_base_packages.py approach) will remain optional in the sense that anyone can always just supply the dependencies in any other way and CCTBX can work with those dependencies. This was discussed in the email thread with David Waterman about this change not affecting how CCP4 would build CCTBX. This is also important for developers that work on systems that have more specialized environments (e.g. large HPC systems).
There should be a list of dependency versions that basically define the API. An outdated list is at
https://github.com/cctbx/conda_build/blob/master/cctbx_dependencies/meta.yaml. For example, we support Python 2.7, so any version of Python 2.7.x should be supported. This is so that CCTBX can be more friendly towards existing conda environments. There will be a separate list (location TBD, probably somewhere in libtbx/auto_build) that has an explicit set of versions. This would be for creating installers and clean developer environments. An example is attached that can be used to create an conda environment compatible with building CCTBX ("conda create -n <environment name> --file env.txt). The procrunner package is missing, but you can install it with pip once you're in the environment.
As for dispatchers, I described how to run commands in issue #85 (
https://github.com/cctbx/cctbx_project/issues/85). There is no need to be inside a conda environment to run commands, but there may be additional logic added to determine if the user is inside a conda environment.
Installers will probably use the conda constructor package (
https://github.com/conda/constructor). It has the benefit of uniformly building the macOS, linux, and Windows installers. The command-line installer would behave like the miniconda installer. I have not tested the .pkg and .exe options for macOS and Windows installers, respectively, yet.
Some features that were brought up, but will not be part of this initial migration is,
1) "conda install cctbx" - Some directory rearrangement will probably be required and we would need to decide on what actually gets installed from the cctbx_project umbrella directory.
2) pip editable-mode - Markus brought this up at the GRC. There is a conda equivalent ("conda develop"), but since conda is not a requirement, this is probably not generally applicable. Also, CCTBX is used in non-Python programs as header files and linked as shared libraries, and it looks like this mode only really works with Python modules.
The current bootstrap.py approach will be kept since it provides a simple way for developers to build their projects. The "base" step will either be replaced by building the dependencies via conda instead of install_base_packages.py, or a different name can be used. Also, a step for updating dependencies could be added. The goal is for developers to not have to change there workflow if they are not interested in using conda explicitly.
Research Scientist, Molecular Biophysics and Integrated Bioimaging
Lawrence Berkeley National Laboratory
1 Cyclotron Road, M/S 33R0345
Berkeley, CA 94720
Tel: (510) 486-5709
Fax: (510) 486-5909