Markus, We want option c. simtbx should be in the cctbx_project directory. No separate git repository. We need to double check we've gotten all the files in cctbx_project, then remove them from the separate repo. James, could you look into this? Nick Nicholas K. Sauter, Ph. D. Senior Scientist, Molecular Biophysics & Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720 (510) 486-5713 On Wed, May 3, 2017 at 7:49 AM, wrote:

Hi James,

It looks that you now have two separate places where simtbx code is kept. One is in the simtbx repository, where you fixed the sign earlier, and the other one is in the cctbx repository, where you just now fixed it again.

Simtbx repo: https://github.com/dials/simtbx

Simtbx folder in cctbx repo: https://github.com/cctbx/ cctbx_project/tree/master/simtbx

I guess suppose there are three options here:

a. move simtbx into the module directory, add simtbx to the bootstrap script so it is initialised correctly.

b. leave simtbx inside the cctbx_project directory and repository, but as its own git repository (using git submodule)

c. leave simtbx inside the cctbx_project directory, delete the separate git repository

NB: This issue is independent of resolving the dials-dependency issue.

-Markus

*From:* [email protected] [mailto:cctbxbb-bounces@ phenix-online.org] *On Behalf Of *James Holton *Sent:* 03 May 2017 15:40 *To:* Winter, Graeme (DLSLtd,RAL,LSCI) ; [email protected] *Subject:* Re: [cctbxbb] simtbx

Yesterday, Aaron and I were trying to set things up so that simtbx would be available to people who aren't James Holton. It's been checked in and working for over a month now, and only a few days ago I discovered that a fresh "bootstrap" has a broken simtbx, despite my "git status" telling me all my files are up to date. I asked Aaron and Nick for help with this.

I thought what we did was make cctbx.xfel depends on simtbx, since fitting simulations to stills is the whole reason for porting my stuff into python.

I set up simtbx with a constructor that takes a dxtbx detector as input, but does that make it depend on dials? I did not expect that.

Sorry, didn't mean to break anything! Would much appreciate any help on this.

-James

On 5/3/2017 1:41 AM, [email protected] wrote:

Hi Folks

We have noticed that simtbx, part of the core cctbx now, depends on dials

This means if you do not have a dials build your cctbx will be broken

This is probably not the way we want things?

I thought simtbx was a separate repo?

Cheerio Graeme

--

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

--

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Same questions here as Markus asked.. What is simtbx? First time hear about it. How cctbx can benefit from it? Pavel On 5/3/17 07:53, [email protected] wrote:

Hi Nick,

Sorry, outsider perspective here. I don’t know what simtbx does or is supposed to do.

What is the benefit of having code in a repository that, according to your earlier mail, is explicitly not used in the project?

Or more bluntly: We (and ccp4) distribute dials and, as part of it, the cctbx_project repository.

Why should we ship dead code?

-Markus

*From:*[email protected] [mailto:[email protected]] *On Behalf Of *Nicholas Sauter *Sent:* 03 May 2017 15:52 *To:* cctbx mailing list *Subject:* Re: [cctbxbb] simtbx

Markus,

We want option c. simtbx should be in the cctbx_project directory. No separate git repository. We need to double check we've gotten all the files in cctbx_project, then remove them from the separate repo. James, could you look into this?

Nick

Nicholas K. Sauter, Ph. D. Senior Scientist, Molecular Biophysics & Integrated Bioimaging Division

Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720 (510) 486-5713

On Wed, May 3, 2017 at 7:49 AM, mailto:[email protected]> wrote:

Hi James,

It looks that you now have two separate places where simtbx code is kept. One is in the simtbx repository, where you fixed the sign earlier, and the other one is in the cctbx repository, where you just now fixed it again.

Simtbx repo: https://github.com/dials/simtbx

Simtbx folder in cctbx repo: https://github.com/cctbx/cctbx_project/tree/master/simtbx

I guess suppose there are three options here:

a.move simtbx into the module directory, add simtbx to the bootstrap script so it is initialised correctly.

b.leave simtbx inside the cctbx_project directory and repository, but as its own git repository (using git submodule)

c.leave simtbx inside the cctbx_project directory, delete the separate git repository

NB: This issue is independent of resolving the dials-dependency issue.

-Markus

*From:*[email protected] mailto:[email protected] [mailto:[email protected] mailto:[email protected]] *On Behalf Of *James Holton *Sent:* 03 May 2017 15:40 *To:* Winter, Graeme (DLSLtd,RAL,LSCI) mailto:[email protected]>; [email protected] mailto:[email protected] *Subject:* Re: [cctbxbb] simtbx

Yesterday, Aaron and I were trying to set things up so that simtbx would be available to people who aren't James Holton. It's been checked in and working for over a month now, and only a few days ago I discovered that a fresh "bootstrap" has a broken simtbx, despite my "git status" telling me all my files are up to date. I asked Aaron and Nick for help with this.

I thought what we did was make cctbx.xfel depends on simtbx, since fitting simulations to stills is the whole reason for porting my stuff into python.

I set up simtbx with a constructor that takes a dxtbx detector as input, but does that make it depend on dials? I did not expect that.

Sorry, didn't mean to break anything! Would much appreciate any help on this.

-James

On 5/3/2017 1:41 AM, [email protected] mailto:[email protected] wrote:

Hi Folks

We have noticed that simtbx, part of the core cctbx now, depends on dials

This means if you do not have a dials build your cctbx will be broken

This is probably not the way we want things?

I thought simtbx was a separate repo?

Cheerio Graeme

--

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

--

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Nick Shower notwithstanding, I think what happened over here is this took out our build, the board went red and surprise was expressed and questions were raised. If I broke Phenix by bundling in xia2 into something, or some other project, I would expect the same level of questioning from LBL coming my way… But sounds like you can resolve this with people in the same building then let us know over here what the conclusion was. Best wishes Graeme From: [email protected] [mailto:[email protected]] On Behalf Of Nicholas Sauter Sent: 03 May 2017 15:59 To: cctbx mailing list Subject: Re: [cctbxbb] simtbx I'm stepping in the shower now. Pavel, I can answer about simtbx later. Never knew that we had to explain ourselves to everyone? What is mmtbx? What is scitbx? Nick Nicholas K. Sauter, Ph. D. Senior Scientist, Molecular Biophysics & Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720 (510) 486-5713 On Wed, May 3, 2017 at 7:57 AM, Pavel Afonine mailto:[email protected]> wrote: Same questions here as Markus asked.. What is simtbx? First time hear about it. How cctbx can benefit from it? Pavel On 5/3/17 07:53, [email protected]mailto:[email protected] wrote: Hi Nick, Sorry, outsider perspective here. I don’t know what simtbx does or is supposed to do. What is the benefit of having code in a repository that, according to your earlier mail, is explicitly not used in the project? Or more bluntly: We (and ccp4) distribute dials and, as part of it, the cctbx_project repository. Why should we ship dead code? -Markus From: [email protected]mailto:[email protected] [mailto:[email protected]] On Behalf Of Nicholas Sauter Sent: 03 May 2017 15:52 To: cctbx mailing list mailto:[email protected] Subject: Re: [cctbxbb] simtbx Markus, We want option c. simtbx should be in the cctbx_project directory. No separate git repository. We need to double check we've gotten all the files in cctbx_project, then remove them from the separate repo. James, could you look into this? Nick Nicholas K. Sauter, Ph. D. Senior Scientist, Molecular Biophysics & Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720 (510) 486-5713tel:(510)%20486-5713 On Wed, May 3, 2017 at 7:49 AM, mailto:[email protected]> wrote: Hi James, It looks that you now have two separate places where simtbx code is kept. One is in the simtbx repository, where you fixed the sign earlier, and the other one is in the cctbx repository, where you just now fixed it again. Simtbx repo: https://github.com/dials/simtbx Simtbx folder in cctbx repo: https://github.com/cctbx/cctbx_project/tree/master/simtbx I guess suppose there are three options here: a. move simtbx into the module directory, add simtbx to the bootstrap script so it is initialised correctly. b. leave simtbx inside the cctbx_project directory and repository, but as its own git repository (using git submodule) c. leave simtbx inside the cctbx_project directory, delete the separate git repository NB: This issue is independent of resolving the dials-dependency issue. -Markus From: [email protected]mailto:[email protected] [mailto:[email protected]mailto:[email protected]] On Behalf Of James Holton Sent: 03 May 2017 15:40 To: Winter, Graeme (DLSLtd,RAL,LSCI) mailto:[email protected]>; [email protected]mailto:[email protected] Subject: Re: [cctbxbb] simtbx Yesterday, Aaron and I were trying to set things up so that simtbx would be available to people who aren't James Holton. It's been checked in and working for over a month now, and only a few days ago I discovered that a fresh "bootstrap" has a broken simtbx, despite my "git status" telling me all my files are up to date. I asked Aaron and Nick for help with this. I thought what we did was make cctbx.xfel depends on simtbx, since fitting simulations to stills is the whole reason for porting my stuff into python. I set up simtbx with a constructor that takes a dxtbx detector as input, but does that make it depend on dials? I did not expect that. Sorry, didn't mean to break anything! Would much appreciate any help on this. -James On 5/3/2017 1:41 AM, [email protected]mailto:[email protected] wrote: Hi Folks We have noticed that simtbx, part of the core cctbx now, depends on dials This means if you do not have a dials build your cctbx will be broken This is probably not the way we want things? I thought simtbx was a separate repo? Cheerio Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

"simtbx" is the name we are giving to my diffraction image simulation codes. You may or may not of heard of their standalone versions as MLFSOM, fastBragg (now nanoBragg), nonBragg and nearBragg. Nick and I thought the capability of creating absolutely realistic diffraction data from a completely independent code base would be an excellent way to generate test data for cctbx, dials, phenix, and even third-party packages. There is also the idea of fitting a simulation directly to diffraction images, which I have had some success doing independently, but we hope this will work better under cctbx. Until recently, all this was standalone c code, with home pages here: http://bl831.als.lbl.gov/~jamesh/mlfsom/ http://bl831.als.lbl.gov/~jamesh/nanoBragg/ http://bl831.als.lbl.gov/~jamesh/nearBragg/ http://bl831.als.lbl.gov/~jamesh/nonBragg/ I have spent the bulk of my time for three months trying to wire these codes into the boost/Python framework of cctbx. I put docstrings into every single one of my objects, but since I am new to Python I don't know where people look for documentation. Again, sorry this broke everything. It's been in there since February and I was surprised to find that a new developer here couldn't access it. This is why I've been asking so many questions about tests I should perform before checking things in, but apparently it was all for naught. -James On 5/3/2017 7:57 AM, Pavel Afonine wrote:

Same questions here as Markus asked.. What is simtbx? First time hear about it. How cctbx can benefit from it? Pavel

On 5/3/17 07:53, [email protected] wrote:

...

Hi Nick,

Sorry, outsider perspective here. I don’t know what simtbx does or is supposed to do.

What is the benefit of having code in a repository that, according to your earlier mail, is explicitly not used in the project?

Or more bluntly: We (and ccp4) distribute dials and, as part of it, the cctbx_project repository.

Why should we ship dead code?

-Markus

*From:*[email protected] [mailto:[email protected]] *On Behalf Of *Nicholas Sauter *Sent:* 03 May 2017 15:52 *To:* cctbx mailing list *Subject:* Re: [cctbxbb] simtbx

Markus,

We want option c. simtbx should be in the cctbx_project directory. No separate git repository. We need to double check we've gotten all the files in cctbx_project, then remove them from the separate repo. James, could you look into this?

Nick

Nicholas K. Sauter, Ph. D. Senior Scientist, Molecular Biophysics & Integrated Bioimaging Division

Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720 (510) 486-5713

On Wed, May 3, 2017 at 7:49 AM, mailto:[email protected]> wrote:

Hi James,

It looks that you now have two separate places where simtbx code is kept. One is in the simtbx repository, where you fixed the sign earlier, and the other one is in the cctbx repository, where you just now fixed it again.

Simtbx repo: https://github.com/dials/simtbx

Simtbx folder in cctbx repo: https://github.com/cctbx/cctbx_project/tree/master/simtbx

I guess suppose there are three options here:

a.move simtbx into the module directory, add simtbx to the bootstrap script so it is initialised correctly.

b.leave simtbx inside the cctbx_project directory and repository, but as its own git repository (using git submodule)

c.leave simtbx inside the cctbx_project directory, delete the separate git repository

NB: This issue is independent of resolving the dials-dependency issue.

-Markus

*From:*[email protected] mailto:[email protected] [mailto:[email protected] mailto:[email protected]] *On Behalf Of *James Holton *Sent:* 03 May 2017 15:40 *To:* Winter, Graeme (DLSLtd,RAL,LSCI) mailto:[email protected]>; [email protected] mailto:[email protected] *Subject:* Re: [cctbxbb] simtbx

Yesterday, Aaron and I were trying to set things up so that simtbx would be available to people who aren't James Holton. It's been checked in and working for over a month now, and only a few days ago I discovered that a fresh "bootstrap" has a broken simtbx, despite my "git status" telling me all my files are up to date. I asked Aaron and Nick for help with this.

I thought what we did was make cctbx.xfel depends on simtbx, since fitting simulations to stills is the whole reason for porting my stuff into python.

I set up simtbx with a constructor that takes a dxtbx detector as input, but does that make it depend on dials? I did not expect that.

Sorry, didn't mean to break anything! Would much appreciate any help on this.

-James

On 5/3/2017 1:41 AM, [email protected] mailto:[email protected] wrote:

Hi Folks

We have noticed that simtbx, part of the core cctbx now, depends on dials

This means if you do not have a dials build your cctbx will be broken

This is probably not the way we want things?

I thought simtbx was a separate repo?

Cheerio Graeme

--

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

--

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Aaron Last command exceedingly useful thank you Will explore Think the dependency was indirect will verify Cheers Graeme On 3 May 2017, at 16:59, Aaron Brewster mailto:[email protected]> wrote: Hi, sorry I thought things were good based on my local tests. However, it looks like we need to fix some compile errors on different platforms. I've reverted the change for now, but I'll be putting it back soon (probably later today) after we've done some more testing. Regarding Graeme's original point: "simtbx, part of the core cctbx now, depends on dials. This means if you do not have a dials build your cctbx will be broken". This is untrue. A core cctbx build means you run bootstrap.py --builder=cctbx. You'll get 'cctbx', 'cbflib', 'scitbx', 'libtbx', 'iotbx', 'mmtbx', 'smtbx', 'dxtbx', 'gltbx', 'wxtbx'. This does means several packages in cctbx_project would not be configured. If instead you ran python /path/to/libtbx/configure.py cctbx in an empty folder, you would get a similar list of packages. About dependencies, there is a circular dependency between xfel and dials. xfel depends on dials and dials depends on xfel. It's been there for a while. The root cause is dials.stills_process depends on xfel code, specifically xfel.mono_simulation. It would be nice to break this cycle, but it would involve shuffling a lot of code around. Instead, xfel is an optional dependency of dials and dials is a required dependency of xfel. simtbx will be an optional dependency of xfel. If you'd like to see the dependency chains, use libtbx.show_module_dependencies xfel libtbx.show_module_dependencies dials libtbx.show_module_dependencies cctbx -Aaron On Wed, May 3, 2017 at 8:28 AM, James Holton mailto:[email protected]> wrote: "simtbx" is the name we are giving to my diffraction image simulation codes. You may or may not of heard of their standalone versions as MLFSOM, fastBragg (now nanoBragg), nonBragg and nearBragg. Nick and I thought the capability of creating absolutely realistic diffraction data from a completely independent code base would be an excellent way to generate test data for cctbx, dials, phenix, and even third-party packages. There is also the idea of fitting a simulation directly to diffraction images, which I have had some success doing independently, but we hope this will work better under cctbx. Until recently, all this was standalone c code, with home pages here: http://bl831.als.lbl.gov/~jamesh/mlfsom/ http://bl831.als.lbl.gov/~jamesh/nanoBragg/ http://bl831.als.lbl.gov/~jamesh/nearBragg/ http://bl831.als.lbl.gov/~jamesh/nonBragg/ I have spent the bulk of my time for three months trying to wire these codes into the boost/Python framework of cctbx. I put docstrings into every single one of my objects, but since I am new to Python I don't know where people look for documentation. Again, sorry this broke everything. It's been in there since February and I was surprised to find that a new developer here couldn't access it. This is why I've been asking so many questions about tests I should perform before checking things in, but apparently it was all for naught. -James On 5/3/2017 7:57 AM, Pavel Afonine wrote: Same questions here as Markus asked.. What is simtbx? First time hear about it. How cctbx can benefit from it? Pavel On 5/3/17 07:53, [email protected]mailto:[email protected] wrote: Hi Nick, Sorry, outsider perspective here. I don’t know what simtbx does or is supposed to do. What is the benefit of having code in a repository that, according to your earlier mail, is explicitly not used in the project? Or more bluntly: We (and ccp4) distribute dials and, as part of it, the cctbx_project repository. Why should we ship dead code? -Markus From: [email protected]mailto:[email protected] [mailto:[email protected]] On Behalf Of Nicholas Sauter Sent: 03 May 2017 15:52 To: cctbx mailing list mailto:[email protected] Subject: Re: [cctbxbb] simtbx Markus, We want option c. simtbx should be in the cctbx_project directory. No separate git repository. We need to double check we've gotten all the files in cctbx_project, then remove them from the separate repo. James, could you look into this? Nick Nicholas K. Sauter, Ph. D. Senior Scientist, Molecular Biophysics & Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720 (510) 486-5713tel:(510)%20486-5713 On Wed, May 3, 2017 at 7:49 AM, mailto:[email protected]> wrote: Hi James, It looks that you now have two separate places where simtbx code is kept. One is in the simtbx repository, where you fixed the sign earlier, and the other one is in the cctbx repository, where you just now fixed it again. Simtbx repo: https://github.com/dials/simtbx Simtbx folder in cctbx repo: https://github.com/cctbx/cctbx_project/tree/master/simtbx I guess suppose there are three options here: a. move simtbx into the module directory, add simtbx to the bootstrap script so it is initialised correctly. b. leave simtbx inside the cctbx_project directory and repository, but as its own git repository (using git submodule) c. leave simtbx inside the cctbx_project directory, delete the separate git repository NB: This issue is independent of resolving the dials-dependency issue. -Markus From: [email protected]mailto:[email protected] [mailto:[email protected]mailto:[email protected]] On Behalf Of James Holton Sent: 03 May 2017 15:40 To: Winter, Graeme (DLSLtd,RAL,LSCI) mailto:[email protected]>; [email protected]mailto:[email protected] Subject: Re: [cctbxbb] simtbx Yesterday, Aaron and I were trying to set things up so that simtbx would be available to people who aren't James Holton. It's been checked in and working for over a month now, and only a few days ago I discovered that a fresh "bootstrap" has a broken simtbx, despite my "git status" telling me all my files are up to date. I asked Aaron and Nick for help with this. I thought what we did was make cctbx.xfel depends on simtbx, since fitting simulations to stills is the whole reason for porting my stuff into python. I set up simtbx with a constructor that takes a dxtbx detector as input, but does that make it depend on dials? I did not expect that. Sorry, didn't mean to break anything! Would much appreciate any help on this. -James On 5/3/2017 1:41 AM, [email protected]mailto:[email protected] wrote: Hi Folks We have noticed that simtbx, part of the core cctbx now, depends on dials This means if you do not have a dials build your cctbx will be broken This is probably not the way we want things? I thought simtbx was a separate repo? Cheerio Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

Yes, that's what it does. If you look at tst_nanoBragg_minimal.py you will see some code for running it. Currently, I have the call to "sfall" commented out for testing purposes, but you can uncomment it to make structure factors from a PDB as shown. Twinning is fairly straightforward. You can cheat by "pre-twinning" the data (square the Fs, re-index, weighted sum, then square root). However, what I generally do for twinning is generate two diffraction images with different crystal orientations and then add them together. My plan is to set this up so you do the image manipulations as flex arrays. This would also allow background to be done separately. Then you would use "noisify" to add the final noise and generate the image files using the new dxtbx output functions. Working on getting those "pythonized" today. In the meantime, it should be a simple matter to generate noiseless test images. Or, if you're in a hurry, I can probably tell you how to do something using the standalone code. Like this twinning example: http://bl831.als.lbl.gov/~jamesh/challenge/twin/ Does that help? -James On 6/5/2017 4:14 AM, R.D. Oeffner wrote:

Dear James,

I'm curious about simtbx. Does it allow me to supply a set of atomic coordinates in an ASU and then spit out corresponding diffraction images? I suppose it can't generate twinned data sets.

Many thanks,

Rob

On 03/05/2017 16:28, James Holton wrote:

...

"simtbx" is the name we are giving to my diffraction image simulation codes. You may or may not of heard of their standalone versions as MLFSOM, fastBragg (now nanoBragg), nonBragg and nearBragg. Nick and I thought the capability of creating absolutely realistic diffraction data from a completely independent code base would be an excellent way to generate test data for cctbx, dials, phenix, and even third-party packages. There is also the idea of fitting a simulation directly to diffraction images, which I have had some success doing independently, but we hope this will work better under cctbx. Until recently, all this was standalone c code, with home pages here:

http://bl831.als.lbl.gov/~jamesh/mlfsom/ http://bl831.als.lbl.gov/~jamesh/nanoBragg/ http://bl831.als.lbl.gov/~jamesh/nearBragg/ http://bl831.als.lbl.gov/~jamesh/nonBragg/

I have spent the bulk of my time for three months trying to wire these codes into the boost/Python framework of cctbx. I put docstrings into every single one of my objects, but since I am new to Python I don't know where people look for documentation.

Again, sorry this broke everything. It's been in there since February and I was surprised to find that a new developer here couldn't access it. This is why I've been asking so many questions about tests I should perform before checking things in, but apparently it was all for naught.

-James

On 5/3/2017 7:57 AM, Pavel Afonine wrote:

...

Same questions here as Markus asked.. What is simtbx? First time hear about it. How cctbx can benefit from it? Pavel

On 5/3/17 07:53, [email protected] wrote:

...

Hi Nick,

Sorry, outsider perspective here. I don't know what simtbx does or is supposed to do.

What is the benefit of having code in a repository that, according to your earlier mail, is explicitly not used in the project?

Or more bluntly: We (and ccp4) distribute dials and, as part of it, the cctbx_project repository.

Why should we ship dead code?

-Markus

*From:*[email protected] [mailto:[email protected]] *On Behalf Of *Nicholas Sauter *Sent:* 03 May 2017 15:52 *To:* cctbx mailing list *Subject:* Re: [cctbxbb] simtbx

Markus,

We want option c. simtbx should be in the cctbx_project directory. No separate git repository. We need to double check we've gotten all the files in cctbx_project, then remove them from the separate repo. James, could you look into this?

Nick

Nicholas K. Sauter, Ph. D. Senior Scientist, Molecular Biophysics & Integrated Bioimaging Division

Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720 (510) 486-5713

On Wed, May 3, 2017 at 7:49 AM, mailto:[email protected]> wrote:

Hi James,

It looks that you now have two separate places where simtbx code is kept. One is in the simtbx repository, where you fixed the sign earlier, and the other one is in the cctbx repository, where you just now fixed it again.

Simtbx repo: https://github.com/dials/simtbx

Simtbx folder in cctbx repo: https://github.com/cctbx/cctbx_project/tree/master/simtbx https://github.com/cctbx/cctbx_project/tree/master/simtbx

I guess suppose there are three options here:

a.move simtbx into the module directory, add simtbx to the bootstrap script so it is initialised correctly.

b.leave simtbx inside the cctbx_project directory and repository, but as its own git repository (using git submodule)

c.leave simtbx inside the cctbx_project directory, delete the separate git repository

NB: This issue is independent of resolving the dials-dependency issue.

-Markus

*From:*[email protected] mailto:[email protected] [mailto:[email protected] mailto:[email protected]] *On Behalf Of *James Holton *Sent:* 03 May 2017 15:40 *To:* Winter, Graeme (DLSLtd,RAL,LSCI) mailto:[email protected]>; [email protected] mailto:[email protected] *Subject:* Re: [cctbxbb] simtbx

Yesterday, Aaron and I were trying to set things up so that simtbx would be available to people who aren't James Holton. It's been checked in and working for over a month now, and only a few days ago I discovered that a fresh "bootstrap" has a broken simtbx, despite my "git status" telling me all my files are up to date. I asked Aaron and Nick for help with this.

I thought what we did was make cctbx.xfel depends on simtbx, since fitting simulations to stills is the whole reason for porting my stuff into python.

I set up simtbx with a constructor that takes a dxtbx detector as input, but does that make it depend on dials? I did not expect that.

Sorry, didn't mean to break anything! Would much appreciate any help on this.

-James

On 5/3/2017 1:41 AM, [email protected] mailto:[email protected] wrote:

Hi Folks

We have noticed that simtbx, part of the core cctbx now, depends on dials

This means if you do not have a dials build your cctbx will be broken

This is probably not the way we want things?

I thought simtbx was a separate repo?

Cheerio Graeme

--

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

--

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________ cctbxbb mailing list [email protected] mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

_______________________________________________ cctbxbb mailing list [email protected] http://phenix-online.org/mailman/listinfo/cctbxbb

HI Markus I think over this side of the project we are more keen on splitting things up i.e. xia2, dials, i19 modules which could all be considered as “one thing” Looks like the DIALS-West side approach is to have all things under one roof i.e. [gw56@ws133 cctbx_project]$ ls boost_adaptbx cma_es dox.sphinx iota prime site.py tbxx cbflib_adaptbx cootbx dxtbx iotbx README.md site.pyc ucif cctbx crys3d fable libtbx rstbx smtbx wxtbx chiltbx cudatbx fftw3tbx mmtbx scitbx sphinx xfel clipper_adaptbx dox gltbx omptbx simtbx spotfinder [gw56@ws133 cctbx_project]$ cd ../ [gw56@ws133 modules]$ ls allsvn ccp4io dials_scratch phenix_regression annlib ccp4io_adaptbx dlstbx scons annlib_adaptbx cctbx_project eigen tntbx bestmx clipper gui_resources xia2 boost cxi_user i19 xia2_regression bpcx_regression dials labelit cbflib dials_regression labelit_regression so more projects in cctbx_project than in my modules directory… To CCP4 / DIALS this could be “dead code” but to cctbx.xfel crew this is bread and butter stuff That said, I can see some argument to bundling fewer things in cctbx_project… Best wishes Graeme From: [email protected] [mailto:[email protected]] On Behalf Of [email protected] Sent: 03 May 2017 15:54 To: [email protected] Subject: Re: [cctbxbb] simtbx Hi Nick, Sorry, outsider perspective here. I don’t know what simtbx does or is supposed to do. What is the benefit of having code in a repository that, according to your earlier mail, is explicitly not used in the project? Or more bluntly: We (and ccp4) distribute dials and, as part of it, the cctbx_project repository. Why should we ship dead code? -Markus From: [email protected]mailto:[email protected] [mailto:[email protected]] On Behalf Of Nicholas Sauter Sent: 03 May 2017 15:52 To: cctbx mailing list mailto:[email protected]> Subject: Re: [cctbxbb] simtbx Markus, We want option c. simtbx should be in the cctbx_project directory. No separate git repository. We need to double check we've gotten all the files in cctbx_project, then remove them from the separate repo. James, could you look into this? Nick Nicholas K. Sauter, Ph. D. Senior Scientist, Molecular Biophysics & Integrated Bioimaging Division Lawrence Berkeley National Laboratory 1 Cyclotron Rd., Bldg. 33R0345 Berkeley, CA 94720 (510) 486-5713 On Wed, May 3, 2017 at 7:49 AM, mailto:[email protected]> wrote: Hi James, It looks that you now have two separate places where simtbx code is kept. One is in the simtbx repository, where you fixed the sign earlier, and the other one is in the cctbx repository, where you just now fixed it again. Simtbx repo: https://github.com/dials/simtbx Simtbx folder in cctbx repo: https://github.com/cctbx/cctbx_project/tree/master/simtbx I guess suppose there are three options here: a. move simtbx into the module directory, add simtbx to the bootstrap script so it is initialised correctly. b. leave simtbx inside the cctbx_project directory and repository, but as its own git repository (using git submodule) c. leave simtbx inside the cctbx_project directory, delete the separate git repository NB: This issue is independent of resolving the dials-dependency issue. -Markus From: [email protected]mailto:[email protected] [mailto:[email protected]mailto:[email protected]] On Behalf Of James Holton Sent: 03 May 2017 15:40 To: Winter, Graeme (DLSLtd,RAL,LSCI) mailto:[email protected]>; [email protected]mailto:[email protected] Subject: Re: [cctbxbb] simtbx Yesterday, Aaron and I were trying to set things up so that simtbx would be available to people who aren't James Holton. It's been checked in and working for over a month now, and only a few days ago I discovered that a fresh "bootstrap" has a broken simtbx, despite my "git status" telling me all my files are up to date. I asked Aaron and Nick for help with this. I thought what we did was make cctbx.xfel depends on simtbx, since fitting simulations to stills is the whole reason for porting my stuff into python. I set up simtbx with a constructor that takes a dxtbx detector as input, but does that make it depend on dials? I did not expect that. Sorry, didn't mean to break anything! Would much appreciate any help on this. -James On 5/3/2017 1:41 AM, [email protected]mailto:[email protected] wrote: Hi Folks We have noticed that simtbx, part of the core cctbx now, depends on dials This means if you do not have a dials build your cctbx will be broken This is probably not the way we want things? I thought simtbx was a separate repo? Cheerio Graeme -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom _______________________________________________ cctbxbb mailing list [email protected]mailto:[email protected] http://phenix-online.org/mailman/listinfo/cctbxbb