Hi Luc,
All the cctbx tests now appear to pass, however I am getting a couple of floating point exceptions running the dials tests:
$ libtbx.python $(libtbx.find_in_repositories dials/test/algorithms/refinement/tst_multi_experiment_refinement.py)
show_stack(1): /Users/rjgildea/software/cctbx/modules/dials/algorithms/refinement/outlier_detection/mcd.py(54) __init__
show_stack(2): /Users/rjgildea/software/cctbx/modules/dials/algorithms/refinement/outlier_detection/outlier_base.py(296) from_parameters_and_colnames
show_stack(3): /Users/rjgildea/software/cctbx/modules/dials/algorithms/refinement/refiner.py(1194) config_refman
show_stack(4): /Users/rjgildea/software/cctbx/modules/dials/algorithms/refinement/refiner.py(461) _build_components
show_stack(5): /Users/rjgildea/software/cctbx/modules/dials/algorithms/refinement/refiner.py(400) from_parameters_data_experiments
show_stack(6): /Users/rjgildea/software/cctbx/modules/dials/test/algorithms/refinement/tst_multi_experiment_refinement.py(269) <module>
libc backtrace (41 frames, most recent call last):
42 ??? 0x0000000000000003 0x0 + 3
41 Python 0x0000000100000d71 Python + 3441
40 Python 0x0000000100000e58 Python + 3672
39 Python 0x00000001000eb6aa Py_Main + 3178
38 Python 0x00000001000d424a PyRun_SimpleFileExFlags + 698
37 Python 0x00000001000d4714 PyRun_FileExFlags + 164
36 Python 0x00000001000aba26 PyEval_EvalCode + 54
35 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690
34 Python 0x00000001000af860 PyEval_EvalFrameEx + 13472
33 Python 0x00000001000b3486 fast_function + 118
32 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690
31 Python 0x00000001000af860 PyEval_EvalFrameEx + 13472
30 Python 0x00000001000b3486 fast_function + 118
29 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690
28 Python 0x00000001000af860 PyEval_EvalFrameEx + 13472
27 Python 0x00000001000b3486 fast_function + 118
26 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690
25 Python 0x00000001000af860 PyEval_EvalFrameEx + 13472
24 Python 0x00000001000b3486 fast_function + 118
23 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690
22 Python 0x00000001000afab2 PyEval_EvalFrameEx + 14066
21 Python 0x0000000100011223 PyObject_Call + 99
20 Python 0x00000001000684ab type_call + 347
19 Python 0x000000010006dfef slot_tp_init + 175
18 Python 0x0000000100011223 PyObject_Call + 99
17 Python 0x000000010001e138 instancemethod_call + 232
16 Python 0x0000000100011223 PyObject_Call + 99
15 Python 0x0000000100036b4c function_call + 364
14 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690
13 Python 0x00000001000af11d PyEval_EvalFrameEx + 11613
12 Python 0x0000000100011223 PyObject_Call + 99
11 libboost_python.dylib 0x0000000102492663 boost::python::objects::function_call(_object*, _object*, _object*) + 83
10 libboost_python.dylib 0x0000000102496e71 boost::python::handle_exception_impl(boost::function0<void>) + 81
9 libboost_python.dylib 0x00000001024929ca boost::detail::function::void_function_ref_invoker0::invoke(boost::detail::function::function_buffer&) + 26
8 libboost_python.dylib 0x000000010249093a boost::python::objects::function::call(_object*, _object*) const + 874
7 dials_refinement_helpers_ext.so 0x000000010649a093 boost::python::detail::caller_arity<2u>::impl >::operator()(_object*, _object*) + 179
6 dials_refinement_helpers_ext.so 0x000000010648312d dials::refinement::qchisq(double, double) + 189
5 dials_refinement_helpers_ext.so 0x000000010648dbe6 double boost::math::detail::gamma_p_inv_imp >(double, double, boost::math::policies::policy const&) + 374
4 Python 0x000000010004c160 PyDict_DelItemString + 64
3 libsystem_platform.dylib 0x00007fff89b59eaa _sigtramp + 26
2 boost_python_meta_ext.so 0x000000010244b3f0 initboost_python_meta_ext + 0
Floating-point error (Python and libc call stacks above)
This crash may be due to a problem in any imported
Python module, including modules which are not part
of the cctbx project. To disable the traps leading
to this message, define these environment variables
(e.g. assign the value 1):
BOOST_ADAPTBX_FPE_DEFAULT
BOOST_ADAPTBX_SIGNALS_DEFAULT
This will NOT solve the problem, just mask it, but
may allow you to proceed in case it is not critical.
Richards-MacBook-Pro:build rjgildea$
Richards-MacBook-Pro:build rjgildea$ libtbx.python $(libtbx.find_in_repositories dials/test/algorithms/refinement/tst_multi_experiment_refinement.py)
show_stack(1): /Users/rjgildea/software/cctbx/modules/dials/algorithms/refinement/outlier_detection/mcd.py(54) __init__
show_stack(2): /Users/rjgildea/software/cctbx/modules/dials/algorithms/refinement/outlier_detection/outlier_base.py(296) from_parameters_and_colnames
show_stack(3): /Users/rjgildea/software/cctbx/modules/dials/algorithms/refinement/refiner.py(1194) config_refman
show_stack(4): /Users/rjgildea/software/cctbx/modules/dials/algorithms/refinement/refiner.py(461) _build_components
show_stack(5): /Users/rjgildea/software/cctbx/modules/dials/algorithms/refinement/refiner.py(400) from_parameters_data_experiments
show_stack(6): /Users/rjgildea/software/cctbx/modules/dials/test/algorithms/refinement/tst_multi_experiment_refinement.py(269) <module>
libc backtrace (41 frames, most recent call last):
42 ??? 0x0000000000000003 0x0 + 3
41 Python 0x0000000100000d71 Python + 3441
40 Python 0x0000000100000e58 Python + 3672
39 Python 0x00000001000eb6aa Py_Main + 3178
38 Python 0x00000001000d424a PyRun_SimpleFileExFlags + 698
37 Python 0x00000001000d4714 PyRun_FileExFlags + 164
36 Python 0x00000001000aba26 PyEval_EvalCode + 54
35 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690
34 Python 0x00000001000af860 PyEval_EvalFrameEx + 13472
33 Python 0x00000001000b3486 fast_function + 118
32 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690
31 Python 0x00000001000af860 PyEval_EvalFrameEx + 13472
30 Python 0x00000001000b3486 fast_function + 118
29 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690
28 Python 0x00000001000af860 PyEval_EvalFrameEx + 13472
27 Python 0x00000001000b3486 fast_function + 118
26 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690
25 Python 0x00000001000af860 PyEval_EvalFrameEx + 13472
24 Python 0x00000001000b3486 fast_function + 118
23 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690
22 Python 0x00000001000afab2 PyEval_EvalFrameEx + 14066
21 Python 0x0000000100011223 PyObject_Call + 99
20 Python 0x00000001000684ab type_call + 347
19 Python 0x000000010006dfef slot_tp_init + 175
18 Python 0x0000000100011223 PyObject_Call + 99
17 Python 0x000000010001e138 instancemethod_call + 232
16 Python 0x0000000100011223 PyObject_Call + 99
15 Python 0x0000000100036b4c function_call + 364
14 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690
13 Python 0x00000001000af11d PyEval_EvalFrameEx + 11613
12 Python 0x0000000100011223 PyObject_Call + 99
11 libboost_python.dylib 0x0000000101e0f663 boost::python::objects::function_call(_object*, _object*, _object*) + 83
10 libboost_python.dylib 0x0000000101e13e71 boost::python::handle_exception_impl(boost::function0<void>) + 81
9 libboost_python.dylib 0x0000000101e0f9ca boost::detail::function::void_function_ref_invoker0::invoke(boost::detail::function::function_buffer&) + 26
8 libboost_python.dylib 0x0000000101e0d93a boost::python::objects::function::call(_object*, _object*) const + 874
7 dials_refinement_helpers_ext.so 0x00000001087a8093 boost::python::detail::caller_arity<2u>::impl >::operator()(_object*, _object*) + 179
6 dials_refinement_helpers_ext.so 0x000000010879112d dials::refinement::qchisq(double, double) + 189
5 dials_refinement_helpers_ext.so 0x000000010879bbe6 double boost::math::detail::gamma_p_inv_imp >(double, double, boost::math::policies::policy const&) + 374
4 Python 0x000000010004c160 PyDict_DelItemString + 64
3 libsystem_platform.dylib 0x00007fff89b59eaa _sigtramp + 26
2 boost_python_meta_ext.so 0x0000000101c803f0 initboost_python_meta_ext + 0
Floating-point error (Python and libc call stacks above)
This crash may be due to a problem in any imported
Python module, including modules which are not part
of the cctbx project. To disable the traps leading
to this message, define these environment variables
(e.g. assign the value 1):
BOOST_ADAPTBX_FPE_DEFAULT
BOOST_ADAPTBX_SIGNALS_DEFAULT
This will NOT solve the problem, just mask it, but
may allow you to proceed in case it is not critical.
Richards-MacBook-Pro:build rjgildea$ libtbx.python $(libtbx.find_in_repositories dials/test/algorithms/statistics/tst_fast_mcd.py)
OK
show_stack(1): /Users/rjgildea/software/cctbx/modules/dials/algorithms/statistics/fast_mcd.py(165) __init__
show_stack(2): /Users/rjgildea/software/cctbx/modules/dials/test/algorithms/statistics/tst_fast_mcd.py(146) test_fast_mcd_small
show_stack(3): /Users/rjgildea/software/cctbx/modules/dials/test/algorithms/statistics/tst_fast_mcd.py(237) <module>
libc backtrace (33 frames, most recent call last):
34 ??? 0x0000000000000003 0x0 + 3
33 Python 0x0000000100000d71 Python + 3441
32 Python 0x0000000100000e58 Python + 3672
31 Python 0x00000001000eb6aa Py_Main + 3178
30 Python 0x00000001000d424a PyRun_SimpleFileExFlags + 698
29 Python 0x00000001000d4714 PyRun_FileExFlags + 164
28 Python 0x00000001000aba26 PyEval_EvalCode + 54
27 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690
26 Python 0x00000001000af860 PyEval_EvalFrameEx + 13472
25 Python 0x00000001000b3486 fast_function + 118
24 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690
23 Python 0x00000001000af11d PyEval_EvalFrameEx + 11613
22 Python 0x0000000100011223 PyObject_Call + 99
21 Python 0x00000001000684ab type_call + 347
20 Python 0x000000010006dfef slot_tp_init + 175
19 Python 0x0000000100011223 PyObject_Call + 99
18 Python 0x000000010001e138 instancemethod_call + 232
17 Python 0x0000000100011223 PyObject_Call + 99
16 Python 0x0000000100036b4c function_call + 364
15 Python 0x00000001000ac0ca PyEval_EvalCodeEx + 1690
14 Python 0x00000001000af11d PyEval_EvalFrameEx + 11613
13 Python 0x0000000100011223 PyObject_Call + 99
12 libboost_python.dylib 0x000000010270f663 boost::python::objects::function_call(_object*, _object*, _object*) + 83
11 libboost_python.dylib 0x0000000102713e71 boost::python::handle_exception_impl(boost::function0<void>) + 81
10 libboost_python.dylib 0x000000010270f9ca boost::detail::function::void_function_ref_invoker0::invoke(boost::detail::function::function_buffer&) + 26
9 libboost_python.dylib 0x000000010270d93a boost::python::objects::function::call(_object*, _object*) const + 874
8 dials_refinement_helpers_ext.so 0x00000001080d4093 boost::python::detail::caller_arity<2u>::impl >::operator()(_object*, _object*) + 179
7 dials_refinement_helpers_ext.so 0x00000001080bd689 dials::refinement::mcd_consistency(double, double) + 89
6 dials_refinement_helpers_ext.so 0x00000001080bd12d dials::refinement::qchisq(double, double) + 189
5 dials_refinement_helpers_ext.so 0x00000001080c7be6 double boost::math::detail::gamma_p_inv_imp >(double, double, boost::math::policies::policy const&) + 374
4 ??? 0x00000001007ba2b0 0x0 + 4303069872
3 libsystem_platform.dylib 0x00007fff89b59eaa _sigtramp + 26
2 boost_python_meta_ext.so 0x00000001025813f0 initboost_python_meta_ext + 0
Floating-point error (Python and libc call stacks above)
This crash may be due to a problem in any imported
Python module, including modules which are not part
of the cctbx project. To disable the traps leading
to this message, define these environment variables
(e.g. assign the value 1):
BOOST_ADAPTBX_FPE_DEFAULT
BOOST_ADAPTBX_SIGNALS_DEFAULT
This will NOT solve the problem, just mask it, but
may allow you to proceed in case it is not critical.
Tracking this down further, I think the problem comes from the function qchisq in:
https://github.com/dials/dials/blob/master/algorithms/refinement/outlier_det...
Specifically, the line that is causing the problem is line 22, the call to quantile in the return statement:
// chi^2 distribution with df d.o.f:
boost::math::chi_squared_distribution<double> chisq(df);
// Find x s.t. F(x; df) = p where F is the CDF of the Chi^2 dist with df d.o.f
return quantile(chisq, p);
Adding debug print statements shows it fails with the following values:
df: 3
p: 0.52
I will try updating to boost 1.60 to see if this fixes things.
Cheers,
Richard
Dr Richard Gildea
Data Analysis Scientist
Tel: +441235 77 8078
Diamond Light Source Ltd.
Diamond House
Harwell Science & Innovation Campus
Didcot
Oxfordshire
OX11 0DE
________________________________________
From: [email protected] [[email protected]] on behalf of Luc Bourhis [[email protected]]
Sent: 07 April 2016 11:05
To: cctbx mailing list
Subject: Re: [cctbxbb] Boost broken on mac?
SCons caches the result of TryRun in .sconf_temp and it gets stale sometimes. In your case, SCons did not track the fact that clang had changed I guess.
I did commit my fix to libtbx/SConscript. Note that it does not fix the error in xray/conversions.h, weirdly enough. So I left my hack in there.
So now, I’ve got some work cut for myself for the next rainy day: come with a reduce test case that I can submit to the clang bugzilla. The problem in xray/conversions.h is clearly the most promising to start with. But still, a lot of work.
On 7 Apr 2016, at 11:43, [email protected] wrote:
Strangely for my development build, the build system appears convinced that it is using clang 6.1.0...
Editing around line 784 of libtbx/SConscript, where clang_version is determined, to read:
from subprocess import call
print call(["which", "clang"])
print call(["clang", "--version"])
flag, output = conf.TryRun("\n".join(test_code), extension='.cpp')
conf.Finish()
assert flag
output = output.replace(":,", ":None,")
print output
I get the following output:
/usr/bin/clang
0
Apple LLVM version 7.3.0 (clang-703.0.29)
Target: x86_64-apple-darwin15.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
0
{"llvm":1, "clang":1, "clang_major":6, "clang_minor":1, "clang_patchlevel":0, "GNUC":4, "GNUC_MINOR":2, "GNUC_PATCHLEVEL":1, "clang_version": "6.1.0 (clang-602.0.53)", "VERSION": "4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)", }
On MacOS, using clang 6.1.0
If I wipe the build directory (or rather, renamed so not wiped completely) then it is now happy that it is using clang 7.3.0:
/usr/bin/clang
0
Apple LLVM version 7.3.0 (clang-703.0.29)
Target: x86_64-apple-darwin15.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
0
{"llvm":1, "clang":1, "clang_major":7, "clang_minor":3, "clang_patchlevel":0, "GNUC":4, "GNUC_MINOR":2, "GNUC_PATCHLEVEL":1, "clang_version": "7.3.0 (clang-703.0.29)", "VERSION": "4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.29)", }
On MacOS, using clang 7.3.0
Any idea what is going on here? Why does it think it is somehow using an older version of the clang to what is actually installed?!
Cheers,
Richard
Dr Richard Gildea
Data Analysis Scientist
Tel: +441235 77 8078
Diamond Light Source Ltd.
Diamond House
Harwell Science & Innovation Campus
Didcot
Oxfordshire
OX11 0DE
________________________________
From: [email protected] [[email protected]] on behalf of Luc Bourhis [[email protected]]
Sent: 07 April 2016 10:23
To: cctbx mailing list
Subject: Re: [cctbxbb] Boost broken on mac?
Could you confirm whether the following two tests fail for you after your latest change?
libtbx.python "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/cctbx/maptbx/boost_python/tst_maptbx.py" [FAIL]
[…]
libtbx.python "/Users/rjgildea/tmp/tst_bootstrap/tmp/modules/cctbx_project/scitbx/lbfgs/tst_lbfgs_fem.py" [FAIL]
[…]
I see those too. Tracking them is too much effort indeed. I followed the route already suggested by Marcin: break -ffast-math in finer options and find the one breaking our code. After a round-trip through Clang source code (I wish they had a proper documentation of *all* their command-line options as it exists for gcc), I came with a fix which keeps all useful optimisations enabled while the 3 failing tests do now pass.
Gonna commit shortly.
Best wishes,
Luc
--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
_______________________________________________
cctbxbb mailing list
[email protected]
http://phenix-online.org/mailman/listinfo/cctbxbb
