easy_mp.multi_core_run function
Hi, I needed a way to do multiprocessing job runs in a simple fashion on a multicore PC. Gabor provided much of the underlying machinery in libtbx for that. But I have often found I'm wasting too much time setting it all up. So I crafted a small function, easy_mp.multi_core_run(), out of his example in Computational Crystallography Newsletter (2013). 4, p21 which does exactly that. Unlike easy_mp.parallel_map() this function continues if one of the jobs crashes catastrophically. It takes as arguments name of function to be parallelised, a list of argument tuples and number of processors to run on. As the jobs completes it returns an iterator of tuples of arguments with corresponding results. Example: def RunMyJob(args1, args2): pass def GatherResults(parmres): pass from libtbx import easy_mp argstuples = [(args1a, args2a), (args1b, args2b), (args1c, args2c) ] for i, parmres in enumerate(easy_mp.multi_core_run( RunMyJob, argstuples, nproc)): GatherResults( parmres ) I'm interested in comments or if this kind of simple function is already available elsewhere. Regards, Rob -- Robert Oeffner, Ph.D. Research Associate, The Read Group Department of Haematology, Cambridge Institute for Medical Research University of Cambridge Cambridge Biomedical Campus Wellcome Trust/MRC Building Hills Road Cambridge CB2 0XY www.cimr.cam.ac.uk/investigators/read/index.html tel: +44(0)1223 763234 mobile: +44(0)7712 887162
participants (1)
-
R.D. Oeffner