Hi Kaushik,

I'm sorry for the problem with parallel autobuilding!

I think that parallel_autobuild is unfortunately not going to do what you want. It is for running many parallel jobs with the same starting model and combining all the results.  It doesn't work with multiple starting models as I think you want it to.

On the error you got, I'm not quite sure what happened but it seems that perhaps the command "qsub" was not found:

RuntimeError: child process stderr output:

  command: 'qsub background=False nproc=20 "~phenixWorkingDir/1yya/part1/8tim/AutoBuild_run_5_/TEMP0/RUN_FILE_1"'

  /bin/sh: qsub: command not found

It is a little surprising that this would happen in parallel_autobuild and not in autobuild.  However perhaps in autobuild you specified that the multiple jobs are to be run with "sh" (run_command=sh) and in parallel_autobuild it was with qsub (run_command=qsub).  

Let me know if that does not help,

All the best,

Tom T

---

From: phenixbb-bounces@phenix-online.org [phenixbb-bounces@phenix-online.org] on behalf of Kaushik Hatti [hskaushik@gmail.com]
Sent: Monday, April 27, 2015 12:05 AM
To: phenixbb@phenix-online.org
Subject: [phenixbb] Autobuild on parallel

Hi,

What is the best way to run Autobuild if I need to run it on hundreds of datasets (through command-line)?  I have a single machine with 24 cores and access to a cluster with 128 cores (on parallel; accessible only through PBS script).  

Each Autobuild job with nproc=4 on a single core machine takes ~12 hours to run (nproc > 4 wouldn't make any difference in the speed of computation?!).  However, I read that I could set nproc=10 (or more) and run on parallel (https://www.phenix-online.org/documentation/reference/autobuild.html#parallel-jobs-nproc-nbatch-number-of-parallel-models-and-how-autobuild-works-in-parallel).  When I tried the same through PBS script, I got the following error,

XXXXXXXXXXXXXX

Running up to  1  jobs in parallel... with total of  3  jobs

Splitting work into 3 jobs and running with 1 processors using qsub background=False nproc=20

background=False in ~phenixWorkingDir/1yya/part1/8tim/AutoBuild_run_5_/TEMP0

Final job will be run with sh  with background=True

Starting job 1...Log will be: ~phenixWorkingDir/1yya/part1/8tim/AutoBuild_run_5_/TEMP0/RUN_FILE_1.log

Traceback (most recent call last):

  File "~opt/phenix-1.9-1692/phenix/phenix/autosol/AutoBaseExtend.py", line 989, in DoNextMethod

    self.CarryOutBest()     # to be obtained after it is finished

  File "~opt/phenix-1.9-1692/phenix/phenix/autosol/AutoBaseExtend.py", line 2147, in CarryOutBest

    getattr(self,str(self.application_method))()  # call this fn

  File "~opt/phenix-1.9-1692/phenix/phenix/wizards/AutoBuild.py", line 2364, in AutoBuild_build_cycle

    self.AutoBuild_rebuild_cycle_run()

  File "~opt/phenix-1.9-1692/phenix/phenix/wizards/AutoBuild.py", line 4647, in AutoBuild_rebuild_cycle_run

    always_reuse_model=always_reuse_model)

  File "~opt/phenix-1.9-1692/phenix/phenix/wizards/AutoBuild.py", line 7452, in AutoBuild_build_refine

    mtz_file=mtz_file,mtz_ref_file=mtz_ref_file)

  File "~opt/phenix-1.9-1692/phenix/phenix/wizards/AutoBuild.py", line 7696, in run_standard_build_in_parallel

    r.run(out=sys.stdout)

  File "~opt/phenix-1.9-1692/phenix/phenix/autosol/run_group_of_wizards.py", line 294, in run

    self.start_run(run_file,last=is_last)

  File "~opt/phenix-1.9-1692/phenix/phenix/autosol/run_group_of_wizards.py", line 604, in start_run

    cmd+" "+self.add_double_quote(run_file,escape_space=False)).raise_if_errors()

  File "~opt/phenix-1.9-1692/cctbx_project/libtbx/easy_run.py", line 37, in raise_if_errors

    raise Error(msg)

RuntimeError: child process stderr output:

  command: 'qsub background=False nproc=20 "~phenixWorkingDir/1yya/part1/8tim/AutoBuild_run_5_/TEMP0/RUN_FILE_1"'

  /bin/sh: qsub: command not found

XXXXXXXXXXXXXX

Please let me know if I can share eff and log files of the run.

Please advice,

Kaushik

--

People living deeply have no fear of death - Anais Nin
Caution: I am still the dumbest person I have ever known :-)