[firedrake] Running helmholtz solver in parallel on ARCHER
Eike Mueller
e.mueller at bath.ac.uk
Mon Sep 29 11:08:18 BST 2014
Dear all,
just to ley you know that it is working now, i.e. I can run the
helmholtz solver on 4 processes on ARCHER.
I needed the local_par-loop branch from the latest version of PyOP2 and
also the multigrid branch from the latest firedrake, so I pulled these
two into my work directory.
I then added
export PYTHONPATH=$WORK/PyOP2:${PYTHONPATH}
export PYTHONPATH=$WORK/firedrake:${PYTHONPATH}
to my job script (attached). However, despite that, python seems to load
the pre-installed PyOP2 module first, below is the output of sys.path.
I could fix this by adding
import sys
sys.path.insert(0,'/work/n02/n02/eike/PyOP2')
[...]
at the top of my main python script. Any ideas why I had to do this?
Thanks,
Eike
['/fs2/n02/n02/eike/git_workspace/firedrake-helmholtzsolver/source',
'/fs2/y07/y07/fdrake/decorator-3.4.0/lib/python2.7/site-packages/decorator-3.4.0-py2.7.egg',
'/work/y07/y07/fdrake/PyOP2/lib/python2.7/site-packages/PyOP2-0.11.0_99_g54eb9ea_dirty-py2.7-linux-x86_64.egg',
'/work/y07/y07/fdrake/ffc/lib/python2.7/site-packages/FFC-1.4.0_-py2.7-linux-x86_64.egg',
'/work/y07/y07/fdrake/decorator-3.4.0/lib/python2.7/site-packages/decorator-3.4.0-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/python_hostlist-1.14-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/pymongo-2.7.2-py2.7-linux-x86_64.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/GridDataFormats-0.2.4-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/setuptools-2.2-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/MDAnalysis-0.8.1-py2.7-linux-x86_64.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/python_hostlist-1.14-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/pymongo-2.7.2-py2.7-linux-x86_64.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/radical.utils-0.7.7-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/saga_python-0.18-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/GridDataFormats-0.2.4-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/apache_libcloud-0.15.1-py2.7.egg',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/setuptools-2.2-py2.7.egg',
'/work/n02/n02/eike/firedrake', '/work/n02/n02/eike/PyOP2',
'/work/n02/n02/eike/firedrake-bench', '/work/n02/n02/eike/pybench',
'/work/y07/y07/fdrake/firedrake/lib/python2.7/site-packages',
'/work/y07/y07/fdrake/PyOP2/lib/python2.7/site-packages',
'/work/y07/y07/fdrake/ufl/lib/python2.7/site-packages',
'/work/y07/y07/fdrake/scientificpython/lib/python2.7/site-packages',
'/work/y07/y07/fdrake/psutil/lib/python2.7/site-packages',
'/work/y07/y07/fdrake/mpi4py/lib/python2.7/site-packages',
'/work/y07/y07/fdrake/instant/lib/python2.7/site-packages',
'/work/y07/y07/fdrake/fiat/lib/python2.7/site-packages',
'/work/y07/y07/fdrake/ffc/lib/python2.7/site-packages',
'/work/y07/y07/fdrake/decorator-3.4.0/lib/python2.7/site-packages',
'/work/y07/y07/fdrake/petsc/arch-linux2-cxx-opt/lib/python2.7/site-packages',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7',
'/opt/cray/sdb/1.0-1.0501.48084.4.48.ari/lib64/py',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python27.zip',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/plat-linux2',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/lib-tk',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/lib-old',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/lib-dynload',
'/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/PIL',
'/opt/cray/sdb/1.0-1.0501.48084.4.48.ari/lib64/py']
On 15/09/14 09:37, Florian Rathgeber wrote:
> On 15/09/14 09:20, Eike Mueller wrote:
>> Hi Florian,
>>
>> I'm having another go at this now.
>>
>>>> export
>>>> PYTHONPATH=${FDRAKE_DIR}/petsc4py/lib/python2.7/site-packages:${PYTHONPATH}
>>>> export
>>>> PYTHONPATH=${FDRAKE_DIR}/ufl/lib/python2.7/site-packages/:${PYTHONPATH}
>>>> export
>>>> PYTHONPATH=${FDRAKE_DIR}/fiat/lib/python2.7/site-packages/:${PYTHONPATH}
>>>> export
>>>> PYTHONPATH=${FDRAKE_DIR}/ffc/lib/python2.7/site-packages/:${PYTHONPATH}
>>>>
>> I added this to the template file on the wiki.
> This *should* be added by loading fdrake-python-env
>
>>> * The https cloning issue I believe is resolved, at least I can't
>>> reproduce it anymore.
>> All seems to work fine for me now.
>>
>>> * The PyOP2 installation issue related to NumPy is a bug [1] in
>>> setuptools 2.2 which comes with anaconda. For now, the best thing to do
>>> is to remove/comment line 138 in PyOP2/setup.py which declares the setup
>>> dependency on NumPy.
>>>
>> I commented out the line 'setup_requires=setup_requires,' in setup.py
>> and the PyOP2 installation goes through without problems.
>>
>>> * The anaconda/environment modules conflict is resolved: I have removed
>>> ANACONDA_LIB from the LD_LIBRARY_PATH and added it to LD_RUN_PATH
>>> instead. This means Python extension modules get an RPATH to that
>>> directory baked in and therefore can find the right libpython2.7.so
>>> without any LD_LIBRARY_PATH being set (which is preferable anyway).
>>>
>> Works for me now, I can run the module command even after loading the
>> firedrake environment.
>>
>>> Unfortunately, Firedrake's setup.py manually sets an RPATH for PETSc and
>>> therefore the LD_RUN_PATH is ignored. This patch is required:
>>>
>>> diff --git a/setup.py b/setup.py
>>> index 00838f8..350d51d 100644
>>> --- a/setup.py
>>> +++ b/setup.py
>>> @@ -70,6 +70,7 @@ setup(name='firedrake',
>>> include_dirs=include_dirs,
>>> libraries=["petsc"],
>>> extra_link_args=["-L%s/lib" % d for d in
>>> petsc_dirs] +
>>> - ["-Wl,-rpath,%s/lib" % d for d in
>>> petsc_dirs]),
>>> + ["-Wl,-rpath,%s/lib" % d for d in
>>> petsc_dirs] +
>>> + ["-Wl,-rpath,%s/lib" % sys.prefix]),
>>> Extension('evtk.cevtk', evtk_sources,
>>> include_dirs=[np.get_include()])])
>>>
>> Can you send me that patch as a separate text file, please? When I
>> copied it out of your email it didn't work.
> I have pushed this to firedrake master.
>
>> Still something not quite right with the python. When I run a 'Hello
>> World!' test, i.e.
>>
>> import sys
>> print 'Hello World!'
>> sys.exit(0)
>>
>> I still get
>>
>> python: error while loading shared libraries: libpython2.7.so.1.0:
>> cannot open shared object file: No such file or directory
> That works for me. Not sure why this would fail since you're not loading
> an extension module. Is this on the login node with anaconda?
>
>> but I guess this is due to the unresolved issue with anaconda that you
>> reported to the helpdesk, so I will wait for what comes back from them.
> Indeed, as reported to helpdesk, it was premature thinking the
> LD_LIBRARY_PATH was no longer required. I'm currently stuck on mpi4py
> even though all the RPATHs are correctly set afaict. For now you'll need
> to add $ANANCONDA_LIBS to LD_LIBRARY_PATH in your firedrake.tpl
>
> Florian
>
>> Thanks,
>>
>> Eike
>>
>>> For FFC it's more tricky, still trying to figure out what's needed.
>>>
>>> Keep us posted on how things are progressing!
>>>
>>> Florian
>>>
>>>> I will go ahead and edit the wiki.
>>>>
>>>> Eike
>>>>
>>>>
>>>> On 8 Sep 2014, at 15:53, Eike Mueller <E.Mueller at bath.ac.uk
>>>> <mailto:E.Mueller at bath.ac.uk>
>>>> <mailto:E.Mueller at bath.ac.uk>> wrote:
>>>>
>>>>> Hi Florian,
>>>>>
>>>>> thanks a lot for your reply and for updating the wiki, I got a bit
>>>>> further now.
>>>>>
>>>>>> You're exactly right again: it seems that anaconda contains a
>>>>>> version of
>>>>>> the TCL libraries which are incompatible with those use by the
>>>>>> environment modules. I'm not sure how to work around this, since we
>>>>>> need
>>>>>> the anaconda libs (as you figured out later) - will open a ticket.
>>>>> ok, I will use the environment script as it is and not worry about the
>>>>> fact that the anaconda library breaks the module command at the
>>>>> moment. Let's see if the support can figure something out.
>>>>>
>>>>>> I've changed the wiki page to clone via https, thanks!
>>>>> It works with the updated instructions on the webpage. Strangely, if I
>>>>> clone a repository into a directory, delete this directory and then
>>>>> try to clone again with exactly the same command, nothing happens.
>>>>>
>>>>>> This seems to happen because (for reasons I can't figure out)
>>>>>> setuptools
>>>>>> doesn't detect that NumPy is already installed and builds it from
>>>>>> source. The primary cause of the error you're seeing is this:
>>>>>> https://stackoverflow.com/a/21621493
>>>>> I will ignore this then, and just call the setup.py script twice in
>>>>> the PyOP2 directory.
>>>>>
>>>>>> This is because you've removed the anaconda lib path, which also
>>>>>> contains the Python libraries themselves, from LD_LIBRARY_PATH
>>>>> I tried a simpler hello-world script where I swap python -> anaconda
>>>>> and run a python script. This does work.
>>>>> Still trying to run the poisson example from the benchmark repository,
>>>>> but the ARCHER queues seem to be terribly slow, even if I just run on
>>>>> one node on the debug queue :-(
>>>>>
>>>>>> Absolutely, feel free to edit the wiki if you find more errors or
>>>>>> things
>>>>>> that are unclear.
>>>>>>
>>>>> I think I already have access to the repository, so will edit if
>>>>> necessary.
>>>>>
>>>>> Thanks a lot,
>>>>>
>>>>> Eike
>>>>>
>>>>>> Florian
>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Eike
>>>>>>>
>>>>>>> *** content of firedrake.env ***
>>>>>>>
>>>>>>> module swap PrgEnv-cray PrgEnv-gnu
>>>>>>> module unload python
>>>>>>> module add anaconda
>>>>>>>
>>>>>>> # Add fdrake module path
>>>>>>> export FDRAKE_DIR=/work/y07/y07/fdrake
>>>>>>> module use "$FDRAKE_DIR/modules"
>>>>>>>
>>>>>>> # Load build environment
>>>>>>> module load fdrake-build-env
>>>>>>> module load fdrake-python-env
>>>>>>>
>>>>>>> LD_LIBRARY_PATH=`echo $LD_LIBRARY_PATH | sed
>>>>>>> 's/\/work\/y07\/y07\/cse\/anaconda\/1.9.2\/lib//g'`
>>>>>>>
>>>>>>> export
>>>>>>> PYTHONPATH=$WORK/firedrake-bench:$WORK/firedrake-bench/pybench:$WORK/firedrake:$WORK/PyOP2:$PYTHONPATH
>
>
> _______________________________________________
> firedrake mailing list
> firedrake at imperial.ac.uk
> https://mailman.ic.ac.uk/mailman/listinfo/firedrake
--
Dr Eike Hermann Mueller
Research Associate (PostDoc)
Department of Mathematical Sciences
University of Bath
Bath BA2 7AY, United Kingdom
+44 1225 38 5803
e.mueller at bath.ac.uk
http://people.bath.ac.uk/em459/
-------------- next part --------------
HTML attachment scrubbed and removed
-------------- next part --------------
#!/bin/bash --login
#PBS -N helmholtz
#PBS -l walltime=0:05:0
#PBS -l select=1
#PBS -A n02-NEJ005576
#PBS -M e.mueller at bath.ac.uk
#PBS -q debug
# Make sure any symbolic links are resolved to absolute path
export PBS_O_WORKDIR=$(readlink -f $PBS_O_WORKDIR)
echo Running in $PBS_O_WORKDIR
cd $PBS_O_WORKDIR
LOGFILE=helmholtz.${PBS_JOBID}.log
module swap PrgEnv-cray PrgEnv-gnu
module unload python
module add anaconda
module use /work/y07/y07/fdrake/modules
module load firedrake
export FIREDRAKE_FFC_KERNEL_CACHE_DIR=$WORK/firedrake-cache
export PYOP2_LAZY=0
export PYOP2_BACKEND_COMPILER=gnu
export PYOP2_SIMD_ISA=avx
export PYOP2_CACHE_DIR=$WORK/pyop2-cache
export LD_LIBRARY_PATH=$ANACONDA_LIB:$LD_LIBRARY_PATH
export PYTHONPATH=$WORK/firedrake-bench:$WORK/pybench:$PYTHONPATH
export PYTHONPATH=$WORK/PyOP2:${PYTHONPATH}
export PYTHONPATH=$WORK/firedrake:${PYTHONPATH}
export PETSC_OPTIONS=-log_summary
# Prevent matplotlib from accessing /home
export HOME=$WORK
export XDG_CONFIG_HOME=''
# MPI (man intro_mpi)
export MPICH_NEMESIS_ASYNC_PROGRESS=MC
export MPICH_MAX_THREAD_SAFETY=multiple
export MPICH_CPUMASK_DISPLAY=1
echo -n Started at | tee -a $LOGFILE
date | tee -a $LOGFILE
echo | tee -a $LOGFILE
echo Running helmholtz 2>&1 | tee -a $LOGFILE
echo | tee -a $LOGFILE
aprun -n 4 -N 4 -S 2 python driver.py parameters.in 2>&1 | tee -a $LOGFILE
echo -n Finished at | tee -a $LOGFILE
date | tee -a $LOGFILE
More information about the firedrake
mailing list