[firedrake] Running helmholtz solver in parallel on ARCHER

Florian Rathgeber f.rathgeber10 at imperial.ac.uk
Mon Sep 29 13:56:20 BST 2014 

On 29/09/14 11:08, Eike Mueller wrote:
> Dear all,
> 
> just to ley you know that it is working now, i.e. I can run the
> helmholtz solver on 4 processes on ARCHER.
> 
> I needed the local_par-loop branch from the latest version of PyOP2 and
> also the multigrid branch from the latest firedrake, so I pulled these
> two into my work directory.
> I then added
> 
> export PYTHONPATH=$WORK/PyOP2:${PYTHONPATH}
> export PYTHONPATH=$WORK/firedrake:${PYTHONPATH}
> 
> to my job script (attached). However, despite that, python seems to load
> the pre-installed PyOP2 module first, below is the output of sys.path.
> 
> I could fix this by adding
> 
> import sys
> sys.path.insert(0,'/work/n02/n02/eike/PyOP2')
> [...]
> 
> at the top of my main python script. Any ideas why I had to do this?

The reason for this is a "feature" of distutils, which generates a file
easy_install.pth containing paths to eggs installed using distutils
which are pushed to the *beginning* of sys.path even before entries in
the $PYTHONPATH. I have fixed this now, so you no longer need to fiddle
with sys.path in your script.

Florian

> Thanks,
> 
> Eike
> 
> 
> ['/fs2/n02/n02/eike/git_workspace/firedrake-helmholtzsolver/source',
> '/fs2/y07/y07/fdrake/decorator-3.4.0/lib/python2.7/site-packages/decorator-3.4.0-py2.7.egg',
> '/work/y07/y07/fdrake/PyOP2/lib/python2.7/site-packages/PyOP2-0.11.0_99_g54eb9ea_dirty-py2.7-linux-x86_64.egg',
> '/work/y07/y07/fdrake/ffc/lib/python2.7/site-packages/FFC-1.4.0_-py2.7-linux-x86_64.egg',
> '/work/y07/y07/fdrake/decorator-3.4.0/lib/python2.7/site-packages/decorator-3.4.0-py2.7.egg',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/python_hostlist-1.14-py2.7.egg',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/pymongo-2.7.2-py2.7-linux-x86_64.egg',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/GridDataFormats-0.2.4-py2.7.egg',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/setuptools-2.2-py2.7.egg',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/extasy.coco-0.1-py2.7.egg',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/radical.pilot-0.18-py2.7.egg',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/MDAnalysis-0.8.1-py2.7-linux-x86_64.egg',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/python_hostlist-1.14-py2.7.egg',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/pymongo-2.7.2-py2.7-linux-x86_64.egg',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/radical.utils-0.7.7-py2.7.egg',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/saga_python-0.18-py2.7.egg',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/GridDataFormats-0.2.4-py2.7.egg',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/apache_libcloud-0.15.1-py2.7.egg',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/setuptools-2.2-py2.7.egg',
> '/work/n02/n02/eike/firedrake', '/work/n02/n02/eike/PyOP2',
> '/work/n02/n02/eike/firedrake-bench', '/work/n02/n02/eike/pybench',
> '/work/y07/y07/fdrake/firedrake/lib/python2.7/site-packages',
> '/work/y07/y07/fdrake/PyOP2/lib/python2.7/site-packages',
> '/work/y07/y07/fdrake/ufl/lib/python2.7/site-packages',
> '/work/y07/y07/fdrake/scientificpython/lib/python2.7/site-packages',
> '/work/y07/y07/fdrake/psutil/lib/python2.7/site-packages',
> '/work/y07/y07/fdrake/mpi4py/lib/python2.7/site-packages',
> '/work/y07/y07/fdrake/instant/lib/python2.7/site-packages',
> '/work/y07/y07/fdrake/fiat/lib/python2.7/site-packages',
> '/work/y07/y07/fdrake/ffc/lib/python2.7/site-packages',
> '/work/y07/y07/fdrake/decorator-3.4.0/lib/python2.7/site-packages',
> '/work/y07/y07/fdrake/petsc/arch-linux2-cxx-opt/lib/python2.7/site-packages',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7',
> '/opt/cray/sdb/1.0-1.0501.48084.4.48.ari/lib64/py',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python27.zip',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/plat-linux2',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/lib-tk',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/lib-old',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/lib-dynload',
> '/work/y07/y07/cse/anaconda/1.9.2/lib/python2.7/site-packages/PIL',
> '/opt/cray/sdb/1.0-1.0501.48084.4.48.ari/lib64/py']
> 
> On 15/09/14 09:37, Florian Rathgeber wrote:
>> On 15/09/14 09:20, Eike Mueller wrote:
>>> Hi Florian,
>>>
>>> I'm having another go at this now.
>>>
>>>>> export
>>>>> PYTHONPATH=${FDRAKE_DIR}/petsc4py/lib/python2.7/site-packages:${PYTHONPATH}
>>>>> export
>>>>> PYTHONPATH=${FDRAKE_DIR}/ufl/lib/python2.7/site-packages/:${PYTHONPATH}
>>>>> export
>>>>> PYTHONPATH=${FDRAKE_DIR}/fiat/lib/python2.7/site-packages/:${PYTHONPATH}
>>>>> export
>>>>> PYTHONPATH=${FDRAKE_DIR}/ffc/lib/python2.7/site-packages/:${PYTHONPATH}
>>>>>
>>> I added this to the template file on the wiki.
>> This *should* be added by loading fdrake-python-env
>>
>>>> * The https cloning issue I believe is resolved, at least I can't
>>>> reproduce it anymore.
>>> All seems to work fine for me now.
>>>
>>>> * The PyOP2 installation issue related to NumPy is a bug [1] in
>>>> setuptools 2.2 which comes with anaconda. For now, the best thing to do
>>>> is to remove/comment line 138 in PyOP2/setup.py which declares the setup
>>>> dependency on NumPy.
>>>>
>>> I commented out the line  'setup_requires=setup_requires,' in setup.py
>>> and the PyOP2 installation goes through without problems.
>>>
>>>> * The anaconda/environment modules conflict is resolved: I have removed
>>>> ANACONDA_LIB from the LD_LIBRARY_PATH and added it to LD_RUN_PATH
>>>> instead. This means Python extension modules get an RPATH to that
>>>> directory baked in and therefore can find the right libpython2.7.so
>>>> without any LD_LIBRARY_PATH being set (which is preferable anyway).
>>>>
>>> Works for me now, I can run the module command even after loading the
>>> firedrake environment.
>>>
>>>> Unfortunately, Firedrake's setup.py manually sets an RPATH for PETSc and
>>>> therefore the LD_RUN_PATH is ignored. This patch is required:
>>>>
>>>> diff --git a/setup.py b/setup.py
>>>> index 00838f8..350d51d 100644
>>>> --- a/setup.py
>>>> +++ b/setup.py
>>>> @@ -70,6 +70,7 @@ setup(name='firedrake',
>>>>                              include_dirs=include_dirs,
>>>>                              libraries=["petsc"],
>>>>                              extra_link_args=["-L%s/lib" % d for d in
>>>> petsc_dirs] +
>>>> -                             ["-Wl,-rpath,%s/lib" % d for d in
>>>> petsc_dirs]),
>>>> +                             ["-Wl,-rpath,%s/lib" % d for d in
>>>> petsc_dirs] +
>>>> +                             ["-Wl,-rpath,%s/lib" % sys.prefix]),
>>>>                    Extension('evtk.cevtk', evtk_sources,
>>>>                              include_dirs=[np.get_include()])])
>>>>
>>> Can you send me that patch as a separate text file, please? When I
>>> copied it out of your email it didn't work.
>> I have pushed this to firedrake master.
>>
>>> Still something not quite right with the python. When I run a 'Hello
>>> World!' test, i.e.
>>>
>>> import sys
>>> print 'Hello World!'
>>> sys.exit(0)
>>>
>>> I still get
>>>  
>>> python: error while loading shared libraries: libpython2.7.so.1.0:
>>> cannot open shared object file: No such file or directory
>> That works for me. Not sure why this would fail since you're not loading
>> an extension module. Is this on the login node with anaconda?
>>
>>> but I guess this is due to the unresolved issue with anaconda that you
>>> reported to the helpdesk, so I will wait for what comes back from them.
>> Indeed, as reported to helpdesk, it was premature thinking the
>> LD_LIBRARY_PATH was no longer required. I'm currently stuck on mpi4py
>> even though all the RPATHs are correctly set afaict. For now you'll need
>> to add $ANANCONDA_LIBS to LD_LIBRARY_PATH in your firedrake.tpl
>>
>> Florian
>>
>>> Thanks,
>>>
>>> Eike
>>>
>>>> For FFC it's more tricky, still trying to figure out what's needed.
>>>>
>>>> Keep us posted on how things are progressing!
>>>>
>>>> Florian
>>>>
>>>>> I will go ahead and edit the wiki.
>>>>>
>>>>> Eike
>>>>>
>>>>>
>>>>> On 8 Sep 2014, at 15:53, Eike Mueller <E.Mueller at bath.ac.uk
>>>>> <mailto:E.Mueller at bath.ac.uk>
>>>>> <mailto:E.Mueller at bath.ac.uk>> wrote:
>>>>>
>>>>>> Hi Florian,
>>>>>>
>>>>>> thanks a lot for your reply and for updating the wiki, I got a bit
>>>>>> further now.
>>>>>>
>>>>>>> You're exactly right again: it seems that anaconda contains a
>>>>>>> version of
>>>>>>> the TCL libraries which are incompatible with those use by the
>>>>>>> environment modules. I'm not sure how to work around this, since we
>>>>>>> need
>>>>>>> the anaconda libs (as you figured out later) - will open a ticket.
>>>>>> ok, I will use the environment script as it is and not worry about the
>>>>>> fact that the anaconda library breaks the module command at the
>>>>>> moment. Let's see if the support can figure something out.
>>>>>>
>>>>>>> I've changed the wiki page to clone via https, thanks!
>>>>>> It works with the updated instructions on the webpage. Strangely, if I
>>>>>> clone a repository into a directory, delete this directory and then
>>>>>> try to clone again with exactly the same command, nothing happens.
>>>>>>
>>>>>>> This seems to happen because (for reasons I can't figure out)
>>>>>>> setuptools
>>>>>>> doesn't detect that NumPy is already installed and builds it from
>>>>>>> source. The primary cause of the error you're seeing is this:
>>>>>>> https://stackoverflow.com/a/21621493
>>>>>> I will ignore this then, and just call the setup.py script twice in
>>>>>> the PyOP2 directory.
>>>>>>
>>>>>>> This is because you've removed the anaconda lib path, which also
>>>>>>> contains the Python libraries themselves, from LD_LIBRARY_PATH
>>>>>> I tried a simpler hello-world script where I swap python -> anaconda
>>>>>> and run a python script. This does work.
>>>>>> Still trying to run the poisson example from the benchmark repository,
>>>>>> but the ARCHER queues seem to be terribly slow, even if I just run on
>>>>>> one node on the debug queue :-(
>>>>>>
>>>>>>> Absolutely, feel free to edit the wiki if you find more errors or
>>>>>>> things
>>>>>>> that are unclear.
>>>>>>>
>>>>>> I think I already have access to the repository, so will edit if
>>>>>> necessary.
>>>>>>
>>>>>> Thanks a lot,
>>>>>>
>>>>>> Eike
>>>>>>
>>>>>>> Florian
>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Eike
>>>>>>>>
>>>>>>>> *** content of firedrake.env ***
>>>>>>>>
>>>>>>>> module swap PrgEnv-cray PrgEnv-gnu
>>>>>>>> module unload python
>>>>>>>> module add anaconda
>>>>>>>>
>>>>>>>> # Add fdrake module path
>>>>>>>> export FDRAKE_DIR=/work/y07/y07/fdrake
>>>>>>>> module use "$FDRAKE_DIR/modules"
>>>>>>>>
>>>>>>>> # Load build environment
>>>>>>>> module load fdrake-build-env
>>>>>>>> module load fdrake-python-env
>>>>>>>>
>>>>>>>> LD_LIBRARY_PATH=`echo $LD_LIBRARY_PATH | sed
>>>>>>>> 's/\/work\/y07\/y07\/cse\/anaconda\/1.9.2\/lib//g'`
>>>>>>>>
>>>>>>>> export
>>>>>>>> PYTHONPATH=$WORK/firedrake-bench:$WORK/firedrake-bench/pybench:$WORK/firedrake:$WORK/PyOP2:$PYTHONPATH

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2980 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://mailman.ic.ac.uk/pipermail/firedrake/attachments/20140929/7901ea85/attachment.p7s>

More information about the firedrake mailing list