[firedrake] Locally installed FFC not found on ARCHER

Fri Jan 30 16:51:17 GMT 2015

Hi Michael,

> In that case I suggest creating your own local module file for the
> python env, starting from fdrake-python-env as a template, where you can
> explicitly select the branch builds you require. The central module,
> however, should always always be self-contained.

I've done that now, and it all works fine again. I.e. if I use my 
locally build firedrake installation I can run the 3d solver code for 
the gravity wave system.

I still use decorator-3.4.0, instant, psutil and scientificpython from 
the central firedrake installation since I can't be bothered to 
reinstall these locally, and it doesn't seem necessary.

> Yes, it should work without this line now. I don't know why these
> components had been disabled in the central module.

I took the firedrake.env and firedrake.tpl from the webpage 
(https://github.com/firedrakeproject/firedrake-bench/wiki/Archer) and 
removed the references to the local PyOP2, petsc4py, firedrake, ufl, 
fiat and ffc. To be precise, the files I use are attached.

I then tried to run the Poisson testcase as described on the webpage and 
it goes through fine. Note that I had to add

export LD_LIBRARY_PATH=$ANACONDA_LIB:${LD_LIBRARY_PATH}

in firedrake.tpl, otherwise I get

python: error while loading shared libraries: libpython2.7.so.1.0: 
cannot open shared object file: No such file or directory
Application 12817509 exit codes: 127
Application 12817509 resources: utime ~0s, stime ~0s, Rss ~4296, 
inblocks ~15, outblocks ~19

Do you have to uncomment

prepend-path LD_LIBRARY_PATH $env(ANACONDA_LIB)

in the central fdrake-python-env again? That's what I did in my local 
module file (i.e. the one I use to run with my local firedrake 
installation) and it doesn't seem to break the module command any more.

Are you happy for me to go ahead and update the firedrake.tpl and 
firedrake.env on the webpage?

Thanks,

Eike

>>
>> Thanks,
>>
>> Eike
>>
>> On 30/01/15 07:21, Michael Lange wrote:
>>> Hi Eike,
>>>
>>> I just updated the "firedrake" module on Archer and it should be fully
>>> functional again. Can you please give this a try and let me know if it
>>> works for you?
>>>
>>> The PETSc problem was due to fdrake-build-env pointing at a very old
>>> petsc build (petsc/dev), not the one Florian re-built yesterday
>>> (petsc/shared). I also pulled the entire mapdes stack forward, so we
>>> should be up-to-date with current master versions.
>>>
>>> Also, as a heads up, I am currently working with Tim Bond and Rupert
>>> Nash from EPCC on rolling out separate petsc-master and petsc4py-master
>>> packages that use the centrally installed (and supported) Python stack.
>>> This is possible after several updates to the Python environment on
>>> Archer and should allow us to get off the relatively unsupported
>>> Anaconda module to avoid more binary incompatibilities in the future.
>>> I'll post to the list once we're ready to switch.
>>>
>>> Thanks,
>>> Michael
>>>
>>>
>>> On 29/01/15 22:54, Florian Rathgeber wrote:
>>>> On 1/29/2015 12:04 PM, Eike Mueller wrote:
>>>>> Hi Florian,
>>>>>
>>>>> I recall having similar issues with incompatible MPI versions before,
>>>>> but can't find anything in my emails which points to a solution.
>>>>> I think the reason in my case was that the CRAY MPICH was updated at
>>>>> some point after the original /work/y07/y07/fdrake installation, but
>>>>> before I installed my own version.
>>>> I would have expected something like this, but afaict we were using
>>>> mpich 7.0.3 throughout.
>>>>
>>>>> Could it be that mpi4py is still using the old MPICH and needs to be
>>>>> reinstalled?
>>>> That is indeed a possibility. I wonder if you could get access to the
>>>> fdrake account since you're pretty much the main user. Michael?
>>>>
>>>> Florian
>>>>
>>>>> Cheers,
>>>>>
>>>>> Eike
>>>>>
>>>>> On 29/01/15 11:51, Florian Rathgeber wrote:
>>>>>> On 29/01/15 10:23, Eike Mueller wrote:
>>>>>>> I need to install my own version of FFC, since the one already
>>>>>>> installed
>>>>>>> in /work/y07/y07/fdrake appears not to be compatible with the
>>>>>>> firedrake
>>>>>>> version I use
>>>>>> I had a go at updating the "official" firedrake package on ARCHER and
>>>>>> dependencies. I updated PETSc and petsc4py to the latest revision
>>>>>> of the
>>>>>> respective firedrake branch. Then I failed however building the
>>>>>> firedrake extension modules due to the following error:
>>>>>>
>>>>>> /work/y07/y07/fdrake/petsc/include/petscsys.h:124:6: error: #error
>>>>>> "PETSc was configured with one MPICH mpi.h v
>>>>>> ersion but now appears to be compiling using a different MPICH mpi.h
>>>>>> version"
>>>>>>
>>>>>> afaict we always use cray-mpich/7.0.3 for building petsc and
>>>>>> everything
>>>>>> else. No idea what triggers this error.
>>>>>>
>>>>>> On another note: I *really* don't have time to continue
>>>>>> maintaining the
>>>>>> fdrake modules on ARCHER and appreciate if someone else (Michael?
>>>>>> Lawrence?) could take on this task. I also don't have time to sort
>>>>>> out
>>>>>> this current issue, sorry about that.
>>>>>>
>>>>>> Florian
>>>>
>>>>
>>>> _______________________________________________
>>>> firedrake mailing list
>>>> firedrake at imperial.ac.uk
>>>> https://mailman.ic.ac.uk/mailman/listinfo/firedrake
>>>
>>>
>>>
>>> _______________________________________________
>>> firedrake mailing list
>>> firedrake at imperial.ac.uk
>>> https://mailman.ic.ac.uk/mailman/listinfo/firedrake
>>>
>>
>> _______________________________________________
>> firedrake mailing list
>> firedrake at imperial.ac.uk
>> https://mailman.ic.ac.uk/mailman/listinfo/firedrake
>
>
> _______________________________________________
> firedrake mailing list
> firedrake at imperial.ac.uk
> https://mailman.ic.ac.uk/mailman/listinfo/firedrake
-------------- next part --------------
module unload PrgEnv-cray
module load PrgEnv-gnu
module unload python
module load anaconda

# Add fdrake module path
export FDRAKE_DIR=/work/y07/y07/fdrake
module use $FDRAKE_DIR/modules

# Load build environment
module load fdrake-build-env
module load fdrake-python-env

export PYTHONPATH=$WORK/git_workspace/firedrake-bench:$WORK/git_workspace/pybench:$PYTHONPATH
-------------- next part --------------
#!/bin/bash --login

#PBS -N %(jobname)s
#PBS -l walltime=%(walltime)s:0
#PBS -l select=%(nodes)d
#PBS -A n02-NEJ005576
#PBS -m eba
#PBS -M e.mueller at bath.ac.uk

# Make sure any symbolic links are resolved to absolute path 
export PBS_O_WORKDIR=$(readlink -f $PBS_O_WORKDIR)

echo Running in $PBS_O_WORKDIR
cd $PBS_O_WORKDIR

LOGFILE=%(jobname)s.${PBS_JOBID}.log

module unload PrgEnv-cray
module load PrgEnv-gnu
module unload python
module load anaconda

# Add fdrake module path
export FDRAKE_DIR=/work/y07/y07/fdrake
module use $FDRAKE_DIR/modules

# Load build environment
module load fdrake-build-env
module load fdrake-python-env

export FIREDRAKE_FFC_KERNEL_CACHE_DIR=$WORK/firedrake-cache
export PYOP2_LAZY=0
export PYOP2_BACKEND_COMPILER=gnu
export PYOP2_CACHE_DIR=$WORK/pyop2-cache
export PYOP2_SIMD_ISA=avx
export PYTHONPATH=$WORK/git_workspace/firedrake-bench:$WORK/git_workspace/pybench:$PYTHONPATH
export PETSC_OPTIONS=-log_summary
# Prevent matplotlib from accessing /home
export HOME=$WORK
export XDG_CONFIG_HOME=''

# MPI (man intro_mpi)
export MPICH_NEMESIS_ASYNC_PROGRESS=MC
export MPICH_MAX_THREAD_SAFETY=multiple
export MPICH_CPUMASK_DISPLAY=1

export LD_LIBRARY_PATH=$ANACONDA_LIB:${LD_LIBRARY_PATH}

echo -n Started at
date

echo
echo Running %(script)s
echo

aprun -n %(ptotal)d -N %(ppn)d -S %(pnuma)d python %(script)s.py %(args)s 2>&1 | tee $LOGFILE

echo -n Finished at
date