[firedrake] Installing Firedrake on an HPC machine

[Lawrence Mitchell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/08/15 18:16, Justin Chang wrote:
> Hi Lawrence,
> 
> After several trial and run experimentations these last few days,
> I still get the warning even with the pre-fork branches. However,
> if I use our system's intel-compiled libraries and compilers, I get
> no such errors when using mpirun. I suspect that this may have
> something to do with the HPC system that I am running on.

Reading between the lines, it seems to be some bad interaction between
OpenMPI, forking, and the infiniband OpenFabrics transport.  I have
filed a bug that will remind us about it: I think our local HPC system
also has the same setup, but I have not run there recently.

> But I am now running into a stranger issue:
> 
> I have attached an RT0 code that I am working with. Basically, it
> takes as input the seed size (aka the number of cells to generate
> in each spatial direction). The command line argument goes like
> this: "mpirun -n 1 python Test_RT0.py <seed>"
> 
> I get strange errors when the seed number changes. I ran all
> problems with -log_trace. For <seed> = 3, i get this:
> 
> $ python Test_RT0.py 3

...

> pyop2.exceptions.CompilationError: Command "['mpicc', '-std=c99', 
> '-fPIC', '-Wall', '-g', '-O3', '-fno-tree-vectorize', 
> '-I/home/jchang23/petsc-dev/include', 
> '-I/home/jchang23/petsc-dev/arch-linux2-c-opt/include', 
> '-I/home/jchang23/firedrake-deps/firedrake/firedrake', 
> '-I/home/jchang23/firedrake-deps/PyOP2/pyop2', '-msse', '-o', 
> '/tmp/pyop2-cache-uid3003/32e0bd01cb649f218f2092c503c1d41f.so.tmp',
>
> 
'/tmp/pyop2-cache-uid3003/32e0bd01cb649f218f2092c503c1d41f.c',
> '-shared', '-L/home/jchang23/petsc-dev/lib', 
> '-L/home/jchang23/petsc-dev/arch-linux2-c-opt/lib', 
> '-Wl,-rpath,/home/jchang23/petsc-dev/lib', 
> '-Wl,-rpath,/home/jchang23/petsc-dev/arch-linux2-c-opt/lib',
> '-lpetsc', '-lm']" returned with error.
> 
> Unable to compile code

This error, I think, comes from using intel compilers, but passing gcc
options.  Can you try with:

parameters["pyop2_options"]["compiler"] = "intel"

which has some prebaked options for the intel compiler.

> For <seed>=5 I get this:

> 
> petsc4py.PETSc.Error: error code 77
> 
> [0] DMPlexGenerate() line 1080 in 
> /home/jchang23/petsc-dev/src/dm/impls/plex/plexgenerate.c
> 
> [0] DMPlexGenerate_CTetgen() line 834 in 
> /home/jchang23/petsc-dev/src/dm/impls/plex/plexgenerate.c
> 
> [0] TetGenTetrahedralize() line 21483 in 
> /home/jchang23/petsc-dev/arch-linux2-c-opt/externalpackages/ctetgen/ctetgen.c
>
>  [0] TetGenMeshDelaunizeVertices() line 12113 in 
> /home/jchang23/petsc-dev/arch-linux2-c-opt/externalpackages/ctetgen/ctetgen.c
>
>  [0] TetGenMeshDelaunayIncrFlip() line 12046 in 
> /home/jchang23/petsc-dev/arch-linux2-c-opt/externalpackages/ctetgen/ctetgen.c
>
>  [0] TetGenMeshInsertVertexBW() line 11559 in 
> /home/jchang23/petsc-dev/arch-linux2-c-opt/externalpackages/ctetgen/ctetgen.c
>
>  [0] TetGenMeshInSphereS() line 5411 in 
> /home/jchang23/petsc-dev/arch-linux2-c-opt/externalpackages/ctetgen/ctetgen.c
>
>  [0] Petsc has generated inconsistent data
> 
> [0] This is wrong
> 
> 
> I am very confused by these strange results. Any explanation for
> these?

This one, I /think/ I've seen before.  I believe that the intel
compiler produces bad code for ctetgen with optimisations on.  I think
this is plausibly a petsc-only problem, try with the same setup but do:

from petsc4py import PETSc
bdy = PETSc.DMPlex().create()
bdy.setDimension(2)
bdy.createCubeBoundary([0, 0, 0], [1, 1, 1], [5, 5, 5])
dm = PETSc.DMPlex().generate(bdy)

I don't have access to the intel compiler stack here, but maybe it's
worth reporting the bug in this form (assuming it shows up) to
petsc-maint.

If you're able to use gmsh to build the appropriate meshes, that is
probably a more robust option.  Alternately you could try (although I
never have) using tetgen (rather than ctetgen).  I think you'll need
to configure PETSc with --with-clanguage=c++ --download-tetgen (rather
than --download-ctetgen).

Cheers,

Lawrence
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJVybRbAAoJECOc1kQ8PEYvvS0IALzJjuMdyjvdn5H4RQtNAuHy
UiMVhPWqE6F9iIeTVBY+Or5YGjlTKFHBXKjZCl5Fs3TgK8GDsn0COuoy+9TWzMR9
Kn/XnxZuN1NYGwuiPlT8plahMGc2jAkI7peiPJsQyAfOfVA6nlBi6mDLiEDHuuvX
7g/9P/ZcWu89D9/9m/mfNTRZq9anF5zcXjIHHJzojdlg8DBWviAFO7gVpqivrD72
Zzr+Q3xaFKQI9deNkjFacjJPJT/OUkBKAf/n4BD5wNAPt6z/TCyTJ4fTdeZ4lY2I
2m6IDyJE4RNlmaPfzqq8D91T/kswy0wbC3IrMwTOGG47GoEYuOXqe45llZ/+M4U=
=HGNY
-----END PGP SIGNATURE-----