[firedrake] Installing Firedrake on an HPC machine

[Justin Chang]

Lawrence,

When I compile everything with MPICH-3.1.4 on this machine, I get no
complains whatsoever. It only happens when I use OpenMPI. I don't like the
default binding options (or lack thereof) for MPICH and would prefer to use
OpenMPI. Could this have something to do with the compilers that I am
using? And/or how I am configuring openmpi and/or python?

I could try this out on another HPC system i have access to (Intel Xeon
E5-2670) to see if I can reproduce the problem, but this other machine has
a firewall and makes the installation process even more troublesome...

Justin

On Tue, Aug 11, 2015 at 3:37 AM, Lawrence Mitchell <
lawrence.mitchell at imperial.ac.uk> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 10/08/15 18:16, Justin Chang wrote:
> > Hi Lawrence,
> >
> > After several trial and run experimentations these last few days,
> > I still get the warning even with the pre-fork branches. However,
> > if I use our system's intel-compiled libraries and compilers, I get
> > no such errors when using mpirun. I suspect that this may have
> > something to do with the HPC system that I am running on.
>
> Reading between the lines, it seems to be some bad interaction between
> OpenMPI, forking, and the infiniband OpenFabrics transport.  I have
> filed a bug that will remind us about it: I think our local HPC system
> also has the same setup, but I have not run there recently.
>
> > But I am now running into a stranger issue:
> >
> > I have attached an RT0 code that I am working with. Basically, it
> > takes as input the seed size (aka the number of cells to generate
> > in each spatial direction). The command line argument goes like
> > this: "mpirun -n 1 python Test_RT0.py <seed>"
> >
> > I get strange errors when the seed number changes. I ran all
> > problems with -log_trace. For <seed> = 3, i get this:
> >
> > $ python Test_RT0.py 3
>
> ...
>
> > pyop2.exceptions.CompilationError: Command "['mpicc', '-std=c99',
> > '-fPIC', '-Wall', '-g', '-O3', '-fno-tree-vectorize',
> > '-I/home/jchang23/petsc-dev/include',
> > '-I/home/jchang23/petsc-dev/arch-linux2-c-opt/include',
> > '-I/home/jchang23/firedrake-deps/firedrake/firedrake',
> > '-I/home/jchang23/firedrake-deps/PyOP2/pyop2', '-msse', '-o',
> > '/tmp/pyop2-cache-uid3003/32e0bd01cb649f218f2092c503c1d41f.so.tmp',
> >
> >
> '/tmp/pyop2-cache-uid3003/32e0bd01cb649f218f2092c503c1d41f.c',
> > '-shared', '-L/home/jchang23/petsc-dev/lib',
> > '-L/home/jchang23/petsc-dev/arch-linux2-c-opt/lib',
> > '-Wl,-rpath,/home/jchang23/petsc-dev/lib',
> > '-Wl,-rpath,/home/jchang23/petsc-dev/arch-linux2-c-opt/lib',
> > '-lpetsc', '-lm']" returned with error.
> >
> > Unable to compile code
>
> This error, I think, comes from using intel compilers, but passing gcc
> options.  Can you try with:
>
> parameters["pyop2_options"]["compiler"] = "intel"
>
> which has some prebaked options for the intel compiler.
>
> > For <seed>=5 I get this:
>
> >
> > petsc4py.PETSc.Error: error code 77
> >
> > [0] DMPlexGenerate() line 1080 in
> > /home/jchang23/petsc-dev/src/dm/impls/plex/plexgenerate.c
> >
> > [0] DMPlexGenerate_CTetgen() line 834 in
> > /home/jchang23/petsc-dev/src/dm/impls/plex/plexgenerate.c
> >
> > [0] TetGenTetrahedralize() line 21483 in
> >
> /home/jchang23/petsc-dev/arch-linux2-c-opt/externalpackages/ctetgen/ctetgen.c
> >
> >  [0] TetGenMeshDelaunizeVertices() line 12113 in
> >
> /home/jchang23/petsc-dev/arch-linux2-c-opt/externalpackages/ctetgen/ctetgen.c
> >
> >  [0] TetGenMeshDelaunayIncrFlip() line 12046 in
> >
> /home/jchang23/petsc-dev/arch-linux2-c-opt/externalpackages/ctetgen/ctetgen.c
> >
> >  [0] TetGenMeshInsertVertexBW() line 11559 in
> >
> /home/jchang23/petsc-dev/arch-linux2-c-opt/externalpackages/ctetgen/ctetgen.c
> >
> >  [0] TetGenMeshInSphereS() line 5411 in
> >
> /home/jchang23/petsc-dev/arch-linux2-c-opt/externalpackages/ctetgen/ctetgen.c
> >
> >  [0] Petsc has generated inconsistent data
> >
> > [0] This is wrong
> >
> >
> > I am very confused by these strange results. Any explanation for
> > these?
>
> This one, I /think/ I've seen before.  I believe that the intel
> compiler produces bad code for ctetgen with optimisations on.  I think
> this is plausibly a petsc-only problem, try with the same setup but do:
>
> from petsc4py import PETSc
> bdy = PETSc.DMPlex().create()
> bdy.setDimension(2)
> bdy.createCubeBoundary([0, 0, 0], [1, 1, 1], [5, 5, 5])
> dm = PETSc.DMPlex().generate(bdy)
>
> I don't have access to the intel compiler stack here, but maybe it's
> worth reporting the bug in this form (assuming it shows up) to
> petsc-maint.
>
> If you're able to use gmsh to build the appropriate meshes, that is
> probably a more robust option.  Alternately you could try (although I
> never have) using tetgen (rather than ctetgen).  I think you'll need
> to configure PETSc with --with-clanguage=c++ --download-tetgen (rather
> than --download-ctetgen).
>
> Cheers,
>
> Lawrence
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
>
> iQEcBAEBAgAGBQJVybRbAAoJECOc1kQ8PEYvvS0IALzJjuMdyjvdn5H4RQtNAuHy
> UiMVhPWqE6F9iIeTVBY+Or5YGjlTKFHBXKjZCl5Fs3TgK8GDsn0COuoy+9TWzMR9
> Kn/XnxZuN1NYGwuiPlT8plahMGc2jAkI7peiPJsQyAfOfVA6nlBi6mDLiEDHuuvX
> 7g/9P/ZcWu89D9/9m/mfNTRZq9anF5zcXjIHHJzojdlg8DBWviAFO7gVpqivrD72
> Zzr+Q3xaFKQI9deNkjFacjJPJT/OUkBKAf/n4BD5wNAPt6z/TCyTJ4fTdeZ4lY2I
> 2m6IDyJE4RNlmaPfzqq8D91T/kswy0wbC3IrMwTOGG47GoEYuOXqe45llZ/+M4U=
> =HGNY
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> firedrake mailing list
> firedrake at imperial.ac.uk
> https://mailman.ic.ac.uk/mailman/listinfo/firedrake
>
-------------- next part --------------
HTML attachment scrubbed and removed