[Nektar-users] memory usage and computation time in Taylor-Green problem

[Vishal Saini]

Dear Spencer,

Thank you for your reply. Yes, I have compiled Nektar++ with MPI (Although
I get a warning on the cluster: WARN  Conflicting CPU frequencies detected,
using: 2900.06. The admin claims it's harmless and they're working on it).

For reference, I'm attaching the output log in compact and verbose mode,
where the code seems to partition the mesh (??).

Command run:
"""










*JobID: 124250======Time: Thu 15 Mar 12:09:00 GMT 2018Running on master
node: node0367Current directory:
/gpfs/home/lboro/ttvs3/work_Saini/cases_Nektar/tgv/Re1600_LES/N5/256cubenumtasks=112,
numnodes=4, mpi_tasks_per_node=28 (OMP_NUM_THREADS=1)Executing
command:==================time mpirun -npernode 28 -np 112
IncNavierStokesSolver -v 2pi_domain_64cubeEle_mesh_copyPaste.xml
tgv_conditions_lowTolerance.xml > log_124250*

"""

It is a bit unlikely that there are 112 instances running in parallel
because it is 256^3~17M degrees of freedom case and each instance most
probably is bigger than 450GB/112~4GB.

Hope these are helpful.

Regards,
Vishal


On Sun, Mar 25, 2018 at 7:09 PM, Sherwin, Spencer J <
s.sherwin at imperial.ac.uk> wrote:

> Dear Vishal,
>
> Sorry for not getting to your comments earlier. Can I first confirm you
> have indeed compiled the code in parallel since what you describe sounds as
> if the code may have been compiled in serial and is running 112 versions of
> the case?
>
> To turn on parallelisation you need to enable the NEKTAR_USE_MPI in the
> cmake step.
>
> Best regards,
> Spencer.
>
>
> On 19 Mar 2018, at 17:21, Vishal Saini <vishal.saini.nitj at gmail.com>
> wrote:
>
> Hi all,
>
> After obtaining exciting error vs. computational cost results from
> relatively heavy 2D computations, I've been setting up the 3D Taylor Green
> vortex (TGV) problem using Nektar++ Incompressible solver. The Reynolds
> number used is 1600 (same as in the TGV tutorial from Nektar++ team). A 3D
> grid of 64^3 elements with NumModes=5 was used to obtain a 256^3 simulation
> (close to a DNS). I ran this problem on Midlands+ Tier 2 machine using 4
> nodes (112 cores, 512 GB RAM in total).
> http://www.hpc-midlands-plus.ac.uk/about/system-description/
>
> The memory consumption during the run was ~450GB and it took ~330
> wall-clock-minutes for 1000 time-steps (with dt=1e-4). For a reference, I
> ran a 256^3 TGV simulation of the same case in OpenFOAM using same
> resources. The memory consumption was ~40GB and it took ~20 minutes for
> 1000 time-steps.
>
> So for this case, I observe high resource consumption in terms of memory
> and computation time. I was not expecting this on the basis of my
> experience with Nektar++ on 2D simulations, at least in terms of
> computation time.
>
> Is this normal? In addition to my set-up file, I'm suspecting my
> installation of Nektar++ on the aforementioned cluster. Could anyone please
> try and run a few hundred time-steps on their machines using the files
> attached? FYI, I'm partitioning on-the-fly. The calculation seems to scale
> well on 8 nodes in terms of wall clock time while increasing the memory
> consumption by ~10%.
>
> Any input is highly appreciated.
>
> Best regards,
> Vishal
>
>
> ---
> * Vishal SAINI*
> Master of Research,
> University of Cambridge.
> <nektar_user_tgv.tar.gz>_______________________________________________
> Nektar-users mailing list
> Nektar-users at imperial.ac.uk
> https://mailman.ic.ac.uk/mailman/listinfo/nektar-users
>
>
> Spencer Sherwin FREng, FRAeS
> Head, Aerodynamics,
> Professor of Computational Fluid Mechanics,
> Department of Aeronautics,
> Imperial College London
> South Kensington Campus,
> London, SW7 2AZ,  UK
> s.sherwin at imperial.ac.uk
> +44 (0)20 7594 5052 <+44%2020%207594%205052>
> http://www.imperial.ac.uk/people/s.sherwin/
>
>
-------------- next part --------------
HTML attachment scrubbed and removed
-------------- next part --------------
A non-text attachment was scrubbed...
Name: log.dat
Type: video/mpeg
Size: 7019 bytes
Desc: not available
URL: <https://mailman.ic.ac.uk/mailman/private/nektar-users/attachments/20180326/373dd50d/attachment.m1v>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: log_verbose.dat
Type: video/mpeg
Size: 10013 bytes
Desc: not available
URL: <https://mailman.ic.ac.uk/mailman/private/nektar-users/attachments/20180326/373dd50d/attachment-0001.m1v>