[firedrake] Hardware counters for Firedrake

[Lawrence Mitchell]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Justin,

On 09/07/15 09:18, Justin Chang wrote:
> Hi everyone,
> 
> I am very new to Firedrake, i recently switched over from FEniCS,
> so pardon me if this is a newbie question.
> 
> If I wanted to run hardware performance counters from PAPI to
> diagnose my firedrake application how would I go about doing this?
> In FEniCS, my application was written in C++ and I included the
> PAPI directives and libraries in the CMakefile. With this I was
> able to specify where in the source code I wanted to start and stop
> the counters.
> 
> Is there a way to somehow replicate this if my code is written
> using Firedrake?

This is somewhat tricky, since we generate all code to spin over the
mesh for assembly at runtime, and the calls into PETSc happen purely
from petsc4py.  There are two approaches which will have slightly
different measurement characteristics:


1. Measure from Python

I wrote some bindings for the high-level PAPI API that are callable
from Python.  They seem to be broadly functional.  See

https://github.com/firedrakeproject/PyPAPI

To use these you do:

import pypapi
import numpy as np

events = np.asarray([pypapi.Event.TOT_CYC], dtype=pypapi.EventType)

counts = np.zeros(1, dtype=pypapi.CountType)

pypapi.start_counters(events)
# Do your computation (for example, assembling an operator)
...
pypapi.stop_counters(counts)

# counts now contains the number of cycles elapsed between start
# and stop

Note that this approach includes cycle/flop counts and timings for the
full stack of computation: that is, it counts work in the python
interpreter as well as the low-level C that does the assembly loop.
In some sense, this is the fairest thing to report (since that's the
work your code is actually doing).

2. Measure in the generated code

This gives you finer-grained control over where you want to actually
count things, and gets you much closer to the metal.  However, it's
(much) trickier to do.

Firedrake defers to PyOP2 to do computation on meshes, and it is PyOP2
which generates the code to do this.  If you want to insert PAPI calls
here there are two things to consider:

1. Where exactly you want to measure.  I guess you'll want to measure
around a complete assembly over the mesh, rather than the execution of
an individual kernel.

2. How to get the information back out again so you can inspect it.

As to the former, I think it should be possible to extend the code
generation to optionally insert the correct PAPI calls.  Returning the
information is also probably not too difficult, we can stuff the
results in a Python dict (or similar) that you can then query from PyOP2.


My recommendation is to try the pure Python approach first.  If you
want finer-grained control, we can provide guidance and/or help on how
to best do that.

Cheers,

Lawrence

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJVnmvxAAoJECOc1kQ8PEYvSqYH/iL2eE6Vz5OF9yG8yIFRAghN
h6VOYZ5GgA420U8k6sjP1fBdHwidAUV8gtVLhBuj7UskTWf/HwSVvfUwf1KbPUie
fAzopYfuDXVolR3uYnUX5GDYDin1gkCCRRbOrk0wU98bSOyaRmcgmg0l3aDp4VTO
zHruPKzxyhgqoaWqcq5y7HFTCa3TYPBfdoJ3mu3zK4U8AWKMnuhKq24BEA6pBZfV
fSf4MvlCMKSmxOFTHYeOuPExmtM3m4y049Me88uxRi0gS3fTb1Yhhuhi9zF9DRXC
YIt1xKjZ+kudwiII22MA35rlXYjqxtShqMDuPC2HZi3xuIuANEWIGAE9uqhx4aY=
=HJsh
-----END PGP SIGNATURE-----