[firedrake] disentangling solver setup costs

[Eike Mueller]

Dear firedrakers,

I’ve been trying to disentangle the setup costs in my various solvers again. Recall that I solve a velocity/pressure system and then precondition it with a Schur-complement preconditioner, which is either AMG or geometric multigrid for the pressure system. However, to solve the velocity/pressure system I need to assemble the u/p operator at some point and this cost will be the same in both solvers.

In my matrix-free code I assemble matrices for the velocity/pressure system and then apply them as follows in my geometric multigrid:

mat = assemble(EXPRESSION) # line 1
mat_handle = M.handle # line 2

with r.dat.vec as v:
  with u.dat.vec_ro as x:
     mat_handle.mult(x,v)

I find that the really expensive bit is line 2. Is this the correct way of doing it? The setup seems to be much more expensive than the setup of the corresponding LinearVariationalSolver which I construct for the PETSc fieldsplit solver.

In the PETSc solver (with AMG preconditioner) on the other hand I also assemble an operator for the mixed velocity/pressure system in the corresponding setup routine (in this routine I assemble the velocity/pressure operator and form a LinearVariationalSolver from it, and tell it to use the fieldsplit preconditoner). If I measure the time for this, it is much less than the setup time in the corresponding geometric multigrid operator. However, I then notice that the first AMG iteration is much more expensive than the next ones, so my suspicion is that this contains the setup costs for both the velocity/pressure system and for the AMG.

To make any sense of my results, I really need to disentangle this somehow, and figure out what is AMG setup time and what is the cost of setting up the mixed operator. I might simply do something wrong in the setup of my mixed system.

Thanks a lot,

Eike