/**
 * This file is part of Minerva Cluster and Distributed Algorithms software.
 * Copyright (c) 2010 Oliver Ford 
 * This software was developed independently and personally by Oliver Ford and 
 * is not currently licensed to anyone else for any use.
 */ 

LGI-DMATH: Linear Gaussian Inversion Distributed (matrix) Maths
---------------------------------------------------------------------------------

This is the system which does distributed matrix maths to calculate the linear gaussian inversion in a completely distributed way.

During the distributed calculation of the responses matrix M, the slaves keep their rows and do not return them to the master (reduces master<-->cluster bandwidth too).
When complete, each slave writes it rows of M to local disk, the master transmits the other required matrices to a chose slave (the primary slave). The primary slave 
save them to it's local disk. The master also sends the hostnames and local storage info (a.k.a "MPI app-file entries") of all the other slaves to the primary,
which uses that info to run the C/F77 'lgidmath' program across the participating slaves with 'mpirun'. The BLACS style distributed matrices are build out of the local parts inside
the MPI environment.

The total memory usage on any single slave should be significantly less than would ever be needed to store an actual (nData x nParams) or (nParams x nParams) matrix, 
allowing huge matrices to be treated across the machines. 

Only the posterior mean is ever returned to the master (since the posteriorCov is probably too big to ever have in one place anyway, let alone shove across the master<-->cluster link.
This in theory allows absurdly huge parameter spaces to be handled at a reasonable speed and without running out of memory on the master.

The data storage, distribution and the maths all use the PBLAS, BLACS and scaLAPACK libraries. Info is all here:
Jack J. Dongarra, David W. Walker, "SOFTWARE LIBRARIES FOR LINEAR ALGEBRA COMPUTATIONS ON HIGH PERFORMANCE COMPUTERS"
http://www.netlib.org/utk/papers/siam-review93/node9.html


Easier said then done ...


To use dMath, each slave needs to have the 'lgidmath' program in the minerva-cluster/csrc/lgidmath dir. There is a Makefile in there but you will need all of these native libraries:
MPI			- The more normal parallel processing communication system, (use openMPI). 
BLAS		- Basic vector/matrix multiplication etc.
LAPACK		- Matrix operations (decompositions etc)
BLACS		- Distributed data storage library.
PBLAS		- Parallel versions of BLAS routines.
scaLAPACK	- Parallel varsions of LAPACK routines.

All of these expect MPI can come from NETLIB.
PBLAS comes inside scaLAPACK, the rest are separate and a huge pain to compile/install. Get the scalapack-installer from NETLIB as it will happily do LAPACK,BLACS and SCALAPACK for you.


The principal then is for each slave, once the calculation is complete, to execute the C program to perform the matrix inversion.

TODOs:

1) The prior cov, which is also in theory a huge (nParams x nParams) array is usually mostly only diagonal, so is stored as a BlockSparseMatrix to make it quicker to
send it across the master<-->cluster link. It is then written out as a full dense matrix to the primary's local disk which transfers it to all other machines. This is slow
and woule probably be faster if C could handle a BlockSparse format of some kind and swap that around first.

2) Need some automatic and optimised way of deciding on the BLACS process grid shape and block size from the number of slaves that are alive (and have lgidmath) and the problem size.

3) Need some automatic way of deciding if dMath is going to be a waster of time (based on problem size) 

Some notes on compilation pain....
----------------------------------

All linux/unix, if anyone ever does it on a windows cluster (are you insane??), please update this with anything you can.

General proceedure:
Repeat until bored or it works:
	compile --> google --> compile --> google --> rearrange libs --> compile --> google.


BLACS only comes as static libs (.a) rather than dynamic (.so), everything else (especially ATLAS BLAS) seems to work better if you use the static ones anyway.

This is the list of libs I have to link ex5.c with to get it to work (in this exact order):
scalapack_installer_0.96/lib/blacsF77.a
scalapack_installer_0.96/lib/blacs.a
scalapack_installer_0.96/lib/blacsC.a
scalapack_installer_0.96/lib/libscalapack.a 
scalapack_installer_0.96/lib/libreflapack.a		# <-- that is scalapack_installer's downloaded reference lapack, which would otherwise be 'liblapack.a'
atlas/lib/libf77blas.a							# <-- because I use ATLAS compiled BLAS, if not you'll  
atlas/lib/libatlas.a							#		just need libblas of some fortran variety(cblas is not quite the same).
atlas/lib/libcblas.a
scalapack_installer_0.96/lib/blacs.a			# <-- Yes, 'blacs.a' does need to be in here twice, in these specific places, for some reason. 
-lmpi_f77

Be careful to use a full version of LAPACK, some (e.g the one that comes in ATLAS) are not complete and scaLAPACK will have undefined references.

The -lmpi_f77 is needed because scalapack uses the fortran BLACS, which uses the F77 MPI bindings. I seem to need to use the F77 BLACS binding myself in ex5.c, rather than the C ones, otherwise it segsfault. Despite that, the blacsC.a still needs to be in the libs list.

"undefined reference to `_gfortran_st_write'":
	You need to add '-lgfortran' because some bits (LAPACK, maybe others) were compiled with gfortran.

"undefined reference to `fio_src_info'":
	You need to add "-lpgftnrtl" because something was compiled with the Portland fortran compiler (e.g. BLAS on the JACs).

		
Now I remember why we use Java!

Atlas
------

For building ATLAS on Core i5:
add -A 16 to configure command.


Some notes on the weird use of MPI
-------------------------------------------

The primary slave will try to start the DMATH job with the mpirun command taken from "minerva.cluster.mpirun" out of minerva-settings.
It will pass it the argument --app {appfile} where appfile is filled in with --hosts for the hostnames given by each of the slaves to the master
and their local "minerva.cluster.linearGuassian.dMathCmd" from minerva-settings.

 

Oli ( codes<at>oliford.co.uk )
 