For higher order interactions, LCBP and LCBPI give bad results, in particular their RNDSEQ
update schemes, which yield unreproducible results of bad quality. For example,

# hoi4.fg
# METHOD                                      CLOCKS     MAX ERROR   AVG ERROR  LOGZ ERROR     MAXDIFF
JTREE_HUGIN                                        0    
HAK_MIN                                            1     1.048e-09   1.000e-09   1.000e-09   1.000e-09
BP_SEQMAX                                          0     1.219e-02   4.162e-03  -2.149e-03   1.000e-09

LCBP_UNICAV_SEQFIX                                 0     2.212e-02   4.227e-03         nan   1.000e-09
LCBPI_UNICAV_SEQFIX                                1     1.601e-02   4.253e-03         nan   1.000e-09
LCBPJ_UNICAV_SEQFIX                                0     1.642e-02   7.639e-03         nan   1.000e-09
LCBPK_UNICAV_SEQFIX                                1     1.000e-09   1.000e-09         nan   1.000e-09
LCBPL_UNICAV_SEQFIX                                2     1.000e-09   1.000e-09         nan   1.000e-09

LCBP_UNICAV_SEQRND                               400     1.340e-01   7.365e-02         nan   8.949e-03
LCBPI_UNICAV_SEQRND                                1     1.680e-01   5.742e-02         nan   1.000e-09
LCBPJ_UNICAV_SEQRND                                0     1.000e-09   1.000e-09         nan   1.000e-09
LCBPK_UNICAV_SEQRND                                2     1.000e-09   1.000e-09         nan   1.000e-09
LCBPL_UNICAV_SEQRND                                1     1.000e-09   1.000e-09         nan   1.000e-09

LCBP_FULLCAV_SEQFIX                                0     1.000e-09   1.000e-09         nan   1.000e-09
LCBPI_FULLCAV_SEQFIX                               1     1.000e-09   1.000e-09         nan   1.000e-09
LCBPJ_FULLCAV_SEQFIX                               0     1.000e-09   1.000e-09         nan   1.000e-09
LCBPK_FULLCAV_SEQFIX                               0     1.000e-09   1.000e-09         nan   1.000e-09
LCBPL_FULLCAV_SEQFIX                               1     1.000e-09   1.000e-09         nan   1.000e-09

LCBP_FULLCAV_SEQRND                              398     1.340e-01   6.971e-02         nan   8.949e-03
LCBPI_FULLCAV_SEQRND                               1     1.655e-01   5.519e-02         nan   1.000e-09
LCBPJ_FULLCAV_SEQRND                               1     1.000e-09   1.000e-09         nan   1.000e-09
LCBPK_FULLCAV_SEQRND                               1     1.000e-09   1.000e-09         nan   1.000e-09
LCBPL_FULLCAV_SEQRND                               2     1.000e-09   1.000e-09         nan   1.000e-09


This is a relatively simple factor graph of 3 factors of connectivity 3 with a
single loop.  Note that the difference between HAK_MIN and BP_SEQMAX is large.
Note that LCBP_{UNI|FULL}CAV_SEQRND and LCBPI_{UNI|FULL}CAV_SEQRND yield
irreproducible results.  What is strange is that LCBPJ_UNICAV_SEQRND gives a
good result, but LCBPJ_UNICAV_SEQFIX not. Note that LCBPK and LCBPL give
good results for both SEQFIX and SEQRND, for FULLCAV and even for UNICAV.

This means that LCBP has to be replaced by LCBPK and LCBPI by LCBPL, which
seem to be the correct higher order generalizations. The other methods are
having problems with degrees freedom, apparently.

Note that HAK_MIN gives strange results if the factor graph is not connected.



We should test the following:
  LCBP_SEQFIXED is always faster than LCBP_SEQRANDOM
  NOINIT is always faster than INIT


alarm.fg

- Everything converges on alarm.fg, except for LCBP-II (because of the zeroes in the factors)
- Starting from complete cavity dists, LCBP-I makes errors *larger*!
  LCBP-III does make them smaller (often at least one order of magnitude, but not always).
- Starting from second order cavity dists, LCBP-I yields really bad results.
  LCBP-III results are very good (except for SEQMAXRES scheme, which stops way too early).

    BP_SEQFIXED                                        0          2.026e-01          8.136e-03
    LCBP-III LCBP_SEQFIXED                          1300          5.444e-04          1.514e-05
    LCBP-III LCBP_SEQFIXED  2ndO                    4051          3.317e-02          9.006e-04
- After defining x/0 = 0, also LCBP-II works! It gives identical results as (but approx 3x than) LCBP-III.
- INITCAVBP3 gives a slightly lower error for the initial cavity distributions and gives faster
  convergence for both LCBP-I and LCBP-II (for LCBP-II almost a factor of 2). INITCAVBP3 does only
  take marginally longer for initializing cavities than INITCAVBP2.


alarm2.fg

- Everything converges on alarm2.fg, except for LCBP-II (becase of the zeroes in the factors)
  and, remarkably, MF.
- Starting from complete cavity dists, LCBP-I makes large errors smaller and makes small errors larger....
  LCBP-III results are very good, sometimes giving an error two magnitudes smaller than LCBP-I!
  Only LCBP-III with SEQMAXRES stops too earyl, apparently.
- Starting from second order cavity dists, LCBP-I yields quite bad results.
  LCBP-III results are very good (except for SEQMAXRES scheme, which stops way too early).

    BP_PAR                                          2          2.031e-01          7.590e-03
    LCBP-III LCBP_SEQFIXED                          1198          6.098e-05          1.678e-06
    LCBP-III LCBP_SEQFIXED  2ndO                    9660          3.468e-02          9.427e-04


alarm.fg vs. alarm2.fg

We can conclude that merging potentials may not be necessary, although it may help to obtain higher accuracy.


testslow1.fg

- MF converges

- BP_SEQFIXED does not converge
- BP_SEQRANDOM does not converge
- BP_SEQMAXRES converges
- BP_PAR does not converge

- GBP (CVM) gets NANs
- HAK (CVM) converges

- LCBP_INITCAVBP *seems* to have no convergence problems for any BP method, but the results and the timings
  are largely inconsistent. Again BP_SEQMAXRES has the smallest errors.
  Unfortunately, the cavity distributions used further on are those of BP_SEQFIXED...


promedas/examples/chain*.fg

- NO_INIT is always faster than INIT (NO_INIT needs 40% to 85% of the time INIT needs).
  To extract this data from the output files of testnew, you can use:
    for i in `seq 1 9`; do tail -70 chain$i.out.2 | grep "INIT[^_]" | sort | awk '{print $4}' | sed '$!N;s/\n/ /' >> inits.dat; done
- INITCAVBP3 always yields a smaller maximum error than INITCAVBP2
    for i in `seq 1 9`; do tail -80 chain$i.out.2 | grep "INITCAVBP[^ ] BP_SEQFIXED     INIT" | awk '{print $5}' | sed '$!N;s/\n/ /'; done
  INITCAVBP3 often yields a smaller average error than INITCAVBP2
    for i in `seq 1 9`; do tail -80 chain$i.out.2 | grep "INITCAVBP[^ ] BP_SEQFIXED     INIT" | awk '{print $6}' | sed '$!N;s/\n/ /'; done
