Hi,

  I am trying to use SphinxTrain to train up some English acoustic models for a project.  I have trained models with SphinxTrain on numerous prior occassions but this time I encountered a new error that I'm unfamiliar with.

  I have about 200hrs of speech data with transcripts and appropriate lexicon, etc., I initially went through a training run with a small fraction of this data (=~2%) just to confirm that things were more or less setup correctly.  This completed successfully so I started a run with the full corpus.

   The full corpus setup made it through the first several steps,

--------------------------------------

    ("$ST::CFG_SCRIPT_DIR/00.verify/verify_all.pl",
     "$ST::CFG_SCRIPT_DIR/02.falign_ci_hmm/slave_convg.pl",
     "$ST::CFG_SCRIPT_DIR/03.force_align/slave_align.pl",
     "$ST::CFG_SCRIPT_DIR/20.ci_hmm/slave_convg.pl",
     "$ST::CFG_SCRIPT_DIR/30.cd_hmm_untied/slave_convg.pl",
     "$ST::CFG_SCRIPT_DIR/40.buildtrees/slave.treebuilder.pl",
     "$ST::CFG_SCRIPT_DIR/45.prunetree/slave.state-tying.pl",

---------------------------------------

   but eventually bombed out on

     "$ST::CFG_SCRIPT_DIR/50.cd_hmm_tied/slave_convg.pl",

   on 4 gaussians, giving the following error,

----------------------------------------------------

....

INFO: corpus.c(1346): Will process 17768 utts starting at 53304
INFO: main.c(622): Reestimation: Baum-Welch
bw: gauden.c:1377: gauden_scale_densities_bwd: Assertion `finite(den[c][j][k])' failed.
....

----------------------------------------------------

  I tracked this down in the source code to,

gauden.c:

gauden_scale_densities_bwd(...){
....
/* BHIKSHA converged g->n_density to g->n_top; possible bugfix, APR 6 98 */
for (k = 0; k < g->n_top; k++) {
/* BHIKSHA converged g->n_density to g->n_top; possible bugfix, END */
den[c][j][k] = EXPF(den[c][j][k] - scl[j]);
assert(finite(den[c][j][k]));
...
}

Where the above assertion seems to be what is causing the failure, however I'm uncertain as to what this implies about my training data, or the process itself.

The only thing I'm doing that is any different from previous training exercises or projects is that I'm using Queue::POSIX and npart > 1, however it seems unlikely that this should be the cause of the problem. 

Any pointers as to what this error implies would be greatly appreciated!

--- (Edited on 8/23/2010 7:42 pm [GMT-0500] by bkbroiler) ---