Hi Mike,

As you sent me the same email as posted here I'll reply here - that save you replying back here.

The ARPA language model format is text only.   This is the format you are expecting.   However, as you can imagine sometimes these get very big (I have one I'm working with right now which is 362G).   In order to make these more usable there are a variety of binary formats.  HTK has chosen to keep the unigrams the same and write the rest in binary, hence you can't read it as text.

You can tell from the header you have a binary LM file because instead of the normal ngram size using a = sign like this:

ngram 2=127902353

you will see

ngram 2~127902353

For more details refer to the HTK book.

Tony

--
** Cantab is hiring: www.cantabResearch.com/openings **
Dr A J Robinson, Founder, Cantab Research Ltd
Phone direct: 01223 977211, office: 01223 794497
St Johns Innovation Centre, Cowley Road, Cambridge, CB4 0W

--- (Edited on 15-December-2013 8:15 am [GMT+0000] by TonyR) ---

Hi, Dr. Robinson,

Thank you so much for your help. I solved this issue with your help. But I noticed that when building a language model, HTK counts <s> and </s>, and computes their probability also. In a bigram, there is even a pair </s> <s> in the list. But in fact <s> and </s> are only starting and ending of a sentence, and means nothing by their probability. So if I use the language model with <s> and </s> in decoding a sentence, will they mess up the result?

Thank you very much.

Mike

--- (Edited on 12/15/2013 8:14 am [GMT-0600] by Visitor) ---

Mike,

Iniitailly you should think that <s> starts an utterance and </s> ends it, irrespective of how many sentences are in the utterence.

Historically we build decoders that worked on one sentence at a time, so an utterence was a sentence.   Now of course we have moved on.   However, if you get things working with "<s> starts an utterance and </s> ends it" then the rest will fall out.

Tony

--
** Cantab is hiring: www.cantabResearch.com/openings **
Dr A J Robinson, Founder, Cantab Research Ltd
Phone direct: 01223 977211, office: 01223 794497
St Johns Innovation Centre, Cowley Road, Cambridge, CB4 0W

--- (Edited on 15-December-2013 2:33 pm [GMT+0000] by TonyR) ---

Hi, Dr Robinson,

Thank you very much for your help. I think I am not too sure about what you mean by " fall out ".

Do you mean having <s> and </s> in the model won't cause any issue to the decoder?

In my experiments, the input to the decoder is still one sentence each time. I just want to make sure that if I use the model with <s> and </s> assigned actual probability when being trained, the decoder will not think of <s> and </s> as actual words and output a decoded sentence that contains them somewhere, and counts them toward the final accuracy of the system. In other words, I just want <s> and </s> to work in the same way as !ENTER and !EXIT to the decoder.

By the way, if the decoder can not identify <s> and </s> as start and end of a sentence, I think I can put an entry in my dictionary <s> [] sil, </s> [] sil, so the decoder will not output anything then.

Thank you very much again.

Mike

--- (Edited on 12/15/2013 1:56 pm [GMT-0600] by weedwind) ---

Yes, you can make <s> and </s> work in just the same way as !ENTER and !EXIT.

You can either use [] so that these aren't output, or you can call HResults with  -e '???' '<s>' -e '???' '</s>' so that they do not contribute to the scores.

Tony

--
** Cantab is hiring: www.cantabResearch.com/openings **
Dr A J Robinson, Founder, Cantab Research Ltd
Phone direct: 01223 977211, office: 01223 794497
St Johns Innovation Centre, Cowley Road, Cambridge, CB4 0W

--- (Edited on 16-December-2013 9:18 am [GMT+0000] by TonyR) ---

Hi, Dr. Robinson,

Thank you very much for your help.  I've generated the trigram language model, and I want to run Hvite to do a N-best decoding for each utterance ( the task is TIMIT phoneme recognition ), and then using the lattice output by this step to do a rescoring with the trigram model. But when I run Hvite with the option -n 4 20 -z ..., a warning says Hvite will do a 1-best decoding, and the warning code means the N-best ( 20-best in my case) can not be done. I tried a couple of other numbers, like 2, 5, etc, they do not work also. Do you know where the issue is?

Thank you very much.

Mike

--- (Edited on 12/17/2013 6:47 pm [GMT-0600] by weedwind) ---

Hi, Dr. Robinson,

When I run HLRescore, it keeps telling me " word 0 not in LM wordlist in HLRescore". I looked up this problem online, someone says that I need to have !NULL in my trigram model. So I added this entry into my model, but then HTK says getfloat error, line 1, colomn 9, char 8 needs to be a float number... I am really confused. Have you ever met the " word 0" error? Do you have any solutions?

Thank you very much.

Mike

--- (Edited on 12/17/2013 11:34 pm [GMT-0600] by weedwind) ---

Yes, I've met the "word 0" error.   Have you found http://speech-research.com/htkSearch/index.php - there are a whole page of answers from the HTK mailing list there.

Tony

-- 

** Cantab is hiring: www.cantabResearch.com/openings **
Dr A J Robinson, Founder, Cantab Research Ltd
Phone direct: 01223 977211, office: 01223 794497
St Johns Innovation Centre, Cowley Road, Cambridge, CB4 0W

--- (Edited on 18-December-2013 8:46 am [GMT+0000] by TonyR) ---

Hi, Dr. Robinson,

I've searched over that list, and I've also tried adding !NULL to the trigram model...It seems that the only way to make it work is to modify the HTK source code as you suggested....

Mike

--- (Edited on 12/18/2013 3:48 am [GMT-0600] by weedwind) ---