6957e7a077
Refactoring to remove some duplication of probability tables between tokenization and detokenization. Change-Id: I2fc6a6497f9c0410021a9b41f828bc58a864e466