I have a string tokenization algorithm written in javascript and requirement is to convert this into Golang and expose REST API. Algorithm is in two parts. First part does segmentation of strings based on the defined rules and second part tokenizes strings based on the dictionary data using FMM (Forward Maximum Matching) algorithm. Will provide source files on your interest. Thank you.