Language Detector
This golang library provides functionality to analyze and recognize language based on text.
The implementation is based on the following paper:
N-Gram-Based Text Categorization
William B. Cavnar and John M. Trenkle
Environmental Research Institute of Michigan P.O. Box 134001
Ann Arbor MI 48113-4001
Language profile
A language profile is a map[string] int
that maps n-gram tokens to its occurrency-rank. So for the most
frequent token 'X' of the analyzed text, map['X'] will be 1.
Usage
Detect
Get the closest language:
The default detector supports the following languages: Arabic, English, French, German, Hebrew, Russian, Turkish
detector := langdet.NewDefaultDetector()
testString := "do not care about quantity"
result := detector.GetClosestLanguage(testString)
fmt.Println(result)
output:
english
by setting the value langdet.MinimumConfidence (0-1), you can set the accepted confidence level. E.g. 0.7 --> if langdet is 70% or higher sure that the language matches, return it, else it returns 'undefined'
Get Language Probabilities
GetClosestLanguage will return the language that most probably matches. To get the result of all analyzed language, you can use GetLanguage, which will return you all analyzed languages and their percentage of matching the input snippet
testString := "ont permis d'identifier"
GetLanguages returns:
french 86 %
english 79 %
german 71 %
turkish 54 %
hebrew 39 %
arabic 8 %
russian 5 %
Use default languages
In order to use default languages, the file default_languages.json must be placed in the same directory as the binary. Alternatively it can be anywhere on the filesystem and initialized by calling InitWithDefault with the filepath.
Analyze new language
For analysing a new language random Wikipedia articles in the target languages are ideal. The result will be a Language object, containing the specified name and the profile example:
language := langdet.Analyze(text_sample, "french")
language.Profile // language profile in form of map[string]int as defined above
language.Name // the name that was given as parameter
Add more languages
New languages can directly be analyzed and added to a detector by providing a text sample:
text_sample := GetTextFromFile("samples/polish.txt")
detector.AddLanguageFrom(text_sample, "polish")
The text sample should be bigger then 200kb and can be "dirty" (special chars, lists, etc.), but the language should not change for long parts.
Alternatively Analyze can be used and the resulting language can added using AddLanguage method:
text_sample := GetTextFromFile("samples/polish.txt")
french := langdet.Analyze(text_sample, "french")
//language can be added selectively to detectors
detectorA.AddLanguage(french)
detectorC.AddLanguage(french)
Contribution
Suggestions and Bug reports can be made through Github issues. Contributions are welcomed, there is currently no need to open an issue for it, but please follow the code style, including descriptive tests with GoConvey.
License
Licensed under Apache 2.0.