As I’ve noted before, it also passes punctuation unmodified, like this: One word, one tag, one lemma, one standard spelling. Recall that MorphAdorner training data usually looks like this: So I wrote a little script to list the tags used in the training data, and immediately discovered the problem. I emailed the helpful folks at Stanford, who pointed out that my training data (from MorphAdorner), in addition to being significantly larger than what they normally use (about 4M words instead of 1M 500+ word maximum sentence length instead of about half that), was using 420+ tags, which made training impossible under 4GB of physical memory.Ĥ20 tags was news to me NUPOS contains about 180 valid tags. The bad news is that the training process keeps crashing out of memory (given 3800 MB to work with) on data sets over about 1M words. The good news is that it works, more or less I can feed training data to it and get a trained model out, which I can then use to tag new text. Where we handle these issues is the main method in module MaxentTaggerServer.After a couple days off, I’ve been trying to run cross-validation on the Stanford tagger. TaggerConfig has been extended with additional parameter (serverPort) to handle the port to which the XML-RPC Service is listening to. Private static String getXMLWords(ArrayList s, int sentNum)įirst we need to do some initializations including loading the model, specifying the output format, encoding etc.Īll these parameters are to be loaded prior to start up the service, this is handled by an instance of TaggerConfig.java Private static void writeXMLSentence(Writer w, ArrayList s, int sentNum) Public void runTagger(BufferedReader reader, BufferedWriter writer, String tagInside, boolean stdin) Identify the relevant methods from MaxentTagger and copy them into the edu/stanford/main/MaxentTaggerServer.java. Create a module in which the XML-RPC service is to be started, in this case it is called edu/stanford/main/MaxentTaggerServer.java. Training has taken place and a model is generated. The service to be exposed is specified as follows: Given a (.txt) document and a model, words in provided document needed to be tagged.Īfter some investigation it turned out that the module. is the starting point for this service. test it with the latest XML-RPC library. Python client.py E:/Temp/stanford-postagger-full-/sample-input.txt 8090 E:\programs\tools\xmlrpc-1.2-b1\ TaggerClient E:\tmp\stanford-postagger-\sample-input.txt 8090 Java -ea TaggerClient E:\tmp\stanford-postagger-\sample-input.txt 8090 Java -ea TaggerClient // Default port 8000 is used Java -cp %TAGGER_RPC% %CLASSPATH% -model models\left3words-distsim-wsj-0-18.tagger -serverPort 8090 Java -cp %TAGGER_RPC% %CLASSPATH% -model See TaggerConfig.java for a list of relevant parameters set TAGGER_RPC=%TAGGER_HOME%\tagger_server.jar %TAGGER_HOME%\stanford-postagger-.jar E:\progtams\stanford-postagger-full-\tagger_server.jar Copy the tagger_server.jar into the stanford-postagger-full- home directory (TAGGER_HOME)Į.g. tagger_server.jar) and copy into stanford-postagger-full- home directory (TAGGER_HOME).
POS TAGGER STANFORD WINDOWS
It has been also tested on Windows XP and Windows 2000.ĭownload the Stanford Postagger XML-RPC Service source.Ĭompile and pack class files to a jar (e.g. The Service is developed in Java 1.6, on Eclipse under Win XP. (MaxentTagger.java requires this, shouldn't really need it since we do not make any changes, (?) maybe a Generic-related issue in the implementaion of the TypesafeMap and/or CoreMap) This is a copy of the module edu/stanford/nlp/ling/CoreAnnotations.java from Stanford POS Tagger, v. 3.0 - extended with serverPort parameterĤ) edu/stanford/nlp/ling/CoreAnnotations.java This is a copy of the module edu/stanford/nlp/tagger/maxent/TaggerConfig.java from Stanford POS Tagger, v. Public static void printErrWordsPerSec(long milliSec, int numWords)ģ) edu/stanford/nlp/tagger/maxent/TaggerConfig.java private static void printErrWordsPerSec(long milliSec, int numWords) Public static String getTsvWords(ArrayList s) private static String getTsvWords(ArrayList s) Public TokenizerFactory chooseTokenizerFactory() protected TokenizerFactory chooseTokenizerFactory() Visibilty for the three following methods have been changed.
This is a copy of the module edu/stanford/nlp/tagger/maxent/MaxentTagger.java from Stanford POS Tagger, v. The service is packed and delivered as a patchġ) edu/stanford/main/MaxentTaggerServer.javaĢ) edu/stanford/nlp/tagger/maxent/MaxentTagger.java 3.0 - ( )Ĭopyright (C) 2010-present, MetaOptimize LLC MaxentTaggerXML-RPC Service applicable to Stanford POS Tagger, v.
0 kommentar(er)
