Skip to content

Morphosyntactic tagger

The Morphosyntactic Tagger is a tool allowing to tag the fields "title_tokens" and "content_tokens" of the tkeir document. This tools is a rest service. Morphosyntactic tagger is based on spacy toolbox.

Morphosyntactic tagger API

Morphosyntactic tagger configuration

Example of Configuration:

mstagger.json
{
    "logger": {
        "logging-level": "{{ project.loglevel }}"
    },
    "morphosyntax": {
        "taggers":[{
            "language":"en",
            "resources-base-path":"{{ project.path }}/resources/modeling/tokenizer/en",
            "mwe": "tkeir_mwe.pkl",
            "pre-sentencizer": true,
            "pre-tagging-with-concept":true,
            "add-concept-in-knowledge-graph":true
        }],
        "network": {
            "host":"0.0.0.0",
            "port":10002,
            "associate-environment": {
                "host":"MSTAGGER_HOST",
                "port":"MSTAGGER_PORT"
            }
        },
        "runtime":{
            "request-max-size":100000000,
            "request-buffer-queue-size":100,
            "keep-alive":true,
            "keep-alive-timeout":500,
            "graceful-shutown-timeout":15.0,
            "request-timeout":600,
            "response-timeout":600,
            "workers":1
        }
    }
}

Morphosyntactic tagger is an aggreation of network configuration, serialize configuration, runtime configuration (in field converter), logger (at top level). The segmenter configuration is a table containing path to Multiple Word Expression entries (MWE):

  • language :the language of tokenizer
  • resources-base-path: the path to the resources (containing file created by tools createAnnotationResources.py
  • mwe : the file containing MWE entries

Configure Morphosyntactic tagger logger

Logger is configuration at top level of json in logger field.

Example of Configuration:

logger configuration
{
    "logger": {
        "logging-level": "debug"
    }    
}

The logger fields is:

  • logging-level

It can be set to the following values:

  • debug for the debug level and developper information
  • info for the level of information
  • warning to display only warning and errors
  • error to display only error
  • critical to display only error

Configure Morphosyntactic tagger Network

Example of Configuration:

network configuration
{
    "network": {
        "host":"0.0.0.0",
        "port":8080,
        "associate-environment": {
            "host":"HOST_ENVNAME",
            "port":"PORT_ENVNAME"
        },
        "ssl":
        {
            "certificate":"path/to/certificate",
            "key":"path/to/key"
        }
    }
}

The network fields:

  • host : hostname

  • port : port of the service

  • associated-environement

: default one. This field is not mandatory.

  • "host" : associated "host" environment variable

  • "port" : associated "port" environment variable

  • ssl : ssl configuration IN PRODUCTION IT IS MANDATORY TO USE CERTIFICATE AND KEY THAT ARE *NOT* SELF SIGNED

  • cert : certificate file

  • key : key file

Configure Morphosyntactic tagger runtime

Example of Configuration:

network configuration
{
    "runtime":{
        "request-max-size":100000000,
        "request-buffer-queue-size":100,
        "keep-alive":true,
        "keep-alive-timeout":5,
        "graceful-shutown-timeout":15.0,
        "request-timeout":60,
        "response-timeout":60,
        "workers":1
    }    
}

The Runtime fields:

  • request-max-size : how big a request may be (bytes)

  • request-buffer-queue-size: request streaming buffer queue size

  • request-timeout : how long a request can take to arrive (sec)

  • response-timeout : how long a response can take to process (sec)

  • keep-alive: keep-alive

  • keep-alive-timeout: how long to hold a TCP connection open (sec)

  • graceful-shutdown_timeout : how long to wait to force close non-idle connection (sec)

  • workers : number of workers for the service on a node

  • associated-environement : if one of previous field is on the associated environment variables that allows to replace the default one. This field is not mandatory.

  • request-max-size : overwrite with environement variable

  • request-buffer-queue-size: overwrite with environement variable
  • request-timeout : overwrite with environement variable
  • response-timeout : overwrite with environement variable
  • keep-alive: overwrite with environement variable
  • keep-alive-timeout: overwrite with environement variable
  • graceful-shutdown_timeout : overwrite with environement variable
  • workers : overwrite with environement variable

Morphosyntactic tagger service

To run the command type simply from tkeir directory:

python3 thot/mstagger_svc.py --config=<path to Morphosyntactic Tagger configuration file>

or if you install tkeir wheel:

tkeir-mstagger-svc --config=<path to Morphosyntactic Tagger configuration file>

A light client can be run through the command

python3 thot/mstagger_client.py --config=<path to Morphosyntactic Tagger configuration file> --input=<input directory> --output=<output directory>
or if you install tkeir wheel:

tkeir-mstagger-client --config=<path to Morphosyntactic Tagger configuration file> --input=<input directory> --output=<output directory>

or if you install tkeir wheel:

Morphosyntactic Tagger Tests

The Morphosyntactic tagger service come with unit and functional testing.

Morphosyntactic Tagger Unit tests

Unittest allows to test Tokenizer classes only.

python3 -m unittest thot/tests/unittests/TestMorphoSyntacticTaggerConfiguration.py
python3 -m unittest thot/tests/unittests/TestMorphoSyntacticTagger.py

Morphosyntactic Tagger Functional tests

python3 -m unittest thot/tests/functional_tests/TestMorphoSyntacticTaggerSvc.py