Leo

Documentation Overview

The documentation is separated into two main sections; in the first sections of this guide you’ll find a description of Leo as well as associated terminology and concepts. The next section goes into detail about setting up your computer, installing and configuring everything needed in order to run a sample Leo Project. Once the Leo project is up and running, you will find a description of the steps needed in order to adapt Leo to your specific project needs. For specific questions about the use of various classes and methods within Leo see the Javadoc page.

If you’re already familiar with UIMA and Maven, you can go to the Download page here for the POM dependencies required. A list of relevant tools and packages used by Leo can be found in the Components and Tools page.

1.1 Leo-Client

The client is used to define the readers and listeners needed to execute a pipeline via the client API for UIMA_AS. In Leo-client will initialize and execute a pipeline via the client API for UIMA AS which has been created via the Service object. Setting up a client is as simple as picking the collection reader and listener required; plugging in the input directory you’re using, and specifying the output listener that you’d like to use. An example of the code requiring your input can be found in the “Understanding and Adapting the Leo-Example Project” section of the User-guide.

DatabaseCollectionReader reader = new DatabaseCollectionReader("com.mysql.jdbc.Driver", "jdbc:mysql://localhost/test",
                                 "testuser", "password", "select id, document from example_document", "id", "document");                       

BatchDatabaseCollectionReader reader = new BatchDatabaseCollectionReader("com.mysql.jdbc.Driver", "jdbc:mysql://localhost/test",
                                      "testuser", "password",
                                      "select id, document from example_document where rowno >= {min} and rowno < {max}",
                                      "id", "document", 0, 100, 10);     

SQLServerPagedDatabaseCollectionReader reader = new SQLServerPagedDatabaseCollectionReader("com.microsoft.sqlserver.jdbc.SQLServerDriver",
                                                "jdbc:mysql://localhost/test",
                                                "testuser", "password",
                                                "select id, document from example_document order by id",
                                                "id", "document", 50000, 0);                                  

FileCollectionReader reader = new FileCollectionReader(new File("data/input"), false);

XmiFileCollectionReader reader = new XmiFileCollectionReader(new File("data/xmi-input"), false);

@Override
public boolean hasNext() throws IOException, CollectionException {
    return currentString < numberOfStrings;
}

@Override
public void getNext(CAS aCAS) throws CollectionException, IOException {
    int length = random.nextInt(MAX_STRING_LENGTH);
    aCAS.setDocumentText(RandomStringUtils.randomAlphanumeric(length));
    currentString++;
}

@Override
public Progress[] getProgress() {
    return new Progress[]{ new ProgressImpl(currentString, numberOfStrings, Progress.ENTITIES) };
}

/**
 * This reader populates a specified number of CAS objects with random strings. Useful primarily as an example of the
 * implementation of the BaseLeoCollectionReader API.
 *
 * User: Thomas Ginter
 * Date: 10/27/14
 * Time: 09:15
 */
public class RandomStringCollectionReader extends BaseLeoCollectionReader {

    /**
     * Number of strings produced so far.
     */
    protected int currentString = 0;
    /**
     * Number of strings that the reader will create.
     */
    @LeoConfigurationParameter(mandatory = true)
    protected int numberOfStrings = 0;
    /**
     * Generates a random length for each string.
     */
    protected Random random = new Random(System.currentTimeMillis());
    /**
     * Maximum length of the generated string.
     */
    protected static final int MAX_STRING_LENGTH = 1024;

    public RandomStringCollectionReader(int numberOfStrings) {
        this.numberOfStrings = numberOfStrings;
    }

    /**
     * @param aCAS the CAS to populate with the next document;
     * @throws org.apache.uima.collection.CollectionException if there is a problem getting the next and populating the CAS.
     * @see LeoCollectionReaderInterface#getNext(org.apache.uima.cas.CAS)
     */
    @Override
    public void getNext(CAS aCAS) throws CollectionException, IOException {
        int length = random.nextInt(MAX_STRING_LENGTH);
        aCAS.setDocumentText(RandomStringUtils.randomAlphanumeric(length));
        currentString++;
    }

    /**
     * @return true if and only if there are more elements available from this CollectionReader.
     * @throws java.io.IOException
     * @throws org.apache.uima.collection.CollectionException
     * @see LeoCollectionReaderInterface#hasNext()
     */
    @Override
    public boolean hasNext() throws IOException, CollectionException {
        return currentString < numberOfStrings;
    }

    /**
     * Gets information about the number of entities and/or amount of data that has been read from
     * this <code>CollectionReader</code>, and the total amount that remains (if that information
     * is available).
     * <p/>
     * This method returns an array of <code>Progress</code> objects so that results can be reported
     * using different units. For example, the CollectionReader could report progress in terms of the
     * number of documents that have been read and also in terms of the number of bytes that have been
     * read. In many cases, it will be sufficient to return just one <code>Progress</code> object.
     *
     * @return an array of <code>Progress</code> objects. Each object may have different units (for
     * example number of entities or bytes).
     */
    @Override
    public Progress[] getProgress() {
        return new Progress[]{ new ProgressImpl(currentString, numberOfStrings, Progress.ENTITIES) };
    }
}

//From reader object
CollectionReader myReader = new MyReader(param1, param2);
ExternalCollectionReader myExternalReader = new ExternalCollectionReader(myReader);

//From descriptor file
ExternalCollectionReader externalReader = new ExternalCollectionReader(new File("path/to/readerDescriptor.xml"));

1.2 Leo-Service

The Leo Service class can be used to define type system and annotators for the pipeline programmatically. The Service class deploys the UIMA AS service from the list of primitive descriptor files or Annotator objects provided by the caller. Any modules that will be executed must be in the class path of the Server process, remote modules being the exception once those are implemented. Setting up the Service, like most aspects of Leo, is as simple as editing a few directories to your specific files.

import gov.va.vinci.leo.ae.LeoBaseAnnotator;
import gov.va.vinci.leo.descriptors.LeoTypeSystemDescription;
import org.apache.uima.analysis_engine.AnalysisEngineProcessException;
import org.apache.uima.jcas.JCas;

public class myAnnotator extends LeoBaseAnnotator {
	
	@LeoConfigurationParameter
	protected String myStringParam = null;
	
	@Override
	public void initialize(UimaContext aContext) throws ResourceInitializationException {
		super.initialize(aContext);
		< additional initialization code >
	}

	@Override
	public void annotate(JCas aJCas) throws AnalysisEngineProcessException {
		< additional processing code >
	}
}

initialize(org.apache.uima.UimaContext aContext);

1.4 Leo-Core

The Leo-Core package contains the tools, descriptors, types, and extra functionalities of Leo that are helpful for individualized tasks.

Installation and Configuration

The guide below will walk you through the steps needed to configure your computer, set up Leo, and start running a simple Leo pipeline. It is exhaustive; if you’re already familiar with UIMA and Maven, you can go to the Download page here for the POM’s required.

2.1 Install Java

Leo uses Java 8 and both the JRE and JDK are required.

2.3 Install Eclipse or Intellij

2.4 Basics of Maven

Leo uses Maven dependencies and as such, there needs to be a maven environment configured on your computer. Maven is a dependency and build system. The core of maven is found in the pom.xml that will reside at the top level of every project. The pom.xml describes the dependencies, as well as build, deployment and reporting aspects of the project. With Leo you typically use maven at the very beginning of your project in order to setup dependencies and builds; eclipse is generally used throughout the rest of the project.

In order to use our maven repository, you need to update your settings.xml file. This file is found in (~/.m2/). If there is no settings.xml file create a blank text file with that name and populate it with this code.

<project ...>
  <repositories>
     <repository>
	 <id>decipher</id>
	 <url>http://decipher.chpc.utah.edu/nexus/content/groups/public</url>
     </repository>
  </repositories>
  ...
</project>

Additionally in the menu bar you can go to Window > Preferences. Select Java > Build Path > Classpath Variables, click on the new button > MR_REPO variable and point it to your Maven repository.

2.5. Add Libraries

Enter the Eclipse Preferences window once again and select Java -> Installed JREs. If you have more than one JRE listed, you may want to do this for each one just to be sure.

Select your JRE and click the “Edit” Button on the right. In the window that pops up, click on the “Add External JARs…” button. Browse to the location where your JDK is installed, looking either for tools.jar or classes.jar (depending on your platform):

• In Windows, look for C:\Program Files\Java\jdk1.8.0_02\lib\tools.jar (the folder version number may vary).
• In Ubuntu Linux, this will likely be something like /usr/lib/jvm/java-8-openjdk/lib/tools.jar
• In Mac OS, look in /System/Library/Frameworks/JavaVM.framework/Classes

Verify that the JAR you just selected (either tools.jar or classes.jar) was added to the list in the “Edit JRE” window and click Finish.

Before closing the Preferences window, select Java -> Build Path -> User Libraries. Click the “New” button, name the library “UIMA-AS-2.9.0”, and click OK. You should now see UIMA-AS-2.9.0 in your list of User Libraries. Select UIMA-AS-2.9.0 and click on the “Add JARs…” button. Now browse to the location where you saved the UIMA library way back at the beginning of this document. Within the UIMA directory, browse into the lib directory and select all of the JARs. After you click OK, you should see many entries under UIMA-AS-2.9.0 in the User Libraries dialog. We still need to add more JARs to it, so select UIMA-AS-2.9.0 again and click the “Add JARs…” button. Under the main UIMA directory, look for a directory called apache-activemq-5.14.0. Browse into it, then browse into its lib directory, and select all of the JARs you find. Now click OK on the User Libraries dialog.

2.6. Install Git

While it is true that you already installed the Git plugin, the command line tools are still necessary as the plugin simply doesn’t give us everything we need.

sudo apt-get install git

2.7. Clone the Leo Example

Now that Git is installed, we can grab the latest version of Leo. Open up the terminal(or Git Bash on Windows) and cd to your workspace directory. For an easy example pipeline, enter this command:

git clone http://decipher.chpc.utah.edu/gitblit/git/examples/leo-example.git

Git has now created an example pipeline that uses Leo, and has the ability to process text, annotate the text through various java classes, and output the data in xmi format. However, the project is not quite ready to be run; at this point you will need to import the project into Eclipse.

2.8. Import Leo into Eclipse or IntelliJ

In the step above, we downloaded Leo into a directory contained in our workspace directory. Please use your IDE of choice to import the project using Maven.

• For IntelliJ use File > New > Project from Existing Sources
• For Eclipse use File > Import > Maven > Existing Maven Projects

2.9 Running the Sample Project

In the Previous section we set up a Leo environment using a Leo-Sample project. The steps to run a project described in this section are the same whether you are using the sample project, or your own project build.

If you’re running the sample project make sure that there is an input and output directory in the “Data” folder on the left side of the eclipse project. If the output folder is not there, create one.

If you’re running your own pipeline using Leo components, be sure to stop here and make sure that your Service and Client are configured.

Now you’re ready to run the example pipeline, or Leo as a whole pipeline with your own or other annotators.

Start Broker

Open terminal/command line, navigate to the UIMA home directory, then CD to the bin sub-directory. Now run the startBroker script(startBroker.bat on Windows, ./startBroker.sh otherwise). This will start an ActiveMQ broker that must be running for Leo to work. Please keep this running in the background.

Run the Service

In your IDE, run the service program by opening its source file (i.e src/main/java/ExampleService.java) and clicking the “Run” button in your IDE.

Run the Client

Now that the service is running, run the client program as well. The client should be found in the same directory as the service. You will now need to open a new terminal window, cd to your UIMA_HOME/bin directory again and run the “annotationViewer” script. A dialog like this will now appear:

Change the first text field “input directory” to the output directory that you created.

The second one is a little trickier. Scroll back through the SampleService output looking for a line that looks something like this:

Aggregate: /var/folders/x9/r87vxc1s1fb6gl80r306hf7h0000gn/T/leoAggregate7687725334298388844.xml

Copy the path to the aggregate descriptor into the second text field of the Annotation Viewer window.

Once both fields are filled in, click the “View” button. A new window with the analyzed documents will pop up. It will contain a list of xmi files, each one corresponding to one of the files in your input directory. To see results of the simple service, double click on any of the xmi files. If an “Annotation Results…” window appears with colorful text, you’re all done! The pipeline has processed the text according to the annotators provided in the client.java program, and given you the xmi files needed for your research!

Adapting the Service

For scalability, multiple instances of a service may be started in their own JVM instances. All services should have the same broker url and service names. In that scenario, the client requests are routed to whichever instance is currently idle. In gov.va.vinci.leo.examples/ExampleService.java you will find the code required to change the broker url and name of the input queue.

The ServerConfig.groovy file requires the user to specify a brokerUrl where the ActiveMQ Broker is running, a CAS pool size to determine how big the input and output queue is, as well as the “endpoint” or project name.

brokerUrl                       = "tcp://localhost:61616"
endpoint                        = "leoExample"
jamQueryIntervalInSeconds       = 60
jamResetStatisticsAfterQuery    = true
jamServerBaseUrl                = "http://localhost:8080/jam" // If not specified, the service will not try to register with JAM. 
jamJmxPort                      =-1  // If -1, the service will pick an open port automatically.

In addition, a service requires a type description and aggregate to actually do work.

 LeoAEDescriptor aggregate =  new ExamplePipeline().getPipeline();
            aggregate.setIsAsync(true);
            aggregate.setNumberOfInstances(1);

Adapting the Pipeline and Selecting Annotators

At this point, the only thing you need to do in order to further adapt your pipeline is to choose which annotators you will be using in gov.va.vinci.leo.examples.pipeline. We will be looking into ExamplePipeline.java. ExampleScalablePipeline provides the structure required if the pipeline you are implementing requires further scalability. PipelineInterface.java is a helper class that allows for interchangeability of multiple pipelines so that the user can easily switch between pipelines if needed. You only need to extend it in order to use the same names from one pipeline to another.

ExamplePipeline.java has the code required to use the regex and Window Annotators. For information on how to use other annotators, go to the “Additional Annotators” section of this User-guide.

public class ExamplePipeline implements PipelineInterface{
    LeoAEDescriptor pipeline = null;

    /**
     * Constructor
     * @throws Exception
     */
    public ExamplePipeline() {
    }

    @Override
    public LeoAEDescriptor getPipeline() throws Exception {
        if(pipeline == null) {
            pipeline = new LeoAEDescriptor();

            /** Add the regex annotator first **/
            pipeline.addDelegate(
                    new RegexAnnotator()
                        .setGroovyConfigFile("config/resources/example-regex.groovy")
                        .setLeoTypeSystemDescription(getLeoTypeSystemDescription())
                        .getLeoAEDescriptor()
            );

            /**
             * Second, add windows around the regexes. The window annotation will extend 1 word (defined as separated by a whitespace
             * character before and after each character string.
             */
            pipeline.addDelegate(
                    new WindowAnnotator(Window.class.getCanonicalName(), RegularExpressionType.class.getCanonicalName())
                        .setWindowSize(1)
                        .setAnchorFeature("Anchor")
                        .setLeoTypeSystemDescription(getLeoTypeSystemDescription())
                        .getLeoAEDescriptor()
            );

        }

        return pipeline;
    }

    @Override
    public LeoTypeSystemDescription getLeoTypeSystemDescription() {
        LeoTypeSystemDescription description = new WindowAnnotator().getLeoTypeSystemDescription();
        description.addTypeSystemDescription(new RegexAnnotator().getLeoTypeSystemDescription());
        return description;
    }
}

In order to create your own annotator. the annotate method has to be developed.

This line of code already assumes that JcasGen has been called.

@Override
public LeoTypeSystemDescription getLeoTypeSystemDescription() {
    LeoTypeSystemDescription description = new WindowAnnotator().getLeoTypeSystemDescription();
    description.addTypeSystemDescription(new RegexAnnotator().getLeoTypeSystemDescription());
    return description;
}

In other projects you will need to call JcasGen in order to generate your java and class files.

if (generateTypes) 
{	
	types.jCasGen("src/main/java", "target/classes");
}

Running your Project

The process used to finally run a project is the same as described at the end of the previous section.

1. Start the Broker
2. Run the Service
3. Run the Client

From here, you can view the annotations created in your pipeline using the default UIMA annotation viewer, or some of the other tools described in the Components and Tools page.

Annotation Librarian

The AnnotationLibrarian is a helper class that performs many of the common functions needed by Annotation Engines. Functionality includes methods such as getAllAnotationsOfType(), getFirstAnnotationOfType(), mergeAnnotations().

Annotation Pattern Annotator(APA)

Building on Annotation Librarian ideas, the AnnotationPatternAnnotator mirrors regular expression functionality to manipulate annotations. It uses a resource file that contains regular expressions that include tagged annotation types combined with standard regular expressions syntax. Additionally, part of the pattern can be separated as an Anchor feature, where a new annotation is created for the tagged snippet.

The behavior of Annotation Pattern is very similar to the Java Regex Package’s Pattern and Matcher classes. The AnnotationPattern Object is created using the simple syntax : AnnotationPattern patternName = AnnotationPattern.compile(regex); where - compile() - is a private static method which takes the annotationPattern to return an AnnotationPattern Object. - regex - is a String that will mention the Annotation annotationPattern which the user is expecting. Below are various examples to show different combination of patterns.

ConceptLink Annotator

Creates output annotations covering related concepts. Includes a MatchMaker annotator for concept value pair annotations.

CONTEXT

This project contains a Leo implementation for the ConText algorithm.

ConText is based on a negation algorithm called NegEx. ConText’s input is a sentence with indexed clinical conditions; ConText’s output for each indexed condition is the value for contextual features or modifiers. The initial versions of ConText determines values for three modifiers:

Negation - affirmed or negated. Temporality - recent, historical, or hypothetical. Experiencer - patient or other. The current version of ContextAnnotator has additional output features.

ContextAnnotator outputs an annotation of the outputType type. The outputType also has an additional feature – Window – which contains the annotation of the window in which the concept is found. The ContextAnnotation will span only the concept itself and not the window. So this feature is a hook to the input type in the output type.

We adopted it as a UIMA annotator, and made several changes to the algorithm. Such as a new rule category “adjacent”, which specifies rules for words that are immediately before or immediately after the anchor word; as well as several new rules to the resource file.

Merge Annotator

The Merge Annotator allows the user to easily merge overlapping or proximal annotations of the same type.

REGEX Annotator

The Regular Expression Annotator provides a UIMA based annotator for searching a corpus using regular Expressions. The Regular Expression Annotator can be configured via a simple configuration file, for processing a single group of patterns with the same configuration for each pattern, or via a more flexible Groovy configuration file that allows for multiple groups of patterns configured with different parameters per group of patterns. For a more detailed synopsis and download instructions, you can go the the Regular Expression site here.

Sentence Annotator

Annotates sentences found in a covering annotation or on the document as a whole.

Sherlock

The Sherlock framework provides the base objects for implementing machine learning (ML) algorithms in a UIMA environment. It does not attempt to provide a specific ML implementation but rather the means to accessing existing implementations within a framework of objects that provide a common interface. This is accomplished by recognizing two primary workflows for learning tasks:

(Training) INPUT -> Compute Feature Vector -> Create Vector Classification -> Aggregate Vectors and Classifications -> Train a Learning Model (Prediction) INPUT -> Compute Feature Vector -> Predict from Trained Model Sherlock was created from the need to be able to perform the above outlines processes within UIMA utilizing previously created annotations as well as the source text as the INPUT for feature vector creation.

White Space Annotator

Create a series of tokens that ignore whitespace and identify the basic building blocks of words, newline, number, and punctuation identifiers. This can be done either character by character using the WhitespaceTokenizer, or regular expressions using RegexWhitespaceTokenizer.

Window Annotator

The Window annotator takes an anchor annotation and builds a new annotation that is larger that the anchor annotation based on specified number of tokens to the left and to the right of the anchor annotation. If only a single window number is specified, the left and write window will be at the equal distance from the anchor annotation. A link to the anchor annotation gets stored in an output annotation as a feature.