Developing and extending the Shepherd Project

The Shepherd Project is an open source Java application to which you can contribute features and fixes as a software developer. You can also independently customize it to meet your specific mark-recapture project. To begin software development with the Shepherd Project, you will need to install and familiarize yourself with the Java JDK 6 or higher, the Git version control system, and the Apache Maven build system. While these are sophisticated tools with significant volumes of documentation, only a few steps are required to create a development environment and start programming.

Java is required to develop for the Shepherd Project. To develop with Java:

1. Download the Java 6 or higher JDK installer.
2. Follow the installer instructions to install Java on your computer. We will use <java_install_dir> to represent the file system location in which you install Java.
3. Add the <java_install_dir>\bin directory to your PATH environment variable. This allows Java to be easily called from the command line in any directory on your file system.
4. Create a JAVA_HOME environment variable and set its value to <java_install_dir>.

These steps create a Java software development foundation for use by other tools.

To start using Git with the Shepherd Project:

1. Install Git on your development computer, which will then create a local Git version control system in which you can manage Shepherd Project code and synchronize with the Shepherd Project space on GitHub.
2. Add your Git executable (e.g., git.exe on Windows) to your system's PATH environment variable, allowing git to be easily executed from any directory.
3. Navigate on your file system to the directory in which you want to create a Shepherd Project development environment. Let's call this directory <shepherd_base_dir>.
4. In <shepherd_base_dir>, clone the Shepherd Project from GitHub by executing the following command line statement:

git clone https://github.com/holmbergius/Shepherd-Project

Next switch to the 2.1.x branch:

git checkout 2.1.x

Note: An Internet connection is required to obtain code from GitHub.

Git documentation can provide you with additional cloning options. If you would prefer visual management of your Git source control system, tools like the SmartGit client are free for non-commercial use and provide a good user interface (on top of an existing command line Git installation) that makes working in Git quite simple.

After cloning the Shepherd Project, create a branch in which to make your changes. An example git command is:

git branch myExperimentalBranch

To start using Maven with the Shepherd Project:

1. Install Maven on your development computer.
2. Add your Maven executable (e.g., mvn.exe on Windows) to your system's PATH environment variable, allowing Maven to be easily executed from the <shepherd_base_dir> directory.
3. Navigate on your file system to <shepherd_base_dir>.
4. In <shepherd_base_dir>, build the Shepherd Project by executing the following command line statement:

mvn clean install

Note: An Internet connection is required for Maven to obtain the Shepherd Project third party dependencies.

Note: <shepherd_base_dir> corresponds to a directory with the Maven project file pom.xml in it. If this file is not present, 'mvn clean install' should not be run from this folder.

Tip: If you have a window open to your project's <shepherd_base_dir>/target folder (or any subfolder) or have an application opening a file in the target folder, the Maven build will fail because it needs to blow away and refresh <shepherd_base_dir>/target during the build.

Maven documentation can provide you with additional Maven options.

Executing 'mvn clean install' generates a .war file in the <shepherd_base_dir>/target folder. This WAR file is the Shepherd Project executable and can be run in popular J2EE containers, such as Tomcat and Jetty.

For example, with Tomcat installed on your laptop or web server, you can place the WAR file into the webapps folder of the base Tomcat installation directory and see the Shepherd Project appear in a browser according to the URL:

http://<my_URL>/<war_file_name_without_extension>

For example, when running locally on a laptop with Shepherd Project 2.1.0-RELEASE (war file name shepherd-2.1.0-RELEASE.war), your URL might be:

http://localhost:8080/shepherd-2.1.0-RELEASE

or on your host web server at www.markandcapture.org, it might be:

http://www.markandcapture.org/shepherd-2.1.0-RELEASE

If you want to use the Eclipse IDE for Shepherd Project development, you can build an Eclipse EE project for the Shepherd Project by executing the following command from <shepherd_base_dir>:

mvn eclipse:eclipse

After installing the Eclipse EE IDE:

1. Open Eclipse.
2. Define the M2_REPO classpath variable in Eclipse.
   1. From the Window menu, select Preferences. The Preferences dialog box appears.
   2. In the Preferences dialog box, select Java > Build Path > Classpath Variables from the tree.
   3. In the Classpath Variables view, click New to add a new variable. The New Variable Entry dialog box appears.
   4. In the New Variable Entry dialog box, enter M2_REPO as the variable Name and the full path to the repository that Maven created for you as the Value. Then click OK. Note that the value you set may vary from operating system to operation system, and you can further specify the location of this repository using Maven (see the relevant Maven 2 documentation). As an example, the default location for a Maven 2 repository on a Windows 7 system is: C:\Users\<username>\.m2\repository
3. From the File menu, select Import.
4. In the Import dialog box, select General, Existing Projects into Workspace and then click Next.
5. In the Import Projects dialog box in the Select root directory field, click Browse to navigate to and select <shepherd_base_dir>.
6. In the Projects field of the Import Projects dialog box, make sure that the Shepherd Project is selected and then click Finish. Eclipse will open the Shepherd Project.

While the Shepherd Project JavaDoc provides a full list of the project's classes, three classes are foundational to development with the Shepherd Project:

• Encounter - This persistent class encapsulates all of the information recorded for a single sighted animal at a specific location at a point in time.
• MarkedIndividual - This persistent class represents a marked individual in a marked-recapture study. A MarkedIndividual instance encapsulates one or more Encounter instances of the same animal.
• Shepherd - provides convenience methods for obtaining groups of persisted objects (e.g., Encounter and MarkedIndividual instances) and manages database transactions and persistence. Shepherd is essentially an interface to the DataNucleus persistence layer.

Understanding how to invoke and manipulate these three objects is the start to adding, extending, or modifying a Shepherd Project instance.

Shepherd is the main information retrieval, processing, and persistence class to be used for all Shepherd Project applications. The Shepherd class interacts directly with the database and all persistent objects stored within it. Any application seeking access to data must invoke an instance of Shepherd first and use it to retrieve any data stored in the database.

While a Shepherd object is easily invoked with a single, simple constructor, no data can be retrieved until a new Transaction has been started. Changes made using the Transaction must be committed (store changes) or rolled back (ignore changes) before the application can finish. Example:

Shepherd myShepherd=new Shepherd();
myShepherd.beginDBTransaction();

//Now make any changes to the database objects that are needed.

myShepherd.commitDBTransaction(); 
myShepherd.closeDBTransaction();

Alternatively, for read-only transactions or to abandon changes:

Shepherd myShepherd=new Shepherd();
myShepherd.beginDBTransaction();

//Now read and process data as needed

myShepherd.rollbackDBTransaction(); 
myShepherd.closeDBTransaction();

The behavior of a Shepherd Project is configured using a set of DataNucleus persistence properties defined in commonConfiguration.properties. Any properties in that file that start with “datanucleus” or “javax.jdo” influence where and how data is pulled by the Shepherd object.

A Shepherd object contains many helper methods to facilitate easy access to mark-recapture data. These include:

• Shepherd.getAllMarkedIndividuals() - returns all marked individuals in the database for iteration.
• Shepherd.getAllEncounters() - returns all encounters in the database.
• Shepherd.getEncounter(String num) - returns data for a specified encounter.
• Shepherd.getMarkedIndividual(String name) - returns data for a specified marked individual.
• Shepherd.getPM() - returns a JDO PersistenceManager. Because the Shepherd Project uses DataNucleus, you can use the JDO API to manipulate and persist data.
• ...many more...

Here are the contents of an example JSP file that invokes a Shepherd object to return the number of encounters and marked individuals in the database.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<%@ page contentType="text/html; charset=utf-8" language="java" import="org.ecocean.*"%>
<html>
<head>
<title>My Sample JSP File Using a Shepherd Object</title>
</head>
<body>
<%
//invoke a Shepherd
Shepherd myShepherd=new Shepherd();
%>

<p>The number of encounters in the database is: <%=myShepherd.getNumEncounters()%></p>
<p>The number of marked individuals in the database is: <%=myShepherd.getNumMarkedIndividuals()%></p>
<%
//rollback the Shepherd as there were no changes
myShepherd.rollbackDBTransaction();
myShepherd.closeDBTransaction();
%>
</body>
</html>

Encounter is a persistent class that represents, at a minimum, the occurrence of one animal at a specific location and point in time. One or more Encounter instances make up a MarkedIndividual in a mark-recapture study.

In the Shepherd Project, the Encounter class is the most commonly stored and accessed object, and therefore it is important that you know how to store, retrieve, and manipulate it.

As the record of the occurrence of an organism in nature, the attributes of Encounter are often derived from the predefined terms of the Darwin Core. The Recorded data section of the Shepherd Project User Manual provides some of the attribute definitions. See the Javadoc listing for the class for a comprehensive list.

An Encounter instance is most often created when a new animal sighting is submitted through the Shepherd Project user interface. However, if you're importing legacy mark-recapture data through a script, you may want to manually create Encounters to store pre-existing data.

The primary key for an Encounter object in the database is its catalogNumber attribute. An Encounter instance is generally retrieved from a Shepherd object.

Shepherd myShepherd=new Shepherd();

//let's start a database transaction
myShepherd.beginDBTransaction();

//let's retrieve Encounter instance "12345"
Encounter enc = myShepherd.getEncounter("12345");

//let's retrieve some sample data
System.out.println("Encounter "+enc.getCatalogNumber()+" was sighted in "+enc.getVerbatimLocality()+" on "+enc.getDate()+".");
System.out.println("Encounter "+enc.getCatalogNumber()+" includes "+enc.getImages().size()+" photos and/or videos.");
System.out.println("Encounter "+enc.getCatalogNumber()+" was subsequently identified as "+enc.getIndividualID()+".");

//we didn't make changes, so let's rollback rather than commit
myShepherd.rollbackDBTransaction();
myShepherd.closeDBTransaction();

As an example, let's make a change to an encounter, moving the state of an Encounter from “unapproved” to “approved for public display”.

Shepherd myShepherd=new Shepherd();

//let's start a database transaction
myShepherd.beginDBTransaction();


//let's retrieve Encounter instance "12345"
Encounter enc = myShepherd.getEncounter("12345");

//let's change the state
boolean changedState=false;
if(enc.getState().equals("unapproved")){
     enc.setState("approved for public display");
     changedState=true;
}

if(changedState){

     //we made a change, so let's commit the transaction
     myShepherd.commitDBTransaction();
     
}
else{
     myShepherd.rollbackDBTransaction();
}
myShepherd.closeDBTransaction();

You can see the example above translated into a full Shepherd Project servlet here in EncounterSetState.java.

A MarkedIndividual is an uniquely identified member of a population and includes one or more reported encounters. It is up to each library and its research staff to determine the minimum amount of data and procedures required for a unique identification (e.g., a distinct ear tag, a visual photo-identification, digital extraction of spot patterning, a distinct DNA pattern, etc.). As the study acquires more and more encounters for each individual in the library, it will be able to build up robust metrics for population analysis, allowing its research team to better understand biology and population trends.

Creating and updating MarkedIndividual objects is the ultimate goal of Shepherd Project libraries.

A marked individual object can be easily retrieved by its individualID attribute, which is also its primary key in the database.

Shepherd myShepherd=new Shepherd();

//let's start a database transaction
myShepherd.beginDBTransaction();

//let's retrieve marked individual instance "A-098"
MarkedIndividual indie = myShepherd.getMarkedIndividual("A-098");

//let's retrieve some sample data
System.out.println("Marked individual "+indie.getIndividualID()+" was sighted "+indie.getEncounters().size()+" times. It was first sighted in "+indie.getEarliestSightingYear()+".");


//we didn't make changes, so let's rollback rather than commit
myShepherd.rollbackDBTransaction();
myShepherd.closeDBTransaction();

One of the most important methods is MarkedIndividual.getEncounters(), which allows you to iterate all of the Encounter objects of this MarkedIndividual.

Shepherd myShepherd=new Shepherd();

//let's start a database transaction
myShepherd.beginDBTransaction();

//let's retrieve marked individual instance "A-001"
MarkedIndividual indie = myShepherd.getMarkedIndividual("A-001");

Vector allEncounters=indie.getEncounters();
for(int i=0;i<allEncounters.size();i++){
     Encounter enc=(Encounter)allEncounters.get(i);
     System.out.println(indie.getIndividualID()+" was sighted on "+enc.getDate()+" at "+enc.getVerbatimLocality()+".");
}

//we didn't make changes, so let's rollback rather than commit
myShepherd.rollbackDBTransaction();
myShepherd.closeDBTransaction();

Consult the Javadoc for additional classes used in the Shepherd Project.

When creating new Java code for use in the Shepherd Project, you should follow the model/view/controller (MVC) software architecture. The Internet contains a lot of good documentation about MVC and about the Java Platform, Enterprise Edition platform (J2EE) on which the Shepherd Project is based. Therefore, our discussion here covers specific implementations details of the Shepherd Project.

Here are the MVC basics:

• model - The Shepherd project's Java classes as persisted in a relational database through DataNucleus (JDO) represents the model.
• view - The Java Server Pages (JSP) files of the Shepherd Project merge Java code and HTML to display data. These pages treat data as read-only, displaying it without changing it.
• controller - The Java servlets of the Shepherd Project represent the control logic that changes or manipulates data. Servlets may also export data to third party applications (e.g., Program MARK) or convert it dynamically to other machine-readable formats for mashups.

So how to implement new code in the Shepherd Project depends on what you want to do. For example:

• Do you want to add a new data field to the Shepherd Project? You need to update the model to hold the field, add a servlet (controller) to handle setting/changing/deleting the field, and update a related JSP file, such as encounter.jsp or individuals.jsp, to display the new attribute.
• Do you want to export data for a third party application? You need to build a new servlet (controller) and add it to the Shepherd Project.
• Do you want to add a new visual feature, such as incorporating a mashup GIS map from ArcGIS Online? You need to add a new JSP file or update an existing one.

TBD

TBD

TBD

Once you have made your changes, you can compile a new Shepherd Project from the base project directory (i.e., the directory containing pom.xml) using the Maven command:

mvn clean install

The new WAR file will be crated in the 'target' subdirectory.

TBD

TBD

To commit your changes to your local repository or Github…TBD

See the Git cheat sheet for a quick list of additional 'git commit' options.

If you have made changes to the Shepherd Project and think they will be valuable to the broader mark-recapture community, please register for a GitHub account and then contact Jason Holmberg to get your GitHub account linked to the repository. After that, you can merge the changes in your branch to the active development branch. We would appreciate your help!

Manual: