The LabKey Server source code is available via enlistment in LabKey's Subversion repository. Creating an enlistment will allow you to monitor, build, and develop against the most current, unreleased LabKey source code as well as released versions of the product. The following topic provides step-by-step instructions for acquiring the LabKey Server source code, installing required components, and building LabKey Server from source.
Checklist
Obtain the LabKey Source Files
The LabKey source files are stored in two repositories: the core modules are stored in an SVN repository, selected modules are stored on GitHub.
The following instructions apply to Windows machines. To install SVN on non-Windows machines see
Enlisting in the Version Control Project.
Install TortoiseSVN
- Download the latest version of TortoiseSVN.
- Install TortoiseSVN on your local computer.
- On the list of features to install, include the command line client tools.
- Add the TortoiseSVN/bin directory to your PATH.
Checkout LabKey Source Files
- Create a new directory in the Windows file system for the source files, for example, C:\dev\labkey\trunk
- In Windows Explorer, right-click the new directory and select SVN Checkout.
- Enter the URL for the LabKey repository: https://hedgehog.fhcrc.org/tor/stedi/trunk
- The user/password is cpas/cpas
- Click OK to checkout the source files. At this point all the LabKey source files, tests, and sample data will be copied to your computer.
Install Java, Tomcat, and a Database
Java
Download the Oracle JDK version 8 and install it.
Tomcat
Download the most recent release of Tomcat 8.5.x. Download a ZIP or TAR.GZ distribution, not the Windows Service Installer.
To install Tomcat, unzip it to a chosen directory (for example, C:\apache\tomcat).
LabKey supports older versions of Tomcat as well; find more information about supported versions
here. If using Tomcat 7.0.x, follow instructions on this page:
Encoding in Tomcat 7
Install a Database
Install one of the following database servers: PostgreSQL
or Microsoft SQL Server
Platform-specific installation instructions:
Environment Variables and System PATH
JAVA_HOME
Create or modify the system environment variable JAVA_HOME so it points to your JDK installation location (for example, %ProgramFiles%\Java\jdk1.8.0_xx).
If you've already set the JAVA_HOME variable to point to your installation of the JRE, you should modify it to point to the JDK.
CATALINA_HOME
Create or modify the system environment variable CATALINA_HOME so that it points to your Tomcat installation (for example, C:\apache\tomcat).
PATH
Add the following locations to your system PATH, where LABKEY_HOME is the root of your SVN enlistment.
- <LABKEY_HOME>/external/ant/bin
- <LABKEY_HOME>/build/deploy/bin (This directory won't exist yet, but add it to the PATH anyways.)
For example, C:\dev\labkey\external\ant\bin;C:\dev\labkey\build\deploy\bin.
These directories contain Apache Ant for building the LabKey source, as well as a number of executable files used by LabKey.
The build process creates build/deploy/bin, so it may not exist initially.
Apache Ant is included in the project as a convenience. If you have a recent version of Ant already installed you can use that instead. Ant 1.9.3 or newer is required to build.
For example, on OSX, place the environment variables in your .bash_profile:
export JAVA_HOME=`/usr/libexec/java_home -v 1.8`
export CATALINA_HOME=$HOME/apps/tomcat
export LABKEY_HOME=$HOME/labkey/trunk
export LABKEY_GWT_USER_OVERRIDE="gwt-user-firefox"
export PATH=$LABKEY_HOME/external/ant/bin:$LABKEY_HOME/build/deploy/bin:$PATH
GWT_HOME
Installing and configuring GWT is required only if you plan to modify existing or develop new GWT components. If you do not plan to develop with GWT you can disable IntelliJ's notifications by going to
File > Project Structure. Click
Facets and disable framework detection (remove the checkmark at the top of the dialog).
Open the LabKey Project in IntelliJ
The LabKey development team develops LabKey using IntelliJ IDEA. You can use the licence-free Community Edition of this tool if you are planning on modifying or extending the LabKey source code. Developers at non-profit organizations that are contributing open source code may
qualify for a free licensed version. Please contact LabKey for more information.
Below we describe how to configure the IntelliJ development environment; we recommend employing the same general principles if you are using a different development environment. Some developers have experimented with Eclipse as the IDE and you can find some set up details on the
Developer Message Board.
Configure the LabKey Project in IntelliJ
- Create the workspace.xml file.
- Copy the file <LABKEY_HOME>/server/.idea/workspace.template.xml. Rename the copy to create a file called <LABKEY_HOME>/server/.idea/workspace.xml
- This file configures the debug information for LabKey project. To review the debug settings go to Run > Edit Configurations in IntelliJ.
- Open the LabKey project.
- Launch IntelliJ.
- If your IntelliJ install is brand new, you will see the "Welcome to IntelliJ" pop up screen. Click Open. If you have previously installed IntelliJ, select File > Open.
- Select the LabKey IntelliJ project directory, <LABKEY_HOME>/server
- Set CATALINA_HOME
- Select File > Settings > Appearance & Behavior > Path Variables.
- Click the green plus icon in the upper right. Set the CATALINA_HOME path variable to the root directory of your Tomcat installation, for example, C:\apache\apache-tomcat-8.0.28.
- Click OK to close the Settings window.
- Set the Classpath
- Select Run > Edit Configurations. (If the menu is greyed-out, wait until IntelliJ finishes indexing the project files.)
- Confirm that LabKey Development is the selected Application in the left panel.
- Confirm that the dropdown labeled Use classpath of module is set to LabKey.
- Click OK, to close the Run/Debug Configurations window.
- Configure the Target JDK
- In IntelliJ, select File > Project Structure.
- Under Project Settings, click Project.
- Under Project SDK click New and then click JDK.
- Browse to the path of your JDK, for example, (C:\Program Files (x86)\Java\jdk1.8.0_66), and click OK.
- Click Edit. Change the name of the JDK to "labkey".
- Click Ok to close the Project Structure window.
- Verify the Target JDK for Ant
- In IntelliJ, select View > Tool Windows > Ant Build.
- In the Ant Build panel (on the far right), click the Properties button (which is directly left of the Help question mark '?' button).
- Click the Execution tab.
- Verify that Use project default Ant is selected.
- Verify that Run under JDK drop-down is set to "Project JDK (labkey)".
- Click OK.
Build and Run LabKey
Configure the Appropriate .properties File
The LabKey source includes two configuration files, one for use with PostgreSQL (
pg.properties) and one for use with Microsoft SQL Server (
mssql.properties), each specifying JDBC settings, including URL, port, user name and password, etc.
- If using PostgreSQL, open the file LABKEY_HOME/server/configs/pg.properties
- If using MS SQL Server, open the file LABKEY_HOME/server/configs/mssql.properties
- Edit the appropriate file, adding your values for the jdbcUser and jdbcPassword. (This password is the one you specified when installing PostgreSQL or MS SQL Server. If your password contains an ampersand or other special XML characters, you will need to escape it in the .properties file, as the value will be substituted into an XML template without encoding. For example, if your JDBC password is "this&that", then use the escaped version "this&that".)
Run pick_pg or pick_mssql
- In a command window, go to the directory LABKEY_HOME/server
- Run "ant pick_pg" or "ant pick_mssql" to configure labkey.xml with the corresponding database settings.
When you build LabKey, the values that you've specified in the
mssql.properties or
pg.properties file are copied into the LabKey configuration file, labkey.xml, overwriting previous values. This file is then copied into TOMCAT_HOME/conf/Catalina/localhost.
Build LabKey
To build LabKey, invoke the Ant build targets from the command line in the <LABKEY_HOME>/server directory.
To control which modules are included in the build, see
Customizing the Build.
The most important targets:
Ant Targetooooooooooooooooo | Description |
---|
ant pick_pg ant pick_mssql | Specify the database server to use. The first time you build LabKey, you need to invoke one of these targets to configure your database settings. If you are running against PostgreSQL, invoke the pick_pg target. If you are running against SQL Server, invoke the pick_mssql target. These Ant targets copy the settings specified in the pg.properties or mssql.properties file, which you previously modified, to the LabKey configuration file, labkey.xml. |
ant build | Build the LabKey Server source for development purposes. This is a fast, development-only build that skips many important steps needed for production environments, including GWT compilation for popular browsers, gzipping of scripts, production of Java & JavaScript API documentation, and copying of important resources to the deployment location. Builds produced by this target will not run in production mode. |
ant <module_name> | For convenience, we've added targets to build each of the standard modules. If your changes are restricted to a single module then building just that module is a faster option than a full build. Examples: 'ant study', 'ant query', or 'ant api'. |
ant production | Build the LabKey Server source for deployment to a production server. This build takes longer than ant build but results in artifacts that are suitable and optimized for production environments. |
ant clean | Delete all artifacts from previous builds. |
ant rebuild | Delete all artifacts from previous builds and build the LabKey Server from source. This build target is sometimes required after certain updates. |
Ant targets can also be invoked from within IntelliJ via the "Ant Build" tab.
If you get an error message along the lines of "Please use Ant 1.8.3 or greater", you can configure IntelliJ to use a compatible version of Ant that is included in the <labkey-home>/external/ant directory. Click on the Properties button in the
Ant Window and use the
Execution tab to define a custom Ant installation.
You can speed up development-time builds by restricting GWT complication to a subset of browsers. (GWT compilation is one of the most time-consuming parts of the LabKey build.) For example, setting "LABKEY_GWT_USER_OVERRIDE=gwt-user-firefox" as an environment variable will cause the build to target FireFox only. Other browsers (e.g., Internet Explorer) will not work properly with the resulting server. (Note: gwt-user-override environment variable is a synonym that works on windows but not Mac/Linux.) If set, this environment variable is respected for development builds (ant build) but ignored for production builds (ant production).
To run and debug LabKey, select
Run > Debug 'LabKey Development' in IntelliJ. If Tomcat starts up successfully, navigate your browser to
http://localhost:8080/labkey to begin debugging (assuming that your local installation of Tomcat is configured to use the Tomcat default port 8080).
While you are debugging, you can usually make changes, rebuild, and redeploy LabKey to the server without stopping and restarting Tomcat. Occasionally you may encounter errors that do require stopping and restarting Tomcat.
Post-installation Steps
Install R
Install and configure the R programming language
Run the Basic Test Suite
Run the command 'ant drt' from within your <labkey-home>/server directory, to initiate automated tests of LabKey's basic functionality. Note that 'R' must first be configured for these tests to run. Other automated tests are available as Ant targets. For details, see
Running Automated Tests.
Modules on GitHub
Many optional modules are available from the
LabKey repository on GitHub. To included these modules in your build, install a Git client and clone individual modules into the LabKey Server source.
Install a Git Client
- Install GitHub, or another Git client:
Clone Modules from LabKey's GitHub Repository
- To add a GitHub module to your build, clone the desired module into trunk/labkey/server/optionalModules. For example, to add the 'workflow' module:
Note that you can get the URL by going to the module page on GitHub (for example,
https://github.com/LabKey/workflow), clicking
Clone or Download, and copying the displayed URL.
Manage GitHub Modules via IntelliJ
Once you have cloned a GitHub module, you can have IntelliJ handle any updates:
To add the GitHub-based module to IntelliJ:
- In IntellJ, go to File > Project Structure.
- Under Project Settings, select Modules.
- Click the green plus sign (top of the second column) and select Import Module.
- Navigate to the module you've cloned from the GitHub repository and select its .iml file, for example, NLP.iml., and click OK.
To update the GitHub-based module using IntelliJ:
- To have IntelliJ handle source updates from GitHub, go to File > Settings.
- Select Version Control.
- In the Directory panel, select the target module and set its VCS source as Git, if necessary.
- Note that IntelliJ will sometimes think that embedded 'test' modules have their sources in SVN instead of Git. You can safely delete these embedded 'test' modules using the Directory panel.
- To sync to a particular GitHub branch: in IntelliJ, go to VCS > Git > Branches. A popup menu will appear listing the available Git modules. Use the popup menu to select the branch to sync to.
Install Optional Components
GWT
Installing and configuring GWT is required only if you plan to modify existing or develop new Google Web Toolkit (GWT) components.
Please see
GWT Integration for instructions on installation and configuration of GWT.
Mass Spec and Proteomics Tools
LabKey Server's mass spectrometry and proteomics binaries are provided as a separate (and optional) enlistment. To add these binaries, follow the instructions in the topic:
Enlisting Proteomics Binaries
Troubleshooting
1. Tomcat
If Tomcat fails to start successfully, check the steps above to ensure that you have configured your JDK and development environment correctly. Some common errors you may encounter include:
org.postgresql.util.PSQLException: FATAL: password authentication failed for user "<username>" or
java.sql.SQLException: Login failed for user '<username>'These error occurs when the database user name or password is incorrect. If you provided the wrong user name or password in the
.properties file that you configured above, LabKey will not be able to connect to the database. Check that you can log into the database server with the credentials that you are providing in this file.
java.net.BindException: Address already in use: JVM_Bind:<port x>:
This error occurs when another instance of Tomcat or another application is running on the same port. Specifically, possible causes include:
- Tomcat is already running under IntelliJ.
- Tomcat is running as a service.
- Microsoft Internet Information Services (IIS) is running on the same port.
- Another application is running on the same port.
In any case, the solution is to ensure that your development instance of Tomcat is running on a free port. You can do this in one of the following ways:
- Shut down the instance of Tomcat or the application that is running on the same port.
- Change the port for the other instance or application.
- Edit the Tomcat server.xml file to specify a different port for your development installation of Tomcat.
java.lang.NoClassDefFoundError: com/intellij/rt/execution/application/AppMain:
or
Error: Could not find or load main class com.intellij.rt.execution.application.AppMain:
In certain developer configurations, you will need to add an IntelliJ utility JAR file to your classpath.
- Edit the Debug Configuration in IntelliJ.
- Under the "VM Options" section, find the "-classpath" argument.
- Find your IntelliJ installation. On Windows machines, this is typically "C:\Program Files\JetBrains\IntelliJ IDEA <Version Number>" or similar. On Mac OSX, this is typically "/Applications/IntelliJ IDEA <Version Number>.app" or similar.
- The required JAR file is in the IntelliJ installation directory, and is ./lib/idea_rt.jar. Add it to the -classpath argument value, separating it from the other values with a ":" on OSX and a ";" on Windows.
- Save your edits and start Tomcat.
2. Database State
If you build the LabKey source yourself from the source tree, you may need to periodically delete and recreate your LabKey database. The daily drops often include SQL scripts that modify the data and schema of your database.
3. IntelliJ Warnings and Errors
- Warning: Class "org.apache.catalina.startup.Bootstrap" not found in module "LabKey": You may ignore this warning in the Run/Debug Configurations dialog in IntelliJ.
- Error: Could not find or load main class org.apache.catalina.startup.Bootstrap on OSX (or Linux): you might see this error in the console when attempting to start LabKey server. Update the '-classpath' VM option for your Run/Debug configuration to have Unix (:) path separators, rather than Windows path separators (;).
- Certain lines in build.xml files and other Ant build files may be incorrectly flagged as errors.
- Can't find workspace.template.xml? On older enlistments of LabKey, for example version 15.3, copy <LABKEY_HOME>/server/LabKey.iws.template to LabKey.iws instead.
4. IntelliJ Slow
You can help IntelliJ run faster by increasing the amount of memory allocated to it. To increase memory:
- Go to C:\Program Files\JetBrains\IntelliJ IDEA <Version Number>\bin, assuming that your copy of IntelliJ is stored in the default location on a Windows machine.
- Right click on the idea.exe.vmoptions file and open it in notepad.
- Edit the first two lines of the file to increase the amount of memory allocated to IntelliJ. For example, on a 2 Gig machine, it is reasonable to increase memory from 32m to 512m. The first two lines of this file then read:
- Save the file
- Restart IntelliJ
1.1.5 JAVA_TOOL_OPTIONS
Most users will not have this problem. However, if you see a build error something like the following:
error: unmappable character for encoding ASCII
then setting this environment variable may fix the problem
export JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8