This topic covers troubleshooting some problems you may encounter setting up a development machine and building LabKey Server from source. Roughly divided into categories, there is some overlap among sections, so searching this page may be the most helpful way to find what you need.

• Building and Starting LabKey Server
• IntelliJ Troubleshooting
• Gradle: Using gradle tasks, plugins, etc.
• Log4j
• LabKey Under Tomcat
• Database State
• System PATH
• Environment Variables
• Ext Libraries
• Renaming Files and Directories

Building and Starting LabKey Server

Linux Service Executable Name

If you are on Linux and see an error similar to this:

labkey_server.service - LabKey Server Application

Loaded: bad-setting (Reason: Unit labkey_server.service has a bad unit file setting.)

Active: inactive (dead)

<DATE><SERVER DETAILS> /etc/systemd/system/labkey_server.service:19: Neither a valid executable name>

It is likely because you need to provide the full path to Java at the head of the "ExecStart" line (line 19) in the labkey_server.service file. The default we send looks like:

ExecStart=$JAVA_HOME/bin/java $JAVA_PRE_JAR_OPS $JAVA_MID_JAR_OPS $LABKEY_JAR_OPS $JAVA_LOG_JAR_OPS $JAVA_FLAGS_JAR_OPS $JAVA_REFLECTION_JAR_OPS -jar /labkey/labkey/labkeyServer.jar

ExecStart=/labkey/apps/jdk-17.0.10+7/bin/java $JAVA_PRE_JAR_OPS $JAVA_MID_JAR_OPS $LABKEY_JAR_OPS $JAVA_LOG_JAR_OPS $JAVA_FLAGS_JAR_OPS $JAVA_REFLECTION_JAR_OPS -jar /usr/local/labkey/labkey/labkeyServer.jar

Unable to Start Gradle Daemon

If you are unable to start any Gradle daemon (i.e. even "./gradlew properties" raises an error), check to confirm that you are using the 64-bit version of the JDK. If you accidentally try to use the 32-bit version, that VM has a max heap size of about 1.4GB, which is insufficient for building LabKey.

Note that "java -version" will specify "64-bit" if it is, but will not specify "32-bit" for that (incorrect) version. Instead the version line reads "Client VM" where you need to see "64-bit". At present the correct result of "java -version" should be:

openjdk version "17.0.10" 2024-01-16
OpenJDK Runtime Environment Temurin-17.0.10+7 (build 17.0.10+7)
OpenJDK 64-Bit Server VM Temurin-17.0.10+7 (build 17.0.10+7, mixed mode, sharing)

An error like this may indicate that you are using the 32 bit version, as it :

FAILURE: Build failed with an exception.

* What went wrong:
Unable to start the daemon process.
This problem might be caused by incorrect configuration of the daemon.
For example, an unrecognized jvm option is used.For more details on the daemon, please refer to https://docs.gradle.org/8.6/userguide/gradle_daemon.html in the Gradle documentation.
Process command line: C:labkeyappsjdk-17binjava.exe -XX:+UseParallelGC --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.prefs/java.util.prefs=ALL-UNNAMED --add-opens=java.base/java.nio.charset=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED -Xmx2048m -Dfile.encoding=windows-1252 -Duser.country=US -Duser.language=en -Duser.variant -cp C:Usersmohara.gradlewrapperdistsgradle-8.6-binafr5mpiioh2wthjmwnkmdsd5wgradle-8.6libgradle-launcher-8.6.jar -javaagent:C:Usersmohara.gradlewrapperdistsgradle-8.6-binafr5mpiioh2wthjmwnkmdsd5wgradle-8.6libagentsgradle-instrumentation-agent-8.6.jar org.gradle.launcher.daemon.bootstrap.GradleDaemon 8.6
Please read the following process output to find out more:
-----------------------
Error occurred during initialization of VM
Could not reserve enough space for 2097152KB object heap

Server Not Starting Using a LabKey-Provided Run/Debug Configs

Problem: Server does not start in using one of the LabKey-provided Run/Debug configurations. In the console window the following error appears:

/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home/bin/java 
-agentlib:jdwp=transport=dt_socket,address=127.0.0.1:59311,suspend=y,server=n -Dcatalina.base=./ 
-Dcatalina.home=./ -Djava.io.tmpdir=./temp -Ddevmode=true -ea -Dsun.io.useCanonCaches=false -Xmx1G 
-XX:MaxPermSize=160M -classpath "./bin/*;/Applications/IntelliJ IDEA.app/Contents/lib/idea_rt.jar" 
-javaagent:/Users/susanh/Library/Caches/IntelliJIdea2016.3/groovyHotSwap/gragent.jar 
-Dfile.encoding=UTF-8 org.apache.catalina.startup.Bootstrap start
Connected to the target VM, address: '127.0.0.1:59311', transport: 'socket'
Error: Could not find or load main class org.apache.catalina.startup.Bootstrap
Disconnected from the target VM, address: '127.0.0.1:59311', transport: 'socket'

Cause: This is most likely caused by an incorrect path separator in the Run/Debug configuration's classpath argument.

Solution: Edit the Run/Debug configuration and change the separator to the one appropriate to your platform (semicolon for Windows; colon for Mac/Linux).

Fatal Error in Java Runtime Environment

Error: When starting LabKey or importing data to a new server, you might see a virtual machine crash similar to this:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000000000000, pid=23893, tid=39779
#
# JRE version: Java(TM) SE Runtime Environment (8.0_45-b14) (build 1.8.0_45-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# C  0x0000000000000000
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

Cause: These are typically bugs in the Java Virtual Machine itself.

Solution: Ensuring that you are on the latest patched release for your preferred Java version is best practice for avoiding these errors. If you have multiple versions of Java installed, be sure that JAVA_HOME and other configuration is pointing at the correct location. If you are running through the debugger in IntelliJ, check the JDK configuration. Under Project Structure > SDKs check the JDK home path and confirm it points to the newer version.

Exception in compressClientLibs task

If you encounter this error when building in production mode:

* What went wrong:
Execution failed for task ':server:modules:platform:query:compressClientLibs'.
> org.xml.sax.SAXException: java.lang.NullPointerException
  java.lang.NullPointerException

It may indicate a parsing error from yuicompressor which handles javascript and css minification. The problem is usually that the code you added is using elements of ES6 (like the spread operator) that the parser doesn't support.

Solution: Fall back to older javascript syntax.

Cannot find class X when hot-swapping

Cause: IntelliJ puts its build output files, which it uses when doing hot-swapping, in different directories than the command-line Gradle build does, so if you have not built the entire project (or at least the jars that the file you are attempting to swap in depends on) within IntelliJ, IntelliJ will not find them.

Solution: Open the Gradle window in IntelliJ and run the root-level "build" task from that window.

IntelliJ Troubleshooting

IntelliJ Version

If you are using IntelliJ and experience errors building or upgrading LabKey from source, check to see if your IntelliJ IDEA is up to date. You can download the the latest version here:

• http://www.jetbrains.com/idea

Build model 'org.jetbrains.plugins.gradle.model.GradleExtensions' for root project 'labkey'

IntelliJ Warnings and Errors

• Warning: Class "org.apache.catalina.startup.Bootstrap" not found in module "LabKey": You may ignore this warning in the Run/Debug Configurations dialog in IntelliJ.
• Error: Could not find or load main class org.apache.catalina.startup.Bootstrap on OSX (or Linux): you might see this error in the console when attempting to start LabKey server. Update the '-classpath' VM option for your Run/Debug configuration to have Unix (:) path separators, rather than Windows path separators (;).
• On Windows, if you are seeing application errors, you can try resetting the winsock if it has gotten into a bad state. To reset:
• Open a command window in administrator mode
• Type into the command window: netsh winsock reset
• When you hit enter, that should reset the winsock and your application error may be resolved. You might need to restart your computer for the changes to take effect.

IntelliJ Slow

You can help IntelliJ run faster by increasing the amount of memory allocated to it. The specific method may vary depending on your version of IntelliJ and OS. Learn more here:

• IntelliJ documentation for increasing memory heap

XML Classes not found after Gradle Refresh

Problem: After doing a Gradle Refresh, some of the classes with package names like org.labkey.SOME_SCOPE.xml (e.g., org.labkey.security.xml) are not found.

Cause: This is usually due to the relevant schema jars not having been built yet.

Solution: Run the deployApp command and then do a Gradle Refresh within IntelliJ. Though you can use a more targeted Gradle task schemaJar to build just the jar file that is missing, you might find yourself running many of these to resolve all missing classes, so it is likely most efficient to simply use deployApp

Gradle Troubleshooting

Understanding how gradle builds the server, and the relationships between building and cleaning can help you diagnose many build problems. See these related topics:

• Build LabKey from Source
• Gradle Cleaning

Update version of gradlePlugins

We maintain release notes for the gradlePlugins that describe changes and bug fixes to the plugins. Check these notes to find out if the problem you are having has already been addressed. Be sure to pay attention to the "Earliest compatible LabKey version" indicated with each release to know if it is compatible with your current LabKey version. If you want to update to a new version of the gradle plugins to pick up a change, you need only update the gradlePluginsVersion property in the root-level gradle.properties file:

gradlePluginsVersion=1.3.6

Could not Resolve All Files for Configuration

Problem: When compiling a module, an error such as the following appears

Execution failed for task ':server:modules:myModule:compileJava'.
> Could not resolve all files for configuration ':server:modules:myModule:compileClasspath'.
   > Could not find XYZ-api.jar (project :server:modules:XYZ).

<ij_msg_gr>Project resolve errors<ij_msg_gr><ij_nav>/Development/labkeyEnlistment/build.gradle<ij_nav><i><b>root project 'labkeyEnlistment': Unable to resolve additional project configuration.</b><eol>Details: org.gradle.api.artifacts.ResolveException: Could not resolve all dependencies for configuration ':server:modules:myModule:compileClasspath'.<eol>Caused by: org.gradle.internal.resolve.ArtifactNotFoundException: Could not find XYZ-api.jar (project :server:modules:XYZ).</i>

Cause: The settings.gradle file has included only a subset of the dependencies within the transitive dependency closure of your project and Gradle is unable to resolve the dependency for the api jar file. In particular, though your settings file likely includes the :server:modules:XYZ project, it likely does not include another project that myModule depends on that also depends on XYZ. That is, you have the following dependencies:

• myModule depends on otherModule
• myModule depends on XYZ
• otherModule depends on XYZ

include ':server:modules:myModule'
include ':server:modules:XYZ'

Solution: Include the missing project in your settings.gradle file

include ':server:modules:myModule'
include ':server:modules:otherModule'
include ':server:modules:XYZ'

Module Not Deployed

gradlew projects

Module Plugins Not Applied or Missing Properties orMethods

FAILURE: Build failed with an exception.

* Where:
Build file '/path/to/Development/labkey/root/server/modules/myModule/build.gradle' line: 6

* What went wrong:
A problem occurred evaluating project ':server:modules:myModule'.
> Could not find method implementation() for arguments com.sun.mail:jakarta.mail:1.6.5 on object of type org.gradle.api.internal.artifacts.dsl.dependencies.DefaultDependencyHandler.

Cause: As of LabKey Server version 21.3.0 we no longer apply the LabKey gradle plugins for all modules under the server/modules directory by default. Each module is responsible for applying the plugins it requires. When running a deployApp command, if we find a gradle project that contains a module.properties file but does not have a task named 'module', a message like the following will be shown:

The following projects have a 'module.properties' file but no 'module' task. These modules will not be included in the deployed server. You should apply either the 'org.labkey.build.fileModule' or 'org.labkey.build.module' plugin in each project's 'build.gradle' file.
        :server:modules:myModule
        :server:modules:otherModule

Solution: Apply the appropriate plugin in the module's build.gradle file by including either

plugins {
   id 'org.labkey.build.fileModule'
}

plugins {
   id 'org.labkey.build.module'
}

Starting Over with Gradle + IntelliJ

If you get into a situation where the IntelliJ configuration has been corrupted or created in a way that is incompatible with what we expect (which can happen if, for example, you say 'Yes' when IntelliJ asks if you want to have it link an unlinked Gradle project), these are the steps you can follow to start over with a clean setup for Gradle + IntelliJ:

• Shut down IntelliJ.
• Revert the <LK_ENLISTMENT>/.idea/gradle.xml file to the version checked in to VCS.
• Remove the <LK_ENLISTMENT>/.idea/modules directory.
• Remove the <LK_ENLISTMENT>/.idea/modules.xml file.
• Start up IntelliJ.
• Open the Gradle window (View > Tool Windows > Gradle) and click the Refresh icon, as described above.

Log4J

log4j2.xml Appender

Problem: The server seems to have started fine, but there are no log messages in the console after server start up.

Cause: The log4j2.xml file that controls where messages are logged does not contain the appropriate appender to tell it to log to the console. This appender is added when you deploy the application in development mode (as a consequence of running the deployApp command).

In log4j2, a custom appender is defined by creating a plugin. The log4j documentation has an explanation with an example for a custom appender here:

• http://logging.apache.org/log4j/2.x/manual/extending.html#Appenders

LabKey Under Tomcat

LabKey Fails to Start

If LabKey fails to start successfully, check the steps above to ensure that you have configured your JDK and development environment correctly. Some common errors you may encounter include:

org.postgresql.util.PSQLException: FATAL: password authentication failed for user "<username>" or java.sql.SQLException: Login failed for user "<username>"

These error occurs when the database user name or password is incorrect. If you provided the wrong user name or password in the .properties file that you configured above, LabKey will not be able to connect to the database. Check that you can log into the database server with the credentials that you are providing in this file.

java.net.BindException: Address already in use: JVM_Bind:<port x>:

This error occurs when another instance of LabKey, Tomcat, or another application is running on the same port. Specifically, possible causes include:

• LabKey is trying to double bind the specified port.
• LabKey (or Tomcat) is already running (as a previous service, under IntelliJ, etc.
• Microsoft Internet Information Services (IIS) is running on the same port.
• Another application is running on the same port.

• Shut down the instance of LabKey or the application that is running on the same port.
• Change the port for the other instance or application.
• Edit the application.properties file to specify a different port for your installation of LabKey.

server.port
context.httpPort

java.lang.NoClassDefFoundError: com/intellij/rt/execution/application/AppMain:
or
Error: Could not find or load main class com.intellij.rt.execution.application.AppMain:

In certain developer configurations, you will need to add an IntelliJ utility JAR file to your classpath.

• Edit the Debug Configuration in IntelliJ.
• Under the "VM Options" section, find the "-classpath" argument.
• Find your IntelliJ installation. On Windows machines, this is typically "C:\Program Files\JetBrains\IntelliJ IDEA <Version Number>" or similar. On OSX, this is typically "/Applications/IntelliJ IDEA <Version Number>.app" or similar.
• The required JAR file is in the IntelliJ installation directory, and is ./lib/idea_rt.jar. Add it to the -classpath argument value, separating it from the other values with a ":" on OSX and a ";" on Windows.
• Save your edits and start LabKey.

Database State Troubleshooting

If you build the LabKey source yourself from the source tree, you may need to periodically delete and recreate your LabKey database. The daily drops often include SQL scripts that modify the data and schema of your database.

Database Passwords Not Working

If your password contains a backslash, it may not get copied correctly from your pg.properties (or mssql.properties) to application.properties during the build and deploy process. Either use a password without a backslash, or double-escape the backslash within the pg.properties (or mssql.properties) file: (e.g. '\\\\').

System PATH

LabKey Server will automatically load certain binaries and tools (graphviz, dot, proteomics binaries) from Artifactory and expects to find them on the users's PATH. Be sure to include the following location in your system PATH. This directory won't exist before you build your server, but add it to the path anyway.

<LK_ENLISTMENT>\build\deploy\bin

For example, C:\labkeyEnlistment\build\deploy\bin.

Environment Variables Troubleshooting

JAVA_TOOL_OPTIONS

Most users will not have this problem. However, if you see a build error something like the following:

error: unmappable character for encoding ASCII

then setting this environment variable may fix the problem

export JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF8

Ext Libraries

Problem: Ext4 is not defined.

The page or a portion of the page is not rendering appropriately. Upon opening the browser console you see an error stating "Ext4 is not defined". This usually occurs due to a page not appropriately declaring client-side code dependencies.

Example of the error as seen in Chrome console.

Solution

Declare a dependency on the code/libraries you make use of on your pages. Depending on the mechanism you use for implementing your page/webpart we provide different hooks to help ensure dependent code is available on the page.

• Wiki: In a wiki you can use LABKEY.requiresScript() with a callback.
• File-based module view: It is recommended that you use file-scoped dependencies by declaring a view.xml and using the <dependencies> attribute.
• Java: In Java you can use ClientDependency.fromPath(String p) to declare dependencies for your view. Note, be sure to declare these before the view is handed off to rendering otherwise your dependency will not be respected.
• JSP: Override JspBase.addClientDependencies(ClientDependencies dependencies) in the .jsp. Here is an example.

Renaming Files and Directories

Occasionally a developer decides a module or file (and possibly associated classes) should be renamed for clarity or readability.

Most file/directory name changes are handled transparently: rename locally through Intellij, verify your build and that tests pass, and commit.

The big exception to this is doing a case-only rename. (e.g., "myfile.java" -> "MyFile.java"). Because Windows file systems are case-insensitive, this causes major headaches. Don't do it as described above.

But what if I really need to do a case-only rename?

Do a renaming to only change casing (ex. Mymodule -> myModule) in two rename steps:

1. First rename to an intermediate name that changes more than the casing: (ex. Mymodule -> myModule2)
2. Commit that change.
3. Then rename to the target final name (ex. myModule2 -> myModule)
4. Commit again.
5. Push your commits.
6. Squash-merge to develop.

Related Topics

• Set Up a Development Machine
• Troubleshoot LabKey Server
• Troubleshoot the Enterprise Pipeline
• Troubleshoot Rlabkey
• Troubleshoot Installation and Configuration