Table of Contents

guest
2018-12-18
Documentation Home
   What's New in 18.3
     What's New in 18.3
   Release Notes 18.3
     Feature Changes: 18.3
   Upcoming Features in 19.1
     Release Notes 19.1
Getting Started
   Try it Now: Data Grids
   Trial Servers
     Explore LabKey Server - Hosted Trial
       Learn About LabKey Server: Trial
       Exploring LabKey Collaboration
       Exploring Laboratory Data
       Exploring LabKey Studies
       Exploring LabKey Security
       Exploring Project Creation
       Extending Your Trial
       Trial Server: Modules
     Virtual Machine Trial Server
     Explore LabKey Biologics - Hosted Trial
       Learn About LabKey Biologics: Trial
   Tutorials
     Set Up for Tutorials: Trial
     Set Up for Tutorials - Non-Trial
     Navigation and UI Basics
   LabKey Server Editions
     Training
     LabKey Support Portal Pages
     Modules in LabKey Server Editions
LabKey Server
   Introduction to LabKey Server
   Navigate Server
   Data Basics
     Data Grids
       Data Grids: Basics
       Import Data
       Sort Data
       Filter Data
         Filtering Expressions
       Saved Filters and Sorts
       Column Summary Statistics
       Select Rows
       Customize Grid Views
       Export Data Grid
       Join Columns from Multiple Tables
         Lookup Columns
       Participant Details View
       Query Scope: Filter by Folder
       Field Properties Reference
         URL Field Property
         String Expression Format Functions
         Conditional Formats
         Date & Number Display Formats
           Date and Number Formats Reference
     Visualizations
       Bar Charts
       Box Plots
       Line Plots
       Pie Charts
       Scatter Plots
       Time Charts
       Column Visualizations
       Quick Charts
     Lists
       Tutorial: Lists
         List Tutorial: Setup
         Create a Joined Grid
         Add a URL Property
       Create Lists
         Create a List Design
         Import a List Archive
       Choose a Primary Key
       Edit a List Design
       Populate a List
       Manage Lists
       Connect Lists
     R Reports
       R Report Builder
       Saved R Reports
       Datasets in R
       Multi-Panel R Plots
       Lattice Plots
       Participant Charts in R
       R Reports with knitr
       Input/Output Substitutions Reference
       FAQs for LabKey R Reports
     RStudio Integration
       RStudio Set Up - Linux and OSX Systems
         Docker and TLS Set Up
       Connect to RStudio Pro
         Set Up RStudio Pro
       Edit R Reports in RStudio
       Export Data to RStudio
       Advanced Initialization of RStudio
       Configure Docker Host
     SQL Queries
       LabKey SQL Tutorial
       SQL Query Browser
       Create a SQL Query
       Edit SQL Query Source
       LabKey SQL Reference
       Lookups: SQL Syntax
       LabKey SQL Utility Functions
       Query Metadata
         Query Metadata: Examples
       Edit Query Properties
       Query Web Part
       SQL Synonyms
       LabKey SQL Examples
         Calculated Columns
         Pivot Queries
         Cross Folder Queries
         Cross Schema Joins
         Parameterized SQL Queries
         Various LabKey SQL Examples
     Reports and Charts
       Report Web Part: Display a Report or Chart
       Data Views Browser
       Query Snapshots
       Attachment Reports
       Link Reports
       Participant Reports
       Query Report
       Manage Data Views
         Manage Dataset/Report Notifications
       Manage Categories
       Manage Thumbnail Images
       Measure and Dimension Columns
       Legacy Reports
         Advanced Reports / External Reports
         Legacy Chart Views
         Crosstab Reports
     External Schemas and Data Sources
       External MySQL Data Sources
       External Oracle Data Sources
       External Microsoft SQL Server Data Sources
       External PostgreSQL Data Sources
       External SAS Data Sources
       External Redshift Data Sources
       Linked Schemas and Tables
       Manage Remote Connections
     Where Should My Data Go?
     LabKey Data Structures
     Preparing Data for Import
     Controlling Data Scope
     Data Quality Control
     Quality Control Trend Reports
       Define QC Trend Report
       Use QC Trend Reports
       QC Trend Report Guide Sets
     Search
       Search Administration
     Spotfire Integration
     LabKey Natural Language Processing (NLP)
       Natural Language Processing (NLP) Pipeline
       Metadata Json Files
       Document Abstraction Workflow
       Automatic Assignment for Abstraction
       Manual Assignment for Abstraction
       Abstraction Task Lists
       Document Abstraction
       Review Document Abstraction
       Review Multiple Result Sets
       NLP Result Transfer
       Configure LabKey NLP
       Run NLP Pipeline
   Assay Data
     Tutorial: Design a General Purpose Assay Type (GPAT)
       Step 1: Assay Tutorial Setup
       Step 2: Infer an Assay Design from Spreadsheet Data
       Step 3: Import Assay Data
       Step 4: Work with Assay Data
       Step 5: Data Validation
       Step 6: Integrate Assay Data into a Study
     Tutorial: ELISA Assay
     ELISpot Assay
       Tutorial: ELISpot Assay Tutorial
         Import ELISpot Data
         Review ELISpot Data
       ELISpot Properties
     Flow Cytometry
       Flow Cytometry Overview
       LabKey Flow Module
         Set Up a Flow Folder
         Tutorial: Import a Flow Workspace
           Step 1: Customize Your Grid View
           Step 2: Examine Graphs
           Step 3: Examine Well Details
           Step 4: Export Flow Data
           Step 5: Flow Quality Control
         Tutorial: Perform a LabKey Flow Analysis
           Set Up Flow For LabKey Analysis
           Step 1: Define a Compensation Calculation
           Step 2: Define an Analysis
           Step 3: Apply a Script
           Step 4: View Results
         Tutorial: Set Flow Background
         Import a Flow Workspace and Analysis
           FCS File Resolution
         Edit Keywords
         Add Sample Descriptions
         Custom Flow Queries
           Add Statistics to FCS Queries
           Calculate Suites of Statistics for Every Well
           Flow Module Schema
         Analysis Archive Format
         Supported FlowJo Versions
       FCS Express
         Tutorial: Import Flow Data from FCS Express
       Add Flow Data to a Study
       FCS keyword utility
       Flow Team Members
     FluoroSpot Assay
     HPLC - High-Performance Liquid Chromatography
     Luminex
       Luminex Assay Tutorial Level I
         Set Up Luminex Tutorial Project
         Step 1: Create a New Luminex Assay Design
         Step 2: Import Luminex Run Data
         Step 3: Exclude Analytes for QC
         Step 4: Import Multi-File Runs
         Step 5: Copy Luminex Data to Study
       Luminex Assay Tutorial Level II
         Step 1: Import Lists and Assay Archives
         Step 2: Configure R, Packages and Script
         Step 3: Import Luminex Runs
         Step 4: View 4pl and 5pl Curve Fits
         Step 5: Track Analyte Quality Over Time
         Step 6: Use Guide Sets for QC
         Step 7: Compare Standard Curves Across Runs
       Track Single-Point Controls in Levey-Jennings Plots
       Import Luminex Runs
       Luminex Calculations
       Luminex QC Reports and Flags
       Luminex Reference
         Review Luminex Assay Design
         Luminex Properties
         Luminex File Formats
         Review Well Roles
         Luminex Conversions
         Customize Luminex Assay for Script
         Review Fields for Script
       Troubleshoot Luminex Transform Scripts and Curve Fit Results
     Microarray
       Microarray Assay Tutorial
       Expression Matrix Assay Tutorial
       Microarray Properties
     Mass Spectrometry
       Panorama - Targeted Proteomics
         Panorama Partners Program
         Configure Panorama Folder
         Panorama Document Revision Tracking
         Panorama QC Dashboard
         Panorama QC Plots
         Panorama QC Plot Types
         Panorama QC Annotations
         Panorama QC Guide Sets
         Panorama: Pareto Plots
         Panorama: Heat Maps
         Panorama: Calibration Curves
         Panorama: Figures of Merit and Pharmacokinetics (PK)
         Skyline Annotation Data
         Working with Small Molecule Targets
       Discovery Proteomics
         Discovery Proteomics Tutorial
           Step 1: Set Up for Proteomics Analysis
           Step 2: Search mzXML Files
           Step 3: View PeptideProphet Results
           Step 4: View ProteinProphet Results
           Step 5: Compare Runs
           Step 6: Search for a Specific Protein
         Work with MS2 Data
           Search MS2 Data Via the Pipeline
             Set Up MS2 Search Engines
               Set Up Mascot
               Set Up Sequest
               Set Up Comet
               Working with mzML Files
             Search and Process MS2 Data
               Configure Common Parameters
               Configure X! Tandem Parameters
               Configure Mascot Parameters
               Configure Sequest Parameters
                 Sequest Parameters
                 MzXML2Search Parameters
                 Examples of Commonly Modified Parameters
               Configure Comet Parameters
             Import Existing Analysis Results
             Trigger MS2 Processing Automatically
             Set Proteomics Search Tools Version
           Explore the MS2 Dashboard
           View an MS2 Run
             Customize Display Columns
               Peptide Columns
               Protein Columns
             View Peptide Spectra
             View Protein Details
             View Gene Ontology Information
             Experimental Annotations for MS2 Runs
           Protein Search
           Peptide Search
           Compare MS2 Runs
             Compare ProteinProphet
           Export MS2 Runs
           Export Spectra Libraries
           View, Filter and Export All MS2 Runs
           Work with Mascot Runs
         Discovery Proteomics Video
         Loading Public Protein Annotation Files
         Using Custom Protein Annotations
         Using ProteinProphet
         Using Quantitation Tools
         MaxQuant Module
         Link Protein Expression Data with Annotations
         Spectra Counts
           Label-Free Quantitation
         Combine XTandem Results
         MS1
           MS1 Pipelines
         Run Groups
       Protein Expression Matrix Assay
       Proteomics Team
     NAb (Neutralizing Antibody) Assays
       Tutorial: NAb Assay
         Step 1: Create a NAb Assay Design
         Step 2: Import NAb Assay Data
         Step 3: View High-Throughput NAb Data
         Step 4: Explore NAb Graph Options
       Work with Low-Throughput NAb Data
       Use NAb Data Identifiers
       NAb Assay QC
       Work with Multiple Viruses per Plate
       NAb Plate File Formats
       Customize NAb Plate Template
       NAb Properties
     Signal Data Assay
     Assay Administrator Guide
       Assay Feature Matrix
       Set Up Folder For Assays
       Assay Designs and Types
       Import Assay Design
       Design a New Assay
         Editing Assay Data
         Assay Design Properties
       Design a Plate-Based Assay
         Edit Plate Templates
       Participant/Visit Resolver Field
       Manage an Assay Design
       Improve Data Entry Consistency & Accuracy
       Set up a Data Transformation Script
       Copy Assay Data into a Study
         Copy-To-Study History
       XAR Archives
         XAR Terminology
         XAR Files
           Uses of XAR.xml Files
           Import a XAR.xml
             Troubleshoot XAR Import
             Import XAR Files Using the Data Pipeline
           Example 1: Review a Basic XAR.xml
           Examples 2 & 3: Describe Protocols
           Examples 4, 5 & 6: Describe LCMS2 Experiments
         Life Science Identifiers (LSIDs)
           LSID Substitution Templates
     Assay User Guide
       Import Assay Runs
       Exclude Assay Data
       Reimport Assay Runs
       Export Assay Data
     Assay Request Tracker
       Premium Resource: Using the Assay Request Tracker
       Premium Resource: Assay Request Tracker Administration
     Electronic Laboratory Notebooks (ELN)
       Tutorial: Electronic Lab Notebook
         Step 1: Create the User Interface
         Step 2: Import Lab Data
         Step 3: Link Assays to Samples
         Step 4: Using and Extending the ELN
     Reagent Inventory
     Sample Sets
       Import Sample Sets
       Samples: Unique IDs
       View SampleSets and Samples
       Link Assay Data to Sample Sets
       Parent Samples: Derivation and Lineage
       Sample Sets: Examples
       'Active' Sample Set
     DataClasses
   Longitudinal Studies
     Study Tour
     Tutorial: Cohort Studies
       Step 1: Install the Sample Study
       Step 2: Study Data Dashboards
       Step 3: Integrate Data from Different Sources
       Step 4: Compare Participant Performance
     Tutorial: Set Up a New Study
       Step 1: Create Study Framework
       Step 2: Add Participants and Data
       Step 3: Identify Cohorts
       Step 4: Integrate and Visualize Data
       Step 5: Add Specimen Data
     Study User Guide
       Study Navigation
       Study Data Browser
       Cohorts
       Participant Groups
       Comments
       Dataset Quality Control States
     Study Administrator Guide
       Study Management
         Additional Study Properties
         Manage Datasets
         Manage Visits or Timepoints
         Study Schedule
         Manage Locations
         Manage Cohorts
         Manage Participant IDs
           Alternate Participant IDs
           Alias Participant IDs
         Manage Comments
         Manage Study Security (Dataset-Level Security)
           Configure Permissions for Reports & Views
           Securing Portions of a Dataset (Row and Column Level Security)
         Manage Dataset QC States
         Manage Study Products
         Manage Treatments
         Manage Assay Schedule
         Assay Progress Report
         Demonstration Mode
       Create a Study
       Create and Populate Datasets
         Create a Dataset from a File
         Create a Dataset by Defining Fields
         Tutorial: Inferring Datasets from Excel and TSV Files
         Import Data to a Dataset
           Import via Copy/Paste
           Import From a Dataset Archive
             Create Pipeline Configuration File
         Import Study Data From REDCap Projects
         Dataset Properties
         Study: Reserved and Special Fields
         Edit Dataset Properties
         Dataset System Fields
       Visits and Dates
         Create Visits Manually
         Edit Visits or Timepoints
         Import Visit Map
         Import Visit Names / Aliases
         Continuous Studies
         Study Visits and Timepoints FAQ
       Import, Export, and Reload a Study
         Export a Study
         Import a Study
         Reload a Study
           Study Archive Files and Formats
           Serialized Elements and Attributes of Lists and Datasets
       Publish a Study
         Publish a Study: Protected Health Information / PHI
         Publish a Study: Refresh Snapshots
       Ancillary Studies
       Shared Datasets and Timepoints
       Data Aliasing
       Study Data Model
         How is Study Data Stored in LabKey Server?
       Create a Vaccine Study Design
         Vaccine Study: Data Storage
     Specimen Tracking
       Tutorial: Specimen Request System
         Step 1: Repository Setup (Admin)
         Step 2: Request System (Specimen Coordinator)
         Step 3: Request Specimens (User)
         Step 4: Track Requests (Specimen Coordinator)
       Specimens: Administrator Guide
         Import Specimen Spreadsheet Data
         Import a Specimen Archive
           Specimen Archive File Reference
           Specimen Archive Data Destinations
           Troubleshoot Specimen Import
           Import FreezerPro Data
         Delete Specimens
         Specimen Properties and Rollup Rules
         Customize Specimens Web Part and Grid Views
         Flag Specimens for Quality Control
         Edit Specimen Data
         Customize the Specimen Request Email Template
         Export a Specimen Archive
       Specimen Coordinator Guide
         Email Specimen Lists
       View Specimen Vials and Grouped Vials
       Generate Specimen Reports
       Laboratory Information Management System (LIMS)
       Using Sample Sets as an Alternative to a Specimen Repository
     Electronic Health Records (EHR)
       EHR: Animal History
       EHR: Animal Search
       EHR: Data Entry
       EHR: Genetics Algorithms
       EHR: Administration
       EHR Team
       EHR Project Setup
     Structured Narrative Datasets
       SND: Packages
       SND: Categories
       SND: Super-packages
       SND: Projects
       SND: Events
       SND: QC and Security
       SND: APIs
       SND: Event Triggers
       SND: UI
       Extending SND Tables
       XML Import of Packages
     Enterprise Master Patient Index Integration
   Collaboration
     Tutorial: Collaboration Tools
       Step 1: Use the Message Board
       Step 2: Collaborate Using a Wiki
       Step 3: Track Issues
     Tutorial: File Repository
       Step 1: Set Up a File Repository
       Step 2: File Repository Administration
       Step 3: Search the Repository
       Step 4: Import Data from the Repository
     Files
       Using the Files Repository
       Share and View Files
       Controlling File Display via the URL
       Import Data from Files
       Linking Assays with Images and Other Files
       Linking Data Records with External Files
       File Metadata
       File Transfer Module / Globus File Sharing
       File Administrator Guide
         Files Web Part Administration
         Upload Files: WebDAV
         Set File Roots
           Troubleshoot File Roots and Pipeline Overrides
         File Terminology
         Integrating S3 Cloud Data Storage
       Data Processing Pipeline
         Set a Pipeline Override
         Pipeline Protocols
         File Watchers
       Enterprise Pipeline
         Install Prerequisites for the Enterprise Pipeline
           JMS Queue
           RAW to mzXML Converters
         Configure LabKey Server to use the Enterprise Pipeline
           Configure the Conversion Service
           Configure Remote Pipeline Server
           Configure Pipeline Path Mapping
         Use the Enterprise Pipeline
         Troubleshoot the Enterprise Pipeline
     Messages
       Use Message Boards
       Administer Message Boards
       Object-Level Discussions
     Wikis
       Wiki Admin Guide
         Copy Wiki Pages
       Wiki User Guide
         Wiki Syntax
         Wiki Syntax: Macros
         Markdown Syntax
         Special Wiki Pages
         Embed Live Content in HTML Pages or Messages
           Examples: Embedded Web Parts
           Web Part Configuration Properties
         Add Screenshots to a Wiki
         Manage Wiki Attachment List
     Issue/Bug Tracking
       Using the Issue Tracker
       Issue Tracker: Administration
     Electronic Data Capture (EDC)
       Tutorial: Survey Designer, Step 1
       Tutorial: Survey Customization, Step 2
       Survey Designer: Reference
       Survey Designer: Example Questions
       REDCap Survey Data Integration
       Medidata / CDISC ODM Integration
     Workflow Module
       Workflow Tutorial
         Step 1: Set Up Workflow Tutorial
         Step 2: Run Sample Workflow Process
         Step 3: Workflow Process Definition
         Step 4: Customize Workflow Process Definition
       Workflow Process Definition
     Adjudication Module
       Set Up an Adjudication Folder
       Initiate an Adjudication Case
       Make an Adjudication Determination
       Monitor Adjudication
       Infection Monitor
       Audit of Adjudication Events
       Role Guide: Adjudicator
       Role Guide: Adjudication Lab Personnel
     Tours for New Users
     Contacts
     How to Cite LabKey Server
   Development
     Set Up a Development Machine
       Enlist in the Version Control Project
       Add the Proteomics Binaries
       Build LabKey from Source
       Customize the Build
       SVN and Git Ignore Configurations
       Build Offline
       Gradle Cleaning
       Gradle Properties
       Gradle: How to Add Modules
       Gradle: Declare Dependencies
       Gradle Tips and Tricks
       Run Selenium Tests
       Create Production Builds
       Gradle: Migration from Ant
       Machine Security
       Notes on Setting up OSX for LabKey Development
       Tomcat 7 Encoding
       Troubleshoot Development Machines
     LabKey Client APIs
       JavaScript API
         Tutorial: Create Applications with the JavaScript API
           Step 1: Create Request Form
           Step 2: Confirmation Page
           Step 3: R Histogram (Optional)
           Step 4: Summary Report For Managers
           Repackaging the App as a Module
         Tutorial: Use URLs to Pass Data and Filter Grids
           Choose Parameters
           Show Filtered Grid
         Tutorial: Visualizations in JavaScript
           Step 1: Export Chart as JavaScript
           Step 2: Embed the Script in a Wiki
           Modify the Exported Chart Script
           Display the Chart with Minimal UI
         JavaScript API - Samples
         Adding a Report to a Data Grid with JavaScript
         Export Data Grid as a Script
         Custom HTML/JavaScript Participant Details View
         Example: Master-Detail Pages
         Custom Button Bars
           Premium Resource: Invoke JavaScript from Custom Buttons
         Premium Resource: Sample Management Demo
         Insert into Audit Table via API
         Programming the File Repository
         Declare Dependencies
         Loading ExtJS On Each Page
         Licensing for the ExtJS API
         Search API Documentation
         Naming & Documenting JavaScript APIs
           Naming Conventions for JavaScript APIs
           How to Generate JSDoc
           JsDoc Annotation Guidelines
       Java API
         LabKey JDBC Driver
         Remote Login API
         Security Bulk Update via API
       Perl API
       Python API
         Premium Resource: Python API Demo
       Rlabkey Package
         Troubleshoot Rlabkey
       SAS Macros
         SAS Setup
         SAS Macros
         SAS Security
         SAS Demos
       HTTP Interface
         Examples: Controller Actions
         Example: Access APIs from Perl
       API Keys
       Compliant Access via Session Key
     Develop Modules
       Tutorial: Hello World Module
       Map of Module Files
       Example Modules
       Modules: Queries, Views and Reports
         Module Directories Setup
         Module Query Views
         Module SQL Queries
         Module R Reports
         Module HTML and Web Parts
       Modules: JavaScript Libraries
       Modules: Assay Types
         Tutorial: Define an Assay Type in a Module
         Assay Custom Domains
         Assay Custom Details View
         Loading Custom Views
         Example Assay JavaScript Objects
         Assay Query Metadata
         Customize Batch Save Behavior
         SQL Scripts for Module-Based Assays
         Transformation Scripts
           Example Workflow: Develop a Transformation Script (perl)
           Example Transformation Scripts (perl)
           Transformation Scripts in R
           Transformation Scripts in Java
           Transformation Scripts for Module-based Assays
           Run Properties Reference
           Transformation Script Substitution Syntax
           Warnings in Tranformation Scripts
       Modules: Folder Types
       Modules: Query Metadata
       Modules: Report Metadata
       Modules: Custom Footer
       Modules: Custom Header
       Modules: SQL Scripts
       Modules: Database Transition Scripts
       Modules: Domain Templates
       Modules: Java
         Module Architecture
         Getting Started with the Demo Module
         Tutorial: Hello World Java Module
         The LabKey Server Container
         Implementing Actions and Views
         Implementing API Actions
         Integrating with the Pipeline Module
         Integrating with the Experiment API
         Using SQL in Java Modules
         GWT Integration
         GWT Remote Services
         Database Development Guide
         Java Testing Tips
         HotSwapping Java classes
         Deprecated Components
       Modules: Custom Login Pages
       ETL: Extract Transform Load
         Tutorial: Extract-Transform-Load (ETL)
           ETL Tutorial: Set Up
           ETL Tutorial: Run an ETL Process
           ETL Tutorial: Create a New ETL Process
         ETL: Define an ETL Using XML
         ETL: User Interface
         ETL: Configuration and Schedules
         ETL: Filter Strategies
         ETL: Column Mapping
         ETL: Queuing ETL Processes
         ETL: Stored Procedures
           ETL: Stored Procedures in MS SQL Server
           ETL: Functions in PostgreSQL
           ETL: Check For Work From a Stored Procedure
         ETL: SQL Scripts
         ETL: Remote Connections
         ETL: Logs and Error Handling
         ETL: All Jobs History
         ETL: Examples
         ETL: Reference
         Premium Resource: ETL Best Practices
       Deploy Modules to a Production Server
       Upgrade Modules
       Main Credits Page
       Module Properties Reference
       Node.js Build Dependency
     Common Development Tasks
       Trigger Scripts
         Availability of Server-side Trigger Scripts
       Script Pipeline: Running R and Other Scripts in Sequence
       LabKey URLs
         URL Actions
       How To Find schemaName, queryName & viewName
       LabKey/Rserve Setup Guide
       Web Application Security
         HTML Encoding
         Cross-Site Request Forgery (CSRF) Protection
       MiniProfiler
       Using loginApi.api
       Configuring IntelliJ for XML File Editing
       Premium Resource: Git Branch Naming
       Premium Resource: Best Practices for Writing Automated Tests
     LabKey Open Source Project
       Release Schedule
       Current Release Details
       Previous Releases
       Open Source Project: Entering Issues
       Branch Policy
       Test Procedures
       Run Automated Tests
       Hotfix Policy
       Submit Contributions
         Confidential Data
         CSS Design Guidelines
         UI Design Patterns
         Documentation Style Guide
         Check in to the Source Project
         Renaming files in Subversion
     Developer Reference
   Administration
     Tutorial: Security
       Step 1: Configure Permissions
       Step 2: Test Security with Impersonation
       Step 3: Audit User Activity
       Step 4: Handle Protected Health Information (PHI)
     Projects and Folders
       Project and Folder Basics
       Site Structure: Best Practices
       Folder Types
       Manage Projects and Folders
         Create a Project or Folder
         Move, Delete, Rename Projects and Folders
         Enable a Module in a Folder
         Export / Import a Folder
         Export and Import Permission Settings
         Manage Email Notifications
         Define Hidden Folders
       Establish Terms of Use
       Workbooks
       Shared Project
     Build User Interface
       Page Admin Mode
       Add Web Parts
       Manage Web Parts
       Web Part Inventory
       Use Tabs
       Add Custom Menus
       Web Parts: Permissions Required to View
       Old/New User Interface Comparison
     Security
       Configure Permissions
       Security Groups
         Global Groups
         Site Groups
         Project Groups
         Guests / Anonymous Users
       Security Roles Reference
         Role / Permissions Matrix
         Administrator Role / Permissions Matrix
         Matrix of Report, Chart, and Grid Permissions
         Developer Roles
       User Accounts
         My Account
         Site Administrator
         Add Users
         Manage Users
         Manage Project Users
       Authentication
         Configure Database Authentication
           Passwords
           Password Reset & Security
         Configure SAML Authentication
         Configure CAS Single Sign On Authentication (SSO)
         Configure CAS Identity Provider
         Configure LDAP Authentication
         LDAP User/Group Synchronization
         Configure Duo Two-Factor Authentication
         Create a netrc file
         HTTP Basic Authentication
       Virus Checking
       Test Security Settings by Impersonation
       Premium Resource: Best Practices for Security Scanning
     Compliance
       Compliance: Overview
       Compliance: Checklist
       Compliance: User Accounts and Login
       Compliance: Setting PHI Levels on Fields
       Compliance: Terms of Use
       Compliance: Security Roles
       Compliance: Configure PHI Data Handling
       Compliance: Logging
       PHI Report
       Electronic Signatures / Sign Data
       GDPR Compliance
       Premium Resource: Configuring LabKey for GDPR Compliance
     Admin Console
       Site Settings
         Usage/Exception Reporting - Details
       Look and Feel Settings
         Branding
         Web Site Theme
       Email Template Customization
       Experimental Features
       Manage Missing Value Indicators / Out of Range Values
       Short URLs
       Configure System Maintenance
       Configure Scripting Engines
       Audit Site Activity
         SQL Query Logging
       Export Diagnostic Information
       Actions Diagnostics
       Cache Statistics
       Dump Heap
       Memory Usage
       Running Threads
       Query Performance
       Site/Container Validation
     Install LabKey
       Install LabKey Manually
         Install Required Components
         Configure the LabKey Web Application
         labkey.xml Configuration File
         Third-Party Components and Licenses (Optional Install)
         Creating & Installing SSL Certificate on Tomcat
       Supported Technologies
       Troubleshoot Server Installation and Configuration
         Installation Error Messages
         Dump Memory and Thread Usage Debugging Information
       Common Install Tasks
         Install Microsoft SQL Server
         Install PostgreSQL (Windows)
         Install PostgreSQL (Linux or Macintosh)
         Notes on Installing PostgreSQL on All Platforms
         Install and Set Up R
           Configure an R Docker Engine (Experimental Feature)
           Determine Available Graphing Functions
         Install SAS/SHARE for Integration with LabKey Server
         Configure Webapp Memory
         Set Up Robots.txt and Sitemaps
         GROUP_CONCAT Install
         PremiumStats Install
         Deploying an AWS Windows Application Firewall
       Default Modules
       Additional Source Modules
       Example Setups and Configurations
         Example Hardware/Software Configuration
         Set up a JMS-Controlled Remote Pipeline Server
         Configure R on Linux
         Configure the Virtual Frame Buffer on Linux
       Server Bootstrap Properties
       Sending Email from Non-LabKey Domains
     Upgrade LabKey
       Manual Upgrade Checklist
       Manual Upgrade Script for Linux, MacOSX, and Solaris
       Upgrade Support Policy
     Backup and Maintenance
       Backup Checklist
       A Sample Backup Plan
       Sample Scripts for Backup Scenarios
       PostgreSQL Maintenance
       Administer the Site Down Servlet
     Staging, Test and Production Servers
       Example of a Large-Scale Installation
       Tips for Configuring a Staging Server
   Troubleshoot LabKey Server
LabKey Biologics
   LabKey Biologics - Videos
   Biologics Tutorial: Navigate and Search the Registry
   Biologics Tutorial: Add Sequences to the Registry
   Biologics Tutorial: Register Samples and Experiments
   Biologics Tutorial: Work with Mixtures and Batches
   Biologics Tutorial: Create a New Biologics Project
   Terminology
   Entity Registration
     Navigate the Registry
     Search the Registry
     Register Nucleotide Sequences
     Register Protein Sequences
       Protein Sequence Annotations
     Register Leaders, Linkers, and Tags
     Register Molecules
     Molecule Sets and Molecular Species
     Vectors, Constructs, Cell Lines, and Expression Systems
     Entity Lineage
     Bulk Registration of Entities
     Use the Registry API
   Biologics: Sample Management
     Samples and Sample Sets Within LabKey Server
     Sample Sets and Detail Pages
     Generate Samples
   Biologics: Assay Management
     Biologics: Upload Assay Data
     Biologics: Set Up Assays
     Biologics: Assay Grids
   Media Registration
     Registering Ingredients and Raw Materials
     Registering Mixtures (Recipes)
     Registering Batches
   Biologics: Workflow Management
     Workflow
     Service Request Tracker Set Up
   Biologics: Experiments
   Biologics Administration
     Configuring Grids, Detail Views, and Entry Forms
     Building Biologics
Premium Resources
   Premium Edition Required
Community Resources
   LabKey Terminology/Glossary
   FAQ - Frequently Asked Questions
   System Integration: Instruments and Software
   Demos
   Videos
   LabKey User Conference Resources
     LabKey User Conference 2018
       LabKey User Workshop 2018
     LabKey User Conference 2017
     LabKey User Conference 2016
     LabKey User Conference 2015
     LabKey User Conference 2014
     LabKey User Conference 2013
     LabKey User Conference 2012
     LabKey User Conference 2011
   Documentation Archive
     Release Notes 18.2
       What's New in 18.2
       Upcoming Feature Changes in 18.3
     Release Notes 18.1
       What's New in 18.1
     Release Notes 17.3
     Release Notes 17.2
     Release Notes 17.1
     Release Notes 16.3
   LabKey Privacy Policy

Documentation Home


Getting Started

LabKey Server Documentation

Biologics Documentation

Premium Resources

Community Resources




What's New in 18.3


We're delighted to announce the release of LabKey Server version 18.3.

Feature Highlights of Version 18.3

See the full Release Notes 18.3




What's New in 18.3


We're delighted to announce the release of LabKey Server version 18.3.

Feature Highlights of Version 18.3

See the full Release Notes 18.3




Release Notes 18.3


R Integration

  • Integrate with RStudio Pro - Use RStudio Pro to view, edit, and export LabKey Server data. (docs)
  • RStudio Data Browser - View data from the LabKey schemas within RStudio or RStudio Pro. (docs)
  • Docker-based R Engines (Experimental Feature) - Connect the server to a Docker-based R engine. (docs)
  • Per-Project R Engines - Configure multiple R engines. Individual projects can choose alternative R engines other than the site default engine. (docs)
  • Sandboxed R Engines - Declare an R engine as sandboxed to isolate it for security purposes. (docs)
  • Rlabkey Helper Functions - Two new functions have been added to the Rlabkey package: labkey.transform.getRunPropertyValue and labkey.transform.readRunPropertiesFile. Use these helper functions within an assay's R transform script to load the run data and properties. Available in Rlabkey version 2.2.4. (docs | docs)
  • Rlabkey: Set and Get Module Properties - New Rlabkey functions, setModuleProperty and setModuleProperty, allow developers to set and get a module property for a specific LabKey Server project. Available in Rlabkey version 2.2.4. (docs)

Administration

  • Antivirus Scanning - Automatically scan uploaded files for viruses using ClamAV. (docs)
  • Analyst and Trusted Analyst Roles - New site-wide roles provide different levels of access for developers writing R scripts and other code. Analysts can write code that runs on the server under their own user account. Trusted Analysts can write code to be run by other users, including custom SQL queries. (docs)
  • Platform Developer Role - A new site-wide role allows trusted users to write and deploy code. Platform developers can create and edit code and queries in folders where they have edit permission. The Platform Developer role replaces and provides the same permissions that the "Site Developers" group previously provided. (docs)
  • LDAP User/Group Synchronization - Synchronize LabKey Server user accounts with an external LDAP system. (docs)
  • Export Diagnostic Information - Download diagnostics to assist LabKey Client Services in resolving problems you encounter. (docs)
  • Connect to Existing Amazon S3 Directories - Connect to an existing S3 directory or create a new one for each LabKey folder. (docs)
  • Custom HTML Header - Add an HTML header to your site for a more custom look. Also available in 18.2. (docs)
  • Diagnostic Logs - Easily bundle and download server logs for diagnostic purposes. (docs)
  • Improved Navigation Menu - The project menu and any custom menus you define have a more consistent interface, and each contain graphical elements signaling that they are interactive elements. You can also access the project menu from the admin console. (docs)
  • Subfolder Web Part - This web part shows the subfolders of the current location; included in new collaboration folders by default. Also available in the 18.2 release. (docs)

LabKey Server Trial Instances

  • Biologics Trial Servers Available - Create a web-based server instance to explore LabKey Biologics. (docs)
  • Virtual Machine Trial Servers - Download a virtual machine image of LabKey Server for on-premise evaluation. (docs | download)

Biologics

  • Experiment Framework - Manage experiments, associated samples and assay data. Add and remove samples from an existing experiment. View the assay data linked to samples in an experiment. (docs)
  • Lineage Grid - View sample lineage in a grid format. (docs)
  • Import Constructs from GenBank Files - Import Constructs using the GenBank file format (.gb, .genbank). (docs)
  • Custom Views - Administrators can define and expose multiple views on Biologics grid data, allowing end users to select from the list of views. (docs)
  • Tutorials - Use these tutorials as a guided tour of LabKey Biologics:

Reporting and Visualization

  • Scatter and Line Plot Enhancements - Specify multiple Y axes. Show all data on a single plot or display separate plots per measure. (docs | docs)
  • Time Chart Enhancements - Hide the trend line or the data points on a time chart. (docs)
  • Add Charts to Participant Views - Include visualizations in participant views within a study. (docs)
  • QC Trend Report Options - In addition to Levey-Jennings plots, apply Moving Range and Cumulative Sum plots to GPAT and file-based assay data. (docs)

Electronic Data Capture (Survey)

  • Date & Time Question Type - An additional question type combining a date picker and time selector has been added. (docs)

Document Abstraction and Annotation

  • Document Abstraction: Multiple Result Set Review - Reviewers can now simultaneously see all abstracted results for a given document and select among differing value assignments. (docs)
  • Transfer Results between Servers - Batches of abstraction/annotation results can be encrypted and transferred between servers. (docs)

Adjudication

  • Adjudication Improvements - Compare determinations regarding HIV-1 and HIV-2 infection separately and raise alerts to infection monitors when there is consensus about a positive diagnosis for any strain. (docs)

Panorama

  • Panorama: Normalized Y-axes in QC Plots - Support for normalizing Levey-Jennings and Moving Range plots using percent of mean or standard deviation as the zero point on the Y-axis. (docs)
  • Improved Figures of Merit performance - Rendering performance for the Figures of Merit report has been improved. (docs)
  • Read Chromatograms Directly from SKYD files - An experimental feature allows you to read chromatograms directly from SKYD files instead of storing them in the database. (docs)

Development and APIs

  • Improvements to Experiment.saveBatch - The Experiment.saveBatch action has been modified to support non-assay backed runs. This allows developers to import assay data not associated with a particular assay design. In earlier versions of the API, identifying a specific assay design was required. This change has been incorporated into both the JavaScript API and the R package Rlabkey. (JS docs | R docs)
  • Modal Dialogs in JS API - A new JavaScript function in the LabKey.Utils API displays modal dialogs on the page that can have dynamic content. (docs)
  • Display PNG Images in Grids - Use a displayColumnFactory to override the display properties of a column to display images and thumbnails. (docs)
  • NaN and Infinity Values - LabKey SQL supports constants NaN, INF, and -INF. (docs)
  • Domain APIs - The Python and R APIs have been updated to allow for the inference, creation, and update of dataset and list schemas. Also available in 18.2. (Python docs | R docs)

Potential Backwards Compatibility Issues

  • Transition to Jackson Serialization - XStream serialization of pipeline jobs has been replaced with Jackson serialization. Modules that currently depend on XStream should transition to Jackson. XStream serialization has also been removed from the API. While it is always recommended to let running pipeline jobs complete prior to upgrading the server, this is especially true for the 18.3 upgrade as any jobs in progress cannot be restarted automatically. (docs)
  • Redshift Driver Upgrade - If you are using Redshift with LabKey, download the latest JDBC 4.2 driver and replace the existing driver in your Tomcat lib directory. (docs)
  • Changes to CSRF Setting - At 18.3 upgrade time, the CSRF checking setting on all servers will be set to "All POST requests". Site administrators will have the ability to revert back to "Admin requests" for deployments that still need to make their external modules or custom pages compatible with this setting. For release 19.1, we plan to remove the setting entirely and check CSRF tokens on every POST (except for specially annotated actions). (docs)
  • ETL Functionality Shipping with Premium Editions - The ETL module is no longer be distributed with the Community Edition. This functionality is now included with Premium Editions of LabKey Server. (docs)
  • returnURL Handling - By default, servers will reject returnURL values that are not identified as being to the local server, either by being a relative path, or because the hostname matches the server's configured base server URL in Site Settings. Rejected URLs will be logged to labkey.log with the message "Rejected external host redirect" along with URL details. Administrators may enable an experimental feature in the Admin Console to allow these URLs through again. This is a stopgap to allow for backwards compatibility, if required. In the future, we will either remove the experimental feature and always reject external URLs, or allow admins to whitelist the sites that should be allowed.

Operations

  • Support for Java 11 - We recommend upgrading your server installation to Java 11. Oracle is expected to end public support for Java 8 in January 2019, and, as a result, LabKey Server will no longer support Java 8 for the 19.1 release. For details see Supported Technologies.
  • Support for PostgreSQL 11 - PostgreSQL 11.1 and above is supported (not the initial PostgreSQL 11.0 release). For details, see Supported Technologies.
  • Remove support for PostgreSQL 9.3 - PostgreSQL 9.3 reached end-of-life in November 2018. We recommend upgrading your PostgreSQL installation to version 10 or later. For details, see Supported Technologies.

Extract-Transform-Load (ETLs)

  • ETL Definition Editor Enhancements - Define a new ETL using an existing one as a template. Autocomplete while entering XML. (docs)

Premium Resources

Premium Resources Available - Expanded documentation and code examples are available with premium editions of LabKey Server, including




Feature Changes: 18.3


This topic is provided to help administrators and developers migrate specific features to the 18.3 release.

Extract-Transform-Load (ETL)

The ETL module has been removed from the Community Edition and the functionality has migrated to the dataintegration module, available with all Premium Editions. Premium clients will not experience any change in functionality or interruption in service.

The announcement made on the Developer Forum about this change can be found here.

Developers with premium clients who have enlisted in the version control project and will be using ETLs in 18.3 and beyond will need to take the following steps in order to continue to use the premium 'dataintegration' module from Git. (Learn more about working with Git modules here).

  1. To obtain access, you will need to provide your LabKey Account Manager with the username of your GitHub account if you have not already done so.
  2. Clone the dataintegration module into your optionalModules directory.
  3. Sync your svn enlistment to remove the old svn module.
  4. Add a line like this to your settings.gradle file:
include ":server:optionalModules:dataintegration"
  1. Execute these two commands in this order to ensure that you clear out the old copy and are solely building and running the module from Git:
gradlew cleanBuild
gradlew deployApp

Create Chart View

The deprecated chart creation tool will be removed in 18.3 and existing chart views created by users will be automatically migrated to the updated implementation.

Premium subscribers have access to a variety of LabKey features outside of the community edition as well as professional support. Learn more about LabKey Premium Editions.

Windows Installer Decommission

The LabKey Server Windows-only Graphical Installer will be decommissioned as part of the 18.3 release in November 2018. Moving forward, virtual machine images will be provided with each release as an alternate method for evaluating LabKey Server on-premise. We encourage new users looking to evaluate LabKey Server to take advantage of the cloud-hosted trial version of LabKey Server.

Documentation will be provided with the 18.3 release to help ensure a smooth transition for teams currently using the graphical installer to update their LabKey Server installations.




Upcoming Features in 19.1


Upcoming Features

Some features we are currently working on for the 19.1 release of LabKey Server:

Recent Documentation Updates

Click the links below to see the most recent changes to the LabKey Server documentation.




Release Notes 19.1


This topic is under construction for the 19.1 release of LabKey Server.
  • Enforce CSRF checking on all POST requests (This is no longer a configurable option.)
  • QC Trend report workflow improvements
  • PERL API updates
  • Antivirus scanning for cloud-based file repositories
  • ETL definitions included in folder export/import: (docs)
  • Docs: Java Module Tutorial
  • Docs: Configuring IntelliJ for XML File Editing
  • Premium Docs: Example Code for Embedding Spotfire Visualizations
  • Premium Docs: Example Code for using R API for QC Trend Reporting
  • LabKey SQL: If you use a HAVING clause without a GROUP BY, all selected rows are treated as a single group for aggregation. (docs)

Biologics

  • Include "unknowns" in bulk registration of Mixtures and Batches. Also available in 18.3. (docs)
  • Bulk Upload Ingredients and Raw Materials. Also available in 18.3.
  • Attach files to Experiments. Also available in 18.3.
  • Display saved LabKey charts within Biologics

Operations

  • Web Application Firewall (WAF) - We have enabled a Web Application Firewall for all cloud and trial customers to improve protection against web attacks.



Getting Started


This topic is under construction.

The LabKey platform provides an integrated data environment for biomedical research. This section will help you get started using LabKey Server or LabKey Biologics.

Additional Topics

LabKey Solutions




Try it Now: Data Grids


The Data Grid Tutorial is a quick hands-on demonstration you can try without any setup or registration. It shows you just a few of the ways that LabKey server can help you:
  • Securely share your data with colleagues through interactive grid views
  • Collaboratively build and explore interactive visualizations
  • Drill down into de-identified data for study participants
  • Combine related datasets using data integration tools
Click here to begin the tour.



Trial Servers


To get started using LabKey Server and understanding the core functionality, you can create a trial instance. Options include:

Cloud-Based LabKey Server Trial - Recommended

LabKey Server Trial instances contain a core subset of features, and sample content to help get you started. Upload your own data, try tutorials, and even create a custom site tailored to your research and share it with colleagues. Your trial lasts 30 days and we're ready to help you with next steps toward incorporating LabKey Server into your research projects.

Start here: Explore LabKey Server - Hosted Trial

Virtual Machine LabKey Server Trial - On-Premise Evaluation

Virtual machine trial servers run on your local machine inside a virtual environment (Oracle VirtualBox). VM instances contain a core subset of features, and some basic content to get you started. Import your own data, try tutorials, and even create a custom site tailored to your research and share it with colleagues.

Start here: Virtual Machine Trial Server

Cloud-Based Biologics Trial

Try the core features of LabKey Biologics using our sample data and tutorial walkthroughs. Your trial lasts 30 days and we're ready to help you with next steps toward incorporating LabKey Biologics into your work.

Start here: Explore LabKey Biologics - Hosted Trial




Explore LabKey Server - Hosted Trial


To get started using LabKey Server and understanding the core functionality, you can create a hosted LabKey Server - Trial instance. Go here to Start Your Free Trial, enter some basic contact information, then go!

In the hosted account manager, select the product LabKey Server - Trial and click Create Site.

Trial server instances contain a subset of features, and some basic content to get you started. Upload your own data, try tutorials, and even create a custom site tailored to your research and share it with colleagues. Your trial lasts 30 days and we're ready to help you with next steps toward incorporating LabKey Server into your research projects.

Tours & Tutorials

Step by step introductions to key functionality of LabKey Server.

Trial Support Forum

If you have questions or need guidance, please reach out in the Trial Support Forum.


Extending Your Trial: When your trial is nearing it's expiration date, you will see a banner message in the server offering you upgrade options. If you need a bit more time to explore, you can extend your trial beyond the initial 30 days.




Learn About LabKey Server: Trial


Welcome to LabKey Server!

This topic helps you get started understanding how LabKey Server works and how it can work for you. It is intended to be used alongside a LabKey Trial Server. You should have another browser window open on the home page of your instance.

Navigation and User Interface

Projects and Folders

The project and folder hierarchy is like a directory tree and forms the basic organizing structure inside LabKey Server. Everything you create or configure in LabKey Server is located in some folder. Projects are the top level folders, with all the same behavior, plus some additional configuration options; they typically represent a separate team or research effort.

The Home project is a special project. On your Trial server, it contains the main welcome banner. To return to the home project at any time, click the LabKey logo in the upper left corner.

The project menu is on the left end of the menu bar and includes the display name of the current project.

Hover over the project menu to see the available projects, and folders within them. Click any project or folder name to navigate there.

Any project or folder with subfolders will show / buttons for expanding and contracting the list shown. If you are in a subfolder, there will be a clickable 'breadcrumb' trail at the top of the menu for quickly moving up the hierarchy. The menu will scroll when there are enough items, with the current location visible and expanded by default.

The project menu always displays the name of the current project, even when you are in a folder or subfolder. A link with the Folder Name is shown near the top of page views like the following, offering easy one click return to the main page of the folder.

For more about projects, folders, and navigation, see Project and Folder Basics.

Tabs

Using tabs within a folder can give you new "pages" of user interface to help organize content. For an example of tabs in action, see the Research Study within the Example Project.

When your browser window is too narrow to display tabs arrayed across the screen, they will be collapsed into a pulldown menu showing the current tab name and a (chevron). Click the name of the tab on this menu to navigate to it.

For more about adding and customizing tabs, see Use Tabs.

Web Parts

Web parts are user interface panels that can be shown on any folder page or tab. Each web part provides some type of interaction for users with underlying data or other content.

There is a main "wide" column on the left and narrower column on the right. Each column supports a different set of web parts. By combining and reordering these web parts, an administrator can tailor the layout to the needs of the users.

For a hands-on example to try right now, explore the Collaboration Workspace project on your Trial Server.

To learn more, see Add Web Parts and Manage Web Parts. For a list of the types of web parts available in a full installation of LabKey Server, see the Web Part Inventory.

Header Menus

In the upper right, icon menus offer:

  • (Search): Click to open a site-wide search box
  • (Admin): Shown only to Admins: Administrative options available to users granted such access. See below.
  • (User): Login and security options; help links to documentation and the trial support forum.

(Admin) Menu

The "first user" of this trial site will always be an administrator and have access to the menu. If that user adds others, they may or may not have the same menu of options available, depending on permissions granted to them.

  • Site >: Settings that pertain to the entire site.
    • Admin Console: In this Trial edition of LabKey Server, some fields are not configurable and may be shown as read-only. See Admin Console for details about options available in the full installation of LabKey.
    • Site Users, Groups, Permissions: Site-level security settings.
    • Create Project: Creates a new project (top-level folder) on the server.
  • Folder >: Settings for the current folder.
    • Permissions: Security configuration for the current folder.
    • Management: General configuration for the current folder.
    • Project Users and Settings: General configuration for the current project.
  • Page Admin Mode: Used to change page layout and add or remove UI elements.
  • Manage Views, Lists, Assays: Configuration for common data containers.
  • Manage Hosted Server Account: Return to the site from which you launched this trial server.
  • Go To Module >: Home pages for the currently enabled modules.

Security Model

LabKey Server has a group and role-based security model. Whether an individual is authorized to see a resource or perform an action is checked dynamically based on the groups they belong to and roles (permissions) granted to them. Learn more here: Security. Try a walkthrough using your Trial Server here: Exploring LabKey Security

Tools for Working Together

Collaborating with teams within a single lab or around the world is made easier when you share resources and information in an online workspace.

  • Message Boards: Post announcements and carry on threaded discussions. LabKey uses message boards for the Support Forums. Learn more here.

  • Issue Trackers: Track issues, bugs, or other workflow tasks (like assay requests) by customizing an issue tracker. LabKey uses an issue tracker to manage development issues. Learn more here.

  • Wikis: Documents written in HTML, Markdown, Wiki syntax, or plain text; they can include images, links, and live content from data tables. You're reading a Wiki page right now. Learn more here.

  • File Repositories: Upload and selectively share files and spreadsheets of data; connect with custom import methods. You can see an example here. Learn more here.

To learn more and try these tools for yourself, navigate to the Example Project > Collaboration Workspace folder of your Trial Server in one browser window, and open the topic Exploring LabKey Collaboration in another.

Tools for Data Analysis

Biomedical research data comes in many forms, shapes, and sizes. LabKey integrates directly with many types of instruments and software systems and with customization can support any type of tabular data.


  • Uploading Data: From dragging and dropping single spreadsheets to connecting a data pipeline to an outside location for incoming data, your options are as varied as your data. Learn about the options here: Import Data.


  • Interpreting Instrument Data: Using assay designs and pipeline protocols, you can direct LabKey Server to correctly interpret complex instrument data during import. Learn more here: Assay Data.


  • Visualizations: Create easy charts and plots backed by live data. Learn more here: Visualizations.

  • Reporting: Generate reports and query snapshots of your data. Use R, JavaScript, and LabKey plotting APIs to present your data in many ways. Learn more here: Reports and Charts.

To tour some example content and try these tools, navigate to the Example Project > Laboratory Data folder of your Trial Server in one browser window, and open the topic Exploring Laboratory Data in another.

Tools for Research Studies

Study folders organize research data about participants over time. There are many different ways to configure and use LabKey studies. Learn more here: Longitudinal Studies.

  • Study Schedules and Navigation: Dashboards for seeing at a glance what work is completed and what is scheduled helps coordinators manage research data collection. Learn more here: Study Navigation.

  • Participant and Date Alignment: By aligning all of your data based on the participant and date information, you can integrate and compare otherwise disparate test results. Explore the breadth of data for a single study subject, or view trends across cohorts of similar subjects. Learn more here: Study User Guide.


To learn more and try these tools, navigate to the Example Project > Research Study folder of your Trial Server in one browser window, and open the topic Exploring LabKey Studies in another.

What's Next?

Explore the example content on your Trial Server using one of these walkthroughs.

Find out more about what a full installation of LabKey Server can do by reading documentation here:



Exploring LabKey Collaboration


LabKey tools for collaboration include message boards, task trackers, file sharing, and wikis. The default folder you create in LabKey is a Collaboration folder which gives you many basic tools for working and sharing data with colleagues.

This topic is intended to be used alongside a LabKey Trial Server. You should have another browser window open to view the Example Project > Collaboration Workspace folder.

Tour

The "Collaboration Workspace" folder shows three web parts on its main dashboard. Web parts are user interface panels and can be customized in many ways.

  • 1. Learn About LabKey Collaboration Folders: a panel of descriptive information (not part of a default Collaboration folder).
  • 2. Messages: show conversations or announcements in a messages web part
  • 3. Task Tracker: LabKey's issue tracker tools can be tailored to a variety of uses.

To help show some of the collaborative options, this project also includes a few sample users with different roles. You can see a message from the "team lead" and there is also a task called "Get Started" listed as a "Todo".

Try It Now

Messages

A message board is a basic tool for communication; LabKey Message Boards can be customized to many use cases - from support forums to announcement panels.

  • Notice the first message, "Hello World".
  • Click View Message or Respond below it.
  • Click Respond.
  • Enter any text you like in the Body field. If you also change the Title, your response will have a new title but the main message thread will retain the existing title.
  • Notice that you can select other options for how your body text is rendered. Options: Plain Text, HTML, Markdown, or Wiki syntax.
  • Notice you could attach a file if you like.
  • Click Submit.
  • You are now viewing the message thread. Notice links to edit or delete. Since you an administrator on this server, you can edit other's messages, but that is not typically true for users.
  • You may have also received an email when you posted your response. By default, you are subscribed to any messages you create or comment on. Click unsubscribe to see how you would reset your preferences if you don't want to receive these emails.
  • Click the Collaboration Workspace link near the top of the page to return to the main folder dashboard. These links are shown any time you are viewing a page within a folder.
  • Notice that on the main folder dashboard, the message board does not show the text of your reply, but just the note "(1 response)".
  • Click New to create a new message.
  • You will see the same input fields as when replying; enter some text and click Submit.
  • When you return to the main folder dashboard, you will see your new message.
  • An administrator can control many display aspects of message boards, including the level of detail, order of messages, and even what "Messages" are called.
  • The (triangle) menu in the upper right corner includes several options for customizing.
    • New: Create a new message.
    • View List: See the message board in list format.
    • Admin: Change things like display order, what messages are called, security, and what options users have when adding messages.
    • Email: Control when and how email notifications are sent about messages on this board.
    • Customize: Select whether this web part shows the "full" information about the message, as shown in our example, or just a simple preview.

More details about using message boards can be found in the Collaboration Tutorial.

Task Tracker

LabKey provides flexible tools for tracking tasks with multiple steps done by various team members. Generically referred to as "issue trackers" the example project includes a simple "Task Tracker".

The basic life cycle of any task or issue moves through three states:

  • Open: someone decides something needs to be done; various steps and reassignments can happen, including prioritization and reassignments
  • Resolved: someone does the thing
  • Closed: the orginal requestor confirms the solution is correct
  • Navigate to the Collaboration Folder.
  • Scroll down to the Task Tracker and click the title of the task, Get Started to open the detail view.
  • The status, assignment, priority, and other information are shown here. You can see that the team leader opened this task and added the first step: assign this task to yourself.
  • Click Update. Used when you are not "resolving" the issue, merely changing assignment or adding extra information.
  • Select your username from the Assigned To pulldown.
  • You could also change other information and provide an optional comment about your update.
  • Click Save.
    • Note: You may receive an email when you do this. Email preferences are configurable. Learn more here.
  • The task is now assigned to you. You can see the sequence of changes growing below the current issue properties.
  • Click Collaboration Folder to return to the main page.
  • Notice the task is assigned to you now.
  • Click New Task to create a new task.
  • Enter your choice of title, notice that the default status "Open" cannot be changed, but you can change the priority, and enter other information. Note that setting the priority field is required.
  • Assign the task to the "lab technician" user.
  • Click Save to open the new issue.
  • When you return to the task list on the Collaboration Workspace page, you will see it listed as issue 2.

To show how the resolution process works, you are an administrator and can use impersonation to take on another user's identity. Learn more here.

  • Select (Your Username) > Impersonate > User.
  • Choose "lab_technician" and click Impersonate. You are now seeing the page as the lab technician would.
  • Click the title of the new task you assigned to the lab technician to open it.
  • Click Resolve.
  • Notice that by default, the resolved task will be assigned to the original user who opened it - in this case you!
  • Select a value for the Resolution field, such as "Fixed".
  • Enter a comment saying you've completed the task, then click Save.
  • Click Stop Impersonating.
  • Open the task again and close it as yourself.
  • Enter a few more tasks to have more data for viewing.
  • When finished, return to the Collaboration Folder main page.

The Task Tracker grid offers many options.

  • Search the contents by ID number or search term using the search box.
  • Sort and filter the list of tasks using the header menu for any column.
  • Create custom grid views (ways to view tasks), such as "All assigned to me" or "All priority 1 issues for the current milestone". Use (Grid Views) > Customize Grid and add filters and sorts here.
  • You could also customize the grid to expose other columns if helpful, such as the name of the user who originally opened the issue.
The generic name for these tools is "issue trackers" and they can be customized to many purposes. LabKey uses one internally for bug tracking, and client portals use them to track client questions and work requests. More details about using issue trackers can be found in the Collaboration Tutorial.

What Else Can I Do?

Add New Web Parts

Web parts are panels of user interface that display content to your users. As an administrator, you can customize what is shown on the page by using page admin mode. To add a new web part to any folder page:

  • Select (Admin) > Page Admin Mode. Note that if you do not see this option, make sure you are logged in as an administrator and not impersonating another user.
  • Notice that <Select Web Part> pulldown menus appear at the bottom of the page. There is a wider "main" column on the left and a narrow column on the right; each column supports a different set of web parts.
    • Note: If both pulldown menus are stacked on the right, make your browser slightly wider to show them on separate sides.
  • Select the type of web part you want to create on the desired side. For example, to create a main panel wiki like the welcome panel shown in the Collaboration Workspace folder, select Wiki on the left.
  • Click Add.
  • The new web part will be added at the bottom of the column. You can reposition it on the page using the (triangle) menu in the web part header. Move up or down as desired.
  • Click Exit Admin Mode in the upper right.

If you later decide to remove a web part from a page, the underlying content is not deleted. The web part only represents the user interface.

For more about web part types and functionality, see Add Web Parts.

Add a Wiki

Wiki documents provide an easy way to display content. They can contain any text or visual information and be formatted in HTML, Markdown, Wiki syntax, or plain text by using "Wiki" format but not including formatting syntax. To create our first wiki, we use a wiki web part.

  • Add a Wiki web part on the left side of the page. If you followed the instructions above, you already did so.
  • Click Create a new wiki page. Note that if a page named "default" already exists in the folder, the new web part will display it. In this case, create a new one by selecting New from the (triangle) menu in the header of the web part.
  • The Name must be unique in the folder.
  • The Title is displayed at the top of the page.
  • Choosing a Parent page lets you organize many wikis into a hierarchy. The table of contents on the right of the page you are reading now shows many examples.
  • Enter the Body of the wiki. To change what formatting is used, use the Convert to... button in the upper right. A short guide to the formatting you select is shown at the bottom of the edit page.
  • Notice that you can attach files and elect whether to show them listed at the bottom of the wiki page.
  • Click Save & Close.
  • Scroll down to see your new wiki web part.
  • To reopen for editing, click the (pencil) icon.

More details about using wikis can be found in the Collaboration Tutorial.

Add a List

A list is a simple table of data. For example, you might store a list of labs you work with.

  • Select (Admin) > Manage Lists.
  • You will see a number of preexisting lists related to the task tracker.
  • Click Create New List.
    • Name: Enter a short name (such as "labs"). It must be unique in the folder.
    • Primary Key: Enter "Name" in this field. If you leave the Primary Key default "Key", the new list will contain a "Key" column. In this example, we will use a column called "Name" as the key.
    • Primary Key Type: Select "Text (String)". The default is "Auto-Increment Integer", but in our example we use a text name.
    • Leave the box to import from file unchecked for this first list.
  • Click Create List.
  • Review the list properties available; for this first example, leave them unchanged.
  • Scroll down to List Fields and notice that "Name" was created and shows a (key) icon. You could give it a new label, like "Lab Name" if you like. Notice a (wrench) icon marks rows with unsaved changes.
  • Click Add Field and enter the name and label for each column you want in your list. In this example, we have added an address and contact person.
  • Click Save, then click Done to create the "labs" list. You will see it in the grid of "Available Lists".
To populate this new empty list:
  • Click the list name ("labs") in the grid. There is no data to show.
  • Select (Insert data) > Insert new row.
  • Enter any values you like.
  • Click Submit.
  • Repeat the process to add a few more rows.

Now your collaboration workspace shows quick contact information on the front page.

Learn more about creating and using lists here and in the Tutorial: Lists.

Add a File Repository

  • Each file can then be downloaded by other users of the collaboration workspace.

See the Tutorial: File Repository for a walkthrough of using file repositories.

More Tutorials

Other tutorials using "Collaboration" folders that you can run on your LabKey Trial Server:

To avoid overwriting this Example Project content with new tutorial content, you could create a new "Tutorials" project to work in. See Exploring Project Creation for a walkthrough.

Explore More on your LabKey Trial Server




Exploring Laboratory Data


Laboratory data can be organized and analyzed using LabKey Assay folders. This topic walks you through the basics using a simple Excel spreadsheet standing in for typically more complex instrument output.

This topic is intended to be used alongside a LabKey Trial Server. You should have another browser window open to view the Example Project > Laboratory Data folder.

Tour

The "Laboratory Data" folder is an Assay folder. You'll see four web parts in our example:

  • 1. Learn About LabKey Assay Folders: a panel of descriptive information (not part of a default Assay folder).
  • 2. Assay List: a list of assays defined in the folder. Here we have "Blood Test Data."
  • 3. Blood Test Data Results: A query web part showing a grid of data.
  • 4. Files: a file repository where you can browse files in this container or upload new ones.

The folder includes a simple assay design created from the "General Purpose Assay Type" (GPAT) representing some blood test results. A single run (spreadsheet) of data has been imported using it. Continue below to try out the assay tools.

Try It Now

Assay List

Using an Assay List web part, you can browse existing assay designs and as an administrator, add new ones. An assay design tells the server how to interpret uploaded data. Click the name of the design in this web part to see the run(s) that have been uploaded.

  • Click the name Blood Test Data in the Assay List to see the "Runs" imported using it.
  • In this case, a single run (spreadsheet) "BloodTest_Run1.xls" has been imported. Click the Assay ID (in this case the file name) to see the results.
  • This example shows blood data for 3 participants over a few different dates. White blood counts (WBC), mean corpuscular volume (MCV), and hemoglobin (HGB) levels have been collected in an Excel spreadsheet.
  • To see the upload process, we can simulate importing the existing run by clicking Re-Import Run above the grid. We will cancel the import before any reimporting actually happens.
  • The first page shows:
    • Assay Properties: These are fixed properties, like the assay design name, that are read only for all imports using this design.
    • Batch Properties: These properties will apply to a set of files uploaded together. For example, here they include specifying how the data is identified (the Participant/Visit setting) and selecting a target study to use if any data is to be copied. Select /Example Project/Research Study (Research Study).
  • Click Next.
  • On the next page, notice the batch properties are now read only.
  • Below them are Run Properties to specify; these could be entered differently for each run in the assay, such as the "Assay ID". If not provided, the name of the file is used, as in our example.
  • Here you also provide the data from the assay run, either by uploading a data file (or reusing the one we already uploaded) or entering your own new data.
  • Click Show Expected Data Fields. You will see the list of fields expected to be in the spreadsheet. When you design the assay, you specify the name, type, and other properties of expected columns.
  • Click Download Spreadsheet Template to generate a blank template spreadsheet. It will be named "data_<datetimestamp>.xls". By opening it in Excel and populating it, you can upload your own data to this server as a new run. Save this file and use it when you get to the Files section below.
  • Since we are just reviewing the process right now, click Cancel after reviewing this page.

Next, review what the assay design itself looks like. You need to be an administrator to perform this step.

  • From the Blood Test Data Runs page, select Manage Assay Design > Edit assay design to open it.

The sections correspond to how a user will set properties as they import a run.

  • Assay Properties: Values that do not vary per run or batch.
  • Batch Fields: Values set once for each batch of runs.
  • Run Fields: Values set per run; in this case none are defined, but as you saw when importing the file, some built in things like "Assay ID" are defined per run.
  • Data Fields: This is the heart of the assay design and where you would identify what columns are expected in the spreadsheet, what labels to use, what their datatype is. If you click a row, you can also set other properties of each column--including things like whether a value can be used as a "measure" in reporting, as shown here for the WBC field.
  • Review the contents of this page, but make no changes and click Cancel.
  • Return to the main folder page by clicking the Laboratory Data link near the top of the page.

Blood Test Data Results

The assay results are displayed in a web part on the main page. Explore some general features of LabKey data grids here:

Column Header Menus: Each column header has a menu of options to apply to the grid based on the values in that column.


Sort: Click the header of the WBC column and select Sort Descending. Notice the (sort) icon that appears in the column header.


Filter: Click the header of the Participant ID column and select Filter... to open a popup.

  • Use the checkboxes to "Choose Values". For this example, uncheck the box for "202". Switching to the filtering tab in this popup would let you use filter expressions instead. The option to choose values is only available when there are a limited number of distinct values.
  • Click OK to apply the filter. Notice the (filter) icon shown in the column header. There is also a new entry in the filter panel above the grid - you can clear this filter by clicking the for the "Participant <> 202" filter.

Summary Statistics: Click the header of the MCV column and select Summary Statistics... to open a popup.
  • Select the statistics you would like to show (the values are previewed in the popup.) In this case, Mean might be most interesting. Click Apply.
  • Notice the new row at the bottom of the grid showing the Mean for this column.

The view of the grid now includes a sort, a filter, and a summary statistic.

Notice the message "This grid view has been modified" was added when you added the summary statistic. The grid you are now seeing is not the original default. You have not changed anything about the underlying imported data table, merely how you see it in this grid. Click Save to make this change persistent and sharable with others.

  • Select Named and give the new grid a name (such as "Grid with Statistic").
  • Check the box for "Make this grid view available to all users."
  • Click Save.
  • Notice that the grid now has Grid With Statistic shown above it.
  • To switch between grid views, use the (Grid Views) menu in the grid header. Switch between "default" and "Grid With Statistic" to see the difference.
  • Return to the "default" view before proceeding with this walkthrough.
Visualizations and Reports:

There is not enough data in this small spreadsheet to make meaningful visualizations or reports, but you can see how the options work:

  • Column Visualizations: Each column header menu has a set of visualizations available that can be displayed directly in the grid view. Options vary based on the data type. For example, from the "WBC" column header menu, choose Box and Whisker.


  • Quick Charts: Choose Quick Chart from the header menu of any column to see a quick best guess visualization of the data from that column. The default is typically a box and whisker plot.
  • While similar to the column chart above, this chart creates a stand alone chart that is named and saved separately from the grid view.
  • It also opens in the "Chart Wizard", described next, which offers many customization options.
  • Note that a chart like this is backed by the "live" data in the source table. If you change the underlying data (either by editing or by adding additional rows to the table) the chart will also update automatically.

  • Plot Editor: More types of charts and many more configuration options are available using the common plot editor, the "Chart Wizard".
  • Return to the grid view if you navigated away.
  • Select Charts > Create Chart above the grid.
  • Select the type of chart you want using the options along the left edge.

Learn more about creating and customizing each chart type in the documentation. You can experiment with this small data set or use the more complex data when Exploring LabKey Studies.

Files

The Files web part lists the single assay run as well as the log file from when it was uploaded.

To explore the abilities of the file browser, we'll create some new assay data to import.

  • Open the template you downloaded earlier. It is an Excel file named "data_<datetimestamp>.xls"
  • Enter some values. Some participant ID values to use are 101, 202, 303, 404, 505, and 606, though any integer values will be accepted. You can ignore the VisitID column and enter any dates you like. The WBC and MCV columns are integers, HGB is a double. The more rows of data you enter, the more "new" results you will have available to "analyze."
  • Save the spreadsheet with the name "data_run2.xls" if you want to match our instructions.
  • You can also click here to download ours if you want your data to match our screenshots: data_run2.xls

  • Drag and drop the "data_run2.xls" file into the Files web part to upload it.
  • Select it using the checkbox and click Import Data.
  • By default, the "Blood Test Data" assay design we predefined is selected. Leave this choice, but notice you could create a new "General Purpose" assay design by clicking that option.
  • Click Import.
  • You will see the Assay Properties and be able to enter Batch Properties; select "Example Project/Research Study" as the Target Study.
  • Click Next.
  • On the Run Properties and Data File page, notice your "data_run2.xls" file listed as the Run Data. You don't need to make any changes on this page.
  • Click Save and Finish.
  • Now you will see the runs grid, with your new run added to our original sample.
  • Click the file name to see the data you created.
  • Notice the (Grid Views) menu includes the custom grid view "Grid with Statistic" we created earlier - select it.
  • Notice the grid also shows a filter "Run = 2" because you clicked the run name to view only results from that run. Click the for the filter to delete it.
  • You now see the combined results from both runs.
  • If you saved any visualizations earlier using the original spreadsheet of data, view them now from the (Charts/Reports) menu, and notice they have been updated to include your additional data.

What Else Can I Do?

Define a New Assay

LabKey can help you create a new assay design from a spreadsheet of your own. Choose something with a few columns of different types, or simply add a few more columns to the "data_run2.xls" spreadsheet you used earlier. If you have existing instrument data that is in tabular format, you can also use that to complete this walkthrough. Here we use the name "mydata.xls" to represent your own spreadsheet of data.

  • Navigate to the main page of the Laboratory Data folder.
  • Drag and drop your "mydata.xls" file to the Files web part.
  • Select the file and click Import Data.
  • In the popup, choose "Create New General Assay Design" and click Import.
  • Give your new assay design a Name, such as "MyData".
  • Notice the default Location is the current project. Select instead the "Current Folder (Laboratory Data)".
  • The Columns for Assay Data section shows the columns imported from your spreadsheet and the data types the server has inferred from the contents of the first few rows of your spreadsheet.
  • If you see a column here that you do not want to import, simply uncheck it. If you want to edit column properties, you can do that using the (triangle) button next to the column name.
  • You can change the inferred data types as necessary. For example, if you have a column that happens to contain whole number values, the server will infer it is of type "Integer". If you want it to be "Number (Double)" instead, select that type after clicking the (caret) button next to the type. If a column happens to be empty in the first few rows, the server will guess "Text (String)" but you can change that as well.
  • Column Mapping below these inferrals is where you would map things like Participant ID and Date information. For instance, if you have a column called "When" that contains the date information, you can tell the server that here. It is not required that you provide mappings at all.
  • When you are satisfied with how the server will interpret your spreadsheet, scroll back up and click Begin Import.
  • You will be simultaneously creating the assay design (for this and future uses) and importing this single run. Enter batch properties if needed, then click Next.
  • Enter run properties if needed, including an Assay ID if you want to use a name other than your file name.
  • Click Show Expected Data Fields to review how your data will be structured. Notice that if you didn't provide mappings, new columns will be created for Specimen, Participant, Visit ID, and Date.
  • Click Save and Finish.

  • You now see your new run in a grid, and the new assay design has been created.
  • Click the filename to see your data.
  • Select Manage Assay Design > Edit assay design to review the assay design.
  • Click null Laboratory Data to return to the main folder page.
  • Notice your new assay design is listed in the Assay List.
  • You can now upload and import additional spreadsheets with the same structure using it. When you import .xls files in the future, your own assay design will be one of the options for importing it.

Understand Copy to Study

You may have noticed the Copied to Research Study column in the Blood Test Data Results web part. The sample assay design includes copying data automatically to the Example Project/Research Study folder on your Trial Server.

"Copying" data to a study does not literally copy the data. What is created is a dynamic link so that the assay data can be integrated with other data in the study about the same participants. When data changes in the original container, the "copied" set in the study is also updated.

Click the View Copy-to-Study History link in the Blood Test Data Results web part.

You will see at least one copy event from the original copy that occurred when our sample data was copied during the startup of your server (shown as being created by the "team lead"). If you followed the above steps and uploaded a second "data_run2.xls" spreadsheet, it will also be listed, created by you.

  • Click View Results to see the result rows.
  • Click one of the links in the Copied to Research Study column. You see what appears to be the same grid of data, but notice by checking the project menu or looking at the URL that you are now in the "Research Study" folder. You will also see the tabs present in a study folder.
  • Click the Clinical and Assay Data tab.
  • Under Assay Data you will see the "Blood Test Data" copied dataset. In the topic Exploring LabKey Studies we will explore how that data can now be connected to other participant information.
  • Using the project menu, return to the Laboratory Data folder.

More Tutorials

Other tutorials using "Assay" folders that you can run on your LabKey Trial Server:

To avoid overwriting this Example Project content with new tutorial content, you could create a new "Tutorials" project to work in. See Exploring Project Creation for a walkthrough.

Explore More on your LabKey Trial Server




Exploring LabKey Studies


Using LabKey Studies, you can integrate and manage data about participants over time. Cohort and observational studies are a typical use case. Learn more in this topic.

This topic is intended to be used alongside a LabKey Trial Server. You should have another browser window open to view the Example Project > Research Study folder.

Tour

The "Research Study" folder contains a simple fictional study. There are 5 tabs in the default LabKey study folder; the main landing page is the Overview tab where you will see three web parts:

  • 1. Learn About LabKey Study Folders: a panel of descriptive information (not part of a default Study folder)
  • 2. Study Data Tools: commonly used tools and settings.
  • 3. Study Overview: displays study properties and quick links to study navigation and management options.

A study has five tabs by default. Each encapsulates functions within the study making it easier to find what you need. Click the tab name to navigate to it. Returning to the Research Study folder returns you to the Overview tab.

Participants: View and filter the participant list by cohort; search for information about individuals

Clinical and Assay Data: Review data and visualizations available; create new joined grids, charts, charts and reports.

The Data Views web part lists several datasets, joined views, and reports and charts.

  • Hover over a name to see a preview and some metadata about it.
  • Click a name or use the (Details) link to open an item.
  • The Access column indicates whether special permissions have been set for an item. Remember that only users granted "Read" access to this folder can see any content here, so seeing "public" means only that it is shared with all folder users. Other options are "private" (only visible to the creator) and custom (shared with specific individuals). See Manage Study Security (Dataset-Level Security) for more information.
Datasets:
    • Demographic datasets contain a single row per participant for the entire study.
    • Clinical datasets can contain many rows per participant, but only one per participant and date combination.
    • Assay datasets, typically results from instrument tests, can contain many rows per participant and date.
Specimen Data: Studies can include the tracking of specimens and data about them. Like assay data, specimen data can have many rows per participant and date. Another key field, typically a specimenID, uniquely identifies the rows.

Manage: Only administrators can access this dashboard for managing the study.

Try It Now

This example study includes a simplified research scenario. Let's explore some key feature areas on each tab:

Display Study Properties

The Study Overview web part shows properties and introductory information about your study.

  • Click the Overview tab.
  • Click the (pencil) icon.
  • Review and change study properties:
    • Label: By default, the folder name, here "Research Study," is used. You can edit to make it more descriptive, such as "HIV Study".
    • Investigator: Enter your own name to personalize this example.
    • Grant/Species: If you enter the grant name and/or species under study, they will be displayed. These fields also enable searches across many study folders to locate related research. Enter a grant name to see how it is shown.
    • Description/Render Type: Our example shows some simple HTML formatted text. You can include as much or as little information here as you like. Select a different Render Type to use Markdown, Wiki syntax, or just plain text.
    • Protocol Documents: Attach documents if you like - links to download will be included in the web part.
    • Timepoint Type: Studies can use several methods of tracking time; this decision is fixed at the time of study creation and cannot be modified here. See Visits and Dates to learn more.
    • Start/End Date: See and change the timeframe of your study if necessary. Note that participants can also have individual start dates. Changing the start date for a study in progress should be done with caution.
    • Subject Noun: By default, study subjects are called "participants" but you can change that here to "subject," "mouse," "yeast," or whatever noun you choose. Try changing these nouns to "Subject" and "Subjects".
    • Subject Column Name: Enter the name of the column in your datasets that contains IDs for your study subjects. You do not need to change this field to match the subject nouns you use.
  • When finished, click Submit and see how this information is displayed. Notice the "Participants" tab name is changed.
  • Reopen the editor and change the subject noun back to "Participant[s]" to restore the original tab and tool names for this walkthrough.

Track Overall Progress

  • Return to the Overview tab if you navigated away.
  • Click the Study Navigator link or the small graphic in the Study Overview web part.
  • The Study Navigator shows you at a glance how much data is available in your study and when it was collected.
    • Rows represent datasets, columns represent timepoints.
  • Use the Participant's current cohort dropdown to see collection by cohort.
  • Use the checkboxes to switch between seeing counts of participants or rows or both.
  • Click the number at the intersection of any row and column to see the data. For example, Lab Results in month two look like this:

View Participant Data

Within a study you can dive deep for all the information about a single participant of interest.

  • Click the Participants tab.
  • If you know the participant ID, you can use the search box to find their information.
  • The Participant List can be quickly filtered using the checkboxes on the left.
  • Use the Filter box to narrow the list if you know part of the participant ID.
  • Hover over a label to see the group member IDs shown in bold. Click a label to select only that filter option in that category. Here we see there are 8 participants enrolled receiving ARV treatment.
  • Click any participant ID to see all the study data about that participant.
  • The details of any dataset can be expanded/collapsed using the and icons.
  • Click the Search For link above the report to search the entire site for other information about that participant. In this case, you will also see results from the "Laboratory Data" folder in this project.

Integrate Data Aligned by Participant and Date

Study datasets can be combined to give a broad picture of trends within groups over time.

  • Click the Clinical and Assay Data tab.
  • There are three primary datasets here: Demographics, Physical Exam, and Lab Results.
  • Click "Joined View: Physical Exam and Demographics" to open an integrated custom grid.
This grid includes columns from two datasets. To see how it was created:
  • Select (Grid Views) > Customize Grid.
  • Scroll down on the Available Fields side to Datasets and click the to expand it. Listed are the two other datasets.
  • Expand the "Demographics" node by clicking the .
  • Scroll down and notice the checked fields like Country and Group Assignment which appear in our joined view.
  • Scroll down on the Selected Fields side to see these fields shown.
  • You can use checkboxes to add more fields to what is shown in the grid, drag and drop to rearrange columns in your view, and use the and icons for selected fields to edit display titles or delete them from the view.
  • Click View Data when finished to see your changes.
  • Notice the message indicating you have unsaved changes. Click Revert to discard them.

Customize Visualizations and Reports

In Exploring Laboratory Data we show you how to create charts and plots based on single columns or tables. With the integration of diverse data in a study, you can easily create visualization and reports across many tables, backed by live data. Our example study includes a few examples.

  • On the Clinical and Assay Data tab, click Bar Chart: Blood Pressure by Cohort and Treatment.
  • Here we see plotted a measure from the "Physical Exam" dataset (Systolic Blood Pressure) against cohort and treatment group data from the "Demographics" dataset.
  • Click Edit in the upper right to open the plot for editing.
  • Click Chart Type to open the chart wizard.
  • Columns on the left can be dragged and dropped into the plot attribute boxes in the middle.
  • Select a different plot type using the options to the right. The plot editor will make a best effort to retain plot attribute column selections, though not all attributes apply to each chart type.
  • Click Box.
    • Notice that the X and Y axis selections are preserved.
    • Drag the column "Study: Cohort" to the "Color" box.
    • Drag the column "Gender" to the "Shape" box.
  • Click Apply to see the new box plot. At present, only the outliers make use of the color and shape selections.
  • Click Chart Layout.
    • Give the plot a new title, like "Systolic - Box Plot"
    • Change Show Points to All.
    • Check the box to Jitter Points; otherwise points will be shown in a single column.
    • Scroll down to see the controls for plot line and fill colors.
    • Choose a different Color Palette from which the point colors will be selected. Shown here, "Dark".
  • Click Apply to see the revised box plot.
  • Click Save As to save this as a new visualization and preserve the original bar chart.
  • Give the new report a name and click Save in the popup.

Manage Your Study

On the Manage tab, you can control many aspects of your study. For example, click Manage Cohorts to review how cohorts were assigned and what the other options are:

  • Assignment Mode: Cohorts can be simple (fixed) or advanced, meaning they can change during the study.
  • Assignment Type: You can manually assign participants to cohorts, or have assignments made automatically based on a dataset.
  • Automatic Participant/Cohort Assignment: Choose the dataset and column to use for assigning cohorts.
Optional
  • You can experiment by changing which column is used to define cohorts: For instance, choose "Country" and click Update Assignments.
  • Notice the new entries under Defined Cohorts and new assignments in the Participant-Cohort Assignments web part.
  • Click the Participants tab.
  • Now you see under Cohorts that both the original cohorts and new ones are listed. Using the hover behavior, notice that the original "Group 1 and Group 2" cohorts are now empty and participants can quickly be filtered by country.
  • Go back in your browser window or click the Manage tab and Manage Cohorts to return to the cohort page.
  • Under Defined Cohorts you can click Delete Unused, then return to the Participants tab to see they are gone.
  • Click the Clinical and Assay Data tab. Click Bar Chart: Blood Pressure by Cohort and Treatment and you will see it has been automatically updated to reflect the new cohort division by country.

  • Return the original cohorts before moving on:
    • On the Manage tab, click Manage Cohorts.
    • Restore the original assignments based on the Demographics/Group Assignment field.
    • Click Update Assignments.
    • Under Defined Cohorts, click Delete Unused.

What Else Can I Do?

Manage Dataset Security

Access to read and edit information in folders is generally controlled by the LabKey role based security model. Within a study, you gain the additional option of dataset level security.

  • On the Manage tab, click Manage Security.
  • Review the Study Security Type: Datasets can be read-only or editable, under either type of security:
    • Basic security: folder permissions determine access to all datasets
    • Custom security:
  • Change the type to Custom security with editable datasets. Notice the warning message that this can change who can view and modify data.
  • Click Update Type.
  • When using custom security, there are two additional web parts on the page:
    • Study Security: Use radio buttons to grant access to groups. Click Update after changing. In the screenshot below, we're changing the "Study Team" and "Example Team Members" groups to "Per Dataset" permissions.
    • Per Dataset Permissions: Once any group is given per-dataset permissions using the radio buttons, you will have the option to individually set permission levels for each group for each dataset.
  • Click Save when finished, or to revert to the configuration before this step, set all for "Example Team Members" to read, and set all for "Study Team" to edit.

Learn About Protecting PHI

When displaying or exporting data, Protected Health Information (PHI) that could be used to identify an individual can be protected in several ways.

Alternate Participant IDs and Aliases

If you want to share data without revealing participant IDs, you can use a system of alternates or aliases so that you can still show a consistent ID for any set of data, but it is not the actual participant ID.

  • On the Manage tab, click Manage Alternate Participant IDs and Aliases.
  • Alternate participant IDs allow you to use a consistent prefix and randomized set of the number of digits you choose.
  • Dates are also offset by a random amount for each participant. Visit-date information could potentially isolate the individual, so this option obscures that without losing the elapsed time between visits which might be relevant to your study.
  • Participant Aliases lets you specify a dataset containing specific aliases to use. For instance, you might use a master set of aliases across all studies to connect related results without positively identifying individuals.

Mark Data as PHI

There are four levels to consider regarding protection of PHI, based on how much information a user will be authorized to see. For PHI protection to be enforced, you must BOTH mark data as PHI and implement viewing and export restrictions on your project. Data in columns can be marked as:

  • Restricted: The most sensitive information not shared with unauthorized users, even those authorized for Full PHI.
  • Full PHI: Only shared when users have permission to see all PHI. The user can see everything except restricted information.
  • Limited PHI: Some information that is less sensitive can be visible here.
  • Not PHI: All readers can see this information.
To mark a column's PHI level:
  • On the Clinical and Assay Data tab, open the dataset of interest. Here we use the Demographics dataset.
  • Above the grid, click Manage.
  • Click Edit Definition.
  • Scroll down to the dataset fields and select the field of interest, here "Status of Infection."
  • In the properties panel, click the Advanced tab.
  • The default PHI Level is "Not PHI". Make another selection from the dropdown to change; in this example, we choose "Full PHI".
  • Adjust PHI levels for as many fields as necessary.
  • Scroll up and click Save.
  • Click View Data to return to the dataset.

Note that these column markings are not sufficient to protect data from view to users with read access to the folder. The levels must be enforced with UI implementation, perhaps including users declaring their PHI level and agreeing to a custom terms of use. For more information, see Compliance.

To export or publish data at a given PHI level:
  • From the Manage tab, click Export Study.
  • On the export page, notice one of the Options is Include PHI Columns. Select what you want included in your export:
    • Restricted, Full, and Limited PHI (default): Include all columns.
    • Full and Limited PHI: Exclude only the Restricted PHI columns.
    • Limited PHI: Exclude the Full PHI and Restricted columns.
    • Uncheck the checkbox to exclude all PHI columns.
  • If you marked the demographics column as "Full PHI" above, select "Limited PHI"to exclude it. You can also simply uncheck the Include PHI Columns checkbox.
  • Click Export.
  • Examine the downloaded archive and observe that the file named "ARCHIVE_NAME.folder.zip/study/datasets/dataset5001.tsv" does not include the data you marked as PHI.

Publish the Study

Publishing a LabKey study allows you to select all or part of your results and create a new published version.

  • Click the Manage tab.
  • Click Publish Study (at the bottom).
  • The Publish Study Wizard will guide you through selecting what to publish.
    • By default, the new study will be called "New Study" and placed in a subdirectory of the current study folder.
    • Select the participants, datasets, timepoints, and other objects to include. On the Datasets step, you can elect to have the study refresh data if you like, either manually or nightly.
    • The last page of the publish wizard offers Publish Options including excluding PHI or randomizing other information that could identify individuals.
  • Click Finish at the end of the wizard to create the new study folder with the selected information.

Explore the new study, now available on the projects menu.

More Tutorials

Other tutorials using "Study" folders that you can run on your LabKey Trial Server:

To avoid overwriting this Example Project content with new tutorial content, you could create a new "Tutorials" project to work in. See Exploring Project Creation for a walkthrough.

Explore More on your LabKey Trial Server




Exploring LabKey Security


Learn more about how LabKey manages security through roles and groups in this topic.

This topic is intended to be used alongside a LabKey Trial Server. You should have another browser window open to view the Example Project folder. This walkthrough also assumes you are the original creator of the trial server and are an administrator there, giving you broad access site-wide.

Tour

Our Example Project contains three subfolders, intended for different groups of users:

  • Collaboration Workspace: The entire team communicates here and shares project-wide information.
  • Laboratory Data: A lab team performs tests and uploads data here, perhaps performing basic quality control.
  • Research Study: A team of researchers is exploring a hypothesis about HIV.
LabKey's security model is based on assignment of permission roles to users, typically in groups.

Project Groups

Groups at the project level allow you to subdivide your project team into functional subgroups and grant permissions on resources to the group as a whole in each folder and subfolder. While it is possible to assign permissions individually to each user, it can become unwieldy to maintain in a larger system.

  • Navigate to the Example Project. You can be in any subfolder.
  • Select (Admin) > Folder > Permissions.
  • Click the tab Project Groups.
  • There are 5 predefined project groups. See at a glance how many members are in each.
  • Click the name to see the membership. Click Example Team Members and see the list of all our example users.
  • Click Done in the popup.
  • Click the Lab Team to see that the two members are the team lead and the lab technician.
  • Click the Study Team to see that the two members are the team lead and the study researcher.

Next we'll review how these groups are assigned different permissions within the project's subfolders.

Permission Roles

Permission roles grant different types of access to a resource. Read, Edit, and Admin are typical examples; there are many more permission roles available in LabKey Server. Learn more here.

  • If you navigated away after the previous section, select (Admin) > Folder > Permissions.
  • Click the Permissions tab.
  • In the left column list of folders, you will see the entire project hierarchy. The folder you are viewing is shown in bold. Click Example Project to see the permissions in the project itself.
  • The "Admin Team" is both project and folder administrator, and the "Example Team Members" group are Editors in the project container.

  • Click Collaboration Workspace and notice that the "Example Team Members" are editors here, too.
  • Click Laboratory Data. In this folder, the "Lab Team" group has editor permissions, and the "Example Team Members" group only has reader permission.
    • Note that when users are members of multiple groups, like in the case of our sample "lab_technician@local.test", they hold the sum of permissions granted through the groups they belong to. This lab_technician has read access with example team membership, but also editor access because of lab team membership, so that user will be able to edit contents here.
  • To see the user membership of any group, click the group name in the permissions UI.
  • To see all permissions granted to a given user, click the Permissions link in the group membership popup.
  • This example lab technician can edit content in the example project, the collaboration workspace folder, and the laboratory data folder. They can read but not edit content in the research study folder.

  • Close any popups, then click Cancel to exit the permissions editing UI.

Try It Now

Impersonation

Using impersonation, an admin can see what another user would be able to see and do on the server.

  • Navigate to the Example Project/Research Study folder using the project menu.
  • Notice that as yourself, the application admin on this server, you can see a (pencil) icon in the header of the Study Overview web part. You would click it to edit study properties. You also see the Manage tab.
  • Select (User) > Impersonate > User.
  • Select "lab_technician@local.test" from the dropdown and click Impersonate.
  • Now you are the lab technician with permission to read the content here, but cannot change it as we saw when reviewing permissions above.
  • Notice the (pencil) icon and Manage tabs are no longer visible. You also no longer have access to the (Admin) menu in the header.
  • Click Stop Impersonating to return to your own "identity".

Impersonate a Group

Impersonation of a group can help you better understand permissions and access. In particular, when configuring access and deciding what groups should include which users, group impersonation can be very helpful.

  • Navigate to the Example Project/Research Study folder using the project menu.
  • Select (User) > Impersonate > Group.
  • Choose the "Study Team" and click Impersonate.
  • Hover over the project menu and notice that the only folder in "Example Project" that members of the "Study Team" can read is this "Research Study" folder.
  • Click Stop Impersonating.
  • To see an error, select (User) > Impersonate > Group, choose the "Lab Team" and click Impersonate. This group does not have access to this folder, so the error "User does not have permission to perform this operation" is shown.
  • Click Stop Impersonating.

Impersonate a Role

You can also directly impersonate roles like "Reader" and "Submitter" to see what access those roles provide.

To learn more about impersonation, see Test Security Settings by Impersonation.

Audit Logging

Actions on LabKey Server are extensively logged for audit and security review purposes. Impersonation is among the events logged.

  • Select (Admin) > Site > Admin Console.
  • Click Admin Console Links.
  • Under Management, click Audit Log.
  • Using the pulldown, select User Events.
  • You will see the impersonation events you just performed. Notice these are paired events - both the action taken by the user to impersonate, and the action performed "on" the impersonatee.
  • Explore other audit logs to see other kinds of events tracked by the server.

What Else Can I Do?

The Security Tutorial walks you through more security features. You can create a new project for tutorials on your trial server, then run the security tutorial there.

Learn More

Explore More on your LabKey Trial Server




Exploring Project Creation


Once you've learned about LabKey and explored your LabKey Trial Server, you can start creating your own projects and custom applications.

This topic assumes you are using a LabKey Trial Server and have it open in another browser window.
As a first project, let's create a "Tutorials" project in which you can run some LabKey tutorials.

Create a Project

To open the project creation wizard you can:

  • Click the "Create" button on the Trial Server home page.
  • OR: Select (Admin) > Site > Create Project.
  • OR: Click the "Create Project" icon at the bottom of the project menu as shown:

The project creation wizard includes three steps.

  1. Give the project a Name and choose the folder type.
  2. Configure users and permissions (or accept the defaults and change them later).
  3. Choose optional project settings (or configure them later).
  • Select (Admin) > Site > Create Project.
  • Step 1: Create Project:
    • Enter the Name: "Tutorials". Project names must be unique on the server.
    • Leave the box checked to use this as the display title. If you unchecked it, you could enter a different display title.
    • Folder Type: Leave the default selection of "Collaboration". Other available folder types are listed. On a trial server, there are a few basic options. Hover to learn more about any type; click Folder Help for the list of folder types available in a full LabKey installation.
    • Click Next.

  • Step 2: Users / Permissions: Choose the initial security configuration. As the admin, you can also change it later.
    • The default option "My User Only" lets you set up the project before inviting additional users.
    • The other option "Copy From Existing Project" is a helpful shortcut when creating new projects to match an existing one.
    • Leave "My User Only" selected and click Next.


  • Step 3: Project Settings:
    • On a LabKey Trial Server, you cannot Choose File Location, so this option is grayed out.
    • Advanced Settings are listed here for convenience, but you do not need to set anything here.
    • Simply click Finish to create your new project.
  • You'll now see the landing page of your new project and can start adding content.

Add Some Content

Let's customize the landing page and make it easier to make and access subfolders.

  • In the Wiki web part, click Create a new wiki page to display in this web part.
  • Leave the Name as "default"
  • Enter the Title "Welcome"
  • In the body field, enter: "Feel free to make a new subfolder and run a tutorial in it."
  • Click Save & Close.
  • Remove extraneous web parts:
    • Select (Admin) > Page Admin Mode.
    • In the web part header menu for both "Messages" and "Pages", select (triangle) > Remove From Page.

You can now use this project as the base for tutorials. Most LabKey tutorials begin with creation of a new subfolder of a specific type.

Add a new web part for displaying the subfolders you will create for tutorials.

  • From the <Select Web Part> menu in the lower left, select Subfolders.
  • Click Add.
  • Click Exit Admin Mode.
There are no subfolders to display yet, but once you add a few, this web part will look something like this, giving you a one click way to return to a tutorial folder later.

Tutorials for your LabKey Trial Server

Try one of these:

Share With Others

Now that you have created a new project and learned more about LabKey Server, consider sharing with a colleague. To invite someone to explore the Trial Server you created, simply add them as a new user:

  • Select (Admin) > Site > Site Users.
  • Click Add Users above the grid of existing site users.
  • Enter one or more email addresses, each on it's own line.
  • If you want to grant the new user(s) the same permissions on the server as another user, such as yourself, check the box for Clone permissions from user: and enter the user ID. See what permissions will be cloned by clicking Permissions.
  • To grant different permissions to the new user, do not check this box, simply click Add Users and configure permission settings for each project and folder individually.
  • Notification email will be sent inviting the new user. You can also see this email by clicking the here link in the UI that will appear.
  • Click Done.
  • You will see the new user(s) listed in the grid. Click the Permissions link for each one in turn and see the permissions that were granted. To change these in any given project or folder, navigate to it and use (Admin) > Folder > Permissions. See Configure Permissions for more information.

What Else Can I Do?

Change the Color Scheme

The color scheme, or theme, is a good way to customize the look of a given project. To see what the other themes look like, see Web Site Theme.

To change what your Tutorials project looks like:

  • Navigate to the Tutorials project, or any subfolder of it.
  • Select (Admin) > Folder > Project Settings.
  • Under Theme, select another option.
  • Scroll down and click Save.
  • Return to the folder and see the new look.

You can also change the theme for the site as a whole by selecting (Admin) > Site > Site Console. Click Admin Console Links, then under Configuration, choose Look and Feel Settings. The same options are available for the Theme setting.

What's Next?

You can create more new projects and folders on your LabKey Trial Server to mock up how it might work for your specific solution. Many features available in a full LabKey installation are not shown in this trial version; you can learn more in our documentation.

Feel free to contact us and we can help you determine if LabKey is the right fit for your research.




Extending Your Trial


When your LabKey Server Hosted Trial is nearing it's expiration date, you will see a banner message in the server offering you upgrade options. If you need a bit more time to explore, you can extend your trial beyond the initial 30 days.

  • From within your trial server, select > Manage Hosted Server Account.
  • Click Extend.

Related Topics




Trial Server: Modules


Your LabKey Server Trial instance contains the following modules:
  • Announcements: Message Board and Discussion Service
  • API: Internal API classes
  • Audit: Audit Log Service
  • CAS: CAS SSO Authentication Service
  • Core: Administration and Essential Services
  • Docker: Docker Module
  • EvaluationContent: Evaluation Server Content Module
  • Experiment: Experiment Service
  • FileContent: File Content Service
  • Internal: Internal Classes and Resources
  • Issues: Issue Tracking Service
  • List: List Service
  • Pipeline: Pipeline Service
  • Query 18.10: Query Service
  • rstudio: Integrate RStudio Functionality
  • Search: Search Service
  • Study: Study and Assay Service
  • Visualization: Visualization Service
  • Wiki: Wiki Service
Site administrators can view the list of modules at > Site > Admin Console. Click Module Information.

Community Edition

Premium Editions

To learn about additional features and modules available with LabKey Premium Editions, review this chart or contact us.




Virtual Machine Trial Server


You can download a virtual machine-based instance of LabKey Server for on-site evaluation purposes.

Set Up the VM Instance

  • In VirtualBox, select File > Import Appliance. In the Virtual Machine Appliance popup dialog, browse to and select the downloaded virtual machine file. Click Next. Accept the defaults and click Import.
  • Once the VM has been added to the list, select it, and click Start > Normal Start.
  • VirtualBox will startup the server VM. (This may take a few minutes to complete.)
  • When the server VM is ready, you will see a new VirtualBox window requesting an ubuntu login. Ignore but do not close this window; you do not need to log in here.

Launch LabKey Server

  • When startup has finished, open a new web browser window or tab (such as Chrome) outside the vm. Go to the following URL location to find the LabKey Server instance running. If you do not see a server running there, wait a few more minutes for startup to complete.
  • When you first invoke the server, you will be prompted to set up an account by entering your email address and password. (This first user account created is the Site Administrator for this LabKey installation.)
  • LabKey Server is now available to use.
  • To begin, see next steps below.

Add External Users / Collaborators

If you want to add other users to your VM instance so they can explore and collaborate, you must first do the following so that the auto-generated email will direct them to the machine and port where your server is running:

  • Select (Admin) > Site > Admin Console.
  • Click Admin Console Links.
  • Under Configuration, click Site Settings.
  • Click Save. The value in this field will now be included in any invite emails that are sent out.
  • Proceed to add users.

Next Steps

  • Explore basic features using the the built-in example project:
    • Click the LabKey logo in the upper left to return to the home page of your server.
    • Click the Example Project icon.
    • Use the introductory topics here to explore the examples: Explore LabKey Server - Hosted Trial

  • The following tutorials provide step by step instructions for exploring even more key functionality:

Troubleshooting

On Windows, if you receive the following error, you need to disable Hyper-v.

VT-x is not available (VERR_VMX_NO_VMX)

To ensure that Hyper-v is disabled, click the Start menu, then type "Turn Windows features on or off". Select this option in the search results. In the popup dialog box, unselect Hyper-V and click OK. You may need to restart your host machine for the changes to take effect.




Explore LabKey Biologics - Hosted Trial


To get started using LabKey Biologics, you can create a trial instance of LabKey Biologics. Go here to Start Your Free Trial, enter some basic contact information, and go!

Biologics trial instances contain some example data and you can run tutorials to explore using LabKey Biologics and adding your own research data. Your trial lasts 30 days and we're ready to help you understand how LabKey Biologics can work for you.</p>

Biologics Tours

Tour key functionality of LabKey Biologics.

Trial Support Forum

If you have questions or need guidance, please reach out in the Trial Support Forum.

Extending Your Trial: When your trial is nearing it's expiration date, you will see a banner message in the server offering you upgrade options. If you need a bit more time to explore, you can extend your trial beyond the initial 30 days



Learn About LabKey Biologics: Trial


Welcome to LabKey Biologics

This topic helps you get started understanding how LabKey Biologics works and how it can work for you. It is intended to be used alongside a Biologics Trial. You should have another browser window open on the home page of your instance.

Navigation and Getting Started

Your new Biologics Trial includes the Biologics application populated with sample data. To launch the application and explore the sample, click Explore Sample Project on your biologics trial.

The home page for Biologics includes a search box and tiles for different aspects of the data.

For more details about each of these options, see Navigate the Registry.

You can also watch the Biologics videos here: LabKey Biologics - Videos.

Biologics Trial Tutorials

Highlights to explore in your biologics trial server:

What's Next?




Tutorials


Tutorials provide a "hands on" introduction to the core features of LabKey Server, giving step-by-step instructions for building solutions to common problems. They are listed roughly from simple to more complex, and you can pick and choose only those that interest you.

In order to run any tutorial, you will need:

This icon indicates whether the tutorial can be completed on a Hosted trial server or a Virtual Machine trial server.

New User Tutorials

Title Description Works on Hosted Trial Server? Works on VM Trial Server?
Data Grid TutorialTake a quick tour of data grids and visualizations. No trial server necessary.  
Tutorial: SecurityLearn how to organize and secure your data using LabKey Server.
Tutorial: File RepositoryManage, search, and share file resources.
Tutorial: Collaboration ToolsLearn how to use LabKey Server's secure, web-based collaboration tools.
Tutorial: ListsWork with data tables.
Tutorial: Electronic Lab NotebookSet up a basic electronic lab notebook and sample inventory system.


Study Tutorials



Assay Tutorials

Title Description Works on Hosted Trial Server? Works on VM Trial Server?
Tutorial: Design a General Purpose Assay Type (GPAT)Import, manage, and analyze assay data. Add assay data to a study.
Tutorial: NAb AssayWork with NAb experiment data from 96-well or 384-well plates 
Tutorial: ELISA AssayImport and analyze ELISA experiment data. 
Tutorial: ELISpot Assay TutorialImport and analyze ELISpot experiment data. 
Discovery Proteomics TutorialStorage and analysis for high-throughput proteomics and tandem mass spec experiments. 
Tutorial: Import a Flow WorkspaceLearn about using LabKey for management, analysis, and high-throughput processing of flow data. 
Tutorial: Perform a LabKey Flow AnalysisAnalyze flow data. 
Tutorial: Import Flow Data from FCS Express  
Tutorial: Set Flow BackgroundLearn about setting metadata and using backgrounds with flow data. 
Luminex Assay Tutorial Level IManage, quality control, analyze, share, integrate and export Luminex immunoassay results. 
Luminex Assay Tutorial Level IIUse advanced features for quality control and analysis. 
Microarray Assay TutorialWork with sample microarray data. 
Expression Matrix Assay TutorialTry an example expression matrix assay. 


Developer Tutorials



Biologics Tutorials






Set Up for Tutorials: Trial


This topic covers the quickest and easiest way to set up to run LabKey tutorials using a free trial of LabKey Server. If you want to run tutorials not supported in the trial environment, or already have access to LabKey Server, see Set Up for Tutorials - Non-Trial.

Tutorials you can run using a free trial of LabKey Server are marked with this badge.

LabKey Trial Server

To run the LabKey Tutorials, you need three things:

1. An account on a running LabKey Server instance. It only takes a few minutes to create your own LabKey Trial Server:

2. A basic familiarity with navigation, folder creation, and utilities like web parts. Use this topic alongside your new instance: 3. A tutorial workspace project where you are an administrator and can create new folders.

  • On the home page of your trial server, click Create.
  • Enter the Name "Tutorials".
    • If you plan to share this trial server with other users, consider using "Tutorials-Username" so you can each have your own workspace.
  • Leave the default folder type selection, "Collaboration," and click Next.
  • Leave the default permission selection, "My User Only," and click Next.
  • Skip the advanced settings and click Finish.
  • (Optional): To enhance this project, you can add some custom content making it easier to use.

Finished

Congratulations, you can now begin running tutorials in this workspace on your trial server.

I'm Ready for the Tutorial List




Set Up for Tutorials - Non-Trial


This topic explains how to set up for the LabKey Tutorials on a non-trial instance of the server. If you are using a hosted trial server, use this topic instead: Set Up for Tutorials: Trial.

To run the LabKey Tutorials you need:

  1. An account on a running LabKey Server instance.
  2. A tutorial workspace project on that server where you are an administrator and can create new folders.
  3. A basic familiarity with navigation, folder creation, and utilities like web parts.
If running tutorials on a 30 day free trial instance of LabKey Server does not meet your needs, the other options for creating a tutorial workspace are:

Local VM Installation

1. You can download and install your own server on a local machine using VM image of the server. Follow the instructions in this topic to register, download, install, and start using LabKey Server.

2. You will be the site administrator on this server. To create a tutorials project:
  • Select (Admin) > Site > Create Project.
  • Enter the name "Tutorials" and choose folder type "Collaboration".
  • Accept all project wizard defaults.

3. Learn the navigation and UI basics in this topic:

Existing Installations

1. & 2. If your organization is already using LabKey server, contact your administrator about obtaining access as an administrator to a project or folder to use. For example, they might create and assign you a "Tutorials-username" project or subfolder. They will provide you the account information for signing in, and the URL of the location to use.

LabKey Server installations in active use may have specialized module sets or other customizations which cause the UI to look different than tutorial instructions. It's also possible that you will not have the same level of admin access that you would have if you installed a local demo installation.

3. Learn the navigation and UI basics in this topic:

The location given to you by your administrator is your tutorial workspace where you can create a subfolder for each tutorial that you run.

Full Local Installation

1. If none of the above are suitable, you may need to complete a full manual installation of LabKey Server. Detailed instructions are provided here:

If you encounter difficulty during this installation, you may find help in our Installation Forum.

2. You will be the site administrator on this server. To create a tutorials project:

  • Select (Admin) > Site > Create Project.
  • Enter the name "Tutorials" and choose folder type "Collaboration".
  • Accept all project wizard defaults.

3. Learn the navigation and UI basics in this topic:

Finished

Congratulations, you can now log in, navigate to your workspace, and begin running tutorials.

Learn More

I'm Ready for the Tutorial List




Navigation and UI Basics


Welcome to LabKey Server!

This topic helps you get started using LabKey Server, understanding the basics of navigation and the user interface.

If you are using a LabKey Trial Server, use this topic instead: Learn About LabKey Server: Trial.

Projects and Folders

The project and folder hierarchy is like a directory tree and forms the basic organizing structure inside LabKey Server. Everything you create or configure in LabKey Server is located in some folder. Projects are the top level folders, with all the same behavior, plus some additional configuration options; they typically represent a separate team or research effort.

The Home project is a special project. It is the default landing page when users log in and cannot be deleted. You can customize the content here to suit your needs. To return to the home project at any time, click the LabKey logo in the upper left corner.

The project menu is on the left end of the menu bar and includes the display name of the current project.

Hover over the project menu to see the available projects, and folders within them. Click any project or folder name to navigate there.

Any project or folder with subfolders will show / buttons for expanding and contracting the list shown. If you are in a subfolder, there will be a clickable 'breadcrumb' trail at the top of the menu for quickly moving up the hierarchy. The menu will scroll when there are enough items, with the current location visible and expanded by default.

The project menu always displays the name of the current project, even when you are in a folder or subfolder. A link with the Folder Name is shown near the top of page views like the following, offering easy one click return to the main page of the folder.

For more about projects, folders, and navigation, see Project and Folder Basics.

Tabs

Using tabs within a folder can give you new "pages" of user interface to help organize content. LabKey study folders use tabs as shown here:

When your browser window is too narrow to display tabs arrayed across the screen, they will be collapsed into a pulldown menu showing the current tab name and a (chevron). Click the name of the tab on this menu to navigate to it.

For more about adding and customizing tabs, see Use Tabs.

Web Parts

Web parts are user interface panels that can be shown on any folder page or tab. Each web part provides some type of interaction for users with underlying data or other content.

There is a main "wide" column on the left and narrower column on the right. Each column supports a different set of web parts. By combining and reordering these web parts, an administrator can tailor the layout to the needs of the users.

For a hands-on example, try the Tutorial: Collaboration Tools.

To learn more, see Add Web Parts and Manage Web Parts. For a list of the types of web parts available in a full installation of LabKey Server, see the Web Part Inventory.

Header Menus and URLs

In the upper right, icon menus offer:

  • (Search): Click to open a site-wide search box
  • (Admin): Shown only to Admins: Administrative options available to users granted such access. See Admin Console for details about options available.
  • (User): Login and security options; help links to documentation and support forums.


Watch the URL at the top of the page as you navigate LabKey Server and explore features on your server. Many elements are encoded in the URL and programmatic access via building URLs is possible with APIs. Learn more here: LabKey URLs.

Security Model

LabKey Server has a group and role-based security model. Whether an individual is authorized to see a resource or perform an action is checked dynamically based on the groups they belong to and roles (permissions) granted to them. Learn more here: Security.

What's Next?




LabKey Server Editions


LabKey Server Editions

  • Community Edition: Free to download and use forever. Best suited for technical enthusiasts and evaluators in non-mission-critical environments. LabKey provides community forums and documentation to help users support each other.
  • Premium Editions: Paid subscriptions that provide additional functionality to help teams optimize workflows, manage complex projects, and explore multi-dimensional data. Premium Editions also include professional support services for the long-term success of your informatics solutions.
For a complete list of features available in each LabKey Server Edition, see LabKey Server Edition Comparison.

Other Products and Modules




Training


Administrator Training

LabKey's administrator training course, LabKey Fundamentals, is included in the Professional and Professional Plus Editions. It provides an introduction to the following topics:

  • LabKey Server Basics: Explains the basic anatomy/architecture of the server and its moving parts. It outlines the basic structures of folders and data containers, and the modules that process requests and craft responses. Best practices for configuring folders is included. The role of Administrators is also described.
  • Security: Describes LabKey Server's role-based security model--and how to use it to protect your data resources. General folder-level security is described, as well as special security topics, such as dataset-level security and Protected Health Information (PHI) features. Practical security information is provided, such as how to set up user accounts, assigning groups and roles, best practices, and testing security configurations using impersonation.
  • Collaboration: Explains how to use the Wiki, Issues, and Messages modules. Branding and controlling the look-and-feel of your server are also covered.
  • Files and the Database: Explains the two basic ways that LabKey Server can hold data: (1) as files and (2) as records in a database. Topics include: full-text search, converting tabular data files into database tables, special features of the LabKey database (such as 'lookups'), the role of SQL queries, adding other databases as external data sources.
  • Instrument Data: Explains how LabKey Server models and captures instrument-derived data, including how to create a new assay "design" from scratch, or how to use a prepared assay design. Special assay topics are covered, such as transform scripts, creating new assay design templates ("types") from simple configuration files, and how to replace the default assay user interface.
  • Clinical/Research Study Data Management: Explains how to integrate heterogeneous data, such as instrument, clinical, and demographic data, especially in the context of longitudinal/cohort studies.
  • Reports: Explains the various ways to craft reports on your data, including R reports, JavaScript reports, and built-in visualizations, such as Time Charts, Box Plots, and Scatter Plots.
  • Specimens: Explains the ways that LabKey Server can model and manage specimen/sample data.
  • Development: A high-level overview of how to extend LabKey Server. The Professional Edition includes support for users writing custom SQL and R scripts. The Professional Plus Edition provides support for users extending LabKey Server with JavaScript/HTML client applications, user-created file modules, and more (see Developer Training below).
  • Operations: Describes best practices from an IT point-of-view, including installing a server, hardware requirements, logging, and how to debug and track down problems with the server.

Developer Training

LabKey's developer training is included in the Professional Plus Edition. It is tailored to your project's specific needs and can cover:

  • Server-to-server integrations
  • Client APIs
  • ETLs
  • Assay transform scripts
  • Remote pipeline processing servers
  • Custom LabKey-based pipelines
  • Module development assistance



LabKey Support Portal Pages


Premium Feature — Available with the Professional, Professional Plus, and Enterprise Editions. Learn more or contact LabKey.

LabKey Premium Edition clients are provided with a custom support portal page giving them easy access to professional services and support directly from LabKey specific to their project. This portal provides quick links to contact information, builds, documentation, support tickets, announcements, and an easy way to share files with LabKey as required.

To find your own support portal:

  • Click the LabKey logo in the upper left to go to the home page.
  • Log in if necessary to see your projects.
  • Click on the project icon to open the portal.

Features:

Build Content Reporting

Client support portals also include a report on the content of each new build detailing changes since the prior build released to that client.

  • From your main support portal, click Server Builds.
  • In the Download the Latest LabKey Server Release web part, Click here for the latest Bug Fix updates included in this build.

The list of issues closed since your prior build is shown, sorted by module. Verified issues listed are those that have been resolved, confirmed, and closed. If "unverified" issues are listed, they have been marked as resolved, but confirmation and closing has not yet been completed. If there are no issues for a given module, it will not be shown on the list.

Click any listed issue to see details in the issue tracker.

  • Note that secure issues are not displayed in this report. If you are waiting for a secure issue to be fixed, please follow up with your LabKey support person.



Modules in LabKey Server Editions


This topic provides the list of modules included in each edition of LabKey Server. For a complete list of features available in each LabKey Server Edition, see LabKey Server Edition Comparison.

 

LabKey Trial Community EditionProfessional EditionProfessional Plus EditionEnterprise Edition
announcementsannouncementsannouncementsannouncementsannouncements
apiapiapiapiapi
  assayrequestassayrequestassayrequest
auditauditauditauditaudit
 bigironbigironbigironbigiron
cas cascascas
   cdisc_ODMcdisc_ODM
   cloudcloud
   compliancecompliance
   complianceActivitiescomplianceActivities
corecorecorecorecore
  dataintegrationdataintegrationdataintegration
docker  dockerdocker
  duoduoduo
 elisaelisaelisaelisa
 elispotassayelispotassayelispotassayelispotassay
evaluationContent    
experimentexperimentexperimentexperimentexperiment
 fcsexpressfcsexpressfcsexpressfcsexpress
filecontentfilecontentfilecontentfilecontentfilecontent
 flowflowflowflow
   freezerprofreezerpro
internalinternalinternalinternalinternal
issuesissuesissuesissuesissues
listlistlistlistlist
 luminexluminexluminexluminex
 microarraymicroarraymicroarraymicroarray
 ms1ms1ms1ms1
 ms2ms2ms2ms2
 nabnabnabnab
    nlp
  oauthoauthoauth
   openEMPIopenEMPI
pipelinepipelinepipelinepipelinepipeline
  premiumpremiumpremium
queryqueryqueryqueryquery
   redcapredcap
rstudio  rstudiorstudio
  samlsamlsaml
searchsearchsearchsearchsearch
studystudystudystudystudy
 surveysurveysurveysurvey
   synonymsynonym
 targetedmstargetedmstargetedmstargetedms
visualizationvisualizationvisualizationvisualizationvisualization
wikiwikiwikiwikiwiki

Related Topics




LabKey Server


This topic is under construction.

LabKey Server is an open-source software platform designed to help research organizations integrate, analyze, and share complex biomedical data. Adaptable to varying research protocols, analysis tools, and data sharing requirements, LabKey Server couples the flexibility of a custom solution with enterprise-level scalability to support scientific workflows.

Introduction

Solutions




Introduction to LabKey Server


This topic is for absolute beginners to LabKey Server. It explains what LabKey Server is for, how it works, and how to build solutions using its many features.

What is LabKey Server?

LabKey Server's features can be grouped into three main areas:

1. Data Repository

LabKey Server lets you bring data together from multiple sources into one repository. These sources can be physically separated in different systems, such as data in Excel spreadsheets, different databases, FreezerPro, REDCap, etc. Or the data sources can be separated "morphologically", having different shapes. For example, patient questionnaires, instrument-derived assay data, medical histories, and specimen inventories all have different data shapes, with different columns names and different data types. LabKey Server can bring all of this data together to form one integrated whole that you can browse and analyze together.

2. Data Showcase

LabKey Server lets you securely present and highlight data over the web. You can present different profiles of your data to to different audiences. One profile can be shown to the general public with no restrictions, while another profile can be privately shared with selected individual colleagues. LabKey Server lets you collaborate with geographically separated teams, or with your own internal team members. In short, LabKey Server lets you create different relationships between data and audiences, where some data is for general viewing, other data is for peer review, and yet other data is for group editing and development.

3. Electronic Laboratory

LabKey Server provides many options for analyzing and inquiring into data. Like a physical lab that inquires into materials and natural systems, LabKey Server makes data itself the object of inquiry. This side of LabKey Server helps you craft reports and visualizations, confirm hypotheses, and generally provide new insights into your data, insights that wouldn't be possible when the data is separated in different systems and invisible to other collaborators.

The LabKey Server Platform

LabKey Server is a software platform, as opposed to an application. Applications have fixed use cases targeted on a relatively narrow set of problems. As a platform, LabKey Server is different: it has no fixed use cases, instead it provides a broad range of tools that you configure to build your own solutions. In this respect, LabKey Server is more like a car parts warehouse and not like any particular car. Building solutions with LabKey Server is like building new cars using the car parts provided. To build new solutions you assemble and connect different panels and analytic tools to create data dashboards and workflows.

The following illustration shows how LabKey Server takes in different varieties of data, transforms into reports and insights, and presents them to different audiences.

How Does LabKey Server Work?

LabKey Server is a web server, and all web servers are request-response machines: they take in requests over the web (typically as URLs through a web browser) and then craft responses which are displayed to the user.

Modules

Modules are the main functionaries in the server. Modules interpret requests, craft responses, contain all of the web parts, and application logic. The responses can take many different forms:

  • a web page in a browser
  • an interactive grid of data
  • a report or visualization of underlying data
  • a file download
  • a long-running calculation or algorithm
LabKey Server uses a database as its main data store. There is always a main database, either PostgeSQL or MS SQL Server, and you can attach any number of other databases to the server. The following databases are supported:
  • PostgreSQL
  • MS SQL Server
  • Oracle
  • My SQL
  • SAS
LabKey Server offers non-disruptive integration with your existing systems and workflows. You can keep your existing data systems in place, using LabKey Server to augment them, or you can use LabKey Server to replace your existing systems. For example, if you already use FreezerPro to manage your specimens, REDCap to collect patient data, and SAS to hold medical histories, LabKey Server can synchronize and combine the data in these systems, so you can build a more complete picture of your research results, without disrupting the workflows you have already built.

The illustration below shows the relationships between web browsers, LabKey Server, and the underlying databases. The modules shown are not a complete set; many other modules are included in LabKey Server.

User Interface

You configure your own user interface by adding panels, aka "web parts", each with a specific purpose in mind. Some example web parts:

  • The Wiki web part displays text and images to explain your research goals and provide context for your audience. (The topic you are reading right now is displayed in a Wiki web part.)
  • The Files web part provides an area to upload, download, and share files will colleagues.
  • The Query web part displays interactive grids of data.
  • The Report web part displays the results of an R or JS based visualization.
Group web parts on separate tabs to form data dashboards.

The illustration below shows a data dashboard formed from tabs and web parts.

Folders and Projects

Folders are the "blank canvases" of LabKey Server, the workspaces where you organize dashboards and web parts. Folders are also important in terms of securing your data, since you grant access to audience members on a folder-by-folder basis. Projects are top level folders: they function like folders, but have a wider scope. Projects also form the center of configuration inside the server, since any setting made inside a project cascades into the sub-folders by default.

Security

LabKey uses "role-based" security to control who has access to data. You assign roles, or "powers", to each user who visits your server. Their role determines how much they can see and do with the data. The available roles include: Administrator (they can see and do everything), Editors, Readers, Submitters, and others. Security is very flexible in LabKey Server. Any security configuration you can imagine can be realized: whether you want only a few select individual to see your data, or if you want the whole world to see your data.

The server also has extensive audit logs built. The audit logs record:

  • Who has logged in and when
  • Changes to a data record
  • Queries performed against the database
  • Server configuration changes
  • File upload and dowload events
  • And many other activities

The Basic Workflow: From Data Import to Reports

To build solutions with LabKey Server, follow this basic workflow: import or synchronize your data, apply analysis tools and build reports on top of the data, and finally share your results with different audiences. Along the way you will add different web parts and modules as needed. To learn the basic steps, start with the tutorials, which provide step-by-step instructions for mastering the basic building blocks available in the server.

Ready to See More?




Navigate Server


This topic covers the basics of navigating LabKey projects and folders on your LabKey site as a user without administrator permissions. The project menu is on the left end of the menu bar and includes the display name of the current project.

In the upper right, icon menus offer:

  • (Search): Click to open a site-wide search box
  • (User): Login and security options; context-sensitive help.
  • (Admin): Shown only to Admins: Administrative options available to users granted such access.

Project and Folder Menu

Hover over the project menu to see the available projects and folders within them. Note that only projects and folders to which you have at least "Read" access are shown.

Any project or folder with subfolders will show and buttons for expanding and contracting the list shown. The menu will scroll when there are enough items, and the current location will be visible and expanded by default.

  • (Permalink URL): Click for a permalink to the current location.
  • Administrators have additional options shown in the bottom row of this menu.

Navigation

Click the name of any project or folder in the menu to navigate to that location. When you are in a folder or subfolder, you will see a "breadcrumb" path at the top of the projects menu, making it easy to step back up one or more levels.

Notice that from within a folder or subfolder, the project menu still displays the name of the project. A link with the Folder Name is shown next to the title offering easy one click return to the main page of the folder.

Context Sensitive Help

Click the (User) icon in the upper right to open the user menu. Options for help include:

Related Topics




Data Basics


LabKey Server lets you explore, interrogate and present your biomedical research data online and interactively in a wide variety of ways.

Topics

An example online data grid:

A scatter plot visualization of the same data:




Data Grids


Data grids display your data as a table composed of columns and rows. (See an interactive example.)

LabKey Server provides sophisticated tools for customizing data grids, including sorting and filtering your data, creating tabular and graphical reports, and exporting your data to other file types.

Data Grid Topics

The following topics explain how to work with data grids:




Data Grids: Basics


Anatomy of a Data Grid

The following image shows a typical data grid.

  • Data grid title: The name of the data grid.
  • QC State filter: Within a LabKey study, when one or more dataset QC states are defined, the button bar will include a menu for filtering among them. The current status of that filter is shown above the grid.
  • Folder location: The folder location. Click to navigate to the home page of the folder.
  • Grid view indicator: Shows which view, or perspective, on the data is being shown. Custom views are created to show a joined set of tables or highlight particular columns in the data. Every dataset has a "default" view that can also be customized if desired.
  • Filter bar: When filters are applied, they appear here. Click the X to remove a filter. When more than one is present, a Clear all button is also shown.
  • Button bar: Shows the different tools that can be applied to your data. A triangle indicates a menu of options.
  • Column headers: Click a column header for a list of options
  • Data records: Displays the data as a 2-dimensional table of rows and columns.
  • Page through data: Control how much data is shown per page and step through pages.

Column Header Options

See an interactive example grid on labkey.org.

Button Bar

The button bar tools available may vary with different types of data grid. Study datasets can provide additional functionality, such as filtering by cohort, that are not available to lists. Assays and proteomics data grids provide many additional features.

Hover over icon buttons to see a tooltip with the text name of the option. Common buttons and menus include:

  • (Filter): Click to open a panel for filtering by participant groups and cohorts.
  • (Grid Views): Pull-down menu for creating, selecting, and managing various grid views of this data.
  • (Charts/Reports): Pull-down menu for creating and selecting charts and reports.
  • (Export): Export the data grid as a spreadsheet, text, or in various scripting formats.
  • (Insert Data): Insert a new single row, or bulk import data.
  • (Delete): Delete one or more rows selected by checkbox. This option is disabled when no rows are selected.
  • Design/Manage: with appropriate permissions, change the structure and behavior of the given list ("Design") or dataset ("Manage").
  • Groups: Select which cohorts, participant groups, or treatment groups to show. Also includes options to manage cohorts and create new participant groups.
  • View Specimens: If one or more rows are selected using the checkboxes, clicking View Specimens will open a grid view of specimens corresponding to the same participant and visit.
  • Print: Generate a printable version of the dataset in your browser.

Page Through Data

In the upper right corner, you will see how your grid is currently paginated. In the above example, there are 484 rows in the dataset and they are shown 100 to a "page", so the first page is rows 1-100 of 484. To step through pages, in this case 100 records at a time, click the arrow buttons. To adjust paging, hover over the row count message (here "1-100 of 484") and it will become a button. Click to open the menu:

  • Show First/Show Last: Jump to the first or last page of data.
  • Show All: Show all rows in one page.
  • Click Paging to see the currently selected number of rows per page. Notice the > caret indicating there is a submenu to open. On the paging submenu, click another option to change pagination.

Examples

The following links show different views of a single data grid:

Related Topics




Import Data


LabKey provides a variety of methods for importing data into a data grid. Depending on the type and complexity of the data, you must first identify the type of data structure in which your data will be stored. Each data structure has a different specific process for designing a schema and then importing data. The general import process for data is similar among many data structures. Specific import instructions for many types are available here:
Note: When data is imported as a TSV, CSV, or text file, it is parsed using UTF-8 character encoding.



Sort Data


This page explains how to sort data in a table or grid.

Sort Data in a Grid

To sort data in a grid view, click on the header for a column name. Choose either:

  • Sort Ascending: Show the rows with lowest values (or first values alphabetically) for this column at the top of the grid.
  • Sort Descending: Show the rows with the highest values at the top.

Once you have sorted your dataset using a particular column, a triangle icon will appear in the column header: for an ascending sort or for a descending sort.

You can sort a grid view using multiple columns at a time as shown above. Sorts are applied in the order they are added, so the most recent sort will show as the primary way the grid is sorted. For example, to sort by date, with same-date results sorted by temperature, you would first sort on temperature, then on date.

Note: By default, LabKey sorting is case-sensitive. If your LabKey installation is running against Microsoft SQL Server, however, sorting is case-insensitive. If you're not sure which database your LabKey installation is running against, ask your system administrator.

Clear Sorts

To remove a sort on an individual column, click the column caption and select Clear Sort.

Advanced: Understand Sorting URLs

The sort specifications are included on the page URL. You can modify the URL directly to change the sorted columns, the order in which they are sorted, and the direction of the sort. For example, the following URL sorts the Physical Exam grid first by ascending ParticipantId, and then by descending Temp_C:

Note that the minus ('-') sign in front of the Temp_C column indicates that the sort on that column is performed in descending order. No sign is required for an ascending sort, but it is acceptable to explicitly specify the plus ('+') sign in the URL.

The %2C hexadecimal code that separates the column names represents the URL encoding symbol for a comma.

Related Topics




Filter Data


You can filter data displayed in a grid to reduce the amount of data shown, or to exclude data that you do not wish to see.

Filter Column Values

  • Click on a column name and select Filter.

Filter by Value

In cases where there are fewer than 250 unique values, the filter popup will open on the Choose Values tab by default. Here, you can directly select one or more individual values using checkboxes. Click on a label to select only a single value, add or remove additional values by clicking on checkboxes.

Note: When there are more than 250 values, the default option will be Choose Filters. Use filtering expressions to filter. Depending on the datatype of the column, you may still be able to select the Choose Values tab, on which you will see a truncated list of the first 250 values.

Filtering Expressions

Filtering expressions available in dropdowns vary by datatype and context. Possible filters include, but are not limited to:

  • Presence or absence of a value for that row
  • Equality or inequality
  • Comparison operators
  • Membership or lack of membership in a named semicolon separated set
  • Starts with and contains operators for strings
  • Between (inclusive) or Not Between (exclusive) two comma separated values
For a full listing, see Filtering Expressions.

  • Switch to the Choose Filters tab, if available.
  • Specify a filtering expression (such as "Is Greater Than"), and value (such as "57") and click OK.

You may add a second filter if desired - the second filter is applied as an AND with the first. Both conditions must be true for a row to be included in the filtered results.

Once you have filtered on a column, the filter icon () appears in the column header. Current filters are listed above the grid, and can be removed by simply clicking the X in the filter panel.

When there are multiple active filters, you can remove them individually or use the link to Clear All that will be shown.

Notes:
  • Leading spaces on strings are not stripped. For example, consider a list filter like Between (inclusive) which takes two comma-separated terms. If you enter range values as "first, second", rows with the value "second" (without the leading space) will be excluded. Enter "first,second" to include such rows.
  • By default, LabKey filtering is case-sensitive. However, if your LabKey installation is running against Microsoft SQL Server, filtering is case-insensitive.

Filter Value Variables

Using filter value variables can help you use context-sensitive information in a filter. For example, use the variable "~me~" (including the tildes) on columns showing user names from the core.Users table to filter based on the current logged in user.

Persistent Filters

Some filters on some types of data are persistent (or "sticky") and will remain applied on subsequent views of the same data. For example, some types of assays have persistent filters for convenience; these are listed in the active filter bar above the grid.

Use Faceted Filtering

When applying multiple filters to a data grid, the options shown as available in the filter popup will respect prior filters. For example, if you first filter our sample demographics dataset by "Country" and select only "Uganda", then if you open a second filter on "Primary Language" you will see only "French" and "English" as options - our sample data includes no patients from Uganda who speak German or Spanish. The purpose is to simplify the process of filtering by presenting only valid filter choices. This also helps you avoid unintentionally empty results.

Understand Filter URLs

Filtering specifications are included on the page URL. The following URL filters the demo study "Physical Exam" dataset to show only rows where weight is greater than 57. The column name, the filter operator, and the criterion value are all specified as URL parameters.

https://www.labkey.org/study/home/Study/demo/dataset.view?datasetId=5004&Dataset.sort=ParticipantId&Dataset.Weight_kg~gt=57

To specify that a grid should be displayed using the user's last filter settings, set the .lastFilter URL parameter to true, as shown:

https://www.labkey.org/Issues/home/Developer/issues/list.view?.lastFilter=true

Filter by Group

Within a study dataset, you may also filter a data grid by participant group. Click the (Filter) icon above the grid to open the filter panel. Select checkboxes in this panel to further filter your data. Note that filters are cumulatively applied and listed in the active filter bar above the data grid.

Related Topics




Filtering Expressions


Filtering expressions available for columns or when searching for subjects of interest will vary by datatype of the column, and not all expressions are relevant or available in all contexts. In the following tables, the "Arguments" column indicates how many data values, if any, should be provided for comparison with the data being filtered.

ExpressionArgumentsExampleXXXXXXXDescription
Has Any Value  Returns all values, including null
Is Blank  Returns blank values
Is Not Blank  Returns non-blank values
Equals1 Returns values matching the value provided
Does Not Equal1 Returns non-matching values
Is Greater Than1 Returns values greater than the provided value
Is Less Than1 Returns values less than the provided value
Is Greater Than or Equal To1 Returns values greater than or equal to the provided value
Is Less Than or Equal To1 Returns values less than or equal to the provided value
Contains1 Returns values containing a provided value
Does Not Contain1 Returns values not containing the provided value
Starts With1 Returns values which start with the provided value
Does Not Start With1 Returns values which do not start with the provided value
Between, Inclusive2, comma separatedExample usage: -4,4Returns values between or matching two comma separated values provided
Not Between, Exclusive2, comma separatedExample usage: -4,4Returns values which are not between and do not match two comma separated values provided
Equals One Of1 or more, semi-colon separatedExample usage: a;b;cReturns values matching any one of a semi-colon separated list
Does Not Equal Any Of1 or more, semi-colon separatedExample usage: a;b;cReturns values not matching a semi-colon separated list
Contains One Of1 or more, semi-colon separatedExample usage: a;b;cReturns values which contain any one of a semi-colon separated list
Does Not Contain Any Of1 or more, semi-colon separatedExample usage: a;b;cReturns values which do not contain any of a semi-colon separated list

Boolean Filtering Expressions

Expressions available for data of type boolean (true/false values):

  • Has Any Value
  • Is Blank
  • Is Not Blank
  • Equals
  • Does Not Equal

Date Filtering Expressions

Date and DateTime data can be filtered with the following expressions:

  • Has Any Value
  • Is Blank
  • Is Not Blank
  • Equals
  • Does Not Equal
  • Is Greater Than
  • Is Less Than
  • Is Greater Than or Equal To
  • Is Less Than or Equal To

Numeric Filtering Expressions

Expressions available for data of any numeric type, including integers and double-precision numbers:

  • Has Any Value
  • Is Blank
  • Is Not Blank
  • Equals
  • Does Not Equal
  • Is Greater Than
  • Is Less Than
  • Is Greater Than or Equal To
  • Is Less Than or Equal To
  • Between, Inclusive
  • Not Between, Exclusive
  • Equals One Of
  • Does Not Equal Any Of

String Filtering Expressions

String type data, including text and multi-line text data, can be filtered using the following expressions:

  • Has Any Value
  • Is Blank
  • Is Not Blank
  • Equals
  • Does Not Equal
  • Is Greater Than
  • Is Less Than
  • Is Greater Than or Equal To
  • Is Less Than or Equal To
  • Contains
  • Does Not Contain
  • Starts With
  • Does Not Start With
  • Between, Inclusive
  • Not Between, Exclusive
  • Equals One Of
  • Does Not Equal Any Of
  • Contains One Of
  • Does Not Contain Any Of



Saved Filters and Sorts


When you are looking at a data grid, you can sort and filter the data as you wish, but those sorts and filters only persist for your current session on that page. Using the .lastFilter parameter on the URL can preserve the last filter, but otherwise these sorts and filters are temporary.

To create a persistent filter or sort, you can save it as part of a custom grid view. If this saved grid view is shared with all users, the saved filters and sorts defined on it are also available to those users.

Define a Saved Sort

  • Navigate to the grid you'd like to modify.
  • Select (Grid Views) > Customize Grid
  • Click the Sort tab.
  • In the left panel, click the checkbox(es) for fields on which you want to sort.
  • In the right panel, specify whether the sort order should be ascending or descending for each sort applied.
  • Click Save.
  • You may save as a new named grid view or as the default and select whether to make it available to all users.

You can also create a saved sort by first sorting your grid directly using the column headers, then opening the grid customizer panel to convert the local sort to a saved one.

  • In the grid view with the saved sort applied above, sort on a second column, in this example we chose 'Lymphs'.
  • Open (Grid Views) > Customize Grid.
  • Click the Sort tab. Note that it shows (2), meaning two sorts are now defined. Until you save the grid view with this additional sort included, it will remain as temporary as sorts usually are.
  • Drag and drop if you want to change the order in which sorts are applied.
  • Remember to Save your grid to save the additional sort.

Define a Saved Filter

The process for defining saved filters is very similar. You can filter locally first or directly define saved filters. An important advantage of using the saved filters interface is that when filtering locally, you are limited to two filters on a given column. Saved filters may include any number of separate filtering expressions for a given column, which are all ANDed together.

  • Select (Grid Views) > Customize Grid.
  • Click the Filter tab.
  • In the left panel, check boxes for the column(s) on which you want to filter.
  • Drag and drop filters in the right panel to change the filtering order.
  • In the right panel, specify filtering expressions for each selected column. Use pulldowns and value entry fields to set expressions, add more using the (Add) buttons.
  • Use the buttons to delete individual filtering expressions.
  • Save the grid; select whether to make it available to other users.

Apply Grid Filter

When viewing a data grid, you can enable and disable all saved filters and sorts using the Apply Grid Filter checkbox in the (Grid Views) menu. All defined filters and sorts are applied at once using this checkbox - you cannot pick and choose which to apply. If this menu option is not available, no saved filters or sorts have been defined.

Interactions Among Filters and Sorts

Users can still perform their own sorting and filtering as usual when looking at a grid that already has a saved sort or filter applied.

  • Sorting: Sorting a grid view while you are looking at it overrides any saved sort order. In other words, the saved sort can control how the data is first presented to the user, but the user can re-sort any way they wish.
  • Filtering: Filtering a grid view which has one or more saved filters results in combining the sets of filters with an AND. That is, new local filters happen on the already-filtered data. This can result in unexpected results for the user in cases where the saved filter(s) exclude data that they are expecting to see. Note that these saved filters are not listed in the filters bar above the data grid, but they can all be disabled by unchecking the (Grid Views) > Apply View Filter checkbox.

Related Topics




Column Summary Statistics


Premium Feature — An enhanced set of summary statistics is available in the Professional, Professional Plus, and Enterprise Editions. Learn more or contact LabKey.

[ Video Overview: Adding Summary Statistics to Data Grids ]

Add Summary Statistics to a Column

  • Click a column header, then select Summary Statistics.
  • The popup will list all available statistics for the given column, including their values for the selected column.
  • Check the box for all statistics you would like to display.
  • Click Apply. The statistics will be shown at the bottom of the column.

The list of statistics available vary based on the edition of LabKey Server you are running, and on the column datatype. Not all functions are available for all column types, only meaningful aggregates are offered. For instance, boolean columns show only the count fields; date columns do not include sums or means. Calculations ignore blank values, but note that values of 0 or "unknown" are not blank values.

All calculations use the current grid view and any filters you have applied. Remember that the grid view may be shown across several pages. Column summary statistics are for the dataset as a whole, not just the current page being viewed. The number of digits displayed is governed by the number format set for the container, which defaults to rounding to the thousandths place.

Summary statistics available in the Community edition include:

  • Count (non-blank): The number of values in the column that are not blank, i.e. the total number of rows for which there is data available.
  • Sum: The sum of the values in the column.
  • Mean: The mean, or average, value of the column.
  • Minimum: The lowest value.
  • Maximum: The highest value.
Additional summary statistics available in Premium editions of LabKey Server include:
  • Count (blank): The number of blank values.
  • Count (distinct): The number of distinct values.
  • Median: Orders the values in the column, then finds the midpoint. When there are an even number of values, the two values at the midpoint are averaged to obtain a single value.
  • Median Absolute Deviation (MAD): The median of the set of absolute deviations of each value from the median.
  • Standard Deviation (of mean): For each value, take the difference between the value and the mean, then square it. Average the squared deviations to find the variance. The standard deviation is the square root of the variance.
  • Standard Error (of mean): The standard deviation divided by the square of the number of values.
  • Quartiles:
    • Lower (Q1) is the midpoint between the minimum value and the median value.
    • Upper (Q3) is the midpoint between the median value and the maximum value. Both Q1 and Q3 are shown.
    • Interquartile Range: The number of values between Q3 and Q1 in the ordered list. Q3-Q1.

Display Multiple Summary Statistics

Multiple summary statistics can be shown at one time for a column and each column may have it's own set. Here is a compound set of statistics on another dataset:

Related Topics




Select Rows


When you work with a grid of data, such as a list or dataset, you often need to select one or more rows. For example, you may wish to visualize a subset of data or select particular rows from an assay to copy into a study. Large data grids are often viewed as multiple pages, adding selection options.

Topics on this page:

Select Rows on the Current Page of Data

  • To select any single row, click the checkbox at the left side of the row.
  • To unselect the row, uncheck the same checkbox.
  • The box at the top of the checkbox column allows you to select or unselect all rows on the current page at once.

Select Rows on Multiple Pages

Using the and buttons in the top right of the grid, you can page forward and back in your data and select as many rows as you like, singly or by page, using the same checkbox selection methods as on a single page. The selection message will update showing the total tally of selected rows.

To change the number of rows per page, select the row count message ("1 - 100 of 677" in the above screencap) to open a menu. Select Paging and make another selection for rows per page. See Page Through Data for more.

Selection Buttons

Selecting the box at the top of the checkbox column also adds a bar above your grid which indicates the number of rows selected on the current page and additional selection buttons.

  • Select All ___ Rows: Select all rows in the dataset, regardless of pagination.
  • Select None: Unselect all currently selected rows.
  • Show All: Show all rows as one "page" to simplify sorting and selection.
  • Show Selected: Show only the rows that are selected in a single page grid.
  • Show Unselected: Show only the rows that are not selected in a single page grid.
Using the Show Selected option can be helpful in keeping track of selections in large datasets, but is also needed for some actions which may only apply to rows on the current page which are selected. For example, View Specimens, Delete, and Copy to Study work this way.

Related Topics




Customize Grid Views


This topic explains how to create custom grid views and tailor them to your needs. The default set of columns displayed are not always what you need to see. Custom grid views allow you to display columns you want in the order you wish, with filters, sorts and aggregate functions applied. Custom grid views can be saved for easy reference, or used as the default display for a dataset. Editors, administrators, and users granted "Shared View Editor" access can create and share customized views with other users.

Customize a Grid View

To customize a grid for a given dataset, open the dataset, then select (Grid Views) > Customize Grid.

  • Tabs: The tabs contain tools for:
  • Available Fields: Shows the fields available for display. Place a checkmark next to a field to display it. Greyed out items are not fields, but table names. Click and buttons to expand/collapse fields in those tables.
  • Selected Fields: Shows the list of fields currently shown in the grid.
  • Delete: Deletes the current grid view. You cannot delete the default grid view.
  • Revert: Returns the grid to its original default state.
  • View Grid: Click to preview your changes. When viewing a modified grid with unsaved changes, the top bar will include revert, edit, and save buttons.
  • Save: Click to save your changes as the default view or as a new named grid. Saved grid views appear on the (Grid Views) pulldown menu.

Add Columns

  • To add a column to the grid, place a checkmark next to the field in the Available Fields pane. The field will be added to the Selected Fields pane.
  • Hover over any selected field name to see a popup with more information about the key and datatype of that field, as well as a description if one has been added.
  • To change the column display name, click the icon.

Notice the checkbox for Show hidden fields. Hidden fields typically contain metadata about the grid displayed. Learn more about common hidden fields for lists and datasets in this topic.

Add Columns from Other Datasets

In the Available Fields panel, a button indicates fields that are linked to other datasets through joins. When two or more datasets share a key field, that field can be used to construct joins between datasets. In this way, you can see columns from two or more datasets in one view. This combined view is the equivalent of a SQL SELECT query with one or more inner joins. For more information on joined views, see Join Columns from Multiple Tables.

  • To add fields from other datasets, expand the plus sign next to the "DataSets" node.
  • Expand the node of the dataset of interest.
  • Check boxes next to the desired fields.

Reorder Fields

To reorder the columns, drag and drop the fields in the Selected Fields pane. Columns will appear left to right as they are listed top to bottom. Note that the display order is changed, but no changes happen to the underlying data table.

Remove Columns

  • To remove a column, hover over the field in the Selected Fields pane, and click the icon.

  • You can also remove a column directly from the grid itself by clicking the column header and selecting Remove Column.

Sort and Filter

You can further refine your grid view by saving filters and sorts with the column configuration developed above.

  • View the data grid.
  • Apply filters and sorts by clicking the column headers and using the direct menu options.
  • When satisfied, select (Grid Views) > Customize Grid.
  • Find your filters and sorts on the Filter and Sort tabs. A number in parentheses will indicate the number present. Here we show 1 filter and 2 sorts.
  • Each sort or filter is represented as the column with a selector for the type of filter or direction in the case of sorts. On the tab in the view customizer, you can reorder by dragging and dropping, or change the direction or filter criteria using the pulldown menus.
  • Use View Grid to preview the grid with your changes.

Important note about filtering: If a grid view is available to all users, whether it's the default view or a custom view, it's possible to filter it in a way that's unexpected to the user. For example, if you filter the Issues grid on all issues whose priority is 0, 1, or 2 (e.g., Pri less than or equal to 2), and the user filters on issues whose priority is 3, no rows will be returned. But this does not necessarily mean that there are no Pri=3 rows in the table, because they are already being filtered out by the pre-defined filter.

Save Grid Views

  • When you are satisfied with your grid, click Save.
  • By default, you will save your changes as the Default grid view for this page.
  • To save an alternate option for users, select Named, and enter a title for the new grid view.
  • By default a customized grid is private to you. If you have "Editor" permissions (or higher) in the current folder, you can make a grid available to all users by checking the box Make this grid view available to all users.
  • In this example, we named the grid "My Custom Grid View" and you can see it was added to the (Grid Views) pulldown menu.

When the list of saved grid views is long, a Filter box is added. Type to narrow the list making it easier to find the grid view you want.

Reset the Default Grid View

  • To set the default view to an existing view, select (Grid Views) > Set Default.
  • Select the grid you prefer from the list available.

Revert to the Original Default Grid View

  • To revert any customizations to the default grid view, open it using (Grid Views) > default.
  • Select (Grid Views) > Customize Grid.
  • Click the Revert button.

Views Web Part

To create a web part listing all the customized views in your folder, an administrator can create an additional web part:

  • Enter > Page Admin Mode
  • In the lower left, select Views from the Select Web Part menu.
  • Click Add.
  • The web part will show saved grid views, reports, and charts sorted by categories you assign. Here we see the new grid view we just created.

Inheritance: Making Custom Grids Available in Child Folders

In some cases custom grids can be made available in child folders.

When saving a custom grid select the checkbox Make this grid view available in child folders.

The following table types support grid view inheritance in this way:

Table TypeSupports Inheritance into Child Folders?
QueryYes (Also see Edit Query Properties)
Linked QueriesYes (Also see Edit Query Properties)
ListsNo
DatasetsNo
Query SnapshotsNo
Assay DataYes
Sample SetsNo

Related Topics




Export Data Grid


LabKey provides a variety of methods for exporting the rows of a data grid. You can export into formats that can be consumed by external applications (e.g., Excel) or into scripts that can regenerate the data grid. You can also choose whether to export the entire set of data or only selected rows. Your choice of export format determines whether you get a static snapshot of the data, or a dynamic reflection that updates as the data changes. The Excel and TSV formats supply static snapshots, while scripts allow you to display dynamically updated data.

(Export) Menu

  • Click the (Export) button above any grid view to open the export panel.
  • Use tabs to choose between Excel, Text and Script exports, each of which carries a number of appropriate options for that type.

After selecting your options, decribed below, and clicking the Export button, you will briefly see visual feedback that the export is in progress:

Export Column Headers

Both Excel and Text exports allow you to choose whether Column Headers are exported with the data, and if so, what format is used. Options:

  • None: Export the data table with no column headers.
  • Caption: (Default) Include a column header row using the currently displayed column captions as headers.
  • Field Key: Use the column name with FieldKey encoding. While less display friendly, these keys are unambiguous and canonical and will ensure clean export and import of data into the same dataset.

Export Selected Rows

If you select one or more rows using the checkboxes on the left, you will activate the Export Selected Rows checkbox in either Excel or Text export mode. When selected, your exported Excel file will only include the selected rows. Uncheck the box to export all rows. For additional information about selecting rows, see Select Rows.

Filter Data Before Export

Another way to export a subset of data records is to filter the grid view before you export it.

  • Filter Data. Clicking a column header in a grid will open a dialog box that lets you filter and exclude certain types of data.
  • Create or select a Custom Grid View. Custom Grids let you store a selected subset as a named grid view.
  • View Data From One Visit. You can use the Study Navigator to view the grid of data records for a particular visit for a particular dataset. From the Study Navigator, click on the number at the intersection of the desired visit column and dataset row.

Export to Excel

When you export your data grid to Excel, you can use features within that software to access, sort and present the data as required. If your data grid includes inline images they will be exported in the cell in which they appear in the grid.

Export to Text

Select Text tab to export the data grid in a text format. Select tab, comma, colon, or semicolon from the Separator pulldown and single or double from the Quote pulldown. The extension of your exported text file will correspond to the separator you have selected, i.e. "tsv" for tab separators.

LabKey Server uses the UTF-8 character encoding when exporting text files.

Export to Script

You can export the current grid to script code that can be used to access the data from any of the supported client libraries. See Export Data Grid as a Script.

The option to generate a Stable URL for the grid is also included on the (Export) > Script tab.

Related Topics




Join Columns from Multiple Tables


By combining data from multiple tables in one grid view, you can create integrated grids and visualizations while avoiding duplication of data. The basis of joined grids is a "lookup column" -- a column that connects the two tables such that one table looks up values in another table, making it possible to join data from both tables in combined grids.

Create a Lookup Field

An administrator first needs to connect the two tables by creating a lookup column between the two tables. Lookup columns can be created in two ways: either through the graphical user interface, as described in Lookup Columns; or through a SQL query, as described in Lookups: SQL Syntax.

Create a Joined Grid View

Once tables are connected by a lookup column, you can create a joined grid view on either table, subject to key complexity rules described below.

For example, suppose you wish to create a grid that brings together columns from both the Demographics and the Languages tables. A lookup column has already been defined in the Demographics table that looks up data in the Language table. To create a grid including data from both tables:

  • Go to the Demographics data grid.
  • Select (Grid Views) > Customize Grid.
  • In Available Fields, expand the Languages node -- this reveals the available columns in the Languages table.
  • To add a column from the Languages table, place a checkmark next to it. Below two columns have been added: Translator Name and Translator Phone. You may need to drag and drop them to the desired order. Here, directly after the Languages column that was already displayed.
  • The columns will be added to Selected Fields.
  • Save the grid, which now includes columns of data from the Languages table.

Troubleshooting

FAQ: In a study, why can't I customize my grid to show a particular field from another dataset?

Background: To customize your grid view of a dataset by adding columns from another dataset, it must be possible to join the two datasets. The columns used for a dataset's key influence how this dataset can be joined to other tables. Certain datasets have more than one key column (in other words, a "compound key"). In a study, you will typically encounter three types of datasets:

  • Demographic datasets use only one column as a key. The key is the participantID. This means that only one line of data (for any date) can be associated with a participant in such a dataset.
  • Clinical or standard datasets use participant/visit pairs as a compound key. This means that each row is uniquely identified by participant/visit pairs, not by participant identifiers alone.
  • Assay datasets copied into a study also use compound keys. Assay datasets use participant/visit/rowID columns as compound keys, so only the trio of these columns uniquely identifies a row. In other words, a participant can have multiple assay rows associated with any individual visit - these are uniquely differentiated only by their rowIDs.
Consequences: When customizing the grid for a table, you cannot join in columns from a table with more key columns. For example, if you are looking at a clinical dataset in a study, you cannot join to an assay dataset because the assay dataset has more columns in its key, and thus can have multiple rows per participant/visit. So there isn't a unique mapping from a participant/visit pair in the 'originating' clinical dataset to a specific row of data in the assay dataset.

Guidance: To create a grid view combining columns from disparate datasets, start with the dataset with more columns in the key. Then select a column from the table with fewer columns in the key. There can be a unique mapping from the compound key to the simpler one - some columns will have repeated values for several rows, but rows will be unique.

Related Topics




Lookup Columns


To create grids that contain data from two different tables, create a lookup column in one of the tables: a column that pulls in data from the other table. Once you have a lookup column established, you can display values from any column in the target table (the "looked up" table). For example, suppose you want to display values from the Languages table, such as the translator information, alongside data from the Demographics table. You would add a lookup column to the Demographics dataset that used values from the Languages list.

Set Up a Lookup Field

To join these tables, an administrator must add a lookup column to the Demographics dataset definition:

  • Go to the dataset or list where you want to show the data from the other source. Here, Demographics.
  • Click Manage, then Edit Definition. (In a list, click Design > Edit Design to reach the same editor interface.)
  • Click Add Field.
    • Enter a Name, used internally, and an optional Label to show in the column header.
    • Click the dropdown under Type and select Lookup.
    • In the popup dialog select the target table of the lookup. For example, the lists schema and the Languages table, as shown below.
  • Click Apply.
  • You may want to relocate your new field using the and buttons.
  • Click Save.
  • The lookup column is now available for grid views and SQL queries on the Demographics table.
  • You can also create grids merging information from any of the columns of both tables. For example, a grid showing which translators are needed for each cohort would make it possible to schedule their time efficiently. Note that the original "Language" column need not be shown.

For more about creating joined grids see Join Columns from Multiple Tables.

Default Display Fields

For lookup fields where the target table has an integer primary key, the server will use the first text field it encounters as the default display column. For example, suppose the Language field is an integer lookup to the Languages table, as below.

In this case, the server uses Language Name as the default display field because it is the first text field it finds in the looked up table. You can see this in the details of the lookup column shown in the example above, as "English", etc, are displayed though the lookup is to an integer key.

Displaying Alternate Fields

To display other fields from the looked up table, go to (Grid Views) > Customize View and select the fields you want to display.

You can also use query metadata to achieve the same result: see Query Metadata: Examples

Validating Lookups: Enforcing Lookup Values on Import

When you are importing data into a table that includes a lookup column, you can have the system enforce the lookup values, i.e. any imported values must appear in the target table. An error will be displayed whenever you attempt to import a value that is not in the lookup's target table.

To set up enforcement:

  • Go to the definition editor of the dataset or list with the lookup column.
  • Select the lookup column in the table.
  • Click the Validators tab.
  • Click the button Add Lookup Validator.
  • Click Save and Close.

Note that pre-existing data is not retroactively validated by turning on the lookup validator. To ensure pre-existing data conforms to the values in the lookup target table, either review entries by hand or re-import to confirm values.

Related Topics




Participant Details View


The default dataset grid displays data for all participants. To view data for an individual participant, click on the participantID in the first column of the grid.

Participant Details View

The participant details view lists all of the datasets that contain data for the current participant, as shown in the image below.

  • Previous/Next Participant: Page through participants. Note that this option is only provided when you navigate from a dataset listing other participants. Viewing a single participant from the Participants tab does not include these options.
  • button: Expand dataset details.
  • button: Collapse dataset details

Add Charts

Expand dataset details by clicking the or name of the dataset of interest. Click Add Chart to add a visualization to the participant view details. The dropdown will show the charts defined for this dataset.

After you select a chart from the dropdown, click the Submit button that will appear.

Once you create a chart for one participant in a participant view, the same chart is displayed for every participant, with that participant's data.

You can add multiple charts per dataset, or different charts for each dataset. To define new charts to use in participant views, use the plot editor. You can limit the size of plots used for participant views using the Chart Layout width and height options.

Customize Participant Details View

You can alter the HTML used to create the default participant details page and save alternative ways to display the data using the Customize View link. You can leverage the LabKey APIs to tailor your custom page. You can also add the participant.html file via a module: for details see Custom HTML/JavaScript Participant Details View.

Click Save to refresh and see the preview below your script. Click Save and Finish when finished.

Related Topics




Query Scope: Filter by Folder


Overview

Certain LabKey queries (such as assay designs, issue trackers, and survey designs) can be defined at the project level. Data associated with such queries may be located within individual subfolders. You can adjust the scope of queries on such data to cover all data on the site, all data for a project, or only data located in particular folders. Scope is controlled using the "Filter by Folder" option on the (Grid Views) menu in the web part.

This allows you to organize your data in folders that are convenient to you at the time of data collection (e.g., folders for individual labs or lab technicians). Then you can perform analyses independently of the folder-based organization of your data. You can analyze data across all folders, or just a branch of your folder tree.

You can set the scope through either the (Grid Views) menu or through the client API. In all cases, LabKey security settings remain in force, so users only see data in folders they are authorized to see.

Folder Filter Interface

To filter by folder through the user interface:

  • Select (Grid Views) > Filter by Folder.
  • Choose one of the options:
    • Current folder
    • Current folder and subfolders
    • All folders (on the site)

Folder Filters in the JavaScript API

The LabKey API provides developers with even finer grain control over the scope.

The containerFilter config property available on many methods on LABKEY.Query (such as LABKEY.Query.executeSql and LABKEY.Query.selectRows) provides fine-grained control over which folders accessed through the query.

For example, the LABKEY.Query.executeSql API allow you to use the containerFilter parameter to run custom queries across data from multiple folders at once. This query might (for example) show the count of NAb runs available in each lab’s subfolder if folders are organized by lab.




Field Properties Reference


Each field in a list or dataset is associated with a set of properties that can be edited using the field property editor shown below. The set of fields composing a list or dataset can also be called a domain, and this tool is sometimes called the domain editor.

Field Properties Editor

  • Use the , , and buttons to the left of each field to reorder or delete fields.
  • Click on the Name, Label or Type for any field to edit them.
  • Selecting a row brings up the field properties editor: the block of tabs to the right.
  • The highlight bar indicates which field is currently selected for editing.
  • A (wrench) icon on the left will indicate a row with unsaved changes.
The interface for editing field properties looks like this:

Basic Properties

Name: This is the name used to refer to the field programmatically. It must start with a character and include only characters and numbers; it may not include spaces.

Label (Optional): The name that users will see displayed for the field. It can be longer than the name and may contain spaces.

Type: Fields come in different types, each intended to hold a different kind of data. Once defined, there are only limited ways you can change it, based on the ability to convert existing data to the new type. To change a field type, you may need to delete and recreate it, reimporting any data. Field types are:

  • Text (String)
  • Multi-Line Text
  • Boolean (True/False)
  • Integer
  • Number (Double) - A decimal number.
  • Date/Time
  • Flag (String)
  • File - The File type is only available for certain types of table types, including assay designs and sample sets, see below for a complete list. When a file has been uploaded into this field, it displays a link to the file; for image files, an inline thumbnail is shown. The uploaded file is stored in the file repository, in the assaydata folder. For General Purpose assays, the File field presents special behavior for image files; for details see Linking Assays with Images and Other Files.
  • Attachment - This type is only available for lists, see below for a complete list. This type allows you to attach documents to individual records in a list. For instance, an image file could be associated with a given row of data in an attachment field, and would show an inline thumbnail. The attached file is not uploaded into the file repository, but is stored as a BLOB field the database. By default, the maximum file size is 50MB, but this can be changed in the Admin Console using the setting Maximum file size, in bytes, to allow in database BLOBs. See Site Settings.
  • User - This type points to registered users of the LabKey Server system, found in the table core.Users.
  • Subject/Participant (String) - This type is only available in a study.
  • Lookup - See below.

Field Types Available by Data Structure

The following table show which fields are available in which sort of table/data structure. Notice that Datasets do not support File or Attachment fields. For a workaround technique, see Linking Data Records with External Files.

Field TypeDatasetListSample SetAssay Design
Text (String)YesYesYesYes
Multi-Line TextYesYesYesYes
BooleanYesYesYesYes
IntegerYesYesYesYes
Number (Double)YesYesYesYes
DateTimeYesYesYesYes
FileNo (workaround)NoYesYes
AttachmentNo (workaround)YesNoNo
UserYesYesYesYes
Subject/Participant(String)YesYesYesYes
LookupYesYesYesYes

Inline Thumbnails for Files and Attachments

Fields of type File and Attachment are available in certain structures including lists, sample sets, and assay run results (see the table above for details). When the file or attachment is an image, such as a .png or .jpg file, the cell in the data grid will display a thumbnail of the image. Hovering reveals a larger version.

When you export a grid containing these inline images to Excel, the thumbnails remain associated with the cell itself.

Bulk Import into the File Field Type

You can bulk import data into the File field type, provided that the files/images are already uploaded to the File Repository. For example suppose you already have a set of images in the file Repository, as shown below.

You can load these images into a File field, if you refer to the images by their full server path in the File Repository. For example, the following shows how an assay upload might refer to these images by their full server path:

ImageNameImageFile
10001.pnghttp://localhost:8080/labkey/_webdav/Tutorials/List%20Tutorial/%40files/NIMH/Images/10001.png
10002.pnghttp://localhost:8080/labkey/_webdav/Tutorials/List%20Tutorial/%40files/NIMH/Images/10002.png
10003.pnghttp://localhost:8080/labkey/_webdav/Tutorials/List%20Tutorial/%40files/NIMH/Images/10003.png
10004.pnghttp://localhost:8080/labkey/_webdav/Tutorials/List%20Tutorial/%40files/NIMH/Images/10004.png

On import, the Assay grid will display the image thumbnail as shown below:

Lookups

You can populate a field with data via lookup into another table. Click on the Type property for a field, select the Lookup option, then select a source Folder, Schema and Table from the drop-down menus in the popup. These selections identify the source location for the data values that will populate this field. For examples, see the Tutorial: Lists and the Online List Example Project.

A lookup appears as a foreign key (<fk>) in the XML schema generated upon export of this study. An example of the XML generated:

<fk>
<fkDbSchema>lists</fkDbSchema>
<fkTable>Reagents</fkTable>
<fkColumnName>Key</fkColumnName>
</fk>

Note that lookups into lists with auto-increment keys may not export/import properly because the rowIds in this type of list may be different in every database.

Display Properties

Display properties for a field are shown on the Display tab and control how and when the field is displayed.

Description: Verbose description of the field. XML schema name: description.

URL: A template for generating hyperlinks for this field. The ${ } syntax may be used to substitute a field's value into the URL. See URL Field Property.

Shown In Display Modes: Checkboxes allow you to choose whether or not the column is displayed in the display modes: Grid, Insert, Update, Details.

Format Properties

Format: You can create custom formats for displaying values in this column. For example, you can specify Date, DateTime or Number Formats to suit your locale. You can also configure date, datetime, and number formats at the folder, project, or site level for broader consistency. See Date & Number Display Formats.

Conditional Formats: Conditional formats let you change the way the data is displayed based on the data value. For details see Conditional Formats

Validators (Field-level Validation)

Field validators ensure that all values entered for a field obey a regular expression and/or fall within a specified range. They can automate checking for reasonableness and catch a broad range of field-level data-entry errors during the upload process.

Validators are applied in the following circumstances:

 On Existing DataOn ImportOn Update
Required ValidatorYesYesYes
Regex ValidatorNoYesNo
Range ValidatorNoYesNo

Required: When required, a field cannot be empty. Defaults to "False."

Add Regex Validator: Define a regular expression that defines what strings are valid.

  • Name: Required. A name for this expression.
  • Description: Optional. A text description of the expression.
  • Regular Expression: Required. A regular expression that this field's value will be evaluated against. All regular expressions must be compatible with Java regular expressions, as implemented in the Pattern class.
  • Error message. Optional. The message that will be displayed to the user in the event that validation fails for this field.
  • Fail when pattern matches. Optional. By default, validation will fail if the field value does not match the specified regular expression. Check this box if you want validation to fail when the pattern matches the field value, which may be an easier way to express the error cases you want to catch.
Add Range Validator:
  • Name: Required. A name for this range requirement.
  • Description: Optional. A text description of the range requirement.
  • First condition: Required. A condition to this validation rule that will be tested against the value for this field.
  • Second condition: Optional. A condition to this validation rule that will be tested against the value for this field. Both the first and second conditions will be tested for this field.
  • Error message: Required. The message that will be displayed to the user in the event that validation fails for this field.
Note: Validators are not included in XML schemas exported from a study.

For information on script-based validation, which can be used to validate rows or entire tables in a programmatic way, see Transformation Scripts.

Reporting Properties

The reporting tab allows you to set attributes used in creating reports.

Measure: A field identified as a measure contains data useful for charting and other analysis. Measures are typically numeric results/observations, like weight, or CD4 count. Only those columns identified as measures will be listed as options for the y-axis of a time chart, for example.

Data Dimension: Data dimensions define logical groupings of measures. For example, 'Gender' could be a dimension for a dataset containing a 'Height' measure, since it may be desirable to study height by gender.

Recommended Variable: Define which fields in this table/query/etc. should be prioritized as 'recommended' variables when creating new charts/reports/etc for datasets containing large numbers of columns.

Default Scale Type: For numeric field types, defines whether linear or log scales will be used by default for this field.

Advanced Properties

Missing Value Indicators: A field marked with 'Missing Value Indicators', can hold special values when values are missing. Unchecked by default. When checked, the following values are permitted:

  • Q: quality control has been applied and removed the value
  • N: data will not be provided (even if it is officially required).
Default Type: Dataset schemas can automatically supply default values when a user is entering values or when imported data tables have missing values. The "Default Type" property sets how the default value for the field is determined. "Last entered" is the automatic choice for this property if you do not alter it. Note: This property is not included in XML schemas exported from a study.

Options:

  • Editable default: An editable default value will be entered for the user. The default value will be the same for every user for every upload.
  • Last entered: An editable default value will be entered for the user's first use of the form. During subsequent uploads, the user will see their last entered value.
Default Value: For either of the "Default Types," you may wish to set a default value. The use of this value varies depending on the "Default Type" you have chosen. Note: This property is not included in XML schemas exported from a study.

Options:

  • If you have chosen "Editable default," you can set the default value through the Set Values option. Each time the user sees the form, they will see this default value.
  • If you have chosen "Last entered" for the default type, the field will show the setting entered previously, but you can still set the initial value of the field through the "Default Value" option.
Import Aliases: A set of alternate field names when importing from external files, in addition to the field's name and label. Multiple aliases may be separated by spaces or commas. To specify an alias that contains spaces, use double-quotes (") around the alias.

In most places where data is imported from files (TSV, Excel, etc) LabKey Server tries to match file column headers to property names. In some cases it also looks for matching property labels. The "Import Aliases" option offers further flexibility by allowing an arbitrary number of alternate column aliases. At import time, LabKey uses these aliases to track down values.

PHI Level: Tag a field as containing some level of PHI (Protected Health Information):

These settings allow an administrator to exclude columns based on the PHI role assigned to the user. Columns marked with a PHI level higher than the current user's maximum PHI role can be excluded. For details see Compliance: Setting PHI Levels on Fields. These are respected in the following scenarios:

  • Viewing a data grid
  • Editing a data record
  • Exporting a data grid
  • Exporting a folder or study
  • Publishing a study
  • Querying data
  • Building reports on data
  • Accessing data via the API
Possible column settings are:
  • Not PHI: (Default) Not protected. Users with access to read the dataset can see this field.
  • Limited PHI: Users with access to limited PHI can see this field.
  • Full PHI: Only users with access to see full PHI can see this field.
  • Restricted PHI: Only users with access to restricted information can see this field.

To tag a column at a specific PHI level:

  • Navigate to the dataset/list that contains the column you want to protect.
  • Edit the dataset/list definition.
  • In the designer, select the column you wish to protect.
  • Click the Advanced tab, select the appropriate level from the PHI Level dropdown.

Exclude From Shifting: If you check this box for a date column, the value will not be shifted on export/publication when the "Shift Participant Dates" option is selected.

Max Text Length: Specify the maximum number of characters allowed in text fields. The default value is 4000 characters. For values over 4000 characters, select Max. Selecting "Max" will utilize the max number of characters allowed by the underlying database implementation, for example, 1GB for PostgreSQL, 2GB for SQLServer. Sample Sets and assay run fields do not support the Max setting. (Key fields and fields in the table exp.objectproperty do not support the Max setting.)

Related Topics




URL Field Property


The URL property of a field changes the display of the field value within a data grid into a hyperlink. The field's value becomes the display text of the hyperlink. The URL property becomes the target address of the hyperlink (also known as the href).

In an application requiring a different link address for each row in a dataset, the ${ } substitution syntax may be used to insert a field's value into the URL. For example, in a set of experimental data where one column is a Gene Symbol, a researcher might wish to quickly compare her results with the information in The Gene Ontology. Generating a URL for the GO website with the Gene Symbol as a parameter will give the researcher an efficient way to "click through" from any row to the correct gene.

An example URL (in this case for the BRCA gene) might look like:

http://amigo.geneontology.org/amigo/search/ontology?q=brca

Since the search_query parameter value is the only part of the URL that changes in each row of the table, the researcher can set the URL property on the GeneSymbol field to use a substitution marker like this:

http://amigo.geneontology.org/amigo/search/ontology?q=${GeneSymbol}

Once defined, the researcher would simply click on "BRCA" in the correct column to be linked to the URL with the search_query parameter applied.

Link Format Types

Three link format types for URL property are supported:

Full URL: Starts with http:// or https://

http://server/path/page.html?id=${Param}

The full URL type is most often used for a link that navigates to another server, as in the Gene Ontology example described above.

Same-server URL, for example:

https://www.mylabkey.org/home/folder/wiki-page.view?name=${Name}

This same-server URL type is like the full URL type, but omits the protocol type and server name. It points to a resource on the current LabKey Server, useful when

  • the link navigates to a different LabKey folder on the current server
  • when the URL is a WebDAV link to a file that has been uploaded to current server
Same-folder LabKey URL: Includes controller and action name, but omits the context path and folder path:

/wiki-page.view?name=${Name}

(See LabKey URLs for an explanation of the parts of a LabKey URL.)

As the name implies, this format of the URL property is useful for links to other pages in the current folder. A key advantage of this format is that the list or query containing the URL property can be moved or copied to another folder and it will still continue to work correctly.

Substitution Markers

A URL property can contain markers in the format ${field-name}, where "field-name" is the name of any field that is part of the current query (i.e. the tabular data object that contains the field that has the URL property). When the query is displayed in a grid, the value of the field-name for the current record is substituted into the URL property string in place of the ${field-name} marker. Note that multiple such markers can be used in a single URL property string, and the field referenced by the markers can be any field within the query.

Note that substitutions are allowed in any part of the URL, either in the main path, or in the query string. For example, here are two different formats for creating links to an article in wikipedia, here using a "CompanyName" field value:

  • as part of the path:
  • as a parameter value:

Built-in Substitution Markers

The following substitutions markers are built-in and available for any query/dataset. They help you determine the context of the current query.

MarkerDescriptionExample Value
${schemaName}The schema where the current query lives.study
${schemaPath}The schema path of the current query.assay.General.MyAssayDesign
${queryName}The name of the current queryPhysical Exam
${dataRegionName}The data region for the current query.Dataset
${containerPath}The LabKey Server folder path, starting with the project/home/myfolderpath
${contextPath}The Tomcat context path/labkey
${selectionKey}Unique string used by selection APIs as a key when storing or retrieving the selected items for a grid$study$Physical Exam$$Dataset

Link Display Text

The display text of the link created from a URL property is just the value of the current record in the field which contains the URL property. So in the Gene Ontology example, since the URL property is defined on the Gene_Symbol field, the gene symbol serves as both the text of the link and the value of the search_query parameter in the link address. In many cases you may want to have a constant display text for the link on every row. This text could indicate where the link goes, which would be especially useful if you want multiple such links on each row.

In the example above, suppose the researcher wants to be able to look up the gene symbol in both Gene Ontology and EntrezGene. Rather than defining the URL Property on the Gene_Symbol field itself, it would be easier to understand if two new fields were added to the query, with the value in the fields being the same for every record, namely "[GO]" and "[Entrez]". Then set the URL property on these two new fields to

for the GOlink field:

http://amigo.geneontology.org/cgi-bin/amigo/search.cgi?search_query=${Gene_Symbol}&action=new-search

for the Entrezlink field:

http://www.ncbi.nlm.nih.gov/gene/?term=${Gene_Symbol}

The resulting query grid will look like:

Note that if the two new columns are added to the underlying list, dataset, or schema table directly, the link text values would need to be entered for every existing record. Changing the link text would also be tedious. A better approach is to wrap the list in a query that adds the two fields as constant expressions. For this example, the query might look like:

SELECT TestResults.SampleID,
TestResults.TestRun,
TestResults.Gene_Symbol,
TestResults.ResultValueN,

'[GO]' AS GOlink,
'[Entrez]' AS Entrezlink

FROM TestResults

Then in the Edit Metadata page of the Schema Browser, set the URL properties on these query expression fields:

URL Encoding Options

You can specify the type of URL encoding for a substitution marker, in case the default behavior doesn't work for the URLs needed. This flexibility makes it possible to have one column display the text and a second column can contain the entire href value, or only a part of the href. The fields referenced by the ${ } substitution markers might contain any sort of text, including special characters such as question marks, equal signs, and ampersands. If these values are copied straight into the link address, the resulting address would be interpreted incorrectly. To avoid this problem, LabKey Server encodes text values before copying them into the URL. In encoding, characters such as ? are replaced by their character code %3F. By default, LabKey encodes all special character values except '/' from substitution markers. If you know that a field referenced by a substitution marker needs no encoding (because it has already been encoded, perhaps) or needs different encoding rules, inside the ${ } syntax, you can specify encoding options as described in the topic String Expression Format Functions.

Links Without the URL Property

If the data field value contains an entire url starting with an address type designator (http:, https:, etc), then the field value is displayed as a link with the entire value as both the address and the display text. This special case could be useful for queries where the query author could create a URL as an expression column. There is no control over the display text when creating URLs this way, however.

Linking To Other Tables

To link two tables, so that records in one table link to filtered views of the other, start with a filtered grid view of the target table, filtering on the target fields of interest. For example, the following URL filters on the fields "WellLocation" and "WellType":

/home/demo%20study/study-dataset.view?datasetId=5018&Dataset.WellLocation~eq=AA&Dataset.WellType~eq=XX

Parameterize by adding substitution markers within the filter. For example, assume that source and target tables have identical field names, "WellLocation" and "WellType":

/home/demo%20study/study-dataset.view?datasetId=5018&Dataset.WellLocation~eq=${WellLocation}&Dataset.WellType~eq=${WellType}

Finally, set the parameterized URL as the URL property of the appropriate column in the source table.

Related Topics

For an example of UI usage, see: Add a URL Property and view an interactive example by clicking here. Hover over a link in the Species column to see the URL, click to view a list filtered to display only demographic data for the specific species.

For examples of SQL metadata XML usage, see: JavaScript API Demo Summary Report and the JavaScript API Tutorial.




String Expression Format Functions


Reference

The following string formatters can be used when building URLs, or creating unique names for sample sets and DataClasses.

 NameSynonymInput TypeDescriptionExample
General
defaultValue(string) anyUse the string argument value as the replacement value if the token is not present or is the empty string.${field:defaultValue('missing')}
passThroughnoneanyDon't perform any formatting.${field:passThrough}
URL Encoding
encodeURIuristringURL encode all special characters except ',/?:@&=+$#' like JavaScript encodeURI()${field:encodeURI}
encodeURIComponenturicomponentstringURL uncode all special characters like JavaScript encodeURIComponent()${field:encodeURIComponent}
htmlEncodehtmlstringHTML encode${field:htmlEncode}
jsString stringEscape carrage return, linefeed, and <>"' characters and surround with a single quotes${field:jsString}
urlEncodepathstringURL encode each path part preserving path separator${field:urlEncode}
String
join(string) collectionCombine a collection of values together separated by the string argument${field:join('/'):encodeURI}
prefix(string) string, collectionPrepend a string argument if the value is non-null and non-empty${field:prefix('-')}
suffix(string) string, collectionAppend a string argument if the value is non-null and non-empty${field:suffix('-')}
trim stringRemove any leading or trailing whitespace${field:trim}
Date
date(string) dateFormat a date using a format string or one of the constants from Java's DateTimeFormatter. If no format value is provided, the default format is 'BASIC_ISO_DATE'${field:date}, ${field:date('yyyy-MM-dd')}
Number
format numberFormat a number using Java's DecimalFormat${field:number('0000')}
Array
first collectionTake the first value from a collection${field:first:defaulValue('X')}
rest collectionDrop the first item from a collection${field:rest:join('_')}
last collectionDrop all items from the collection except the last${field:last:suffix('!')}

Examples

FunctionApplied to...Result
${Column1:defaultValue('MissingValue')}nullMissingValue
${Array1:join('/')}[apple, orange, pear]apple/orange/pear
${Array1:first}[apple, orange, pear]apple
${Array1:first:defaultValue('X')}[(null), orange, pear] X



Conditional Formats


Conditional formats change how data is displayed depending on the value of the data. For example, if temperature value goes above a certain value, than you can specify that the value be displayed in bold, italic, red, etc. Conditional formats are declared on the Format tab of the field editor, found by editing the definition or design of your dataset, list, or assay.

Specify a Conditional Format

To specify a conditional format, open the field property editor, select a field, click the Format tab and click Add Conditional Format.

First identify the condition under which you want the conditional format applied. Specifying a condition is just like specifying a filter. If you specify two expressions here, both will be AND-ed together to determine whether a single conditional format is displayed.

Click OK.

Next, you can specify how the field should be formatted when that condition is met. The options are:

  • Bold
  • Italic
  • Strikethrough
  • Color: click to popup a dialog for setting foreground and background colors

Multiple Conditional Formats

Multiple conditional formats are supported in a single column. Click Add Conditional Format again to specify another conditional format. Unlike the definition of multiple expressions in the same condition, this additional condition can have a different type of conditional format applied.

If a data cell fulfills multiple conditions, then the first condition satisfied is applied, and conditions lower on the list are ignored.

For example, suppose you have specified two conditional formats on one field:

  • If the value is 40 degrees or greater, then display in bold text.
  • If the value is 38 degrees or greater, then display in italic text.
Although the value 40 fulfills both conditions, only the first condition to apply is considered, resulting in bold display. A sorted list of values would be displayed as shown below:

41
40
39
38
37

Specify Conditional Formats as Metadata XML

Conditional formats can be specified (1) as part of a table definition and/or (2) as part of a table's metadata XML. When conditional formats are specified in both places, the metadata XML takes precedence over the table definition.

You can edit conditional formats as metadata xml source. In the metadata editor, click Edit Source. The sample below shows XML that specifies that values greater than 37 in the Temp_C column should be displayed in bold.

<tables xmlns="http://labkey.org/data/xml">
<table tableName="Physical Exam" tableDbType="NOT_IN_DB">
<columns>
<column columnName="Temp_C">
<conditionalFormats>
<conditionalFormat>
<filters>
<filter operator="gt" value="37"/>
</filters>
<bold>true</bold>
</conditionalFormat>
</conditionalFormats>
</column>
</columns>
</table>
</tables>

Example: Conditional Formats for Human Body Temperature

In the following example, values out of the normal human body temperature range are highlighted with color if too high and shown in italics if too low. In this example, we use the Physical Exam dataset that is included with the importable demo study.

  • In a grid view of the Physical Exam dataset, click Manage.
  • Click Edit Definition.
  • Select a field (in this case Temp_C), click the Format tab, and click Add Conditional Format.
  • In the popup, choose Filter Type "Is Greater Than", enter 37.8, and click Ok.
  • Click the black/white square icon to select colors:
  • Select colors in the foreground text panel.
  • Click Ok.
  • Click Add Conditional Format again.
  • Choose Filter Type: "Is Less Than", enter 36.1, and click Ok.
  • Check the middle checkbox, under the italicized I.
  • Scroll back up and click Save.
  • Click View Data to return to the data grid.

Now temperature values above 37.8 degrees are highlighted with shading and those below 36.1 are displayed in italics. Note that other conditional formats may be applied to other columns. In this screencap, there is conditional formatting in the Systolic Blood Pressure column also: orange for high, and red for very high readings.

When you hover over a formatted value, a pop up dialog will appear explaining the rule behind the format.

Related Topics




Date & Number Display Formats


LabKey Server provides flexible formatting for dates, times, and numbers, so you can control how they are displayed to users. Using formatting you can:
  • Specify how dates are displayed, for example:
    • 04/05/2016
    • May 4 2016
    • Wednesday May 4
  • Specify how times are displayed, for example
    • 01:23pm
    • 1:23pm Pacific Standard Time
  • Specify how numbers are displayed, for example
    • 1.1
    • 1.10
  • Determine granularity of the date/time display, for example:
    • June 4 2016
    • June 4 2016 1pm
    • June 4 2016 1:20pm
  • Set up formats that apply to the entire site, an entire project, or a folder.
  • Override more generally prescribed formats in a particular context, for example, specify that a particular field or folder should follow a different format than the parent container.
Note that date formatting described in this topic is different from date parsing. Formatting determines how date and time data are displayed by the server. Parsing determines how the server interprets date strings.

You can customize how dates, times and numbers are displayed on a field-by-field basis, or specify default formats on a folder-level, project-level or site-wide basis. The server decides which format to use for a particular field by looking first at the properties for that field. If no format property is found at the field-level, it checks the container tree, starting with the folder then up the folder hierarchy to the site level. In detail, decision process goes as follows:

  • The server checks to see if there is a field-level format set on the field itself. If it finds a field-specific format, it uses that format. If no format is found, it looks to the folder-level format. (To set a field-specific format, see Set Formats on a Per-Field Basis.)
  • If a folder-level format is found, it uses that format. If no folder-level format is found, it looks in the parent folder, then that parent's parent folder, etc. until the project level is reached and it looks there. (To set a folder-level default format, see Set Folder Display Formats)
  • If a project-level format is found, it uses that format. If no project-level format is found, it looks to the site-level format. (To set a project-level default format, see Set Project Display Formats.)
  • To set the site-level format, see Set Formats Globally (Site-Level). Note that if no site-level format is specified, the server will default to these formats:
    • Date field: yyyy-MM-dd
    • Date-time field: yyyy-MM-dd HH:mm
When LabKey Server is first installed, it uses these initial formatting patterns:
  • Date fields: Year-Month-Date, which is the standard ISO date format, for example 2010-01-31. The Java string format is yyyy-MM-dd
  • Date-time fields: Year-Month-Date Hour:Minute, for example 2010-01-31 9:30. The Java string format is yyyy-MM-dd HH:mm
  • For date-time fields where displaying a time value would be superfluous, the system overrides the site-wide initial format (Year-Month-Date Hour:Minute) and instead uses a date-only format (Year-Month-Date). Examples include visit dates in study datasets, SpecimenDrawDate in specimens, and BirthDate in various patient schemas. To change the format behavior on these fields, override the query metadata -- see Query Metadata.
A standard Java format string specifies how dates, times, and numbers are displayed. For example, the format string

yyyy-MM-dd

specifies dates to be displayed as follows

2000-01-01

For details on format strings, see Date and Number Formats Reference.

Set Formats Globally (Site-Level)

An admin can set formats at the site level by managing look and feel settings.

  • Select (Admin) > Site > Admin Console.
  • Click the Admin Console Links tab.
  • Under Configuration, click Look and Feel Settings.
  • Scroll down to Customize date and number formats.
  • Enter format strings as desired for date, date-time, or number fields.
  • Click Save.

Set Project Display Formats

An admin can standardize display formats at the project level so they display consistently in the intended scope, which does not need to be consistent with site-wide settings.

  • Navigate to the target project.
  • Select (Admin) > Folder > Project Settings.
  • On the Properties tab, scroll down to Customize date and number formats.
  • Enter format strings as desired for date, date-time, or number fields.
  • Click Save.

Set Folder Display Formats

An admin can standardize display formats at the folder level so they display consistently in the intended scope, which does not need to be consistent with either project or site settings.

  • Navigate to the target folder.
  • Select (Admin) > Folder > Management.
  • Click the Formats tab.
  • Enter format strings as desired for date, date-time, or number fields.
  • Click Save.

Set Formats on a Per-Field Basis

To do this, you use the field property editor:

  • Open a grid view of the dataset of interest.
  • Click Manage.
  • Click Edit Definition.
  • Select the field of interest.
  • In the field property management panel, select the Format tab.
  • Enter the desired format string directly, or use the shortcuts described below.
  • Click Save.

Date Format Shortcuts

At the field-level, instead of providing a specific format string, you can use one of the following shortcut values to specify a standard format. A shortcut value tells the server to use the current folder's format setting (a format which may be inherited from the project or site setting).

Format Shortcut StringDescription
DateUse the folder-level format setting, specified at (Admin) > Folder > Management > Formats tab > Default display format for dates.
DateTimeUse the folder-level format setting, specified at (Admin) > Folder > Management > Formats tab > Default display format for date-times.
TimeCurrently hard-coded to "HH:mm:ss"




Date and Number Formats Reference


The following reference accompanies the topic Date & Number Display Formats.

Date and DateTime Format Strings

Format strings used to describe dates and date-times on the LabKey platform must be compatible with the format accepted by the Java class SimpleDateFormat. For more information see the java documentation. The following table has a partial guide to pattern symbols.

LetterDate/Time ComponentExamples
GEra designatorAD
yYear1996; 96
MMonth in yearJuly; Jul; 07
wWeek in year27
WWeek in month2
DDay in year189
dDay in month10
FDay of week in month2
EDay in weekTuesday; Tue
aAm/pm markerPM
HHour in day (0-23)0
kHour in day (1-24)24
KHour in am/pm (0-11)0
h .......Hour in am/pm (1-12) .......12 .......
mMinute in hour30
sSecond in minute33
SMillisecond978
zTime ZonePacific Standard Time; PST; GMT-08:00
ZTime Zone-0800
XTime Zone-08; -0800; -08:00

To control whether an internationally ambiguous date string such as 04/06/2014 should be interpreted as Day-Month-Year or Month-Day-Year, an admin can set the date parsing format at the site level.

Note that the LabKey date parser does not recognize time-only date strings. This means that you need to enter a full date string even when you wish to display time only. For example, you might enter a value of "2/2/09 4:00 PM" in order to display "04 PM" when using the format string "hh aa".

Format Shortcuts

At the field level, instead of providing a specific format string, you can use a shortcut value for commonly used formats. For details, see Date & Number Display Formats

Number Format Strings

Format strings for Number (Double) fields must be compatible with the format that the java class DecimalFormat accepts. A valid DecimalFormat is a pattern specifying a prefix, numeric part, and suffix. For more information see the java documentation. The following table has an abbreviated guide to pattern symbols:

SymbolLocationLocalized?Meaning
0NumberYesDigit
#NumberYesDigit, zero shows as absent
.NumberYesDecimal separator or monetary decimal separator
-NumberYesMinus sign
,NumberYesGrouping separator

Examples

The following examples apply to Number (Double) fields.

Format StringDisplay Result
yyyy-MM-dd HH:mm2008-05-17 01:45
yyyy-MM-dd HH:mmaa2008-05-17 01:45PM
MMMM dd yyyyMay 17 2008
hh:mmaa zzzz01:45PM Pacific Daylight Time
<no string>85.0
085
000085
.0085.00
000.000085.000
000,000085,000
-000,000-085,000

Java Reference Documents

Dates: http://docs.oracle.com/javase/8/docs/api/java/text/SimpleDateFormat.html

Numbers: http://docs.oracle.com/javase/8/docs/api/java/text/DecimalFormat.html

Related Topics

Date & Number Display Formats




Visualizations


Visualizations

You can visualize, analyze and display data using a range of plotting and reporting tools.

Plot Editor

When viewing a data grid, select (Charts) > Create Chart menu to open the plot editor and create new:

Column Visualizations

To generate a quick visualization on a given column in a dataset, select an option from the column header:

Open Saved Visualizations

Once created and saved, visualizations will be re-generated by re-running their associated scripts on live data. You can access saved visualizations either through the (Reports or Charts) pulldown menu on the associated data grid, or directly by clicking on the name in the Data Views web part.

Related Topics




Bar Charts


Create a Bar Chart

  • Navigate to the data grid you want to visualize.
  • Select (Charts) > Create Chart to open the editor. Click Bar (it is selected by default).
  • The columns eligible for charting from your current grid view are listed.
  • Select the column of data to use for separating the data into bars and drag it to the X Axis Categories box.
  • Only the X Axis Categories field is required to create a basic bar chart. By default, the height of the bar shows the count of rows matching each value in the chosen category. In this case the number of participants from each country.
  • To use a different metric for bar height, select another column (here "Lymphs") and drag it to the box for the Y Axis column. Notice that you can select the aggregate method to use. By default, SUM is selected and the label reads "Sum of Lymphs". Here we change to "Mean"; the Y Axis label will update automatically.
  • Click Apply.

Bar Chart Customizations

  • To remove the large number of "blank" values from your chart:
    • Click View Data.
    • Click the Country column header, then select Filter.
    • Click the checkbox for "Blank" to deselect it.
    • Click OK in the popup.
    • Click View Chart to return to the chart which is now missing the 'blank' bar.

To make a more complex grouped bar chart, we'll add data from another column.

  • Click Chart Type to reopen the creation dialog.
  • Drag a column to the Split Categories By selection box, here "Gender".
  • Click Apply to see grouped bars. The "Split" category is now shown along the X axis with a colored bar for each value in the "X Axis Categories" selection chosen earlier.
  • Further customize your visualization using the Chart Type and Chart Layout links in the upper right.
  • Chart Type reopens the creation dialog allowing you to:
    • Change the "X Axis Categories" column (hover and click the X to delete the current election).
    • Remove or change the Y Axis metric, the "Split Categories By" column, or the aggregation method.
    • You can also drag and drop columns between selection boxes to change how each is used.
    • Note that you can also click another chart type on the left to switch how you visualize the data with the same axes when practical.
    • Click Apply to update the chart with the selected changes.

Change Layout

  • Chart Layout offers the ability to change the look and feel of your chart.

There are 3 tabs:

    • General:
      • Provide a Title to show above your chart. By default, the dataset name is used; at any time you can return to this default by clicking the (refresh) icon for the field.
      • Provide a Subtitle to print under the chart title.
      • Specify the width and height.
      • You can also customize the opacity, line width, and line color for the bars.
      • Select one of three palettes for bar fill colors: Light, Dark, or Alternate. The array of colors is shown.
    • X-Axis/Y-Axis:
      • Change the display labels for the axis (notice this does not change which column provides the data).
      • The range applied to the Y-axis can also be specified - the default is automatic. Select manual and specify the range if desired.
  • Click Apply to update the chart with the selected changes.

Save and Export Charts

  • When your chart is ready, click Save.
  • Name the chart, enter a description (optional), and choose whether to make it viewable by others. You will also see the default thumbnail which has been auto-generated, and can choose whether to use it. As with other charts, you can later attach a custom thumbnail if desired.

Once you have created a bar chart, it will appear in the Data Browser and on the (charts) menu for the source dataset. You can manage metadata about it as described in Manage Data Views.

Export Chart

Hover over the chart to reveal export option buttons in the upper right corner:

Export your completed chart by clicking an option:

  • PDF: generate a PDF file.
  • PNG: create a PNG image.
  • Script: pop up a window displaying the JavaScript for the chart which you can then copy and paste into a wiki. See Tutorial: Visualizations in JavaScript for a tutorial on this feature.

Videos

Related Topics




Box Plots


Create a Box Plot

  • Navigate to the data grid you want to visualize.
  • Select (Charts) > Create Chart to open the editor. Click Box.
  • The columns eligible for charting from your current grid view are listed.
  • Select the column to use on the Y axis and drag it to the Y Axis box.

Only the Y Axis field is required to create a basic single-box plot, but there are additional options.

  • Select another column (here "Study: Cohort") and choose how to use this column:
    • X Axis Categories: Create a plot with multiple boxes along the x-axis, one per value in the selected column.
    • Color: Display values in the plot with a different color for each column value. Useful when displaying all points or displaying outliers as points.
    • Shape: Change the shape of points based on the value in the selected column.
  • Here we make it the X-Axis Category and click Apply to see a box plot for each cohort.
  • Click View Data to see, filter, or export the underlying data.
  • Click View Chart to return. If you applied any filters, you would see them immediately reflected in the plot.

Box Plot Customizations

  • Customize your visualization using the Chart Type and Chart Layout links in the upper right.
  • Chart Type reopens the creation dialog allowing you to:
    • Change any column selection (hover and click the X to delete the current election). You can also drag and drop columns between selection boxes to change positions.
    • Add new columns, such as to group points by color or shape. Here we've chosen "Country" and "Gender", respectively.
    • Click Apply to see your changes and switch dialogs.
  • Chart Layout offers options to change the look of your chart, including these changes to make our color and shape distinctions clearer:
    • Set Show Points to All and
    • Check Jitter Points to spread the points out horizontally.
    • Click Apply to update the chart with the selected changes.
Here we see a plot with all data shown as points, jittered to spread them out colors vary by country and points are shaped based on gender. Notice the legend in the upper right. You may also notice that the outline of the overall box plot has not changed from the basic fill version shown above. This enhanced chart is giving additional information without losing the big picture of the basic plot.

Change Layout

  • Chart Layout offers the ability to change the look and feel of your chart.
  • There are 4 tabs:
    • General:
      • Provide a Title to show above your plot. By default, the dataset name is used, and you can return to this default at an time by clicking the refresh icon.
      • Provide a Subtitle to show below the title.
      • Specify the width and height.
      • Elect whether to display single points for all data, only for outliers, or not at all.
      • Check the box to jitter points.
      • You can also customize the colors, opacity, width and fill for points or lines.
    • X-Axis/Y-Axis:
      • Change the display labels for the axes (notice this does not change which columns provide the data).
      • For the Y-axis, choose log or linear scale, and if desired, apply a Range: the default is automatic. Select manual and specify the range.
    • Developer: Only available to users that are members of the "Site Developers" permission group.
      • Provide a JavaScript function that will be called when a data point in the chart is clicked.
  • Click Apply to update the chart with the selected changes.

Save and Export Plots

  • When your chart is ready, click Save.
  • Name the plot, enter a description (optional), and choose whether to make it viewable by others. You will also see the default thumbnail which has been auto-generated. You can elect None. As with other charts, you can later attach a custom thumbnail if desired.

Once you have created a box plot, it will appear in the Data Browser and on the (charts) menu for the source dataset. You can manage metadata about it as described in Manage Data Views.

Export Chart

Hover over the chart to reveal export option buttons in the upper right corner:

Export your completed chart by clicking an option:

  • PDF: generate a PDF file.
  • PNG: create a PNG image.
  • Script: pop up a window displaying the JavaScript for the chart which you can then copy and paste into a wiki. See Tutorial: Visualizations in JavaScript for a tutorial on this feature.

Rules Used to Render the Box Plot

The following rules are used to render the box plot. Hover over a box to see a pop-up.

  • Min/Max are the highest and lowest data points still within 1.5 of the interquartile range.
  • Q1 marks the lower quartile boundary.
  • Q2 marks the median.
  • Q3 marks the upper quartile boundary.
  • Values outside of the range are considered outliers and are rendered as dots by default. The options and grouping menus offer you control of whether and how single dots are shown.

Video

Related Topics




Line Plots


Create a Line Plot

  • Navigate to the data grid you want to visualize. We will use the Lab Results dataset from the example study for this walkthrough.
  • Select (Charts) > Create Chart. Click Line.
  • The columns eligible for charting from your current grid view are listed.
  • Select the X Axis column by drag and drop, here "Date".
  • Select the Y Axis column by drag and drop, here "Lymphs".
  • Only the X and Y Axes are required to create a basic line plot. Other options will be explored below.
  • Click Apply to see the basic plot.

This basic line chart plots a point for every Lymph value measured for each date, as in a scatter plot, then draws a line between them. When all values for all participants are mixed, this data isn't necessarily useful; we might want to separate by participant to see if any patterns emerge for individuals.

Line Plot Customizations

Customize your visualization using the Chart Type and Chart Layout links in the upper right.

  • Chart Type reopens the creation dialog allowing you to:
    • Change the X or Y Axis column (hover and click the X to delete the current selection).
    • Select a Series column (optional). The series measure is used to split the data into one line per distinct value in the column.
    • Note that you can also click another chart type on the left to switch how you visualize the data with the same axes when practical.
  • For this walkthrough, drag "Participant ID" to the Series box.
  • Click Apply.

Now the plot draws series' lines between values for the same subject, but is unusably dense. Let's filter to a subset of interest.

  • Click View Data to see and filter the underlying data.
  • Click the ParticipantID column header and select Filter.
    • Click the "All" checkbox in the popup to unselect all values. Then, check the boxes for the first 6 participants, 101-606.
    • Click OK.
  • Click View Chart to return. Now there are 6 lines showing values for the 6 participants, clearly divided into upward and downward trending values.

This is a fairly small set of example data, but we can use the line plot tool to check a hypothesis about cohort correlation.

  • Click Chart Type to reopen the editor.
  • Drag "Study: Cohort" to the Series box. Notice it replaces the prior selection.
  • Click Apply.

Now the line plot shows a line series of all points for the participants in each cohort. Remember that this is a filtered subset of (fictional) participants, but in this case it appears there is a correlation. You could check against the broader dataset by returning to view the data and removing the filter.

Add a Second Y Axis

To plot more data, you can add a second Y axis and display it on the right.

  • Click Chart Type to reopen the editor.
  • Drag the "CD4+" column to the Y Axis box. Notice it becomes a second panel and does not replace the prior selection (Lymphs).
  • Click the (circle arrow) to set the Y Axis Side for this measure to be on the right.
  • Click Apply.
  • You can see the trend line for each measure for each cohort in a single plot.

Change Chart Layout

The Chart Layout button offers the ability to change the look and feel of your chart.

There are four tabs:

  • General:
    • Provide a title to display on the plot. The default is the name of the source data grid.
    • Provide a subtitle to display under the title.
    • Specify a width and height.
    • Control the point size and opacity, as well as choose the default color, if no "Series" column is set.
    • Control the line width.
    • Hide Data Points: Check this box to display a simple line instead of showing shaped points for each value.
    • Number of Charts: Select whether to show a single chart, or a chart per measure, when multiple measures are defined.
  • X-Axis/Y-Axis:
    • Change the display labels for the axis (notice this does not change which column provides the data).
    • Specify a manual range if desired.
  • Developer: Only available to users that are members of the "Site Developers" permission group.
    • Provide a JavaScript function that will be called when a data point in the chart is clicked.

Save and Export Plots

  • When your plot is finished, click Save.
  • Name the chart, enter a description (optional), and choose whether to make it viewable by others. You will also see the default thumbnail which has been auto-generated, and can choose whether to use it. As with other charts, you can later attach a custom thumbnail if desired.

Once you have saved a line plot, it will appear in the Data Browser and on the (charts) menu for the source dataset. You can manage metadata about it as described in Manage Data Views.

Export Chart

Hover over the chart to reveal export option buttons in the upper right corner:

Export your completed chart by clicking an option:

  • PDF: generate a PDF file.
  • PNG: create a PNG image.
  • Script: pop up a window displaying the JavaScript for the chart which you can then copy and paste into a wiki. See Tutorial: Visualizations in JavaScript for a tutorial on this feature.

Related Topics




Pie Charts


Create a Pie Chart

  • Navigate to the data grid you want to visualize.
  • Select (Charts) > Create Chart to open the editor. Click Pie.
  • The columns eligible for charting from your current grid view are listed.
  • Select the column to visualize and drag it to the Categories box.
  • Click Apply. The size of the pie wedges will reflect the count of rows for each unique value in the column selected.
  • Click View Data to see, filter, or export the underlying data.
  • Click View Chart to return. If you applied any filters, you would see them immediately reflected in the chart.

Pie Chart Customizations

  • Customize your visualization using the Chart Type and Chart Layout links in the upper right.
  • Chart Type reopens the creation dialog allowing you to:
    • Change the Categories column selection.
    • Note that you can also click another chart type on the left to switch how you visualize the data using the same selected columns when practical.
    • Click Apply to update the chart with the selected changes.

Change Layout

  • Chart Layout offers the ability to change the look and feel of your chart.
  • Customize any or all of the following options:
    • Provide a Title to show above your chart. By default, the dataset name is used.
    • Provide a Subtitle. By default, the categories column name is used. Note that changing this label does not change which column is used for wedge categories.
    • Specify the width and height.
    • Select a color palette. Options include Light, Dark, and Alternate. Mini squares showing the selected palette are displayed.
    • Customizing the radii of the pie chart allows you to size the graph and if desired, include a hollow center space.
    • Elect whether to show percentages within the wedges, the display color for them, and whether to hide those annotations when wedges are narrow. The default is to hide percentages when they are under 5%.
    • Use the Gradient % slider and color to create a shaded look.
  • Click Apply to update the chart with the selected changes.

Save and Export Charts

  • When your chart is ready, click Save.
  • Name the chart, enter a description (optional), and choose whether to make it viewable by others. You will also see the default thumbnail which has been auto-generated, and can choose whether to use it. As with other charts, you can later attach a custom thumbnail if desired.

Once you have created a pie chart, it will appear in the Data Browser and on the (charts) menu for the source dataset. You can manage metadata about it as described in Manage Data Views.

Export Chart

Hover over the chart to reveal export option buttons in the upper right corner:

Export your completed chart by clicking an option:

  • PDF: generate a PDF file.
  • PNG: create a PNG image.
  • Script: pop up a window displaying the JavaScript for the chart which you can then copy and paste into a wiki. See Tutorial: Visualizations in JavaScript for a tutorial on this feature.

Videos

Related Topics




Scatter Plots


Create a Scatter Plot

  • Navigate to the data grid you want to visualize.
  • Select (Charts) > Create Chart. Click Scatter.
  • The columns eligible for charting from your current grid view are listed.
  • Select the X Axis column by drag and drop.
  • Select the Y Axis column by drag and drop.
  • Only the X and Y Axes are required to create a basic scatter plot. Other options will be explored below.
  • Click Apply to see the basic plot.
  • Click View Data to see, filter, or export the underlying data.
  • Click View Chart to return. If you applied any filters, you would see them immediately reflected in the plot.
  • Customize your visualization using the Chart Type and Chart Layout links in the upper right.
  • Chart Type reopens the creation dialog allowing you to:
    • Change the X or Y Axis column (hover and click the X to delete the current selection).
    • Add a second Y Axis column (see below) to show more data.
    • Optionally select columns for grouping of points by color or shape.
    • Note that you can also click another chart type on the left to switch how you visualize the data with the same axes and color/shape groupings when practical.
    • Click Apply to update the chart with the selected changes.
  • Here we see the same scatter plot data, with colors varying by cohort and points shaped based on gender. Notice the key in the upper right.

Change Layout

The Chart Layout button offers the ability to change the look and feel of your chart.

There are four tabs:

  • General:
    • Provide a title to display on the plot. The default is the name of the source data grid.
    • Provide a subtitle to display under the title.
    • Specify a width and height.
    • Choose whether to jitter points.
    • Control the point size and opacity, as well as choose the default color palette. Options: Light (default), Dark, and Alternate. The array of colors is shown under the selection.
    • Number of Charts: Select either "One Chart" or "One Per Measure".
    • Group By Density: Select either "Always" or "When number of data points exceeds 10,000."
    • Grouped Data Shape: Choose either hexagons or squares.
    • Density Color Palette: Options are blue & white, heat (yellow/orange/red), or select a single color from the dropdown to show in graded levels. These palettes override the default color palette and other point options in the left column.
  • X-Axis/Y-Axis:
    • Change the display labels for the axis (notice this does not change which column provides the data).
    • Choose log or linear scale for the Y-axis, and specify a range if desired.
  • Developer: Only available to users that are members of the "Site Developers" permission group.
    • Provide a JavaScript function that will be called when a data point in the chart is clicked.

Add Second Y Axis

You can add more data to a scatter plot by selecting a second Y axis column. Reopen a chart for editing, click Chart Type, then drag another column to the Y Axis field. The two selected fields will both have panels. On each you can select the side for the Y Axis using the arrow icons.

For this example, we've removed the color and shape columns to make it easier to see the two axes in the plot. Click Apply.

If you use the Chart View > Number of Charts > One Per Measure, you will see two separate charts, still respecting the Y Axis sides you set.

Example: Heat Map

Displaying a scatter plot as a heatmap is done by changing the layout of a chart. Very large datasets are easier to interpret as heatmaps, grouped by density (also known as point binning).

  • Click Chart Layout and change Group By Density to "Always".
  • Select Heat as the Density Color Palette and leave the default Hexagon shape selected
  • Click Apply to update the chart with the selected changes. Shown here, the number of charts was reset to one, and only a single Y axis is included.
  • Notice that when binning is active, a warning message will appear reading: "The number of individual points exceeds XX. The data is now grouped by density which overrides some layout options." XX will be either 10,000 or 1, if you selected "Always" as we did. Click Dismiss to remove that message from the plot display.

Save and Export Plots

  • When your plot is finished, click Save.
  • Name the plot, enter a description (optional), and choose whether to make it viewable by others. You will also see the default thumbnail which has been auto-generated, and can choose whether to use it. As with other charts, you can later attach a custom thumbnail if desired.

Once you have saved a scatter plot, it will appear in the Data Browser and on the (charts) menu for the source dataset. You can manage metadata about it as described in Manage Data Views.

Export Plot

Hover over the chart to reveal export option buttons in the upper right corner:

Export your completed chart by clicking an option:

  • PDF: generate a PDF file.
  • PNG: create a PNG image.
  • Script: pop up a window displaying the JavaScript for the chart which you can then copy and paste into a wiki. See Tutorial: Visualizations in JavaScript for a tutorial on this feature.

Video

Related Topics




Time Charts



Time charts provide rich time-based visualizations for datasets and are available in LabKey study folders. In a time chart, the X-axis shows a calculated time interval or visit series, while the Y-axis shows one or more numerical measures of your choice. With a time chart you can:
  • Individually select which study participants, cohorts, or groups appear in the chart.
  • Refine your chart by defining data dimensions and groupings.
  • Export an image of your chart to a PDF or PNG file.
  • Export your chart to Javascript (for developers only).
Note: Only properties defined as measures in the dataset definition can be plotted on time charts.

Note: In a visit-based study, visits are a way of measuring sequential data gathering. To create a time chart of visit based data, you must first create an explicit ordering of visits in your study. In a continuous study, there are no calculated intervals for measures used for generating time charts.

Create a Time Chart

Time charts are only available in study folders.

  • Navigate to the dataset, view, or query of interest. In this example, we use the Lab Results dataset in the example study.
  • Select (Charts) > Create Chart. Click Time.
  • Whether the X-axis is date based or visit-based is determined by the study type. For a date-based study:
    • Choose the Time Interval to plot: Days, Weeks, Months, Years.
    • Select the desired Interval Start Date from the pulldown menu. All eligible date fields are listed.
  • At the top of the right panel is a drop down from which you select the desired dataset or query. Time charts are only supported for datasets/queries in the "study" schema which include columns designated as 'measures' for plotting. Queries must also include both the 'ParticipantId' and 'ParticipantVisit' columns to be listed here.
  • The list of columns designated as measures available in the selected dataset or query is shown in the Columns panel. Drag the desired selection to the Y-Axis box.
    • By default the axis will be shown on the left; click the right arrow to switch sides.
  • Click Apply.
  • The time chart will be displayed.
  • Use the checkboxes in the Filters panel on the left:
    • Click a label to select only that participant.
    • Click a checkbox to add or remove that participant from the chart.
  • Click View Data to see the underlying data.
  • Click View Chart(s) to return.

Time Chart Customizations

  • Customize your visualization using the Chart Type and Chart Layout links in the upper right.
  • Chart Type reopens the creation dialog allowing you to:
    • Change the X Axis options for time interval and start date.
    • Change the Y Axis to plot a different measure, or plot multiple measures at once. Time charts are unique in allowing cross-query plotting. You can select measures from different datasets or queries within the same study to show on the same time chart.
      • Remove the existing selection by hovering and clicking the X. Replace with another measure.
      • Add a second measure by dragging another column from the list into the Y-Axis box.
      • For each measure you can specify whether to show the Y-axis for it on the left or right.
      • Open and close information panels about time chart measures by clicking on them.
    • Click Apply to update the chart with the selected changes.

Change Layout

  • Chart Layout offers the ability to change the look and feel of your chart.

There are at least 4 tabs:

  • On the General tab:
    • Provide a Title to show above your chart. By default, the dataset name is used.
    • Specify the width and height of the plot.
    • Use the slider to customize the Line Width.
    • Check the boxes if you want to either Hide Trend Line or Hide Data Points to get the appearance you prefer. When you check either box, the other option becomes unavailable.
    • Number of Charts: Choose whether to show all data on one chart, or separate by group, or by measure.
    • Subject Selection: By default, you select participants from the filter panel. Select Participant Groups to enable charting of data by groups and cohorts using the same checkbox filter panel. Choose at least one charting option for groups:
      • Show Individual Lines: show plot lines for individual participant members of the selected groups.
      • Show Mean: plot the mean value for each participant group selected. Use the pull down to select whether to include range bars when showing mean. Options are: "None, Std Dev, or Std Err".
  • On the X-Axis tab:
    • Customize the Label shown on the X-axis. Note that changing this text will not change the interval or range plotted. Use the Chart Type settings to change what is plotted.
    • Specify a range of X values to plot, or use the default automatic setting.
  • There will be one Y-Axis tab for each side of the plot if you have elected to use both the left and right Y-axes. For each side:
    • Customize the Label shown on that Y-axis. Note that changing this text will not change the measure or range plotted.
    • Select whether to use a linear or log scale on this axis.
    • Range: Specify a range of values or use the default automatic setting.
    • For each Measure using that Y-axis:
      • Choose an Interval End Date. The pulldown menu includes eligible date columns from the source dataset or query.
      • Choose a column if you want to Divide Data Into Series by another measure.
      • When dividing data into series, choose how to display duplicate values (AVG, COUNT, MAX, MIN, or SUM).
  • On the Developer tab, users with developer access can provide a JavaScript function that will be called when a data point in the chart is clicked.
  • Click Apply to update the chart with the selected changes. In this example, we now plot data by participant group. Note that the filter panel now allows you to plot trends for cohorts and other groups. This example shows a plot combining trends for two measures, lymphs and viral load, for two study cohorts.

Save Chart

  • When your chart is ready, click Save.
  • Name the chart, enter a description (optional), and choose whether to make it viewable by others. You will also see the default thumbnail which has been auto-generated, and can choose whether to use it. As with other charts, you can later attach a custom thumbnail if desired.
  • Click Save.

Once you have created a time chart, it will appear in the Data Browser and on the charts menu for the source dataset.

Data Dimensions

By adding dimensions for a selected measure, you can further refine the timechart. You can group data for a measure on any column in your dataset that is defined as a "data dimension". To define a column as a data dimension:

  • Open a grid view of the dataset of interest.
  • Click Manage.
  • Click Edit Definition.
  • In the Dataset Fields section, select a column.
  • Select the Reporting tab.
  • Place a checkmark next to Data Dimension.
  • Click Save.

To use the data dimension in a time chart:

  • Click View Data to return to your grid view.
  • Create a new time chart, or select one from the (Charts) menu and click Edit.
  • Click Chart Layout.
  • Select the Y-Axis tab for the side of the plot you are interested in (if both are present.
    • The pulldown menu for Divide Data Into Series By will include the dimensions you have defined.
  • Select how you would like duplicate values displayed. Options: Average, Count, Max, Min, Sum.
  • Click Apply.
  • A new section appears in the filters panel where you can select specific values of the new data dimension to further refine your chart.

Export Chart

Hover over the chart to reveal export option buttons in the upper right corner:

Export your completed chart by clicking an option:

  • PDF: generate a PDF file.
  • PNG: create a PNG image.
  • Script: pop up a window displaying the JavaScript for the chart which you can then copy and paste into a wiki. See Tutorial: Visualizations in JavaScript for a tutorial on this feature.

Related Topics




Column Visualizations


Click a column header to see a list of Column Visualizations, small plots and charts that apply to a single column. When selected, the visualization is added to the top of the data grid. Several can be added at a time.
  • Bar Chart - Histogram displayed above the grid.
  • Box & Whisker - Distribution box displayed above the grid.
  • Pie Chart - Pie chart displayed above the grid.

Visualizations will be updated to reflect updates to the underlying data and any filters added to the data grid.

Column Visualizations are persisted within a saved custom view. When you come back to the saved view, the Column Visualizations will appear again.

To remove a chart, hover over the chart and click the 'X' in the upper right corner.

Available visualization types are determined by datatype as well as whether the column is a Measure and/or a Dimension.

  • The box plot option is shown for any column marked as a Measure.
  • The bar and pie chart options are shown for any column marked as a Dimension.
Column visualizations are simplified versions of standalone charts of the same types. Click any chart to open it within the plot editor which allows you to make many additional customizations and save it as a new standalone chart.

Bar Chart

A histogram of the Weight column.

Box and Whisker Plot

A basic box plot report. You can include several column visualizations above a grid simultaneously.

Pie Chart

A pie chart showing prevalence of ARV Regimen types.

Filters are also applied to the visualizations displayed. If you filter to exclude 'blank' ARV treatment types, the pie chart will update.

Related Topics




Quick Charts


Quick Charts provide a quick way to assess your data without deciding first what type of visualization you will use.

Create a Quick Chart

  • Navigate to a data grid you wish to visualize.
  • Click a column header and select Quick Chart. Depending on the content of the column, LabKey Server makes a best guess at the type and arrangement of chart to use as a starting place. A numeric column in a cohort study, for example, might be quickly charted as a box and whisker plot using cohorts as categories.
  • You can then alter and refine the chart in the following ways:
    • View Data: Toggle to the data grid, potentially to apply filters to the underlying data. Filters are reflected in the plot upon re-rendering.
    • Export: Export the chart as a PDF, PNG, or Script.
    • Help: Documentation links.
    • Chart Type: Click to open the plot editor. You can change the plot type to any of the following and the options for chart layout settings will update accordingly
    • Chart Layout: Click to customize the look and feel of your chart; options available vary based on the chart type. See individual chart type pages for a descriptions of options.
    • Save: Click to open the save dialog.

Related Topics




Lists


A List is a flexible, user-defined table that is defined and managed via the LabKey Server web UI. Lists can be used for a variety of purposes:
  • A place to store and edit data entered by users via forms or editable grids
  • Defined vocabularies, which can be used to constrain choices during completion of fields in data entry forms
  • Simple workflows that incorporate discussions, documents, and states
  • Read-only resources that users can search, filter, sort, and export
The design, or schema, of a list is the set of fields (columns and types) and can be defined in several ways. Lists can be linked via lookups and joins to draw data from many sources. Lists can be indexed for search, including optional indexing of any attachments added to fields. Populated lists can be exported and imported as archives for easy transfer between development, staging and production folders or servers.

Topics

List Web Parts

You need to be an administrator to create and manage lists. You can directly access the list manager by selecting (Admin) > Manage Lists. To make the set of lists visible to other users, and create a one click shortcut for admins to manage lists, add a Lists web part to your project or folder.

Lists Web Part

  • Enter > Page Admin Mode.
  • Choose Lists from the <Select Web Part> pulldown at the bottom of the page.
  • Click Add.
  • Click Exit Admin Mode.

List-Single Web Part

To display the contents of a single list, add a List - Single web part, name it and choose the list and view to display.




Tutorial: Lists


This tutorial introduces you to the simplicity of lists, plus the power of lookups, URL properties and joins for generating insights into your results.

This tutorial can be completed using a free 30-day trial version of LabKey Server.

This "Rats of NIMH" demo uses four lists containing information about animal subjects and blood samples from those animals. Samples were subdivided into portions, then into slides, on which experiments were run. Gaining insight into these results requires joining them with information about samples and demographics to get the full picture.

Lookups and URL properties help you to navigate your data more easily. A lookup can let you display the name of the subject instead of displaying an obscure SubjectID. Lookups also let you link sample data to other information (such as demographic data). URL properties can help you link directly to relevant visualizations or outside sources from subject data. Joins help you gain insight across different lists of data by letting you view data from related lists together in common views.

Completing this tutorial requires administrative permissions, which you will have if you create your own server trial instance in the first step. The covered features, however, are not restricted to admin users.

To begin:

Tutorial Steps

View the interactive example of the lists used in this tutorial.

Related Topics

First Step




List Tutorial: Setup


Set Up

You should be signed into your server and in the new "List Tutorial" folder you created.

Import a List Archive

  • In the Lists web part, click Manage Lists.
  • Click Import List Archive.
  • Click Choose File and select [LabKeyDemoFiles]/Lists/NIMH/ListDemo_NIMH_ListArchive.lists.zip from the unzipped demo files.
  • Click Import List Archive.

The list archive includes four lists interconnected with some specific fields defined as lookups to other lists. You can continue the tutorial using the newly imported set of lists.

Related Topics

  • Connect Lists: Walkthrough the process used to create the list archive you imported.

Start Over | Next Step




Create a Joined Grid


The imported list archive includes four lists interconnected with some lookups between them. These lookups allow us to create joins between these lists which pays off when we investigate our experimental results in the NIMHSlides list. The Slides list shows an interesting pattern in positive/negative stains, but you won't see this pattern unless you first join in some information from another list (Demographics).

  • Click the NIMHSlides list from the Lists web part (or Available Lists grid).

The grid shows all the uploaded slide data. You may or may not be able to spot a pattern at this point. Familiarity with the novel might help in this very simple example, but we can also use additional data to help us.

Create a Joined Grid

  • On the NIMHSlides grid, select (Grid Views) > Customize Grid.
  • In the Available Fields panel, expand the node SubjectID by clicking on the icon next to it.
  • Place checkmarks next to the Name, Family, and Species fields. (Note that these fields are added to the Selected Fields panel.)
  • Click Save.
  • In the Save Custom Grid View popup menu, select Named and name this view NIMHSlideDataJoinedView. Click Save.
  • You'll now see additional columns in the grid view. To view an interactive example, see: NIMHSlideDataJoinedView

Find a Pattern in the Data

Do positive/negative stains correlate with any other characteristics in our subjects?

First, let's do the simplest possible data analysis: sorting the data.

  • Click the column header Stain Positive to bring up a menu of sorting and filter options.
  • Choose Sort Descending.
  • Examine the results and notice that almost all of the positive stains came from field mice.
  • Looks like we have a pattern worth further investigation.

Previous Step | Next Step




Add a URL Property


In this step, we explore two ways to use the URL property.

Create Links to Filtered Results

It can be handy to generate an active filtering link in a list or dataset. For example, here we use a URL property to turn the values in the species column into links to a filtered subset of the data. When you click one species value, you get a grid showing only subjects of the same species.

  • Click the List Tutorial link above the grid to return to the main folder page.
  • From the Lists web part, click NIMHDemographics to open the grid view of the list. Notice that in the "Species" column, the values are just text, not links (yet).
  • Click the column header Species, then Filter.
  • Click the label Rat to select only that single value.
  • Click OK.
  • Notice the URL in your browser, which might look something like this - the full path and your list ID number may vary, but the filter you applied is encoded at the end. This will show only rows where the species is Rat.

Next we'll modify the design of the list to create the links.

  • Click Design.
  • Click Edit Design.
  • Select the Species field by clicking in its Name box.
  • On the Display tab in the field properties box, enter this value for the URL field:
/list/grid.view?name=NIMHDemographics&query.Species~eq=${Species}
    • The filter portion of this URL replaces "Rat" with the substitution string "${Species}". (If we were to specify "Rat", clicking any species link in the list would filter the list to only show the rats!)
    • The "listId" portion of the URL has been replaced with "name=NIMHDemographics."
  • Scroll up and click Save.
  • Click Done.
  • Clear the "Species=Rat" filter using the in the filter bar, or clicking the column header Species and then clicking Clear Filter.

Now notice that the values in the "Species" column are all links. When you click one, you will get the filtering behavior we just defined.

  • Click Field mouse in any row and you will see the demographics list filtered to display only rows for field mice.

Create Links to Files

A column value can also include link to a file. All the values in a column could link to a fixed file (such as to a protocol document) or you can make row-specific links to files where a portion of the filename matches a value in the row such as the Subject ID in this example. Open this link in a new browser window:

Edit the URL, incrementing the file name 20023.png, 20024.png, 20025.png. These are simply cartoon images of our sample characters stored on a public server, but in actual use you might have slide images or other files of interest stored on your local machine and named by subjectId.

Here is a generalized version, using substitution syntax for the URL property, that you can use in the list design:

This generalized version is already included in the list design in our example archive.

  • Click the List Tutorial link near the top of the page, then click the NIMHDemographics list in the Lists web part.
  • Click Design.
  • Click Edit Design.
  • Select the CartoonAvailable field by clicking its Name box.
  • Notice the URL property for this field. It is the generalized version of the link above.
  • Scroll up to click Cancel, then Done to return to the grid for this list.
  • Observe that clicking true in the CartoonAvailable column opens an image of the 'subject' in that row. Click the 'true' link for the SubjectID 20022 to see the same image you opened earlier.

Related Topics

Previous Step




Create Lists


A list is a basic way of storing information. The list design, or schema, is the set of columns and types, which forms the structure of the list. The design can be created and the list data populated in several ways.

Related Topics




Create a List Design


A list is a simple structure for storing data. The design, or schema, for a list is the set of fields (columns and types) which comprise the list. If you are an administrator, you can create a new list design by: You can also simultaneously create the design and populate the list from a spreadsheet|populateList#infer].

Once you have created a list, you can add data to it and can also export the design as a set of fields that can be used like a template for designing another list.

Manually Define Fields

  • In the project or folder where you want the list, select (Admin) > Manage Lists.
  • Click Create New List.
  • Name the list, i.e. "Technicians" in this example.
  • Choose the appropriate Primary Key and Primary Key Type. Here we use the defaults.
  • Do not select the Import From File checkbox this time.
  • Click Create List.

Set Properties and Add Fields

  • Set the List Properties appropriately. We leave the defaults unchanged for this example.
  • The only currently defined data field is "Key," the default we left in place as the Primary Key.
  • Use the Add Field button below it to add additional fields to your list design. If you add an extraneous field, just click the "X" button to the left of the field row you would like to delete; you cannot delete the primary key field.
  • In this example, we've added three fields.
  • The Name, Label and Type for the three new fields:
    • Name: FirstName Label: First Name Type: String
    • Name: LastName Label: Last Name Type: String
    • Name: ID Label: ID Type: Integer
  • If desired, you can also set properties of list fields as you add them.
  • Scroll back to the top of the page and click Save.
  • Click Done to return to the grid of available lists.

Infer Fields from a Spreadsheet

Instead of creating the list fields one-by-one you can infer the list design from the column headers of a sample spreadsheet. Either upload the file or cut and paste the contents using this method. In this process, the data is not imported from the spreadsheet, merely the structure of it.

  • Click here to download this file: Technicians.xls
  • Select (Admin) > Manage Lists and click Create New List.
  • Name the list, i.e. "Technicians2" so you can compare it to the list created above.
  • Leave the key defaults, and click Create List.
  • Scroll down to the "List Fields", and click Infer Fields From File to open the popup as shown below.
    • Upload the sample file directly or paste the contents into the box.
  • Click Submit.
  • Scroll back to the top of the page and click Save.
  • Click Done to return to the grid of available lists.
  • Click "Technicians2" to see that the field names and types are inferred forming the header row, but no data was imported from the spreadsheet.

Note: Importing fields from a file will overwrite the existing list design, including deleting any data that has already been added to the list. Use only for new list creation. A warning message appears in the popup box to remind you.

Shortcut: Infer Fields and Populate a List from a Spreadsheet

If you want to both infer the fields to design the list and populate the new list with the data from the spreadsheet, follow this shortcut process:

  • Select (Admin) > Manage Lists.
  • Click Create New List.
  • Name the list, i.e. "Technicians3".
  • Leave the default key information again, but check the box for Import from File this time.
  • Click Create List.
  • Click Browse or Choose File and select the Technicians.xls file you downloaded.
  • You will see a preview of the fields that will be inferred, along with the first few rows of data. You may change types or labels of columns here if needed, or ignore a column when importing by unchecking the box next to the column name.
  • Click Import.
  • You will see the column headers, matching those in the other two lists you created above, as well as the three lines of data.

Import/Export Fields for a List Design

Once you have created and saved a list design, whether or not you have added data to the list itself, you can export the fields, such as to use as a template when creating another list. The new list could have changes or additional fields, but starting from an exported set of fields can reduce data entry and improve consistency.

  • Select (Admin) > Manage Lists.
  • Click the name of a list in the Lists web part (here, any one of the "Technicians" lists).
  • Click Design.
  • Scroll down to the List Fields section and click Export Fields.
  • Copy the contents of the popup window to your browser clipboard or a notepad. This representation of the schema can be used as a template when creating a new list.
  • Click Done in the popup, then Done in the list editor.

Use the exported set of fields to create a new list design.

Note: Once a list contains data, importing a new set of fields in this way will completely overwrite the list and cause any existing data to be deleted. This import option is intended for list creation, not for adding additional fields to a list.

  • Select (Admin) > Manage Lists.
  • Click Create New List, name it "Technicians4" and leave the key settings at their defaults. Do not check "Import from file".
  • Click Create List.
  • In the List Fields section, click Import Fields.
  • Paste the schema you exported above into the provided window:
  • Click Import and notice the list fields are populated as when you created them by other methods.
  • You could modify, add, or delete fields for the new list design if needed.
  • Click Save to save the new design.

Related Topics




Import a List Archive


You can copy all lists in a folder to another folder or another server using export and import. Export packages up all your lists into a list archive: a .lists.zip file that conforms to the LabKey list export format. The process is similar to study export/import/reload. Information on the list serialization format is covered as part of Study Archive Files and Formats.

Export

To export all the lists in a folder to a list archive:

  • In the folder that contains lists of interest, select (Admin) > Manage Lists.
  • Select Export List Archive.
  • All lists in the current folder are exported into a zip archive.

Import

To import a list archive:

  • In the folder where you would like to import the list archive, select (Admin) > Manage Lists.
  • Select Import List Archive.
  • Browse to the .zip file that contains your list archive and select it.
  • Click Import List Archive.
  • The imported lists will then be displayed in the Lists web part.

Note: Existing lists will be replaced by lists in the archive with the same name; this could result in data loss and cannot be undone.

Auto-Increment Key Considerations

Exporting a list with an auto-increment key may result in different key values on import. If you have lookup lists make sure they use an integer or string key instead of an auto-increment key.

Related Topics

The Tutorial: Lists uses a list archive preconfigured to connect several lists with lookups between them. A walkthrough of the steps involved in creating that archive from independent lists is covered in Connect Lists.




Choose a Primary Key


Every item in a list has a key value that uniquely identifies the item. It could be a basic auto-incremented integer or a number or string uniquely assigned to the list items.

When creating a list, choose the:

  • Primary Key, the name of the column that holds the unique key. If data imported to the list does not already include a column by this name, it will be added. Default: "Key".
  • Primary Key Type:
    • Auto-Increment Integer (Default)
    • Integer
    • Text (String)
Once your list has been created, the field that holds the key is marked by a key in the list design editor. In the example list below (NIMHDemographics, from the Tutorial: Lists), the key is "SubjectID":




Edit a List Design


Editing the list design allows you change the structure and functions of a list, whether or not it has been populated with data. To see the list design, click Design above the grid view of the list. To edit the design, then click Edit Design. If you do not see these options, you do not have permission to edit the given list.

List Properties

The list properties contain metadata about the list and enable various actions including export and search.

The properties of the NIMHDemographics list in the List Tutorial Demo look like this:

  • Name: The displayed name of the list.
  • Description: An optional description of the list.
  • Title Field: Identifies the field (i.e., the column of data) that is used when other lists or datasets do lookups into this list. You can think of this as the "lookup display column." Select a specific column from the dropdown or leave the default "<AUTO>" selection, which uses this process:
    • Use the first non-lookup string column (this could be the key).
    • If there are no string fields, use the key.
  • Discussion Links: Optionally allow discussions to be associated with each list item. Such links will be exposed as a "discussion" link on the details view of each list item. Select one of:
    • None (Default)
    • Allow one discussion per item
    • Allow multiple discussions per item
  • Allowable Actions: These checkboxes determine whether Delete, Upload, Export and Print are allowed for the list. All are checked by default.
  • Full-Text Search Indexing. Determines how the list data, metadata, and attachments are indexed for full-text searching.

List Fields

You can add, delete or edit the fields of your list in this section. See Field Properties Reference.

Example. The field editor for the NIMHDemographics list in the List Tutorial Demo looks like this:

Customize the Order of List Fields

By default, the order of fields in the default grid is used to order the fields in insert, edit and details for a list. All fields that are not in the default grid are appended to the end. To see the current order, click Insert New for an existing list.

To change the order of fields, modify the default grid by selecting (Grid Views) > Customize Grid. See Customize Grid Views for further details.

List Metadata and Hidden Fields

In addition to the fields you define, there is list metadata associated with every list. To see and edit it, use the schema browser. Select (Admin) > Developer Links > Schema Browser. Click lists and then select the specific list. Click Edit Metadata.

List metadata includes the following fields in addition to any user defined fields.

NameDatatypeDescription
Createddate/timeWhen the list was created
CreatedByint (user)The user who created the list
Modifieddate/timeWhen the list was last modified
ModifiedByint (user)The user who modified the list
containerfolderThe folder or project where the list is defined
lastIndexeddate/timeWhen the list was last indexed
entityIdtextA unique identifier for this list

Finally, there are several built in hidden fields in every list. To see them, open (Grid Views) > Customize Grid and check the box for Show Hidden Fields.

NameDatatypeDescription
Last Indexeddate/timeWhen this list was last indexed.
KeyintThe key field.
Entity IdtextThe unique identifier for the list itself.
FolderobjectThe container where this list resides.

Full-Text Search Indexing

You can control how your list is indexed for search depending on your needs. Choose one or more of the options:

  • Index each item as a separate document means that each item in the list will appear as a separate search result.
  • Index entire list as a single document means that the list as a whole will appear as a search result. Select one option:
    • Metadata only (name and description of list and fields)
    • Data only Note: for large lists with frequent updates, updating any item will cause re-indexing of the entire list.
    • Metadata and data Note: for large lists with frequent updates, updating any item will cause re-indexing of the entire list.
  • Index file attachments: Indexes the contents of documents uploaded to attachment fields.
When indexing either the entire list or each item separately, you also specify how to display the title and which fields to index. Note that you may choose to index *both* the entire list and each item, potentially specifying different values for each of these options.
  • Use the radio buttons to select how to display the list title:
    • Standard Title: The standard search result title is <List Name> - <Value of Title Field>
    • Custom Title: Enter a custom title in the box provided. The format can use substitution syntax to create a template that includes your choice of fields, for example: NIMHDemographics - ${SubjectID} ${Name}
  • Next specify which fields in the list should be indexed. Warning: Do not include fields that contain PHI or PII. Full text search results could expose this information.
    • Index all text fields: Values in all text fields will be indexed.
    • Index all fields (text, number, date and boolean): Values in all fields will be indexed.
    • Index using custom template: Choose the exact set of fields to index, for example: ${SubjectID} ${Name} ${Family} ${Mother} ${Father}.

Related Topics




Populate a List


Once you have created a list, there are a variety of options for populating it, designed to suit different types and complexity of data entry.

Insert Single Rows

One option for simple lists is to add a single row at a time:

  • Select (Insert data) > Insert new row.
  • Enter the values for each column in the list.
  • Click Submit.

Import Bulk Data

Paste Multiple Rows

You can also import multiple rows at once by cutting and pasting, by typing tab separated values directly into the window, or by uploading a file. To ensure the format is compatible, particularly for complex lists, you can first download a template, then populate it with your data prior to upload.

  • Select (Insert data) > Import bulk data.
  • Type the following into the Import Data text box, with a tab between fields. Include the header row. To enter comma-separated values, simply change the Format selection.
First NameLast NameID
JohnDoe1
JaneDoe2
JohnSmith3

  • Click Submit.
  • 3 data rows will be added to the list.

Import a File

Another way to upload data is to directly upload an .xlsx, .xls, .csv, or .txt file containing data. Toggle between the import data upload methods using the and buttons on the right as shown below:

Simultaneously Create and Populate a List from a Spreadsheet

If you want to both infer the fields to design the list and populate the new list with the data from the spreadsheet, follow this shortcut process:

  • Click Create New List.
  • Enter a name and choose the appropriate primary key information. If you specify a key field that is not present in the imported file, it will be added as an additional field.
  • Check the box to Import from File.
  • Click Browse or Choose File and select the file you downloaded.
  • You will see a preview of fields inferred - you may change types or labels here if needed.
  • Click Import.

Import Lookups By Alternate Key

When importing data into a list, you can use the checkbox to Import Lookups by Alternate Key. This allows lookup target rows to be resolved by values other than the target's primary key. It will only be available for lookups that are configured with unique column information. For example, tables in the "samples" schema (representing Sample Sets) use the RowId column as their primary key, but their Name column is guaranteed to be unique as well. Imported data can use either the primary key value (RowId) or the unique column value (Name). This is only supported for single-column unique indices. See Import Sample Sets.

View the List

Your list is now populated. You can see the contents of the list by clicking on the name of the list in the Lists web part. An example:

Hovering over a row will reveal icon buttons in the first column:

    • Edit the list item
    • View details about the list item.

Related Topics




Manage Lists


A list is a flexible, user-defined table. To manage lists, Select (Admin) > Manage Lists, or click Manage Lists in the Lists web part.

Manage Lists

An example list management page from an example HIV study:

  • (Grid Views): Customize how this grid is displayed.
  • Reports: Add a JavaScript or Crosstab Report about the grid of lists.
  • Create New List
  • (Delete): Select one or more lists using the checkboxes and click the button. Both the data and the list design are removed permanently from your server.
  • (Export)
  • Import List Archive
  • (Print): Print the grid of Lists.

Manage a Specific List

For each list shown in the Lists web part, you can:

  • View Design: View fields and properties that define the list, including allowable actions and indexing.
  • View History: See a record of all list events and design changes.
  • Click the Name of the list to see all contents of the list shown as a grid. Options offered include:
    • Create custom views and charts
    • Insert data into the list
    • Delete, export, or print the entire list.
    • Click Design to see the same information as when clicking View Design above.
    • Click Delete All Rows to empty the data from the list without actually deleting the list structure itself.

Related Topics:




Connect Lists


In the Tutorial: Lists, we provide an importable list archive as a shortcut to setting up multiple lists and lookups. This page describes how you could manually create the same archive from the same sample data package downloaded in the setup step.

Import Files into Lists

Create the lists individually by importing spreadsheets. If you already imported the tutorial list archive, you can skip this step, or create a new set of lists in another folder.

  • Click Manage Lists link in the Lists web part.
  • Click Create new list
    • Name: "NIMHDemographics"
    • Primary Key: "SubjectID"
    • Primary Key Type: Integer
    • Import from file: Check box.
  • Click Create List.
  • Browse or Choose File and locate [LabKeyDemoFiles]/Lists/NIMH/NIMHDemographics.xls.
  • Assume the inferred fields are correct and click Import.
  • Click Lists > above the grid and repeat the Create New List steps for each of the following:
    • Name: "NIMHPortions"
      • Primary Key: "PortionID"
      • Primary Key Type: Integer
      • Import from file: Check box.
      • Select and import the file named: NIMHPortions.xls
    • Name: "NIMHSamples"
      • Primary Key: "SampleID"
      • Primary Key Type: Integer
      • Import from file: Check box.
      • Select and import the file named: NIMHSamples.xls
    • Name: "NIMHSlides"
      • Primary Key: "SlideID"
      • Primary Key Type: Integer
      • Import from file: Check box.
      • Select and import the file named: NIMHSlides.xls

Set Up Lookups

There are columns in common between our lists that can be used as the basis for joins between these lists. In order to setup the lists for joins, we need to identify the columns in common.

The steps for editing a list design to make a field into a lookup of a value from another table are as follows. If you are already working with the tutorial list archive, simply examine the list designs where you will already see these changes.

  • In the Lists web part, click NIMHSamples.
  • Click Design then Edit Design.
  • Scroll down and for the SubjectID field, click on the Type property and select Lookup.
    • Folder: [current folder].
    • Schema: lists.
    • Table: NIMHDemographics(Integer).
    • Click Apply.
  • Scroll back up and click Save.

Repeat this process for each of the following lookups. You can add lookups to multiple fields of a given list at once before clicking Save for the list.

ListFieldTable for Lookup
NIMHPortionsSubjectIDNIMHDemographics(Integer)
NIMHPortionsSampleIDNIMHSamples(Integer)
NIMHSlidesSubjectIDNIMHDemographics(Integer)
NIMHSlidesSampleIDNIMHSamples(Integer)
NIMHSlidesPortionIDNIMHPortions(Integer)

If you view, for example, the NIMHSlides lists, you will now see hyperlinks in the SubjectID, SampleID and PortionID columns where you have set up lookups.

Export List Archive

Now that you have created a useful package of interconnected lists, you can package them as an archive for use elsewhere.

  • From the Lists web part, click Manage Lists.
  • Click Export List Archive.
  • All the lists in the web part will be included in the zip file that is downloaded.

Import a List Archive

Once you have created and exported an archive, you can reuse it later (as we did in the tutorial example) by importing the archive:

  • In the Lists web part, click Manage Lists.
  • Click Import List Archive.
  • Browse to and select the list archive: ListDemo_NIMH_ListArchive_XXXX.lists.zip in [LabKeyDemoFiles]/Lists/NIMH.
  • Click Import List Archive.




R Reports


You can leverage the full power of the R statistical programming environment to analyze and visualize datasets on LabKey Server. The results of R scripts can be displayed in LabKey reports that reflect live data updated every time the script is run. Reports may contain text, tables, or charts created using common image formats such as jpeg, png and gif. In addition, the Rlabkey package can be used to insert, update and/or delete data stored on a LabKey Server using R, provided you have sufficient permissions to do so.

An administrator must install and configure R on LabKey Server and grant access to users to create and run R scripts on live datasets. Loading of additional packages may also be necessary, as described in the installation topic. Configuration of multiple R engines on a server is possible, but within any folder only a single R engine configuration can be used.

Topics

Related Topics




R Report Builder


Creating R Reports requires that the user have both editor and developer access in the container. Learn more here: Developer Roles.

Create an R Report from a Data Grid

R reports are ordinarily associated with individual data grids. Choose the dataset of interest and further filter the grid as needed. Only the portion of the dataset visible within this data grid become part of the analyzed dataset.

To use the sample dataset we describe in this tutorial, please Tutorial: Set Up a New Study if you have not already done so. Alternately, you may simply add the PhysicalExam.xls demo dataset to an existing study for completing the tutorial. You may also work with your own dataset, in which case steps and screencaps will differ.

  • View the "Physical Exam" dataset in a LabKey study.
  • If you want to filter the dataset and thus select a subset or rearrangement of fields, select or create a custom grid view.
  • Select (Charts/Reports) > Create R Report.

If you do not see the "Create R Report" menu, check to see that R is installed and configured on your LabKey Server. You also need to have the correct permissions to create R Reports. See Configure Scripting Engines for more information.

Create an R Report Independent of any Data Grid

R reports do not necessarily need to be associated with individual data grids. You can also create an R report that is independent of any grid:

  • Select (Admin) > Manage Views.
  • Select Add Report > R Report.

R reports associated with a grid automatically load the grid data into the object "labkey.data". R reports created independently of grids do not have access to labkey.data objects. R reports that pull data from additional tables (other than the associated grid) must use the Rlabkey API to access the other table(s). For details on using Rlabkey, see Rlabkey Package. By default, R reports not associated with a grid are listed under the Uncategorized heading in the list on the Manage Views page.

Review the R Report Builder

The R report builder opens on the Source tab which looks like this. Enter the R script for execution or editing into the Script Source box. Notice the options available below the source entry panel, describe below.

Report Tab

When you select the Report tab, you'll see the resulting graphics and console output for your R report. If the pipeline option is not selected, the script will be run in batch mode on the server.

Data Tab

Select the data tab to see the data on which your R report is based. This can be a helpful resource as you write or refine your script.

Source Tab

When your script is complete and report is satisfactory, return to the Source tab, scroll down, and click Save to save both the script and the report you generated.

A saved report will look similar to the results in the design view tab, minus the help text. Reports are saved on the LabKey Server, not on your local file system. They can be accessed through the Reports drop-down menu on the grid view of you dataset, or directly from the Data Views web part.

The script used to create a saved report becomes available to source() in future scripts. Saved scripts are listed under the “Shared Scripts” section of the LabKey R report builder.

Help Tab

This Syntax Reference list provides a quick summary of the substitution parameters for LabKey R. See Input/Output Substitutions Reference for further details.

Additional Options

On the Source Tab you can expand additional option sections. Not all options are available to all users, based on permission roles granted.

  • Make this report available to all users: Enables other users to see your R report and source() its associated script if they have sufficient permissions. Only those with read privileges to the dataset can see your new report based on it.
    • If you choose to share your report with others, you may also opt to Show source tab to all users if desired.
  • Make this report available in child folders: Make your report available in data grids in child folders where the schema and table are the same as this data grid.
  • Run this report in the background as a pipeline job: Execute your script asynchronously using LabKey’s Pipeline module. If you have a big job, running it on a background thread will allow you to continue interacting with your server during execution.
If you choose the asynchronous option, you can see the status of your R report in the pipeline. Once you save your R report, you will be returned to the original data grid. From the Reports drop-down menu, select the report you just saved. This will bring up a page that shows the status of all pending pipeline jobs. Once your report finishes processing, you can click on “COMPLETE” next to your job. On the next page you’ll see "Job Status." Click on Data to see your report.

Note that reports are always generated from live data by re-running their associated scripts. This makes it particularly important to run computationally intensive scripts as pipeline jobs when their associated reports are regenerated often.

  • Knitr: select None, HTML, or Markdown processing of HTML source; include semicolon separated list of dependencies if needed.
  • Report Thumbnail: Choose to auto-generate a default thumbnail if desired. You can later edit the thumbnail or attach a custom image. See Manage Views.
  • Shared Scripts: Once you save a View, its associated script becomes available to execute using source(“<Script Name>.R”) in future scripts. Check the box next to the appropriate script to make it available for execution.
  • Study Options: Participant Chart: A participant chart shows measures for only one participant at a time. Select the participant chart checkbox if you would like this chart to be available for review participant-by-participant.
  • Study Options: Enable automatic caching of this report for faster reloading.

Example

Regardless of where you have accessed the R report builder, you can create a first R report which is data independent. This sample was adapted from the R help files.

  • Paste the following into the Source tab of the R report builder.
options(echo=TRUE);
# Execute 100 Bernoulli trials;
coin_flip_results = sample(c(0,1), 100, replace = TRUE);
coin_flip_results;
mean(coin_flip_results);
  • Click the Report tab to run the source and see your results, in this case the coin flip outcomes.

Echo to Console

By default, most R commands do not generate output to the console as part of your script. To enable output to console, use the following line at the start of your scripts:

options(echo=TRUE);

Note that when the results of functions are assigned, they are also not printed to the console. To see the output of a function, assign the output to a variable, then just call the variable. For further details, please see the FAQs for LabKey R Reports.

Related Topics




Saved R Reports


Saved R reports may be accessed from the source data grid or from the Data Views web part. Once saved, reports are generated by re-running their associated scripts on live data. This is a good thing because it produces updated current views, but it also requires computational resources each time the view is opened. If your script is computationally intensive, you would be wise to make sure it runs as a pipeline job such that it does not preoccupy your server when selected for viewing. See R Report Builder for details on how to set scripts up to run as background, pipeline jobs.

Edit an R Report Script

Open your saved R report by clicking the name in the data views web part or by selecting it from the (Charts and Reports) menu above the data grid on which it is based. This opens the R report builder interface on the Data tab. Select the Source tab to edit the script and manage other options. Click Save when finished.

Share an R Report

Saved R Reports can be shared with other users, either with all users of the folder, or individually with specific users.

If the report was saved with "Share this report with all users" checked, it is already available to any users with access to the view where it was created.

If the report was shared with the "Share" box unchecked, it is "private" but can still be explicitly shared with other individual users who have access to the source data. When sharing a report, you are indicating that you trust the recipient(s), and your recipients confirm that they trust you when they accept it. Sharing of R reports is audited and can be tracked in the "Study events" audit log.

  • Open an R Report, from the Data Views web part, and click the (Share Report) button.
  • Enter the Recipients email addresses, one per line.
  • The default Message Subject is "A report has been sent: <REPORT_TITLE>" but you may modify this title here if you like.
  • The default Message Body can also be customized as needed.
  • The Message Link is shown; you can click Preview Link to see what the recipient will see.
  • Click Submit to share the report. You will be taken to the permissions page.
  • On the Report and View Permissions page, you can see which groups and users have access to your otherwise "private" R report.

Recipients will receive a notification with a link to the report, so that they may view it. If the recipient has the proper permissions, they will also be able to edit and save their own copy of the report. If the author makes the source tab visible, recipients of a shared report will be able to see the source as well as the report contents. Note that if the recipient has a different set of permissions, they may see a different set of data. Modifications that the original report owner makes to the report will be reflected in the link as viewed by the recipient.

When an R report has been shared, the data browser will show access as "custom", indicating that it has been shared with specific users. Click custom to open the Report Permissions page, where you can see the list of groups and users with whom the report was shared.

Delete an R Report

You can delete a saved report by first clicking the pencil icon at the top of the Data Views web part, then click the pencil to the left of the report name. In the popup window, click Delete. You can also multi-select R reports for deletion on the Manage Views page.

Note that deleting a report eliminates its associated script from the "Shared Scripts" list in the R report interface. Make sure that you don’t delete a script that is called (sourced) by other scripts you need.

Related Topics




Datasets in R


Access Your Dataset as “labkey.data”

LabKey Server automatically reads your chosen dataset into a data frame called labkey.data; using Input Substitution.

A data frame can be visualized as a list with unique row names and columns of consistent lengths. Columns may also be named and their types may differ. You can see the column names for the labkey.data frame by calling:

options(echo=TRUE);
names(labkey.data);

Just like any other data.frame, data in a column of labkey.data can be referenced by the column’s name, preceded by a $:

labkey.data$<column name>

For example: labkey.data$pulse; provides all the data in the Pulse column of the Physical Exam sample dataset. Note that the examples in this section frequently include column names. If you are using your own data or a different version of LabKey sample data, you may need to retrieve column names and edit the code examples given.

Use Pre-existing R Scripts

To use a pre-existing R script with LabKey data, try the following procedure:

  • Open the R Report Builder. Open the dataset of interest ("Physical Exam" for example) and select > Create Chart.
  • Paste the script into the Source tab.
    • Identify the LabKey data columns that you want to be represented by the script, and load those columns into vectors. The following loads the Systolic Blood Pressure and Diastolic Blood Pressure columns into the vectors x and y:
x <- labkey.data$diastolicbloodpressure
y <- labkey.data$systolicbloodpressure

png(filename="${imgout:myscatterplot}", width = 650, height = 480);
plot(x, y, main="Scatterplot Example", xlab="X Axis ", ylab="Y Axis", pch=19)
abline(lm(y~x), col="red") # regression line (y~x)
  • Click the Report tab to see the result:

Find Simple Means

Once you have loaded your data, you can perform statistical analyses using the functions/algorithms in R and its associated packages. For example, calculate the mean Pulse for all participants.

options(echo=TRUE);
names(labkey.data);
labkey.data$pulse;
a <- mean(labkey.data$pulse, na.rm= TRUE);
a;

Find Means for Each Participant

The following simple script finds the average values of a variety of physiological measurements for each study participant.

# Get means for each participant over multiple visits;

options(echo=TRUE);
participant_means <- aggregate(labkey.data, list(ParticipantID = labkey.data$participantid), mean, na.rm = TRUE);
participant_means;

We use na.rm as an argument to aggregate in order to calculate means even when some values in a column are NA.

Create Functions in R

This script shows an example of how functions can be created and called in LabKey R scripts. Before you can run this script, the Cairo package must be installed on your server. See Install and Set Up R for instructions.

Note that the second line of this script creates a "data" copy of the input file, but removes all participant records that contain an NA entry. NA entries are common in study datasets and can complicate display results.

library(Cairo);
data= na.omit(labkey.data);

chart <- function(data)
{
plot(data$pulse, data$pulse);
};

filter <- function(value)
{
sub <- subset(labkey.data, labkey.data$participantid == value);
#print("the number of rows for participant id: ")
#print(value)
#print("is : ")
#print(sub)
chart(sub)
}

names(labkey.data);
Cairo(file="${imgout:a}", type="png");
layout(matrix(c(1:4), 2, 2, byrow=TRUE));
strand1 <- labkey.data[,1];
for (i in strand1)
{
#print(i)
value <- i
filter(value)
};
dev.off();

Access Data in Another Dataset

You can access data in another dataset (a dataset not loaded into labkey.data) through the Rlabkey library's selectRows, for example:

suppressMessages(library(Rlabkey))

mydata <- labkey.selectRows(
baseUrl="http://localhost:8080/labkey",
folderPath="/home/Demo Study",
schemaName="study.NAbAssay",
queryName="Data",
viewName="",
containerFilter=NULL)



Multi-Panel R Plots


The scripts on this page take the analysis techniques introduced in Datasets in R one step further, still using the Physical Exam sample dataset. This page covers a few more strategies for finding means, then shows how to graph these results and display least-squares regression lines.

Find Mean Values for Each Participant

Finding the mean value for physiological measurements for each participant across all visits can be done in various ways. Here, we cover three alternative methods.

For all methods, we use "na.rm=TRUE" as an argument to aggregate in order to ignore null values when we calculate means.

DescriptionCode
Aggregate each physiological measurement for each participant across all visits; produces an aggregated list with two columns for participantid.
data_means <- aggregate(labkey.data, list(ParticipantID = 
labkey.data$participantid), mean, na.rm = TRUE);
data_means;
Aggregate only the pulse column and display two columns: one listing participantIDs and the other listing mean values of the pulse column for each participant
aggregate(list(Pulse = labkey.data$pulse), 
list(ParticipantID = labkey.data$participantid), mean, na.rm = TRUE);
Again, aggregate only the pulse column, but here results are displayed as rows instead of two columns.
participantid_factor <- factor(labkey.data$participantid);
pulse_means <- tapply(labkey.data$pulse, participantid_factor,
mean, na.rm = TRUE);
pulse_means;

Create Single Plots

Next we use R to create plots of some other physiological measurements included in our sample data.

All scripts in this section use the Cairo package. To convert these scripts to use the png() function instead, eliminate the call "library(Cairo)", change the function name "Cairo" to "png," change the "file" argument to "filename," and eliminate the "type="png"" argument entirely.

Scatter Plot of All Diastolic vs All Systolic Blood Pressures

This script plots diastolic vs. systolic blood pressures without regard for participantIDs. It specifies the "ylim" parameter for plot() to ensure that the axes used for this graph match the next graph's axes, easing interpretation.

library(Cairo);
Cairo(file="${imgout:diastol_v_systol_figure.png}", type="png");
plot(labkey.data$diastolicbloodpressure, labkey.data$systolicbloodpressure,
main="R Report: Diastolic vs. Systolic Pressures: All Visits",
ylab="Systolic (mm Hg)", xlab="Diastolic (mm Hg)", ylim =c(60, 200));
abline(lsfit(labkey.data$diastolicbloodpressure, labkey.data$systolicbloodpressure));
dev.off();

The generated plot, where the identity of participants is ignored, might look like this:

Scatter Plot of Mean Diastolic vs Mean Systolic Blood Pressure for Each Participant

This script plots the mean diastolic and systolic blood pressure readings for each participant across all visits. To do this, we use "data_means," the mean value for each physiological measurement we calculated earlier on a participant-by-participant basis.

data_means <- aggregate(labkey.data, list(ParticipantID = 
labkey.data$participantid), mean, na.rm = TRUE);
library(Cairo);
Cairo(file="${imgout:diastol_v_systol_means_figure.png}", type="png");
plot(data_means$diastolicbloodpressure, data_means$systolicbloodpressure,
main="R Report: Diastolic vs. Systolic Pressures: Means",
ylab="Systolic (mm Hg)", xlab="Diastolic (mm Hg)", ylim =c(60, 200));
abline(lsfit(data_means$diastolicbloodpressure, data_means$systolicbloodpressure));
dev.off();

This time, the plotted regression line for diastolic vs. systolic pressures shows a non-zero slope. Looking at our data on a participant-by-participant basis provides insights that might be obscured when looking at all measurements in aggregate.

Create Multiple Plots

There are two ways to get multiple images to appear in the report produced by a single script.

Single Plot Per Report Section

The first and simplest method of putting multiple plots in the same report places separate graphs in separate sections of your report. Use separate pairs of device on/off calls (e.g., png() and dev.off()) for each plot you want to create. You have to make sure that the {imgout:} parameters are unique. Here's a simple example:

png(filename="${imgout:labkeyl_png}");
plot(c(rep(25,100), 26:75), c(1:100, rep(1, 50)), ylab= "L", xlab="LabKey",
xlim= c(0, 100), ylim=c(0, 100), main="LabKey in R: Report Section 1");
dev.off();

png(filename="${imgout:labkey2_png}");
plot(c(rep(25,100), 26:75), c(1:100, rep(1, 50)), ylab= "L", xlab="LabKey",
xlim= c(0, 100), ylim=c(0, 100), main="LabKey in R: Report Section 2");
dev.off();

Multiple Plots Per Report Section

There are various ways to place multiple plots in a single section of a report. Two examples are given here, the first using par() and the second using layout().

Example: Four Plots in a Single Section: Using par()

This script demonstrates how to put multiple plots on one figure to create a regression panel layout. It uses standard R libraries for the arrangement of plots, and Cairo for creation of the plot image itself. It creates a single graphics file but partitions the ‘surface’ of the image into multiple sections using the mfrow and mfcol arguments to par().

library(Cairo);
data_means <- aggregate(labkey.data, list(ParticipantID =
labkey.data$participantid), mean, na.rm = TRUE);
Cairo(file="${imgout:multiplot.png}", type="png")
op <- par(mfcol = c(2, 2)) # 2 x 2 pictures on one plot
c11 <- plot(data_means$diastolicbloodpressure, data_means$weight, ,
xlab="Diastolic Blood Pressure (mm Hg)", ylab="Weight (kg)",
mfg=c(1, 1))
abline(lsfit(data_means$diastolicbloodpressure, data_means$weight))
c21 <- plot(data_means$diastolicbloodpressure, data_means$systolicbloodpressure, ,
xlab="Diastolic Blood Pressure (mm Hg)",
ylab="Systolic Blood Pressure (mm Hg)", mfg= c(2, 1))
abline(lsfit(data_means$diastolicbloodpressure, data_means$systolicbloodpressure))
c21 <- plot(data_means$diastolicbloodpressure, data_means$pulse, ,
xlab="Diastolic Blood Pressure (mm Hg)",
ylab="Pulse Rate (Beats/Minute)", mfg= c(1, 2))
abline(lsfit(data_means$diastolicbloodpressure, data_means$pulse))
c21 <- plot(data_means$diastolicbloodpressure, data_means$temp, ,
xlab="Diastolic Blood Pressure (mm Hg)",
ylab="Temperature (Degrees C)", mfg= c(2, 2))
abline(lsfit(data_means$diastolicbloodpressure, data_means$temp))
par(op); #Restore graphics parameters
dev.off();

Example: Three Plots in a Single Section: Using layout()

This script uses the standard R libraries to display multiple plots in the same section of a report. It uses the layout() command to arrange multiple plots on a single graphics surface that is displayed in one section of the script's report.

The first plot shows blood pressure and weight progressing over time for all participants. The lower scatter plots graph blood pressure (diastolic and systolic) against weight.

library(Cairo);
Cairo(file="${imgout:a}", width=900, type="png");
layout(matrix(c(3,1,3,2), nrow=2));
plot(weight ~ systolicbloodpressure, data=labkey.data);
plot(weight ~ diastolicbloodpressure, data=labkey.data);
plot(labkey.data$date, labkey.data$systolicbloodpressure, xaxt="n",
col="red", type="n", pch=1);
points(systolicbloodpressure ~ date, data=labkey.data, pch=1, bg="light blue");
points(weight ~ date, data=labkey.data, pch=2, bg="light blue");
abline(v=labkey.data$date[3]);
legend("topright", legend=c("bpsys", "weight"), pch=c(1,2));
dev.off();

Related Topics




Lattice Plots


The Lattice package provides presentation-quality, multi-plot graphics. This page supplies a simple script to demonstrate the use of Lattice graphics in the LabKey R environment.

Before you can use the Lattice package, it must be installed on your server. You will load the lattice package at the start of every script that uses it:

library("lattice");

Display a Volcano

The Lattice Documentation provides a Volcano script to demonstrate the power of Lattice. The script below has been modified to work on LabKey R:

library("lattice");  

p1 <- wireframe(volcano, shade = TRUE, aspect = c(61/87, 0.4),
light.source = c(10,0,10), zlab=list(rot=90, label="Up"),
ylab= "North", xlab="East", main="The Lattice Volcano");
g <- expand.grid(x = 1:10, y = 5:15, gr = 1:2);
g$z <- log((g$x^g$g + g$y^2) * g$gr);

p2 <- wireframe(z ~ x * y, data = g, groups = gr,
scales = list(arrows = FALSE),
drape = TRUE, colorkey = TRUE,
screen = list(z = 30, x = -60));

png(filename="${imgout:a}", width=500);
print(p1);
dev.off();

png(filename="${imgout:b}", width=500);
print(p2);
dev.off();

The report produced by this script will display two graphs that look like the following:

Related Topics




Participant Charts in R


You can use the Participant Chart checkbox in the R Report Builder to create charts that display results on a participant-by-participant basis.

Create and View Simple Participant Charts

  • Open the "Physical Exam" dataset in a demo study.
  • Select (Reports) > Create R Report.
  • On the Source tab, begin with a script that shows data for all participants. Paste the following in place of the default content.
png(filename="${imgout:a}", width=900);
plot(labkey.data$systolicbloodpressure, labkey.data$date);
dev.off();
  • Click the Report tab to view the scatter plot data for all participants.
  • Return to the Source tab.
  • Scroll down and click the triangle to open the Study Options section.
  • Check Participant Chart.
  • Click Save.
  • Name your report "Participant Systolic" or another name you choose.

The participant chart option subsets the data that is handed to an R script by filtering on a participant ID. You can later step through per participant charts using this option. The labkey.data dataframe may contain one, or more rows of data depending on the content of the dataset you are working with. Next, reopen the R report:

  • Return to the data grid of the "Physical Exam" dataset.
  • Select (Reports) > Participant Systolic (or the name you gave your report).
  • Click Previous Participant.
  • You will see Next Participant and Previous Participant links that let you step through charts for each participant:

Advanced Example: Create Participant Charts Using Lattice

You can create a panel of charts for participants using the lattice package. If you select the participant chart option on the source tab, you will be able to see each participant's panel individually when you select the report from your data grid.

The following script produces lattice graphs for each participant showing systolic blood pressure over time:

library(lattice);
png(filename="${imgout:a}", width=900);
plot.new();
xyplot(systolicbloodpressure ~ date| participantid, data=labkey.data,
type="a", scales=list(draw=FALSE));
update(trellis.last.object(),
strip = strip.custom(strip.names = FALSE, strip.levels = TRUE),
main = "Systolic over time grouped by participant",
ylab="Systolic BP", xlab="");
dev.off();

The following script produces lattice graphics for each participant showing systolic and diastolic blood pressure over time (points instead of lines):

library(lattice);
png(filename="${imgout:b}", width=900);
plot.new();

xyplot(systolicbloodpressure + diastolicbloodpressure ~ date | participantid,
data=labkey.data, type="p", scales=list(draw=FALSE));
update(trellis.last.object(),
strip = strip.custom(strip.names = FALSE, strip.levels = TRUE),
main = "Systolic & Diastolic over time grouped by participant",
ylab="Systolic/Diastolic BP", xlab="");
dev.off();

After you save these two R reports with descriptive names, you can go back and review individual graphs participant-by-participant. Use the (Reports) menu available on your data grid.




R Reports with knitr


In this topic: Interweave R code into either HTML or Markdown pages to create dynamic reports using the knitr visualization package.

Install R and knitr

  • If you haven't already installed R, follow these instructions: Install R.
  • Open the R graphical user interface. On Windows, a typical location would be: C:\Program Files\R\R-3.0.2\bin\i386\Rgui.exe
  • Select Packages > Install package(s).... Select a mirror site, and select the knitr package.
    • OR
  • Enter the following: install.packages('knitr', dependencies=TRUE)
  • Select a mirror site and wait for the knitr installation to complete.

Develop knitr Reports

  • Go to the dataset you wish to visualize.
  • Select (Charts/Reports) > Create R Report.
  • On the Source tab, enter your HTML or Markdown page with knitr code. (Scroll down for example pages.)
  • Specify which source to process with knitr. Under knitr Options, select HTML or Markdown.
  • Select the Report tab to see the results.

R/knitr Scripts in Modules

R script knitr reports are also available as custom module reports. The script file must have either a .rhtml or .rmd extension, for HTML or markdown documents, respectively. For a file-based module, place the .rhtml/.rmd file in the same location as .r files, as shown below. For module details, see Map of Module Files.

MODULE_NAME
reports/
schemas/
SCHEMA_NAME/
QUERY_NAME/
MyRScript.r -- R report
MyRScript.rhtml -- R/knitr report
MyRScript.rmd -- R/knitr report

Declaring Script Dependencies

To fully utilize the report designer (called the "R Report Builder" in the LabKey user interface), you can declare JavaScript or CSS dependencies for knitr reports. This ensures that the dependencies are downloaded before R scripts are run on the "reports" tab in the designer. If these dependencies are not specified then any JavaScript in the knitr report may not run correctly in the context of the script designer. Note that reports that are run in the context of the Reports web part will still render correctly without needing to explicitly define dependencies.

Reports can either be created via the LabKey Server UI in the report designer directly or included as files in a module. Reports created in the UI are editable via the Source tab of the designer. Open Knitr Options to see a text box where a semi-colon delimited list of dependencies can be entered. Dependencies can be external (via HTTP) or local references relative to the labkeyWebapp path on the server. In addition, the name of a client library may be used. If the reference does not have a .js or .css extension then it will be assumed to be a client library (somelibrary.lib.xml). The .lib.xml extension is not required. Like local references, the path to the client library is relative to the labkeyWebapp path.

File based reports in a module cannot be edited in the designer although the "source" tab will display them. However you can still add a dependencies list via the report's metadata file. Dependencies can be added to these reports by including a <dependencies> section underneath the <R> element. A sample metadata file:

<?xml version="1.0" encoding="UTF-8"?>
<ReportDescriptor xmlns="http://labkey.org/query/xml">
<label>My Knitr Report</label>
<description>Relies on dependencies to display in the designer correctly.</description>
<reportType>
<R>
<dependencies>
<dependency path="http://external.com/jquery/jquery-1.9.0.min.js"/>
<dependency path="knitr/local.js"/>
<dependency path="knitr/local.css"/>
</dependencies>
</R>
</reportType>
</ReportDescriptor>

The metadata file must be named <reportname>.report.xml and be placed alongside the report of the same name under (modulename/resources/reports/schemas/...).

HTML Example

To use this example:

  • Install the R package ggplot2
  • Install the Demo Study.
  • Create an R report on the dataset "Physical Exam"
  • Copy and paste the knitr code below into the Source tab of the R Report Builder.
  • Scroll down to the Knitr Options node, open the node, and select HTML.
  • Click the Report tab to see the knitr report.
<table>
<tr>
<td align='center'>
<h2>Scatter Plot: Blood Pressure</h2>
<!--begin.rcode echo=FALSE, warning=FALSE
library(ggplot2);
opts_chunk$set(fig.width=10, fig.height=6)
end.rcode-->
<!--begin.rcode blood-pressure-scatter, warning=FALSE, message=FALSE, echo=FALSE, fig.align='center'
qplot(labkey.data$diastolicbloodpressure, labkey.data$systolicbloodpressure,
main="Diastolic vs. Systolic Pressures: All Visits",
ylab="Systolic (mm Hg)", xlab="Diastolic (mm Hg)", ylim =c(60, 200), xlim=c(60,120), color=labkey.data$temp);
end.rcode-->
</td>
<td align='center'>
<h2>Scatter Plot: Body Temp vs. Body Weight</h2>
<!--begin.rcode temp-weight-scatter, warning=FALSE, message=FALSE, echo=FALSE, fig.align='center'
qplot(labkey.data$temp, labkey.data$weight,
main="Body Temp vs. Body Weight: All Visits",
xlab="Body Temp (C)", ylab="Body Weight (kg)", xlim=c(35,40), color=labkey.data$height);
end.rcode-->
</td>
</tr>
</table>

The rendered knitr report:

Markdown v2

Administrators can enable Markdown v2 when enlisting an R engine through the Views and Scripting Configuration page. When enabled, Markdown v2 will be used when rendering knitr R reports. If not enabled, Markdown v1 is used to execute the reports.

Independent installation is required of the following:

This will then enable using the Rmarkdown v2 syntax for R reports. The system does not currently perform any verification of the user's setup. If the configuration is enabled when enlisting the R engine, but the packages are not properly setup, the intended report rendering will fail.

Syntax differences are noted here: http://rmarkdown.rstudio.com/authoring_migrating_from_v1.html

Markdown v1 Example

Scatter Plot: Blood Pressure
----------------------------

>The chart below shows data from all participants

```{r setup, echo=FALSE}
# set global chunk options: images will be 7x5 inches
opts_chunk$set(fig.width=7, fig.height=5)
```

```{r graphic1, echo=FALSE}
plot(labkey.data$diastolicbloodpressure, labkey.data$systolicbloodpressure,
main="Diastolic vs. Systolic Pressures: All Visits",
ylab="Systolic (mm Hg)", xlab="Diastolic (mm Hg)", ylim =c(60, 200));
abline(lsfit(labkey.data$diastolicbloodpressure, labkey.data$systolicbloodpressure));
```

Scatter Plot: Body Temp vs. Body Weight
---------------------------------------

>The chart below shows data from all participants.

```{r graphic2, echo=FALSE}
plot(labkey.data$temp, labkey.data$weight,
main="Temp vs. Weight",
xlab="Body Temp (C)", ylab="Body Weight (kg)", xlim=c(35,40));
```

Related Links




Input/Output Substitutions Reference


Your R script uses input substitution parameters to generate the names of input files and to import data from your chosen data grid. It then uses output substitution parameters to either directly place image/data files in your report or to include download links to these files. Substitutions take the form of: ${param} where 'param' is the substitution. You can find the substitution syntax directly in the R Report UI by selecting the Help tab.

Input and Output Substitution Parameters

Valid Substitutions: 
input_data: <name>The input datset, a tab-delimited table. LabKey Server automatically reads your input dataset (a tab-delimited table) into the data frame called labkey.data. If you desire tighter control over the method of data upload, you can perform the data table upload yourself. The 'input data:' prefix indicates that the data file for the grid and the <name> substitution can be set to any non-empty value:
# ${labkey.data:inputTsv}
labkey.data <- read.table("inputTsv", header=TRUE, sep="\t");
labkey.data
imgout: <name>An image output file (such as jpg, png, etc.) that will be displayed as a Section of a View on LabKey Server. The 'imgout:' prefix indicates that the output file is an image and the <name> substitution identifies the unique image produced after you call dev.off(). The following script displays a .png image in a View:
# ${imgout:labkey1.png}
png(filename="labkeyl_png")
plot(c(rep(25,100), 26:75), c(1:100, rep(1, 50)), ylab= "L", xlab="LabKey",
xlim= c(0, 100), ylim=c(0, 100), main="LabKey in R")
dev.off()
tsvout: <name>A TSV text file that is displayed on LabKey Server as a section within a report. No downloadable file is created. For example:
# ${tsvout:tsvfile}
write.table(labkey.data, file = "tsvfile", sep = "\t",
qmethod = "double", col.names="NA")
txtout: <name>A text file that is displayed on LabKey Server as a section within a report. No downloadable file is created. For example:
# ${txtout:tsvfile}
write.csv(labkey.data, file = "csvfile")
pdfout: <name>A PDF output file that can be downloaded from LabKey Server. The 'pdfout:' prefix indicates that he expected output is a pdf file. The <name> substitution identifies the unique file produced after you call dev.off().
# ${pdfout:labkey1.pdf}
pdf(file="labkeyl_pdf")
plot(c(rep(25,100), 26:75), c(1:100, rep(1, 50)), ylab= "L", xlab="LabKey",
xlim= c(0, 100), ylim=c(0, 100), main="LabKey in R")
dev.off()
psout: <name>A postscript output file that can be downloaded from LabKey Server. The 'psout:' prefix indicates that the expected output is a postscript file. The <name> substitution identifies the unique file produced after you call dev.off().
# ${psout:labkeyl.eps}
postscript(file="labkeyl.eps", horizontal=FALSE, onefile=FALSE)
plot(c(rep(25,100), 26:75), c(1:100, rep(1, 50)), ylab= "L", xlab="LabKey",
xlim= c(0, 100), ylim=c(0, 100), main="LabKey in R")
dev.off()
fileout: <name>A file output that can be downloaded from LabKey Server, and may be of any file type. For example, use fileout in the place of tsvout to allow users to download a TSV instead of seeing it within the page:
# ${fileout:tsvfile}
write.table(labkey.data, file = "tsvfile", sep = "\t",
qmethod = "double", col.names=NA)
htmlout: <name>A text file that is displayed on LabKey Server as a section within a View. The output is different from the txtout: replacement in that no html escaping is done. This is useful when you have a report that produces html output. No downloadable file is created:
txt <- paste("<i>Click on the link to visit LabKey:</i>
<a target='blank' href='http://www.labkey.org'>LabKey</a>"
)
# ${htmlout:output}
write(txt, file="output")
svgout: <name>An svg file that is displayed on LabKey Server as a section within a View. htmlout can be used to render svg outputs as well, however, using svgout will generate a more appropriate thumbnail image for the report. No downloadable file is created:
# ${svgout:output.svg}
svg("output.svg", width= 4, height=3)
plot(x=1:10,y=(1:10)^2, type='b')
dev.off()

Implicit Variables

Each R script contains implicit variables that are inserted before your source script. Implicit variables are R data types and may contain information that can be used by the source script.

Implicit variables: 
labkey.dataThe data frame which the input dataset is automatically read into. The code to generate the data frame is:
# ${input_data:inputFileTsv} 
labkey.data <- read.table("inputFileTsv", header=TRUE, sep="\t",
quote="", comment.char="")
labkey.url.pathThe path portion of the current URL which omits the base context path, action and URL parameters. The path portion of the URL: http://localhost:8080/labkey/study/home/test/begin.view would be: /home/test/
labkey.url.baseThe base portion of the current URL. The base portion of the URL: http://localhost:8080/labkey/study/home/test/begin.view would be: http://localhost:8080/labkey/
labkey.url.paramsThe list of parameters on the current URL. The parameters are represented as a list of key / value pairs.
labkey.user.emailThe email address of the current user

Using Regular Expressions with Replacement Token Names

Sometimes it can be useful to have flexibility when binding token names to replacement parameters. This can be the case when a script generates file artifacts but does not know the file names in advance. Using the syntax: regex() in the place of a token name (where LabKey server controls the token name to file mapping) will result the following actions:

  • Any script generated files not mapped to a replacement will be evaluated against the file's name using the regex.
  • If a file matches the regex, it will be assigned to the replacement and rendered accordingly.
<replacement>:regex(<expression>)The following example will find all files generated by the script with the extension : '.gct'. If any are found they will be assigned and rendered to the replacement parameter (in this case as a download link).//
#${fileout:regex(.*?(\.gct))}

Cairo or GDD Packages

You may need to use the Cairo or GDD graphics packages in the place of jpeg() and png() if your LabKey Server runs on a "headless" Unix server. You will need to make sure that the appropriate package is installed in R and loaded by your script before calling either of these functions.

GDD() and Cairo() Examples. If you are using GDD or Cairo, you might use the following scripts instead:

library(Cairo);
Cairo(file="${imgout:labkeyl_cairo.png}", type="png");
plot(c(rep(25,100), 26:75), c(1:100, rep(1, 50)), ylab= "L", xlab="LabKey",
xlim= c(0, 100), ylim=c(0, 100), main="LabKey in R");
dev.off();

library(GDD);
GDD(file="${imgout:labkeyl_gdd.jpg}", type="jpeg");
plot(c(rep(25,100), 26:75), c(1:100, rep(1, 50)), ylab= "L", xlab="LabKey",
xlim= c(0, 100), ylim=c(0, 100), main="LabKey in R");
dev.off();

Deprecated LabKey-specific Syntax

Prior to release 18.1, file substitutions used a LabKey-specific syntax. The use of standard R syntax expected by RStudio as described above means the following syntax is deprecated and should be updated. You can also find this syntax within the R Report UI by selecting the Help tab, then Inline Syntax (Deprecated).

(Deprecated) Valid Substitutions: 
input_dataLabKey Server automatically reads your input dataset (a tab-delimited table) into the data frame called labkey.data. For tighter control over the method of data upload, or to modify the parameters of the read.table function, you can perform the data table upload yourself:
labkey.data <- read.table("${input_data}", header=TRUE);
labkey.data;
imgout: <name>An image output file (such as jpg, png, etc.) that will be displayed as a Section of a report on LabKey Server. The 'imgout:' prefix indicates that the output file is an image and the <name> substitution identifies the unique image produced after you call dev.off(). The following script displays a .png image in a report:
png(filename="${imgout:labkeyl_png}");
plot(c(rep(25,100), 26:75), c(1:100, rep(1, 50)), ylab= "L", xlab="LabKey",
xlim= c(0, 100), ylim=c(0, 100), main="LabKey in R");
dev.off();
tsvout: <name>A TSV text file that is displayed on LabKey Server as a section within a report. No downloadable file is created. For example:
write.table(labkey.data, file = "${tsvout:tsvfile}", sep = "\t", 
qmethod = "double");
txtout: <name>A text file that is displayed on LabKey Server as a section within a report. No downloadable file is created. A CSV example:
write.csv(labkey.data, file = "${txtout:csvfile}");
pdfout: <name>A PDF output file that can be downloaded from LabKey Server.
pdf(file="${pdfout:labkeyl_pdf}");
plot(c(rep(25,100), 26:75), c(1:100, rep(1, 50)), ylab= "L", xlab="LabKey",
xlim= c(0, 100), ylim=c(0, 100), main="LabKey in R");
dev.off();
psout: <name>A postscript output file that can be downloaded from LabKey Server.
postscript(file="${psout:labkeyl_eps}", horizontal=FALSE, onefile=FALSE);
plot(c(rep(25,100), 26:75), c(1:100, rep(1, 50)), ylab= "L", xlab="LabKey",
xlim= c(0, 100), ylim=c(0, 100), main="LabKey in R");
dev.off();
fileout: <name>A file output that can be downloaded from LabKey Server, and may be of any file type. For example, use fileout in the place of tsvout to allow users to download a TSV instead of seeing it within the page:
write.table(labkey.data, file = "${fileout:tsvfile}", sep = "\t", qmethod = "double", col.names=NA);
 Another example shows how to send the output of the console to a file:
options(echo=TRUE);
sink(file = "${fileout:consoleoutput.txt}");
labkey.data;
htmlout: <name>A text file that is displayed on LabKey Server as a section within a report. The output is different from the txtout: replacement in that no html escaping is done. This is useful when you have a report that produces html output. No downloadable file is created:
txt <- paste("<i>Click on the link to visit LabKey:</i>
<a target='blank' href='http://www.labkey.org'>LabKey</a>"
)
write(txt, file="${htmlout:output}");
svgout: <name>An svg file that is displayed on LabKey Server as a section within a report. htmlout can be used to render svg outputs as well, however, using svgout will generate a more appropriate thumbnail image for the report. No downloadable file is created:
svg("${svgout:svg}", width= 4, height=3)
plot(x=1:10,y=(1:10)^2, type='b')
dev.off()

(Deprecated) Implicit Variables: 
labkey.dataThe data frame which the input dataset is automatically read into. The code to generate the data frame is:
labkey.data <- read.table("${input_data}", header=TRUE, sep="\t",
quote="", comment.char="")

Additional Reference

Documentation and tutorials about the R language can be found at the R Project website.




FAQs for LabKey R Reports


Overview

This page aims to answer common questions about configuring and using the LabKey Server interface for creating R Reports. Remember, an administrator must install and configure R on LabKey Server before users can create and run R scripts on live datasets.

Topics:

  1. library(), help() and data() don’t work
  2. plot() doesn’t work
  3. jpeg() and png() don’t work
  4. Does my report reflect live, updated data?
  5. Output is not printed when I source() a file or use a function
  6. Scripts pasted from documentation don't work in the LabKey R Script Builder
  7. LabKey Server becomes very, very slow when scripts execute
  8. Does R create security risks?
  9. Any good sources for advice on R scripting?

1. library(), help() and data() don’t work

LabKey Server runs R scripts in batch mode. Thus, on Windows machines it does not display the pop-up windows you would ordinarily see in R’s interpreted/interactive mode. Some functions that produce pop-ups (e.g., library()) have alternatives that output to the console. Some functions (e.g., help() and some forms of data()) do not.

Windows Workaround #1: Use alternatives that output to the console

library(): The library() command has a console-output alternative. To see which packages your administrator has made available, use the following:

installed.packages()[,0]
Windows Workaround #2: Call the function from a native R window

help(): It’s usually easy to keep a separate, native R session open and call help() from there. This works better for some functions than others. Note that you must install and load packages before asking for help() with them. You can also use the web-based documentation available on CRAN or search the R mailing list for help.

data(): You can also call data() from a separate, native R session for some purposes. Calling data() from such a session can tell you which datasets are available on any packages you’ve installed and loaded in that instance of R, but not your LabKey installation.

2. plot() doesn’t work

Did you open a graphics device before calling plot()?

LabKey Server executes R scripts in batch mode. Thus, LabKey R never automatically opens an appropriate graphics device for output, as would R when running in interpreted/interactive mode. You’ll need to open the appropriate device yourself. For onscreen output that becomes part of a report, use jpeg() or png() (or their alternatives, Cairo(), GDD() and bitmap()). In order to output a graphic as a separate file, use pdf() or postscript().

Did you call dev.off() after plotting?

You need to call dev.off() when you’re done plotting to make sure the plot object gets printed to the open device.

3. jpeg() and png() don’t work

R is likely running in a headless Unix server. On a headless Unix server, R does not have access to the appropriate X11 drivers for the jpeg() and png() functions. Your admin can install a display buffer on your server to avoid this problem. Otherwise, in each script you will need to load the appropriate package to create these file formats via other functions (e.g., GDD or Cairo). See also: Determine Available Graphing Functions for help getting unstuck.

4. Does my report reflect live, updated data?

Yes. In general, LabKey always re-runs your saved script before displaying its associated report. Your script operates on live, updated data, so its plots and tables reflect fresh data.

In study folders, you can set a flag for any script that prevents the script from being re-run unless changes have occurred. This flag can save time when scripts are time-intensive or datasets are large making processing slow. When this flag is set, LabKey will only re-run the R script if:

  • The flag is cleared OR
  • The dataset associated with the script has changed OR
  • Any of the attributes associated with the script are changed (script source, options etc.)
To set the flag, check the "Automatically cache this report for faster reloading" checkbox under "Study Options" on the Source tab of the R report builder.

5. Output is not printed when I source() a file or use a function

The R FAQ explains:

When you use… functions interactively at the command line, the result is automatically printed...In source() or inside your own functions you will need an explicit print() statement.

When a command is executed as part of a file that is sourced, the command is evaluated but its results are not ordinarily printed. For example, if you call source(scriptname.R) and scriptname.R calls installed.packages()[,0] , the installed.packages()[,0] command is evaluated, but its results are not ordinarily printed. The same thing would happen if you called installed.packages()[,0] from inside a function you define in your R script.

You can force sourced scripts to print the results of the functions they call. The R FAQ explains:

If you type `1+1' or `summary(glm(y~x+z, family=binomial))' at the command line the returned value is automatically printed (unless it is invisible()). In other circumstances, such as in a source()'ed file or inside a function, it isn't printed unless you specifically print it.
To print the value 1+1, use
print(1+1);
or, instead, use
source("1plus1.R", echo=TRUE);
where "1plus1.R" is a shared, saved script that includes the line "1+1".

6. Scripts pasted from documentation don't work in the LabKey R report builder

If you receive an error like this:

Error: syntax error, unexpected SYMBOL, expecting 'n' or ';'
in "library(Cairo) labkey.data"
Execution halted
please check your script for missing line breaks. Line breaks are known to be unpredictably eliminated during cut/paste into the script builder. This issue can be eliminated by ensuring that all scripts have a ";" at the end of each line.

7. LabKey Server becomes very, very slow when scripts execute

You are probably running long, computationally intensive scripts. To avoid a slowdown, run your script in the background via the LabKey pipeline. See R Report Builder for details on how to execute scripts via the pipeline.

8. Does R Create Security Risks?

Allowing the use of R scripts/reports on a server can be a security risk. A developer could write a script that could read or write any file stored in any SiteRoot, fileroot or pipeline root despite the LabKey security settings for that file.

A user must have at least the Author role and be part of the Developer group to write a R script or report to be used on the server.

R should not be used on a "shared server", that is, a server where users with admin/developer privileges in one project do not have permissions on other projects. Running R on the server could pose a security threat if the user attempts to access the server command line directly. The main way to execute a system command in R is via the 'system(<system call>)' method that is part of the R core package. The threat is due to the permission level of a script being run by the server possibly giving unwanted elevated permissions to the user.

9. Any good sources for advice on R scripting?

R Graphics Basics: Plot area, mar (margins), oma (outer margin area), mfrow, mfcol (multiple figures)

  • Provides good advice how to make plots look spiffy
R graphics overview
  • This powerpoint provides nice visuals for explaining various graphics parameters in R
Bioconductor course materials
  • Lectures and labs cover the range - from introductory R to advanced genomic analysis
Statistical R graphics overview
  • Also links to useful example figures and code from several R books

10. Graphics File Formats

If you don’t know which graphics file format to use for your plots, this link can help you narrow down your options.

.png and .gif

Graphics shared over the web do best in png when they contain regions of monotones with hard edges (e.g., typical line graphs). The .gif format also works well in such scenarios, but it is not supported in the default R installation because of patent issues. The GDD package allows you to create gifs in R.

.jpeg

Pictures with gradually varying tones (e.g., photographs) are successfully packaged in the jpeg format for use on the web.

.pdf and .ps or .eps

Use pdf or postscript when you aim to output a graph that can be accessed in isolation from your R report.



RStudio Integration


Premium Feature — This feature is part of the Professional Plus and Enterprise Editions. Learn more or contact LabKey for set up assistance.

The RStudio module lets you design and save reports using RStudio, instead of LabKey Server's built in R report designer.

There are two options for using RStudio with LabKey.

Once either RStudio or RStudio Pro is configured and enabled using one of the above methods, you can use it to edit R reports. You can also directly export data into a named variable. Learn more in these topics:

Related Topics




RStudio Set Up - Linux and OSX Systems


Premium Feature — This feature supports RStudio, which is part of the Professional Plus and Enterprise Editions. Learn more or contact LabKey for details.

This topic explains how administrators can set up communication between LabKey Server and an RStudio instance packaged inside a Docker image. As part of the configuration, we recommend that you allow Docker, not LabKey Server, to manage the RStudio data using Docker Volumes. This avoids a number of issues with file sharing, permissions, path mapping, and security, making setup much easier.

Install Docker

Connect Using TCP

Set up TLS following these steps:

  • See https://docs.docker.com/engine/security/https/
  • An example TLS and certificate configuration for LabKey Server is available here.
  • After certificates are set up, find the values of the following environment variables. You will use these values in the Configure Docker step below. (On Linux call 'run env | grep DOCKER'.)
    • DOCKER_CERT_PATH
    • DOCKER_HOST
  • On an Ubuntu system modify the "/etc/default/docker" file to include the following:
DOCKER_TLS_VERIFY=1

DOCKER_HOST=tcp://$HOST:2376

DOCKER_CERT_PATH=<YOUR_CERT_PATH>

DOCKER_OPTS="--dns 8.8.8.8 --tlsverify --tlscacert=$DOCKER_CERT_PATH/ca.pem --tlscert=$DOCKER_CERT_PATH/server-cert.pem --tlskey=$DOCKER_CERT_PATH/server-key.pem -H=0.0.0.0:2376"

Build the LabKey/RStudio Image

cd docker-rstudio/images/labkey/rstudio
  • Build the image from the source files:
./make
  • Test by running the following:
docker run labkey/rstudio
  • Ctrl-C to get to return to the command prompt once you see the 'done' message.
  • Run the following to get the container name:
docker ps
  • To stop the container, run:
docker stop <container-name>

Enable Session Keys

Configure Docker

For details, see Configure Docker Host.

Configure RStudio

When configuring RStudio, you can skip the prerequisite check for user home directories. The configuration page will show a list of existing volumes and corresponding user emails. The email field is blank if the volume doesn't correspond to an existing user.

  • Go to > Site > Admin Console.
  • Click Admin Console Links.
  • Under Premium Features click RStudio Settings.
  • If you have successfully created the Docker image, it will be listed under Docker Volumes.
  • Configure the remaining fields as appropriate for your environment, and click Save.
    • Docker Image Name - labkey/rstudio (or a custom image derived from labkey/rstudio)
    • Port = 8787
    • Mount Volume = /home/rstudio
    • Host library directory - leave blank
    • User for Containers - leave blank
    • AppArmor Profile - leave blank
    • Local IP address from LabKey Server - set to the IP address of the labkey server from within your docker container. This may be left blank if the normal DNS lookup works.
    • Resource Configuration: Customize defaults if needed.

Enable RStudio in Folders

The modules docker and rstudio must be enabled in each folder where you want RStudio to be available. Select (Admin) > Folder > Management, select the Folder Type tab and make sure both modules are checked in the right column. Click Update Folder to save any changes.

Once RStudio is enabled, you can use the instructions in these topics to edit reports and export data:

Related Topics




Docker and TLS Set Up


Premium Feature — Docker supports RStudio, which is part of the Professional Plus and Enterprise Editions. Learn more or contact LabKey for details.

This topic outlines an example TLS and certificate configuration used for LabKey Server.

Installation Instructions for Docker Daemon

First, identify where the server will run Docker Daemon on the Test Environment

The Docker documentation includes the following recommendation: "...if you run Docker on a server, it is recommended to run exclusively Docker in the server, and move all other services within containers controlled by Docker. Of course, it is fine to keep your favorite admin tools (probably at least an SSH server), as well as existing monitoring/supervision processes (e.g., NRPE, collectd, etc)."

The options available include:

  1. Create New Instance in Test Environment for the Docker Daemon. This option is preferred in an ideal world, as it is the most secure and follows all Docker best practices. If a runaway RStudio process overwhelms the instances resources it will not affect Rserve and the Web Server processes and thus the other applications will continue to function properly.
  2. Install and Run Docker Daemon on the Rserve instance. This allows quick installation and configuration of Docker in the Test Environment.
  3. Install and Run Docker Daemon on the Web Server.
For the Test Environment option #2 is recommended as this allows us to:
    • Test the RStudio features using a configuration similar to what will be used in Production
    • If during the testing, we begin to see resource contention between RStudio and Rserve processes, we can change the Docker Daemon configuration to limit resources used by RStudio or request a new instance to run Docker.
    • If a runaway RStudio process overwhelms the instances resources it will not affect the Web Server processes and thus the other applications will continue to function properly

Install the Docker Daemon

Install Docker Daemon on the Rserve instance by running the following commands

sudo apt-get update 
sudo apt-get install apt-transport-https ca-certificates
sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D

sudo vi /etc/apt/sources.list.d/docker.list
[Added]
deb https://apt.dockerproject.org/repo ubuntu-trusty main

sudo apt-get update
sudo apt-get purge lxc-docker
sudo apt-get install linux-image-extra-$(uname -r) linux-image-extra-virtual
sudo apt-get install docker-engine

IMPORTANT: Do not add any users to the docker group. (members of Docker group can access dockerd Linux socket.

Create TLS Certificates

The TLS certificates are used by the LabKey Server to authenticate to the Docker Daemon process.

Create the directory that will hold the CA certificate/key and the Client certificate/key. You can use a different directory if you want than the one shown below this is the value of "DOCKER_CERT_PATH"

sudo su - 
mkdir -p /labkey/apps/ssl
chmod 700 /labkey/apps/ssl

Create the Certificate Authority private key and certificate

Create the CA key CA certificate. Configure the certificate to expire in 10 years.

openssl genrsa -out /labkey/apps/ssl/ca-key.pem 4096
openssl req -x509 -new -nodes -key /labkey/apps/ssl/ca-key.pem -days 3650 -out /labkey/apps/ssl/ca.pem -subj '/CN=docker-CA'

Create openssl.cnf configuration file to be used by the CA.

vi /labkey/apps/ssl/openssl.cnf 
[added]

[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth

Create the Client TLS certificate

Create the client certificates and key. The client certificate will be good for 10 years.

openssl genrsa -out /labkey/apps/ssl/client-key.pem 4096
openssl req -new -key /labkey/apps/ssl/client-key.pem -out /labkey/apps/ssl/client-cert.csr -subj '/CN=docker-client' -config /labkey/apps/ssl/openssl.cnf
openssl x509 -req -in /labkey/apps/ssl/client-cert.csr -CA /labkey/apps/ssl/ca.pem -CAkey /labkey/apps/ssl/ca-key.pem -CAcreateserial -out /labkey/apps/ssl/client-cert.pem -days 3650 -extensions v3_req -extfile /labkey/apps/ssl/openssl.cnf

Create the TLS certificates to be used by the Docker Daemon

Create the directory from which the docker process will read the TLS certificate and key files.

mkdir /etc/docker/ssl 
chmod 700 /etc/docker/ssl/

Copy over the CA certificate.

cp /labkey/apps/ssl/ca.pem /etc/docker/ssl

Create the openssl.cnf configuration file to be used by the Docker Daemon. Note: Values for Test and Production are shown separated by "|" pipes. Keep only the option needed.

vi /etc/docker/ssl/openssl.cnf 
[added]

[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names

[alt_names]
DNS.1 = yourtestweb | yourprodweb
DNS.2 = yourtestrserve | yourprodrserve
IP.1 = 127.0.0.1
IP.2 = 10.0.0.87 | 10.10.0.37

Create the key and certificate to be used by the Docker Daemon.

openssl genrsa -out /etc/docker/ssl/daemon-key.pem 4096
openssl req -new -key /etc/docker/ssl/daemon-key.pem -out /etc/docker/ssl/daemon-cert.csr -subj '/CN=docker-daemon' -config /etc/docker/ssl/openssl.cnf
openssl x509 -req -in /etc/docker/ssl/daemon-cert.csr -CA /etc/docker/ssl/ca.pem -CAkey /labkey/apps/ssl/ca-key.pem -CAcreateserial -out /etc/docker/ssl/daemon-cert.pem -days 3650 -extensions v3_req -extfile /etc/docker/ssl/openssl.cnf

Set the correct permission on the certificates.

chmod 600 /etc/docker/ssl/*

Change Docker Daemon Configuration

Note: Values for Test and Production are shown separated by "|" pipes. Keep only the option needed.

Change the Docker Daemon to use your preferred configuration. The changes are:

  • Daemon will listen Linux socket at "/var/run/docker.sock" and "tcp://10.0.1.204 | 10.10.1.74:2376"
  • Use TLS certificates on the TCP socket for encryption and authentication
  • Turn off inter-container communication
  • Set default limits on container usage and enable userland-proxy
These options were set creating the daemon.json configuration file at:
/etc/docker/daemon.json

vi /etc/docker/daemon.json 
[added]
{
"icc": false,
"tls": true,
"tlsverify": true,
"tlscacert": "/etc/docker/ssl/ca.pem",
"tlscert": "/etc/docker/ssl/daemon-cert.pem",
"tlskey": "/etc/docker/ssl/daemon-key.pem",
"userland-proxy": false,
"default-ulimit": "nofile=50:100",
"hosts": ["unix:///var/run/docker.sock", "tcp://10.0.1.204 | 10.10.1.74:2376"]
}

Start the Docker daemon by running:

service docker start

We can test if docker is running:

docker info

If it is running, you will see something similar to:

Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.12.1
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 0
Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: host overlay bridge null
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-92-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 489.9 MiB
Name: myubuntu
ID: WK6X:HLMO:K5IQ:MENK:ALKP:JN4Q:ALYL:32UC:Q2OD:ZNFG:XLZJ:4KPA
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8

We can test if the TLS configuration is working by running:

docker -H 10.0.1.204 | 10.10.1.74:2376 --tls --tlscert=/labkey/apps/ssl/client-cert.pem --tlskey=/labkey/apps/ssl/client-key.pem  ps -a

This should output:

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

Install AppArmor Configuration to be used by Docker Containers

Create new custom profile named "docker-labkey-myserver".

vi /etc/apparmor.d/docker-labkey-myserver
[added]

#include <tunables/global>


profile docker-labkey-myserver flags=(attach_disconnected,mediate_deleted) {

#include <abstractions/base>

network inet tcp,
network inet udp,
deny network inet icmp,
deny network raw,
deny network packet,
capability,
file,
umount,

deny @{PROC}/* w, # deny write for all files directly in /proc (not in a subdir)
# deny write to files not in /proc/<number>/** or /proc/sys/**
deny @{PROC}/{[^1-9],[^1-9][^0-9],[^1-9s][^0-9y][^0-9s],[^1-9][^0-9][^0-9][^0-9]*}/** w,
deny @{PROC}/sys/[^k]** w, # deny /proc/sys except /proc/sys/k* (effectively /proc/sys/kernel)
deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w, # deny everything except shm* in /proc/sys/kernel/
deny @{PROC}/sysrq-trigger rwklx,
deny @{PROC}/mem rwklx,
deny @{PROC}/kmem rwklx,
deny @{PROC}/kcore rwklx,

deny mount,

deny /sys/[^f]*/** wklx,
deny /sys/f[^s]*/** wklx,
deny /sys/fs/[^c]*/** wklx,
deny /sys/fs/c[^g]*/** wklx,
deny /sys/fs/cg[^r]*/** wklx,
deny /sys/firmware/efi/efivars/** rwklx,
deny /sys/kernel/security/** rwklx,


# suppress ptrace denials when using 'docker ps' or using 'ps' inside a container
ptrace (trace,read) peer=docker-labkey-myserver,


# Rules added by LabKey to deny running executables and accessing files
deny /bin/dash mrwklx,
deny /bin/bash mrwklx,
deny /bin/sh mrwklx,
deny /usr/bin/top mrwklx,
deny /usr/bin/apt* mrwklx,
deny /usr/bin/dpkg mrwklx,

deny /bin/** wl,
deny /boot/** wl,
deny /dev/[^null]** wl,
deny /lib/** wl,
deny /lib64/** wl,
deny /media/** wl,
deny /mnt/** wl,
deny /opt/** wl,
deny /proc/** wl,
deny /root/** wl,
deny /sbin/** wl,
deny /srv/** wl,
deny /sys/** wl,
deny /usr/[^local]** wl,
deny /teamcity/** rwklx,
deny /labkey/** rwklx,
deny /share/files/** rwklx,

}

Load the new profile.

apparmor_parser -r -W /etc/apparmor.d/docker-labkey-myserver

Now that the profile is loaded, we can force the container to use the profile by specifying it at run time, for example:

docker run --security-opt "apparmor=docker-labkey-myserver" -i -t  ubuntu /bin/bash

When starting a new Docker container manually, you should always use the "docker-labkey-myserver" profile. This is done by specifying the following option whenever a container is started:

--security-opt "apparmor=docker-labkey-myserver"

Install New Firewall Rules

Install file containing new firewall rules to block certain connections from running containers Note: Values for Test and Production are shown separated by "|" pipes. Keep only the option needed.

mkdir /etc/iptables.d
vi /etc/iptables.d/docker-containers
[added]

*filter
:INPUT ACCEPT
:FORWARD ACCEPT
:OUTPUT ACCEPT

-A INPUT -i eth0 -p tcp --destination-port 2376 -s 10.0.0.87 | 10.10.0.37/32 -j ACCEPT
-A INPUT -i docker0 -p tcp --destination-port 2376 -s 172.17.0.0/16 -j REJECT
-A INPUT -p tcp --destination-port 2376 -j REJECT
-A INPUT -i docker0 -p tcp --destination-port 6312 -s 172.17.0.0/16 -j REJECT
-A INPUT -i docker0 -p tcp --destination-port 22 -s 172.17.0.0/16 -j REJECT
-A INPUT -i docker0 -p tcp --destination-port 111 -s 172.17.0.0/16 -j REJECT
-A INPUT -i docker0 -p udp --destination-port 111 -s 172.17.0.0/16 -j REJECT
-A INPUT -i docker0 -p tcp --destination-port 1110 -s 172.17.0.0/16 -j REJECT
-A INPUT -i docker0 -p udp --destination-port 1110 -s 172.17.0.0/16 -j REJECT
-A INPUT -i docker0 -p tcp --destination-port 2049 -s 172.17.0.0/16 -j REJECT
-A INPUT -i docker0 -p udp --destination-port 2049 -s 172.17.0.0/16 -j REJECT
-A INPUT -i docker0 -p tcp --destination-port 4045 -s 172.17.0.0/16 -j REJECT
-A INPUT -i docker0 -p udp --destination-port 4045 -s 172.17.0.0/16 -j REJECT
-I DOCKER 1 -i eth0 ! -s 10.0.0.87/32 -j DROP
-I FORWARD 1 -i docker0 -p tcp --destination-port 80 -j ACCEPT
-I FORWARD 2 -i docker0 -p tcp --destination-port 443 -j ACCEPT
-I FORWARD 3 -i docker0 -p tcp --destination-port 53 -j ACCEPT
-I FORWARD 4 -i docker0 -p upd --destination-port 53 -j ACCEPT
-I FORWARD 5 -i docker0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
-I FORWARD 6 -i docker0 -j REJECT
COMMIT

Create the Upstart Configuration

Create the upstart configuration file to to ensure iptables rules are applied at startup.

vi /etc/init/iptables.service.conf 
[added]
# Load iptables firewall rules. These rules are required to
# run the Docker Daemon on this host in a secure manner.
#
# Create by LabKey Software

description "Load iptables firewall rules"

start on started docker
stop on runlevel [!2345]

script
exec /sbin/iptables-restore --noflush /etc/iptables.d/docker-containers
end script

Start the new service.

initctl reload-configuration
initctl start iptables.service

Changes to RServe Instance to Support RStudio Usage

In order to support file sharing between the RStudio process and the LabKey Server we will need to:

  1. Create new user accounts on Rserve instance
  2. Create new groups on Rserve instance
  3. Modify permission on the /share/users directory

Add new groups and user accounts Rserve instance

Create the new group named "labkey-docker" this group is created to facilate the file sharing.

sudo groupadd -g 6000 labkey-docker
Create the new RStudio user. This will be the OS user which runs the RStudio session
sudo useradd -u 6005 -m -G labkey-docker rstudio

Create the directory which will hold the Home Directory Fileroots used by each server

We will use acls to ensure that newly created files/directories have the correct permissions. The ACLs only need to be set on the Rserve operating system. They do not need to specified or changed on the container.

These ACLs should only be specified on the LabKey Server Home Directory FileRoot. Never set these ACLs on any other directory on the Rserve. Sharing files between the docker host and container should only be done in the LabKey Server Home Directory FileRoot.

Install required package:

sudo apt-get install acl

Create the home dir fileroot and set the correct permissions using the newly created accounts. These instructions assume:

  1. Tomcat server is being run by the user "myserver"
  2. LabKey Server Home Directory FileRoot is located at "/share/users"
Run the following commands:

setfacl -R -m u:myserver:rwx /share/users
setfacl -Rd -m u:myserver:rwx /share/users
setfacl -Rd -m u:rstudio:rwx /share/users
setfacl -Rd -m g:labkey-docker:rwx /share/users

Changes to Web Server Instance to Support RStudio Usage

In order to support file sharing between the RStudio process and the LabKey Server we will need to:

  1. Create new user accounts on Rserve instance
  2. Create new groups on Rserve instance
  3. Ensure the NFS share "/share" is mounted with "acl" option.

Add new groups and user accounts Rserve instance

Create the new group named "labkey-docker" to facilitate file sharing.

sudo groupadd -g 6000 labkey-docker

Create the new RStudio user. This will be the OS user which runs the RStudio session.

sudo useradd -u 6005 -m -G labkey-docker rstudio



Connect to RStudio Pro


Premium Feature — This feature supports RStudio, which is part of the Professional Plus and Enterprise Editions. Learn more or contact LabKey for details.

Set Up RStudio Pro

To set up RStudio Pro to connect to LabKey, follow the instructions and guidance in this topic:

Once RStudio Pro is correctly configured, RStudio Pro Admin enables proxy authentication on RStudio to use LabKey as the authenticator. Some key steps:

1. Customize Proxy Token

sudo sh -c "echo 'X-Secret-User-Header' > /etc/rstudio/secure-proxy-user-header"
sudo chmod 0600 /etc/rstudio/secure-proxy-user-header

2. Edit rserver.conf

Add the following two lines to /etc/rstudio/rserver.conf:

auth-proxy=1
auth-proxy-sign-in-url=http://LabKeyServerBaseUrl/login/home/login.view&returnUrl=contextPath/rstudio-startPro.view?method=LAUNCH

3. Restart

Restart RStudio Pro so that these configuration changes are effective by running

$sudo rstudio-server restart

Configure RStudio Pro in the Admin Console

  • Select (Admin) > Site > Admin Console.
  • Click Admin Console Links.
  • Under Configuration, click Site Settings.
  • Set the Base Server URL (i.e. not "localhost").
  • Click Save.
  • Click Admin Console Links
  • Under Premium Features, click RStudio Settings.
  • Select Use RStudio Pro.

  • Enter the Configuration values:
    • RStudio Pro URL: The absolute URL of RStudio Pro.
    • User Email Suffix: Used for LabKey to RStudio Pro user mapping. The RStudio Pro user name is derived from removing this suffix (Ex. "@labkey.com") from the user's LabKey Server email address.
    • Security Token: The HTTP header name that RStudio is configured to use to indicate the authenticated user. Default is 'X-RStudio-Username'.
    • Initialize Schema Viewer: Check the box to enable the viewing of the LabKey schema from within RStudio. See Advanced Initialization of RStudio for more information.

  • Click Test RStudio Pro to confirm a working configuration.
  • Click Save to save.

Testing RStudio Pro Connection

Troubleshooting unsuccessful results from Test RStudio Pro.

  • An error response may indicate the incorrect URL for RStudio Pro was entered, or that URL is unreachable. Try reaching the URL in a browser window outside of LabKey.
  • A warning response may have several possible causes:
    • Configuration is correct, but user is not a valid RStudio user. A user might be a valid Linux user, but not an RStudio user. Does the user have the home directory set? See Add RStudioPro Users for help.
    • A bad user email suffix
    • A bad security token
  • A success response means the configuration is good and current LabKey users can now log in to the RStudio Pro server.

Launch RStudio Pro

  • Select (Admin) > Developer Links > RStudio Server.

When you launch RStudio Pro, you will see your home page. If you have multiple sessions, click on one to select it.

Now you can use RStudio Pro to edit R reports and export data. Learn more in these topics:

Related Topics




Set Up RStudio Pro


Premium Feature — This feature supports RStudio, which is part of the Professional Plus and Enterprise Editions. Learn more or contact LabKey for details.
This topic covers setting up and configuring RStudio Pro. Once you have completed these steps, configure LabKey as described in Connect to RStudio Pro.

Prerequisites

AWS Notes

  1. If you're setting this server up in AWS, you will want to set it up in a public subnet in the same VPC as the destination server. It will need a public-facing IP/EIP.
  2. For AWS, the following ports need to be open, inbound and outbound, for this install to work in AWS. Set your security group/NACLs accordingly:
    • 80
    • 443
    • 1080 (license server)
    • 8787 (initial connection)
    • 1024-65535 (license server)

Non-Linux Host OS

  1. VM: (Optional, for non linux host OS) Use Virtual Box to set up a local Linux environment.
  2. Linux: (Optional, for non linux host OS)
  • Install Linux OS:
    • Ubuntu: Download and install latest Ubuntu; version 12.04 or higher is required) in the VM.
  • Configure VM network so that VM has an IP and can be accessed by the host
    • Ex: VM > Settings > Network > Attached To > Bridged Adapter

Set Up R (for Ubuntu)

Use the following steps to set up R for Ubuntu. For other versions of linux, follow the instructions here instead: http://docs.rstudio.com/ide/server-pro/index.html#installation

1. Configure CRAN Repository. Add the following entry to your /etc/apt/sources.list file, substituting the correct name for your Ubuntu version where this example uses 'xenial' (the name for 16.x):

If you are not sure what name your Ubuntu version is using, you can find it in other lines of the sources.list file.

Linux Tip: to open a system file in linux for editing, use:
$ gksudo gedit  /etc/apt/sources.list
To add edit permission to a file:
$ chmod 777 /etc/apt/sources.list

2. Install latest version of R

apt-get update
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys E084DAB9
sudo apt-get install r-base

3. Install Rlabkey and other R packages

  • If a library is to be used by RStudio as a system library (instead of a user library), run with 'lib' set as shown here:
    $ sudo R
    > install.packages("Rlabkey", lib="/usr/local/lib/R/site-library")
    > q()
  • If a library has dependencies:
    > install.packages("package", lib="/usr/local/lib/R/site-library", dependencies=TRUE)
    > q()
Trouble installing Rlabkey on Linux? Try installing dependencies explicitly:
> sudo apt-get install libssl-dev
> sudo apt-get install libcurl4-openssl-dev

Install RStudio Pro

1. Download and install RStudio Pro as instructed when you purchased it. For demo purposes, you can use the free 45 day trial license and download from the RStudio download page.

$ sudo apt-get install gdebi-core
wget https://download2.rstudio.org/rstudio-server-pro-1.1.447-amd64.deb
(nb: this URL may not be current. Check the RStudio download page for latest version)

gdebi rstudio-server-pro-1.1.447-amd64.deb (substitute current version)

2. Activation: (Optional)

$ sudo rstudio-server license-manager activate ACTIVATION_KEY

This function requires high ports to be world-accessible. If this command fails, check your security groups or NACLs; if this fails with a timeout, check them first.

3. Verify RStudio Pro Installation

  1. Create a ubuntu user:
    useradd -m -s /bin/bash username
    passwd username
  2. Get Ubuntu’s ip address by:
    $ ifconfig
  3. In VM browser, go to localhost:8787, or ip:8787
  4. You should be on the RStudio Pro login page, login using your ubuntu login
4. Verify accessing RStudio pro from the host machine

In the host machine’s browser, go to http://ubuntuVmIPAddress:8787, you should see the RStudio login page and being able to log in.

Troubleshoot: did you configure VM to use Bridged Adapter?

5. Create a Route 53 DNS name for the server. You will need this for the next steps.

Configure RStudio Pro Proxy

1. You will need to create an SSL cert:

  1. openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 3650 -out certificate.pem
  2. Copy the key.pem file to /etc/ssl/private/
  3. Copy the certificate.pem file to /etc/ssl/certs
  4. Edit /etc/rstudio/rstudio-server.conf and add the following lines:
    ssl-enabled=1
    ssl-certificate=/etc/ssl/certs/cert.pem
    ssl-certificate-key=/etc/ssl/private/key.pem
  5. rstudio-server stop
  6. rstudio-server start
2. Use LabKey server as proxy authenticator for RStudio Pro by adding the following lines to /etc/rstudio/rserver.conf:
auth-proxy=1
auth-proxy-sign-in-url=https://nihs-stage.app.labkey.host/rstudio-startPro.view?method=LAUNCH (change the server name to whatever server you want to point at)
3. rstudio-server restart 4. Follow the instructions here for downloading the cert: https://stackoverflow.com/questions/21076179/pkix-path-building-failed-and-unable-to-find-valid-certification-path-to-requ 5. Upload the saved cert to the destination server 6. On the destination Labkey server (nb: this will require restarting the LabKey instance):
  1. Log in.
  2. sudo su -
  3. keytool -import -keystore /usr/lib/jvm/jdk1.8.0_151/jre/lib/security/cacerts -alias <RStudio hostname, but not the FQDN> -file <whatever you named the saved cert>.crt (nb: you will be prompted for a password; ask Ops for it)
  4. service tomcat_lk restart
7. Verify Proxy redirect Go to the RStudio Pro url to verify the page redirects to LabKey. Note: Once SSL is enabled, 8787 no longer works.

8. (Optional) Custom security token

$sudo sh -c "echo 'X-LabKey-RStudio-Header' > /etc/rstudio/secure-proxy-user-header"
$sudo chmod 0600 /etc/rstudio/secure-proxy-user-header

(Optional) Add RStudio Pro Users

LabKey users are mapped to RStudio users by removing the configured "email suffix". For example: LabKey user XYZ@email.com is mapped to RStudio user name XYZ

Adding RStudio Pro Users

RStudio Pro uses Linux’s PAM authentication so any users accounts configured for Linux will be able to access RStudio Pro. Users can be added for RStudio by the adduser Linux command.

The following commands add a user “rstudio1” to Linux (and hence RStudio pro).

$ sudo useradd -m -d /home/rstudio1 rstudio1
$ sudo passwd rstudio1

IMPORTANT: Each user must have a home directory associated with them otherwise they won’t be able to access RStudio since RStudio works on this home directory.

Verify Access

Verify access to RStudio Pro for the newly added user:

Go to ip:8787, enter user and pw for the newly created user, verify the user can be logged in to RStudio successfully.

Troubleshoot for permission denied error on login:

  1. Run pamtester to rule out PAM issue
    $ sudo pamtester --verbose rstudio <username> authenticate acct_mgmt
  2. Try adding the following line to /etc/sssd/sssd.conf (you may need to install sssd):
    ad_gpo_map_service = +rstudio
  3. Did you create the user with a specified home directory?
  4. Make sure you didn’t configure the RStudio Pro server to limit user access by user group (the default is to allow all; check /etc/rstudio/rserver.conf), or if so make sure your user is added to an allowed group.

General Troubleshooting

After making any change to the RStudio Pro config, always run the following for the change to take effect:

$sudo rstudio-server restart

Related Topics




Edit R Reports in RStudio


Premium Feature — This feature supports RStudio, which is part of the Professional Plus and Enterprise Editions. Learn more or contact LabKey for details.

After LabKey is configured to support either RStudio or RStudio Pro it can be enabled in any folder where users want to be able to edit R Reports using RStudio.

Note: To edit R reports in RStudio, you must have Rlabkey v2.2.2.

Within LabKey, open the R Report Builder. There are several paths:

  1. Edit an existing R report (shown below).
  2. Create a new report from a grid by selecting (Charts/Reports) > Create R Report.
  3. From the Data Views web part, select Add Report > R Report.
  • Open the R Report Builder.
  • Click the Source tab and notice at the bottom there is an Edit in RStudio button.
  • When you click Edit in RStudio, you will see a popup giving the information that your report is now open for editing in RStudio. Click Go To RStudio to see it.

This may be in a hidden browser window or tab. This popup will remain in the current browser until you complete work in RStudio and click Edit in LabKey to return to the LabKey report editor.

Edit in RStudio

When you open a report to edit in RStudio, the editor interface will open. After making your changes, click the Save button (circled). Changes will be saved to LabKey.

Edit in RStudio Pro

When you are using RStudio Pro, if you have multiple sessions running, you will land on the home page. Select the session of interest and edit your report there.

Remember to save your report in RStudio Pro before returning to the LabKey Server browser window and clicking Edit in LabKey.

Related Topics




Export Data to RStudio


Premium Feature — This feature supports RStudio, which is part of the Professional Plus and Enterprise Editions. Learn more or contact LabKey for details.

Once RStudio is configured and enabled in a container, or LabKey server is configured to connect to RStudio Pro users can export data from grids for use in their RStudio environment.

On a data grid:
  • Select Export > RStudio. If you don't see this option, your server is not set up to use either RStudio or RStudio Pro.
  • Enter a Variable Name and click Export to RStudio.
  • This will launch RStudio with your data loaded into the named variable.

Exporting to RStudio Pro

When you are using RStudio Pro and have multiple sessions running, when you click Export to RStudio you will first see your dashboard of open sessions. Select the session to which to export the data.

Related Topics




Advanced Initialization of RStudio


Premium Feature — This feature supports RStudio, which is part of the Professional Plus and Enterprise Editions. Learn more or contact LabKey for details.

This topic describes the process for initializing RStudio to enable the viewing of the LabKey schema within the RStudio or RStudio Pro workspace. By default this option is enabled.

Note: To use this feature, you must have the latest version of Rlabkey.

Configuration

Follow the steps for configuring LabKey to access your RStudio or RStudio Pro instance following the instructions in one of these topics:

On the Configure RStudio page for either type of integration, scroll down and check the box for Initialize Schema Viewer.

Use the LabKey Schema within RStudio

The RStudio viewer will now allow you to browse the LabKey schema. Select a project from the menu circled in this screenshot, then select a folder to see the schema it contains, and a schema to see the queries it contains.

Extend Initialization

An RStudio system administrator can add or overwrite variables and settings during initialization by including an R function in the Rprofile.site file which calls the hidden/named function "labkey.rstudio.extend". When performing this initialization, a one time "requestId" is used, rather than a more persistent api key.

As an example, the following screenshot shows a simple call of this function which will print a 'Welcome' message:

RStudio executes the hidden function upon initialization:

Related Topics




Configure Docker Host


This topic explains how to configure a connection between LabKey Server and a Docker container. You can use this connection to:
  • Go to (Admin) > Site > Admin Console.
  • Click Admin Console Links.
  • Under Premium Features click Docker Host Settings.
    • Enable Docker Settings: select On.
    • Docker Host: Enter the URI for your docker host.
    • Docker Certification Path: Enter a valid file system path to the Docker certificates. Required for TCP in production mode.
    • Container Port Range: In most cases, leave these values blank.
    • User File Management: Select Docker Volume for a server running in production mode.
  • Click Save and then Test Saved Host to check your connectivity.



SQL Queries


LabKey Server provides rich tools for working with databases and SQL queries. By developing SQL queries you can:
  • Create filtered grid views of data.
  • Join data from different tables.
  • Group data and compute aggregates for each group.
  • Add a calculated column to a query.
  • Format how data is displayed using query metadata.
  • Create staging tables for building reports.
Special features include:
  • An intuitive table-joining syntax called lookups. Lookups use a convenient syntax of the form "Table.ForeignKey.FieldFromForeignTable" to achieve what would normally require a JOIN in SQL. For details see LabKey SQL Reference.
  • Parameterized SQL statements. Pass parameters to a SQL query via a JavaScript API.

Topics

Examples

Related Topics




LabKey SQL Tutorial


SQL queries are a powerful way to shape different ways to view data. In LabKey Server, queries are the main way to surface data from the database: using a query, you pick the columns you want to retrieve, and optionally apply any filters and sorts. SQL Queries behave like "peers" to other tables in the database: queries are added to the database schema alongside the original, core tables. You can query one table at a time, or create a query that combines data from multiple tables. Queries also provide staging for reports: start with a base query and build a report on top of that query. Queries can be created through the graphical user interface (as shown in this topic) or through a file-based module.

LabKey Server provides a number of mechanisms to simplify SQL syntax:

  • LabKey SQL: LabKey SQL is a SQL dialect that translates your queries into the native syntax of the SQL database underlying your server, whether it is PostgreSQL or Microsoft SQL Server. This lets you write in one SQL dialect but communicate with many SQL database implementations.
  • Lookups: Lookups join tables together using an intuitive syntax.
  • Query Metadata: Add additional properties to a query using metadata xml files, such as: column captions, relationships to other tables or queries, data formatting, and links
The following step-by-step tutorial shows you how to create a SQL query and begin working with it.

Create a SQL Query

In this example, we will create a query based on the Users table in the core schema.

  • Select (Admin) > Developer Links > Schema Browser.
  • Open the core schema in the left hand pane and select the Users table. Then click the button Create New Query. (This tells LabKey Server to create a query based on the table core.Users.)
  • On the New Query page:
    • Provide a name for the query (in the field "What do you want to call the new query?").
    • Confirm that the Users table is selected (in the field "Which query/table do you want this new query to be based on?")
    • Click Create and Edit Source.
  • LabKey Server will provide a "starter query" of the Users table -- a basic SELECT statement for all of the fields in the table -- essentially a duplicate of the Users table. Typically, you would modify this "starter query" to fit your needs, adding WHERE clauses, JOINs to other tables, or substituting an entirely new SQL statement. But for this tutorial, we will just use the "starter query" unchanged.
  • Click Save and Finish.
  • The results of the query are displayed in a data grid, similar to the grid shown below -- though yours will show different data.
  • Click core Schema to return to the query browser view of this schema.
  • Notice that your new query appears under user-defined queries.

Query Metadata

Each query has accompanying XML that defines properties, or metadata, for the query. In this step we will add properties to the query by editing the accompanying XML. In particular we will:

  • Change the data type of the UserId column, making it a lookup into the Users table. By showing a clickable name instead of an integer value, we can make this column more human-readable. We will accomplish this by using the graphical user interface, which will write out the XML automatically.
  • Modify the way it is displayed in the grid. We will accomplish this by editing the XML directly.
  • Click Edit Metadata.
  • On the UserId row, click in the Type column where it shows the value Integer
  • In the Choose Field Type dialog, select User. This will create a lookup between your query and the Users table.
  • Click Apply.
  • Click View Data, and click Save to confirm your changes.
  • Notice that the values in the User Id column are no longer integers, but text links -- this reflects the fact that User Id is now a lookup into the Users table. Click a value in the User Id column to see the corresponding record in the Users table (where you can see the actual user ID integer is unchanged).
This lookup is defined in the XML metadata document. Click back in your browser to return to the query, and let's see what the XML looks like.
  • Click core Schema to return to the Query Browser.
  • Click Edit Source and then select the XML Metadata tab.
  • The XML metadata will appear in a text editor. Notice the XML between the <fk>...</fk> tags. This tells LabKey Server to create a lookup (aka, a "foreign key") to the Users table in the core schema.
  • Next we will modify the XML directly to hide the "Display Name" column in our query. We don't need this column any longer because the User Id column already displays this information.
  • Add the following XML to the document, directly after the </column> tag (i.e directly before the ending </columns> tag.
<column columnName="DisplayName">
<isHidden>true</isHidden>
</column>
  • Click Save.
  • Click the Data tab to see the results without leaving the query editor.
  • Notice that the Display Name column is no longer shown.
  • Click the XML Metadata tab and now add the following XML immediately after the column section for hiding the display name. This section will display the Email column values in red.
<column columnName="Email">
<conditionalFormats>
<conditionalFormat>
<filters>
<filter operator="isnonblank"/>
</filters>
<textColor>FF0000</textColor>
</conditionalFormat>
</conditionalFormats>
</column>
  • Click Save & Finish.
  • Now that you have a SQL query, you can display it directly by using a query web part, or use it as the basis for a report, such as an R report or a visualization.

Related Topics




SQL Query Browser



The schema browser is the dashboard for browsing all the database data in a LabKey Server folder. It also provides access to key schema-related functionality. (A schema is a named collection of tables and queries.) Using the schema browser, you can:

  • Browse the tables and queries
  • Add new SQL queries
  • Discover table relationships to help write queries
  • Define external schemas to access new data
  • Generate scripts for bulk insertion of data into a new schema

Browse and Navigate the Data Model

To open the Query Schema Browser go to (Admin) > Developer Links > Schema Browser.

The browser displays a list of the available schemas, including external schemas and data sources you have added. Each schema contains a collection of queries and tables. User-defined queries are grouped together separately from the built-in queries. Your permissions within the folder determine which tables and queries you can see here.

Schemas live in a particular container (project or folder) on LabKey Server, but can be marked as inheritable, in which case they are accessible in child folders. (For more information on controlling schema heritability in child folders, see Query Metadata.)

You can browse column names by clicking on a particular table or query. The image below shows how to discover the column names of the Comments table.

Show All Columns vs. Columns in the Default Grid View

For a particular table or query, the browser shows two separate lists.

  • All columns in this table: the top section shows all of the columns in the table/query.
  • Columns in your default view of this query: Your default grid view of the table/query may not contain all of the columns present in the table. It also may contain additional "columns" when there are lookups (or foreign keys) defined in the default view. See Step 1: Customize Your Grid View to learn about changing the contents of the "default" view.
Below these lists, the Indices of the table are listed.

Validate Queries

When you upgrade to a new version of LabKey, change hardware, or database software, you may want to validate your SQL queries. You can perform a validation check of your SQL queries by pressing the Validate Queries button, on the top row of buttons in the Query Schema Browser. Validation runs against all queries in the current folder and checks to see if the SQL queries parse and execute without errors.

Generate Schema Export / Migrate Data to Another Schema

If you wish to move data from one LabKey Server schema to another LabKey Server schema, you can do so by generating a migration script. The system will read the source schema and generate:

  1. a set of tab-separated value (TSV) files, one for each table in the source. (Each TSV file is packaged as a .tsv.gz file)
  2. a script for importing these tables into a target schema.
Note that the script only copies data, it does not create the target schema itself. The target schema must already exist for the import script to work. Also, the script must be used as an update script included in a module.

To generate the TSV files and the associated script:

  • Go the Schema Browser: (Admin) > Developer Links > Schema Browser.
  • Click Generate Schema Export.
  • Select the data source and schema name.
  • Enter a directory path where the script and TSV files will be written, for example: C:\temp\exports. This directory must already exist on your machine for the export to succeed.
  • Click Export.
  • The file artifacts will be written to the path you specified.

Field Descriptions

Field NameDescription
Source Data SourceThe data source where the data to export resides.
Source SchemaThe schema you want to export.
Target SchemaThe schema where you want to import the data.
Path in ScriptOptional. If you intend to place the import script and the data files in separate directories, specify a path so that the import script can find the data.
Output DirectoryDirectory on your local machine where the import script and the data files will be written. This directory must already exist on your machine, it will not be created for you.

The generated script consists of a series of bulkImport calls that open the .tsv.gz data files and insert them into the target schema, in this case the target schema is 'assaydata'.

SELECT core.bulkImport('assaydata', 'c17d97_pcr_data_fields', 'dbscripts/assaydata/c17d97_pcr_data_fields.tsv.gz');
SELECT core.bulkImport('assaydata', 'c15d80_rna_data_fields', 'dbscripts/assaydata/c15d80_rna_data_fields.tsv.gz');
SELECT core.bulkImport('assaydata', 'c2d326_test_data_fields', 'dbscripts/assaydata/c2d326_test_data_fields.tsv.gz');
SELECT core.bulkImport('assaydata', 'c15d77_pcr_data_fields', 'dbscripts/assaydata/c15d77_pcr_data_fields.tsv.gz');

Now you can re-import the data by adding the generated .sql script and .tsv.gz files to a module as a SQL upgrade script. For details on adding SQL scripts to modules, see Modules: SQL Scripts.

Related Topics




Create a SQL Query


Creating a custom SQL query gives you the ability to flexibly present the data in a table in any way you wish using SQL features like calculated columns, aggregation, formatting, filtering, joins and lookups. To create a custom SQL query, you must be logged on to your LabKey Server as an Admin. The following steps guide you through creating a custom SQL query and view on a data table.

Create a Custom SQL Query

  • Select (Admin) > Developer Links > Schema Browser.
  • From the schema list, select the schema that includes your data tables of interest.
  • Click the Create New Query button.
  • In the field What do you want to call the new query?, enter a name for your new query.
  • In the drop-down field Which query/table do you want this new query to be based on?, select the base query/table for your query.
  • Click Create and Edit Source.
  • LabKey Server will generate an SQL query for the selected table.
  • Refine the source of this query as desired in the SQL source editor.

Related Topics




Edit SQL Query Source


To edit the SQL source for a query:

  • Go to (Admin) > Developer Links > Schema Browser.
  • Using the navigation tree on the left, browse to the target query and then click Edit Source.
  • The SQL source appears in the source editor.
  • Select the Data tab or click Execute Query to see the results.
  • Return to the Source tab to make your desired changes.
  • Clicking Save will check for syntax errors (such as the trailing comma in the above source). Remove the comma and click Save again.
  • Return to the Data tab or click Execute Query again to see the revised results.
  • Click Save and Finish to save your changes when complete.



LabKey SQL Reference


This topic is under construction for the 19.1 release of LabKey Server. For current documentation, click here.

LabKey SQL 

LabKey SQL is a SQL dialect that supports (1) most standard SQL functionality and (2) provides extended functionality that is unique to LabKey, including:

  • Lookup columns. Lookup columns use an intuitive syntax to access data in other tables to achieve what would normally require a JOIN statement. For example: "SomeTable.ForeignKey.FieldFromForeignTable" For details see Lookups. The special lookup column "Datasets" is injected into each study dataset and provides a syntax shortcut when joining the current dataset to another dataset that has compatible join keys. See example usage.
  • Security. Before execution, all SQL queries are checked against the user's security role/permissions.  
  • Parameterized SQL statements. the PARAMETERS keyword lets you define parameters for a query.  An associated API gives you control over the parameterized query from JavaScript code. See Parameterized SQL Queries.
  • Pivot tables. the PIVOT...BY expression provides an intuitive syntax for creating pivot tables. See Create a Pivot Query.
  • User-related functions. USERID() and ISMEMBEROF(groupid) lets you control query visibility based on the user's group membership. 

Keywords

KeywordDescription
ASAliases can be explicitly named using the AS keyword. Note that the AS keyword is optional: the following select clauses both create an alias called "Name":

    SELECT LCASE(FirstName) AS Name
    SELECT LCASE(FirstName) Name

Implicit aliases are automatically generated for expressions in the SELECT list.  In the query below, an output column named "Expression1" is automatically created for the expression "LCASE(FirstName)":

    SELECT LCASE(FirstName)
    FROM PEOPLE

ASCENDING, ASCReturn results in ascending value order.

    ORDER BY Weight ASC 
CAST(AS)CAST(R.d AS VARCHAR)

Defined valid datatype keywords which can be used as cast/convert targets, and to what java.sql.Types name each keyword maps. Keywords are case-insensitive.

    BIGINT
    BINARY
    BIT
    CHAR
    DECIMAL
    DATE
    DOUBLE
    FLOAT
    GUID
    INTEGER
    LONGVARBINARY
    LONGVARCHAR
    NUMERIC
    REAL
    SMALLINT
    TIME
    TIMESTAMP
    TINYINT
    VARBINARY
    VARCHAR

Examples:

CAST(TimeCreated AS DATE)

CAST(WEEK(i.date) as INTEGER) as WeekOfYear,

DESCENDING, DESCReturn results in descending value order.
DISTINCTReturn distinct, non duplicate, values.

    SELECT DISTINCT Country
    FROM Demographics 
EXISTSReturns a Boolean value based on a subquery. Returns TRUE if at least one row is returned from the subquery.

The following example returns any plasma samples which have been assayed with a score greater than 80%. Assume that ImmuneScores.Data.SpecimenId is a lookup field (aka foreign key) to Plasma.Name. 

    SELECT Plasma.Name
    FROM Plasma
    WHERE EXISTS
        (SELECT *
        FROM assay.General.ImmuneScores.Data
        WHERE SpecimenId = Plasma.Name
        AND ScorePercent > .8)
FALSE 
FROMThe FROM clause in LabKey SQL must contain at least one table. It can also contain JOINs to other tables. Commas are supported in the FROM clause:

    FROM TableA, TableB
    WHERE TableA.x = TableB.x

Nested joins are supported in the FROM clause:

    FROM TableA LEFT JOIN (TableB INNER JOIN TableC ON ...) ON...

To refer to tables in LabKey folders other than the current folder, see Cross-Folder Queries.
GROUP BYUsed with aggregate functions to group the results.  Defines the "for each" or "per".  The example below returns the number of records "for each" participant:

    SELECT ParticipantId, COUNT(Created) "Number of Records"
    FROM "Physical Exam"
    GROUP BY ParticipantId

HAVINGUsed with aggregate functions to limit the results.  The following example returns participants with 10 or more records in the Physical Exam table:

    SELECT ParticipantId, COUNT(Created) "Number of Records"
    FROM "Physical Exam"
    GROUP BY ParticipantId
    HAVING COUNT(Created) > 10

HAVING can be used without a GROUP BY clause, in which case all selected rows are treated as a single group for aggregation purposes.
JOIN,
RIGHT JOIN,
LEFT JOIN,
FULL JOIN
Example:

    SELECT *
    FROM "Physical Exam"
    FULL JOIN "Lab Results"
    ON "Physical Exam".ParticipantId = "Lab Results".ParticipantId 
LIMITLimits the number or records returned by the query.  The following example returns the 10 most recent records:

    SELECT *
    FROM "Physical Exam"
    ORDER BY Created DESC LIMIT 10

ORDER BYOften used with LIMIT to improve performance:

    SELECT ParticipantID,
    Height_cm AS Height
    FROM "Physical Exam"
    ORDER BY Height DESC LIMIT 5

Troubleshooting: "Why is the ORDER BY clause being ignored?"

When authoring queries in LabKey SQL, ORDER BY clauses may appear to not be respected in the results displayed to the user. This is because a LabKey SQL query is typically processed as a subquery within a parent query, and the parent's sort order overrides the ORDER BY clause in the subquery.

Two recommended solutions:
(1) Define the sort in the parent query using the grid view customizer.
(2) Use the "config.sort" property of the API call.

PARAMETERSQueries can declare parameters using the PARAMETERS keyword. Default values data types are supported as shown below:

    PARAMETERS (X INTEGER DEFAULT 37)
    SELECT *
    FROM "Physical Exam"
    WHERE Temp_C = X

Parameter names will override any unqualified table column with the same name.  Use a table qualification to disambiguate.  In the example below, R.X refers to the column while X refers to the parameter:

    PARAMETERS(X INTEGER DEFAULT 5)
    SELECT *
    FROM Table R
    WHERE R.X = X

Supported data types for parameters are: BIGINT, BIT, CHAR, DECIMAL, DOUBLE, FLOAT, INTEGER, LONGVARCHAR, NUMERIC, REAL, SMALLINT, TIMESTAMP, TINYINT, VARCHAR

Parameter values can be passed via JavaScript API calls to the query. For details see Parameterized SQL Queries.

PIVOT BYRe-visualize a table by rotating or "pivoting" a portion of it, essentially promoting cell data to column headers. See Write a Pivot Query for examples.
SELECTSELECT queries are the only type of query that can currently be written in LabKey SQL.  Sub-selects are allowed both as an expression, and in the FROM clause.

Aliases are automatically generated for expressions after SELECT.  In the query below, an output column named "Expression1" is automatically generated for the expression "LCASE(FirstName)":

    SELECT LCASE(FirstName) FROM...

TRUE 
UNION, UNION ALLThe UNION clause is the same as standard SQL.  LabKey SQL supports UNION in subqueries. 
WHEREFilter the results for certain values. Example:

    SELECT *
    FROM "Physical Exam"
    WHERE YEAR(Date) = '2010' 
WITH

Define a "common table expression" which functions like a subquery or inline view table. Especially useful for recursive queries.

Usage Notes: If there are UNION clauses that do not reference the common table expression (CTE) itself, the server interprets them as normal UNIONs. The first subclause of a UNION may not reference the CTE. The CTE may only be referenced once in a FROM clause or JOIN clauses within the UNION. There may be multiple CTEs defined in the WITH. Each may reference the previous CTEs in the WITH. No column specifications are allowed in the WITH (as some SQL versions allow).

Exception Behavior: Testing indicates that PostgreSQL does not provide an exception to LabKey Server for a non-ending, recursive CTE query. This can cause the LabKey Server to wait indefinitely for the query to complete. MS SQL Server does provide an exception to the server which allows the LabKey Server to end the query attempt. 

A non-recursive example:

WITH AllDemo AS
  (
     SELECT *
     FROM "/Studies/Study A/".study.Demographics
     UNION
     SELECT *
     FROM "/Studies/Study B/".study.Demographics
  )
SELECT ParticipantId from AllDemo

A recursive example:

                    -- In a table that holds parent/child information, this query returns all of the 
-- children and grandchildren (recursively down the generations),
-- for a given "Source" parent.

PARAMETERS
(
    Source VARCHAR DEFAULT NULL
)

WITH Derivations AS 
( 
   -- Anchor Query. User enters a 'Source' parent 
   SELECT Item, Parent
   FROM Items 
   WHERE Parent = Source
   UNION ALL 
   -- Recursive Query. Get the children, grandchildren, ... of the source parent
   SELECT i.Item, i.Parent 
   FROM Items i INNER JOIN Derivations p  
   ON i.Parent = p.Item 
) 
SELECT * FROM Derivations

 

Constants

Constants are an upcoming feature supported in 18.3.

The following constant values can be used in LabKey SQL queries.

ConstantDescription
CAST('Infinity' AS DOUBLE) Represents positive infinity.
CAST('-Infinity' AS DOUBLE) Represents negative infinity.
CAST('NaN' AS DOUBLE) Represents "Not a number".
TRUE Boolean value.
FALSE Boolean value.


Operators

OperatorDescription
String Operators 
 ||String concatenation. For example:    
   
    SELECT ParticipantId,
    City || ', ' || State AS CityOfOrigin
    FROM Demographics
 LIKE 
 NOT LIKE 
Arithmetic Operators 
 +Add
 -Subtract
 *Multiply
 /Divide
Comparison operators 
 =Equals
 !=Does not equal
 <> Does not equal
 > Is greater than
 < Is less than
 >=Is greater than or equal to
 <=Is less than or equal to
 IS NULLIs NULL
 IS NOT NULLIs NOT NULL
 BETWEENBetween two values. Values can be numbers, strings or dates.
 INExample: WHERE City IN ('Seattle', 'Portland')
 NOT INExample: WHERE City NOT IN ('Seattle', 'Portland')
Bitwise Operators 
 &Bitwise AND
 |Bitwise OR
 ^Bitwise exclusive OR
Logical Operators 
 ANDLogical AND
 ORLogical OR
 NOTExample: WHERE NOT Country='USA'

Operator Order of Precedence

Order of PrecedenceOperators
 1 - (unary) , + (unary), CASE
 2 *, / (multiplication, division) 
 3 +, -, & (binary plus, binary minus)
 4 & (bitwise and)
 5 ^ (bitwise xor)
 6 | (bitwise or)
 7 || (concatenation)
 8 <, >, <=, >=, IN, NOT IN, BETWEEN, NOT BETWEEN, LIKE, NOT LIKE 
 9 =, IS, IS NOT, <>, !=  
10 NOT
11 AND
12 OR

Aggregate Functions

FunctionDescription
COUNTThe special syntax COUNT(*) is supported as of LabKey v9.2.
MINMinimum
MAXMaximum
AVGAverage
SUMSum 
STDDEVStandard deviation
GROUP_CONCATAn aggregate function, much like MAX, MIN, AVG, COUNT, etc. It can be used wherever the standard aggregate functions can be used, and is subject to the same grouping rules. Like the built-in MySQL functionality, it will return a string value which is comma-separated list of all of the values for that grouping. A custom separator, instead of the default comma, can be specified.  The example below specifies a semi-colon as the separator:

    SELECT Participant, GROUP_CONCAT(DISTINCT Category, ';') AS CATEGORIES FROM SomeSchema.SomeTable

To use a line-break as the separator, use the following:

    SELECT Participant, GROUP_CONCAT(DISTINCT Category, chr(10)) AS CATEGORIES FROM SomeSchema.SomeTable  

SQL Functions

Many of these functions are similar to standard SQL functions -- see the JBDC escape syntax documentation for additional information.

FunctionDescription
abs(value)Returns the absolute value.
acos(value)Returns the arc cosine.
age(date1, date2)

Supplies the difference in age between the two dates, calculated in years.

age(date1, date2, interval)

The interval indicates the unit of age measurement, either SQL_TSI_MONTH or SQL_TSI_YEAR.

age_in_months(date1, date2)Behavior is undefined if date2 is before date1.
age_in_years(date1, date2)Behavior is undefined if date2 is before date1.
asin(value)Returns the arc sine.
atan(value)Returns the arc tangent.
atan2(value1, value2)Returns the arctangent of the quotient of two values.
case

LabKey SQL parser sometimes requires the use of additional parentheses within the statement.

    CASE (value) WHEN (test1) THEN (result1) ELSE (result2) END
    CASE WHEN (test1) THEN (result1) ELSE (result2) END

Example:

    SELECT "StudentName",
    School,
    CASE WHEN (Division = 'Grades 3-5') THEN (Scores.Score*1.13) ELSE Score END AS AdjustedScore,
    Division
    FROM Scores 

ceiling(value)Rounds the value up.
coalesce(value1,...,valueN)Returns the first non-null value in the argument list. Use to set default values for display.
concat(value1,value2)Concatenates two values. 
contextPath()Returns the context path starting with “/” (e.g. “/labkey”). Returns the empty string if there is no current context path. (Returns VARCHAR.)
cos(radians)Returns the cosine.
cot(radians)Returns the cotangent.
curdate()Returns the current date.
curtime()Returns the current time
dayofmonth(date)Returns the day of the month (1-31) for a given date.
dayofweek(date)Returns the day of the week (1-7) for a given date. (Sun=1 and Sat=7)
dayofyear(date)Returns the day of the year (1-365) for a given date.
degrees(radians)Returns degrees based on the given radians.
exp(n)Returns Euler's number e raised to the nth power. e = 2.71828183 
floor(value)Rounds down to the nearest integer.
folderName()LabKey SQL extension function. Returns the name of the current folder, without beginning or trailing "/". (Returns VARCHAR.)
folderPath()LabKey SQL extension function. Returns the current folder path (starts with “/”, but does not end with “/”). The root returns “/”. (Returns VARCHAR.)
greatest(a, b, c, ...)Returns the greatest value from the list expressions provided. Any number of expressions may be used. The expressions must have the same data type, which will also be the type of the result. The LEAST() function is similar, but returns the smallest value from the list of expressions. GREATEST() and LEAST() are not implemented for SAS databases.

When NULL values appear in the list of expressions, different database implementations as follows:

- PostgreSQL & MS SQL Server ignore NULL values in the arguments, only returning NULL if all arguments are NULL.
- Oracle and MySql return NULL if any one of the arguments are NULL. Best practice: wrap any potentially nullable arguments in coalesce() or ifnull() and determine at the time of usage if NULL should be treated as high or low.

Example:

SELECT greatest(score_1, score_2, score_3) As HIGH_SCORE
FROM MyAssay 

hour(time)Returns the hour for a given date/time.
ifdefined(column_name)IFDEFINED(NAME) allows queries to reference columns that may not be present on a table. Without using IFDEFINED(), LabKey will raise a SQL parse error if the column cannot be resolved. Using IFDEFINED(), a column that cannot be resolved is treated as a NULL value. The IFDEFINED() syntax is useful for writing queries over PIVOT queries or assay tables where columns may be added or removed by an administrator.
ifnull(testValue, defaultValue)If testValue is null, returns the defaultValue.  Example: IFNULL(Units,0)
isequalLabKey SQL extension function. ISEQUAL(a,b) is equivalent to (a=b OR (a IS NULL AND b IS NULL))
ismemberof(groupid)LabKey SQL extension function. Returns true if the current user is a member of the specified group.
javaConstant(fieldName)LabKey SQL extension function. Provides access to public static final variable values.  For details see LabKey SQL Utility Functions.
lcase(string)Convert all characters of a string to lower case.
least(a, b, c, ...)Returns the smallest value from the list expressions provided. For more details, see greatest() above.
left(string, integer)Returns the left side of the string, to the given number of characters. Example: SELECT LEFT('STRINGVALUE',3) returns 'STR'
length(string)Returns the length of the given string.
locate(substring, string) locate(substring, string, startIndex)Returns the location of the first occurrence of substring within string.  startIndex provides a starting position to begin the search. 
log(n)Returns the natural logarithm of n.
log10(n)Base base 10 logarithm on n.
lower(string)Convert all characters of a string to lower case.
ltrim(string)Trims white space characters from the left side of the string. For example: LTRIM('     Trim String')
minute(time)Returns the minute value for the given time. 
mod(dividend, divider)Returns the remainder of the division of dividend by divider.
moduleProperty(module name,  property name)

LabKey SQL extension function. Returns a module property, based on the module and property names. For details see LabKey SQL Utility Functions

month(date)Returns the month value (1-12) of the given date. 
monthname(date)Return the month name of the given date.
now()Returns the system date and time.
overlapsLabKey SQL extension function. Supported only when PostgreSQL is installed as the primary database.    
   
    SELECT OVERLAPS (START1, END1, START2, END2) AS COLUMN1 FROM MYTABLE

The LabKey SQL syntax above is translated into the following PostgreSQL syntax:    
   
    SELECT (START1, END1) OVERLAPS (START2, END2) AS COLUMN1 FROM MYTABLE
pi()Returns the value of π.
power(base, exponent)Returns the base raised to the power of the exponent. For example, power(10,2) returns 100.
quarter(date)Returns the yearly quarter for the given date where the 1st quarter = Jan 1-March 31, 2nd quarter = April 1-June 30, 3rd quarter = July 1-Sept30, 4th quarter = Oct 1-Dec 31
radians(degrees)Returns the radians for the given degrees.
rand(), rand(seed)Returns a random number between 0 and 1.
repeat(string, count)Returns the string repeated the given number of times. SELECT REPEAT('Hello',2) returns 'HelloHello'.
round(value, precision)Rounds the value to the specified number of decimal places. ROUND(43.3432,2) returns 43.34
rtrim(string)Trims white space characters from the right side of the string. For example: RTRIM('Trim String     ')
second(time)Returns the second value for the given time.
sign(value)Returns the sign, positive or negative, for the given value. 
sin(value)Returns the sine for the given value.
startswith(string, prefix)Tests to see if the string starts with the specified prefix. For example, STARTSWITH('12345','2') returns FALSE.
sqrt(value)Returns the square root of the value.
substring(string, start, end)Returns a portion of the string as specified by the start and endlocations.
tan(value)

Returns the tangent of the value.

timestampadd(interval, number_to_add, timestamp)

Adds an interval to the given timestamp value. The interval value must be surrounded by quotes. Possible values for interval:

SQL_TSI_FRAC_SECOND
SQL_TSI_SECOND
SQL_TSI_MINUTE
SQL_TSI_HOUR
SQL_TSI_DAY
SQL_TSI_WEEK
SQL_TSI_MONTH
SQL_TSI_QUARTER
SQL_TSI_YEAR

Example: TIMESTAMPADD('SQL_TSI_QUARTER', 1, "Physical Exam".date) AS NextExam

timestampdiff(interval, timestamp1, timestamp2)

Finds the difference between two timestamp values at a specified interval. The interval must be surrounded by quotes.

Example: TIMESTAMPDIFF('SQL_TSI_DAY', SpecimenEvent.StorageDate, SpecimenEvent.ShipDate)

Note that PostgreSQL does not support the following intervals:

SQL_TSI_FRAC_SECOND
SQL_TSI_WEEK
SQL_TSI_MONTH
SQL_TSI_QUARTER
SQL_TSI_YEAR

As a workaround, use the 'age' functions defined above.

truncate(numeric value, precision)Truncates the numeric value to the precision specified. This is an arithmetic truncation, not a string truncation.
  TRUNCATE(123.4567,1) returns 123.4
  TRUNCATE(123.4567,2) returns 123.45
  TRUNCATE(123.4567,-1) returns 120.0 
ucase(string), upper(string)Converts all characters to upper case.
userid()LabKey SQL extension function. Returns the userid, an integer, of the logged in user. 
username()LabKey SQL extension function. Returns the current user display name. VARCHAR
week(date)Returns the week value (1-52) of the given date.
year(date)Return the year of the given date.  Assuming the system date is March 4 2023, then YEAR(NOW()) return 2023.

PostgreSQL Specific Functions

LabKey SQL supports the following PostgreSQL functions.
See the  PostgreSQL docs for usage details.

 PostgreSQL Function  Docs 
 ascii(value)Returns the ASCII code of the first character of value.   
 btrim(value,
  trimchars)
Removes characters in trimchars from the start and end of string. trimchars defaults to white space.

BTRIM(' trim    ') returns TRIM 
BTRIM('abbatrimabtrimabba', 'ab') returns trimabtrim

 character_length(value),  char_length(value)

Returns the number of characters in value.
 chr(integer_code)Returns the character with the given integer_code.  

CHR(70) returns F
 decode(text,
  format)
See the PostgreSQL docs.
 encode(binary,
  format)
See the PostgreSQL docs.
 initcap(string)Converts the first character of each separate word in string to uppercase and the rest to lowercase. 
 lpad(string, 
  int,
  fillchars)
Pads string to length int by prepending characters fillchars
 md5(text)Returns the hex MD5 value of text
 octet_length(string) Returns the number of bytes in string.
 overlapsSee above for syntax details.
 quote_ident(string)Returns string quoted for use as an identifier in an SQL statement. 
 quote_literal(string)Returns string quoted for use as a string literal in an SQL statement.
 regexp_replace See PostgreSQL docs for details: reference doc, example doc
 repeat(string, int)Repeats string the specified number of times.
 replace(string, 
  matchString, 
  replaceString)
Searches string for matchString and replaces occurrences with replaceString.
 rpad(string, 
  int,
  fillchars)
Pads string to length int by postpending characters fillchars.
 split_part(string,
  delmiter,
  int)
Splits string on delimiter and returns fragment number int (starting the count from 1).

SPLIT_PART('mississippi', 'i', 4) returns 'pp'.
 strpos(string,
  substring)
Returns the position of substring in string. (Count starts from 1.)
 substr(string,
 fromPosition,
 charCount)

Extracts the number of characters specified by charCount from string starting at position fromPosition.

SUBSTR('char_sequence', 5, 2) returns '_s' 

 to_ascii(string,
  encoding) 
Convert string to ASCII from another encoding.
 to_hex(int)Converts int to its hex representation.
 translate(text,
  fromText,
  toText) 
Characters in string matching a character in the fromString set are replaced by the corresponding character in toString.
 to_charSee Data Type Formatting Functions in the PostgreSQL docs.
 to_date(textdate,
  format) 
See Data Type Formatting Functions in the PostgreSQL docs. 
 to_timestampSee Data Type Formatting Functions in the PostgreSQL docs.
 to_numberSee Data Type Formatting Functions in the PostgreSQL docs.

MS SQL Server Specific Functions

LabKey SQL supports the following SQL Server functions.
See the SQL Server docs for usage details.

MS SQL Server FunctionDescription
ascii(value)Returns the ASCII code of the first character of value.  
char(int), chr(int)Returns an character for the specified ascii code int
charindex(string, 
 searchString,
 index) 
Returns the position of searchString in string, starting the search at index.
difference(string,string)Returns the difference between the soundex values of two expressions as an integer.
See the MS SQL docs.
isnumeric(expression)Determines whether an expression is a valid numeric type. See the MS SQL docs.
len(string)Returns the number of characters in string. Trailing white space is excluded. 
patindex(pattern,string)Returns the position of the first occurrence of pattern in string. See the MS SQL docs
quotenameSee the MS SQL docs.

replace(string,pattern, replacement)

Replaces all occurences of pattern with replacement in the string provided. See the MS SQL docs.
replicate(string,int)Replicate string the specified number of times.
reverse(string)Returns the string in reverse character sequence.
right(string,index)Returns the right part of string to the specified index.
soundexSee the MS SQL docs.
space(int)Returns a string of white space characters.
str(float,length,decimal)See the MS SQL docs
stuff(string,
 start,
 length,
 replaceWith)
Inserts replaceWith into string. Deletes the specified length of characters in string at the start position and then inserts replaceWith. See the MS SQL docs.

General Syntax

Syntax ItemDescription
Case SensitivitySchema names, table names, column names, SQL keywords, function names are case-insensitive in LabKey SQL.
Comments Comments that use the standard SQL syntax can be included in queries. '--' starts a line comment. Also, '/* */' can surround a comment block:

-- line comment 1
-- line comment 2
/* block comment 1
    block comment 2 */
SELECT ... 

IdentifiersIdentifiers in LabKey SQL may be quoted using double quotes. (Double quotes within an identifier are escaped with a second double quote.) 

SELECT "Physical Exam".*
... 
LookupsLookups columns are columns that see data in other tables.  They are essentially foreign key columns that can be managed through an intuitive user interface.  See Lookups for details on creating lookup columns. Lookups use a convenient syntax of the form "Table.ForeignKey.FieldFromForeignTable" to achieve what would normally require a JOIN in SQL. Example:

Issues.AssignedTo.DisplayName

String LiteralsString literals are quoted with single quotes ('). Within a single quoted string, a single quote is escaped with another single quote.

   SELECT * FROM TableName WHERE FieldName = 'Jim''s Item' 

Date/Time Literals

Date and Timestamp (Date&Time) literals can be specified using the JDBC escape syntax

{ts '2001-02-03 04:05:06'}

{d '2001-02-03'}




Lookups: SQL Syntax


Lookups simplify data integration and SQL queries with an intuitive table linking syntax. LabKey Server understands foreign key columns as "lookups" to columns in other tables and provides a syntax to capture this relationship. Also note that lookups are secure -- before execution, all references in a query are checked against the user's security role/permissions, including lookup target tables.

Lookup SQL Syntax

Lookups have the general form:

Table.ForeignKey.FieldFromForeignTable

Example #1

The following query uses the Datasets table to lookup values in the Demographics table, joining them to the Physical Exam table.

SELECT "Physical Exam".ParticipantId,
"Physical Exam".date,
"Physical Exam".Height_cm,
"Physical Exam".Weight_kg,
Datasets.Demographics.Gender AS GenderLookup,
FROM "Physical Exam"

It replaces the following JOIN statement.

SELECT "Physical Exam".ParticipantId,
"Physical Exam".date,
"Physical Exam".Height_cm,
"Physical Exam".Weight_kg,
Demographics.Gender AS GenderJoin
FROM "Physical Exam"
INNER JOIN Demographics ON "Physical Exam".ParticipantId = Demographics.ParticipantId

Example #2

The following lookup expression shows the Issues table looking up data in the Users table, retrieving the Last Name.

Issues.UserID.LastName

The following expressions show the Demographics table looking up values in the Languages table.

SELECT Demographics.ParticipantId,  
Demographics.StartDate,
Demographics.Language.LanguageName,
Demographics.Language.TranslatorName,
Demographics.Language.TranslatorPhone
FROM Demographics

It replaces the following JOIN statement.

SELECT Demographics.ParticipantId,  
Demographics.StartDate,
Languages.LanguageName,
Languages.TranslatorName,
Languages.TranslatorPhone
FROM Demographics LEFT OUTER JOIN lists.Languages
ON Demographics.Language = Languages.LanguageId;

Other lookup examples:

...
WHERE VialRequest.Request.Status.SpecimensLocked
AND VialRequest.Vial.Visit.SequenceNumMin = ClinicalData.SequenceNum
AND VialRequest.Vial.ParticipantId = ClinicalData.ParticipantId
...

Discover Lookup Column Names

To discover lookup relationships between tables:

  • Go to (Admin) > Developer Links > Schema Browser.
  • Select a schema and table of interest.
  • Browse lookup fields by clicking the icon next to a column name which has a lookup table listed.
  • In the image below, the column study.Demographics.Language looks up the lists.Languages table joining on the column LanguageId.
  • Available columns in the Languages table are listed (in the red box). To reference these columns in a SQL query, use the lookup syntax: Demographics.Language."col_in_lookup_table", i.e. Demographics.Language.TranslatorName, Demographics.Language.TranslatorPhone, etc.
  • Note that the values are shown using the slash-delimited syntax, which is used in the selectRows API. For details on using the query API, see LABKEY.Query.

Adding Lookups to Table/List Definitions

Before lookup columns can be used, they need to be added to definition of a dataset/list. The process of setting up lookup relationships in the property designer is described here: Lookup Columns.

Creating a Lookup to a Non-Primary-Key Field

When you select a schema, all tables with a primary key of the type matching the field you are defining are listed by default. If you want to create a lookup to a column that is not the primary key, you must first be certain that it is unique (i.e. could be a key), then take an additional step to mark the desired lookup field as a key.

  • First create a simple query to wrap your list:
    SELECT * FROM list.MyList
  • You could also add additional filtering, joins, or group by clauses as needed.
  • Next, annotate the query with XML metadata to mark the desired target column as "isKeyField". On the XML Metadata tab of the query editor, enter:
    <tables xmlns="http://labkey.org/data/xml">
    <table tableName="MyList" tableDbType="NOT_IN_DB">
    <columns>
    <column columnName="TargetColumn">
    <isKeyField>true</isKeyField>
    </column>
    </columns>
    </table>
    </tables>
  • Save your query, and when you create lookups from other tables, your list will appear as a valid target.

Related Topics




LabKey SQL Utility Functions


LabKey SQL provides function extensions to help Java module developers and others writing custom LabKey queries access various properties.

moduleProperty(MODULE_NAME, PROPERTY_NAME)

Returns a module property, based on the module and property names. Arguments are strings, so use single quotes not double.

moduleProperty values can be used in place of a string literal in a query, or to supply a container path for Cross Folder Queries.

Examples

moduleProperty('EHR','EHRStudyContainer')

You can use the virtual "Site" schema to specify a full container path, such as '/home/someSubfolder' or '/Shared':

SELECT *
FROM Site.{substitutePath moduleProperty('EHR','EHRStudyContainer')}.study.myQuery

The virtual "Project." schema similarly specifies the name of the current project.

javaConstant(FULLY_QUALIFIED_CLASS_AND_FIELD_NAME)

Provides access to public static final variable values. The argument value should be a string.

Fields must be either be on classes in the java.lang package, or tagged with the org.labkey.api.query.Queryable annotation to indicate they allow access through this mechanism. All values will be returned as if they are of type VARCHAR, and will be coerced as needed. Other fields types are not supported.

Examples

javaConstant('java.lang.Integer.MAX_VALUE')
javaConstant('org.labkey.mymodule.MyConstants.MYFIELD')

To allow access to MYFIELD, tag the field with the annotation @Queryable:

public class MyConstants
{
@Queryable
public static final String MYFIELD = "some value";
}



Query Metadata


Tables and queries can have associated XML files that carry additional metadata information about the columns in the query. Example uses of query metadata include:
  • Data display formatting
  • Define the column display title
  • Add URLs to data in cells
  • Add custom buttons menu items that navigate to other pages or call JavaScript methods.
  • Disable the standard insert, update, and delete buttons.
  • Color coding for values that fall within a numeric range
  • Configure lookups on columns
  • Create an alias field, a duplicate of an existing field to which you can apply independent metadata properties
You can edit or add to this metadata either using:

Edit Metadata using the User Interface

The metadata editor offers a subset of the features available in the field properties editor and works in the same way.

  • Open the schema browser via (Admin) > Developer Links > Schema Browser.
  • Select an individual query/table in the Query Schema Browser and click Edit Metadata.
  • When you click anywhere along the row for a field, you activate that field for editing and open the properties editor to the right, which includes three tabs:
  • To change a column's displayed title, edit its Label property.
  • In the image above, the displayed text for the column has been changed to read "Average Temperature" (instead of "Average Temp"). Notice the wrench icon on the left indicating unsaved changes.
  • You could directly Edit Source or View Data from this interface.
  • If you were viewing a built-in table or query, you would also see an Alias Field button -- this lets you "wrap" a field and display it with a different "alias" field name. This feature is only available for built-in queries.
  • Click Save when finished.

Edit Metadata XML Source

The other way to specify and edit query metadata is directly in the source editor. When you set field properties and other options in the UI, the necessary XML is generated for you and you may further edit in the source editor. However, if you wanted to apply a given setting or format to several fields, it might be most efficient to do so directly in the source editor. Changes made to in either place are immediately reflected in the other.

  • Click Edit Source to open the source editor.
  • The Source tab shows the SQL query.
  • Select the XML Metadata tab (if it not already open).
  • In the screenshot below a conditional format has been applied to the Temp_C column -- if the value is over 37, display the value in red.
  • Click the Data tab to see some values displayed in red, then return to the XML Metadata tab.
  • You could make further modifications by directly editing the metadata here. For example, change the 37 to 39.
  • Click the Data tab to see the result -- fewer red values, if any.
  • Restore the 37 value, then click Save and Finish.

If you were to copy and paste the entire "column" section with a different columnName, you could apply the same formatting to a different column with a different threshold. For example, paste the section changing the columnName to "Weight_kg" and threshold to 80 to show the same conditional red formatting in that data. If you return to the GUI view, and select the format tab for the Weight field, you will now see the same conditional format displayed there.

Another example: the following XML metadata will hide the "Date" column:

<tables xmlns="http://labkey.org/data/xml"> 
<table tableName="TestDataset" tableDbType="NOT_IN_DB">
<columns>
<column columnName="date">
<isHidden>true</isHidden>
</column>
</columns>
</table>
</tables>

Other metadata elements and attributes are listed in the tableInfo.xsd schema available in the XML Schema Reference.

Note that it is only possible to add/alter references to metadata entities that already exist in your query. For example, you can edit the "columnTitle" (aka the "Title" in the query designer) because this merely changes the string that provides the display name of the field. However, you cannot edit the "columnName" because this entity is the reference to a column in your query. Changing "columnName" breaks that reference.

For additional information about query metadata and the order of operations, see Modules: Query Metadata.

Alias Fields

Alias fields are wrapped duplicates of an existing field in a query, to which you can apply independent metadata properties.

You can use alias fields to display the same data in alternative ways, or alternative data, such as mapped values or id numbers. Used in conjunction with lookup fields, an alias field lets you display different columns from a target table. An example use case is described in the following support board discussion: Specimen Repository Alias GlobalUniqueID.

To create an alias field:

  • Click Alias Field on the Edit Metadata page.
  • In the popup dialog box, select a field to wrap and click Ok.
  • A wrapped version of the field will be added to the field list.
    • Rename the field, change its data type, and set properties as desired.

Related Topics




Query Metadata: Examples


This topic provides examples of query metadata.

Auditing Level

Set the level of detail recorded in the audit log. The example below sets auditing to "DETAILED" on the Physical Exam table.

<tables xmlns="http://labkey.org/data/xml">
<table tableName="Physical Exam" tableDbType="NOT_IN_DB">
<auditLogging>DETAILED</auditLogging>
<columns>
...
</columns>
</table>
</tables>

Conditional Formatting

The following adds a yellow background color to any cells showing a value greater than 72.

<tables xmlns="http://labkey.org/data/xml">
<table tableName="Physical Exam" tableDbType="NOT_IN_DB">
<columns>
<column columnName="Pulse">
<columnTitle>Pulse</columnTitle>
<conditionalFormats>
<conditionalFormat>
<filters>
<filter operator="gt" value="72" />
</filters>
<backgroundColor>FFFF00</backgroundColor>
</conditionalFormat>
</conditionalFormats>
</column>
</table>
</tables>

Wrapping/Surfacing Calculated Columns

The following example shows how to surface columns of calculated values in other datasets.

Suppose you have weight data recorded in one dataset (Physical Exam) and height data recorded in another dataset (Demographics). You want to accomplish three things with this data: (1) convert the height data, which is records in inches, into meters, (2) calculate the BMI (Body Mass Index) from the height and weight data, and (3) surface these calculated columns in the Physical Exam dataset.

The following query accomplishes the first two tasks, by including two calculated columns:

  • Height_m (which converts the base data in inches to meters)
  • BMI (which uses the formula Height/Weight^2 to calculate the BMI)
BMI Query
SELECT "Physical Exam".ParticipantId,
"Physical Exam".LSID,
"Physical Exam".Weight_kg,
Datasets.Demographics.Height_inch*0.0254 AS Height_m,
Weight_kg/POWER(Datasets.Demographics.Height_inch*0.0254, 2) AS BMI
FROM "Physical Exam"

How do you accomplish the third task, surfacing the calculated columns so they can be displayed in the Physical Exam table?

To do this, use query metadata to wrap the BMI query as a lookup (= foreign key) from the Physical Exam dataset. Notice that we have included the LSID column in the BMI query to function as a key column. The following metadata on Physical Exam adds a wrapped column over LSID as a lookup to the