Compliance: Logging

 2024-04-15
Premium Feature — Available in the Enterprise Edition of LabKey Server. Learn more or contact LabKey.

Compliance module logging is designed to answer questions broader than those that can be answered using the main audit log, such as:

  • Which users have seen a given patient's data? What data was viewed by each user?
  • Which patients have been seen by a particular user? What data was viewed for each patient?
  • Which roles and PHI levels were declared by each user? Were those declarations appropriate to their job roles & assigned responsibilities?
  • Was all data the user accessed consistent with the user's declarations?
To configure Compliance Logging, use the (Admin) > Folder > Management > Compliance tab.

Topics

What Gets Logged

The default behavior is to log only those queries that access PHI columns.

To open the Audit Log:

  • Select (Admin) > Site > Admin Console.
  • Under Management click Audit Log.
  • The following compliance-related views are available on the dropdown:
    • Compliance Activity Events - Shows the Terms of Use, IRB, and PHI level declared by users on login.
    • Logged query events - Shows the SQL query that was run against the data.
    • Logged select query events - Lists specific columns and identified data relating to explicitly logged queries, such as a list of participant id's that were accessed, as well as the set of PHI-marked columns that were accessed.
    • Site Settings events - Logs compliance-related configuration changes to a given folder, that is, changes made on a folder's Compliance tab.
    • User events - Records login and impersonation events.

To change the logging behavior of a folder, see Compliance: Configure PHI Data Handling.

ParticipantID Determination and PHI

In order to log query events, it must be clear what data for which specific participantIDs has been accessed. This means that queries must be able to conclusively identify the participantID whose data was accessed. In situations where the participantID cannot be determined, queries will fail because they cannot complete the logging required.

At the higher Log all query access level, all queries must conform to these expectations. When Log only query access including PHI columns, queries that do not incorporate any columns marked as containing PHI will have more flexibility.

Query scenarios that can be successfully logged for compliance:

  • SELECT queries that include the participantID and do not include any aggregation or PIVOT.
  • SELECT queries in a study where every data row is associated with a specific participantID, and there is no aggregation, whether the participantID is specifically included in the query or not.
  • SELECT queries with any aggregation (such as MAX, MIN, AVG, etc.) where the participantID column is included in the query and also included in a GROUP BY clause.
Query scenarios that will not succeed when compliance logging is turned on:
  • SELECT queries with aggregation where the participantID column is not included.
  • PIVOT queries, which also aggregate data for multiple participants, when they access any PHI columns. Learn more below.

Filter Behavior

When using compliance logging, you cannot filter by values in a column containing PHI, because the values themselves are PHI that aren't associated with individual participant IDs.

When you open the filter selector for a PHI column, you will see Choose Filters and can use a filtering expression. If you switch to the Choose Values tab you will see a warning:

Linked Schemas and Compliance Logging

Compliance logging across linked schemas behaves as follows, where "Source folder" means the folder where the data resides and "Usage folder" means the folder where a linked schema exposes portions of the source folder. For the most complete compliance logging, configure the logging in both the Usage and Source folders.

If you are viewing the compliance logs from the site-wide Admin Console > Audit Log, access in all containers is combined into one set of tables for convenience. You may expose the "Container" column if desired. If you are using the API to query these logs, you may need to know up front which container will contain the records:

  • The creation of the linked schema itself (in the Usage folder) is audited as a "ContainerAuditEvent" in the Usage folder.
  • The access of data is logged in the Source folder, not in the Usage folder.

PIVOT Queries, PHI, and Compliance Logging

PIVOT queries that involve PHI columns cannot be used with compliance logging of query access. Logging is based on data (and/or PHI) access being checked by row linked to a participant. Because PIVOT queries aggregate data from multiple rows, and thus multiple participants, this access cannot be accurately logged. A pivot query involving PHI, run in a folder with the Compliance module running, will raise an error like: 

Saved with parse errors: ; Pivot query unauthorized.

Related Topics