Add DICOM SR Support for additional/common annotations

[dannyrb]

I think on first pass we should support:

• Length (done)
• Bidirectional (WIP)
• arrow annotation (…is this doable, could be a single point + a descriptive label?)
• Probe (control point)
• Ellipse
• Rectangle

@pieper has kindly provided some guidance on how to approach these mappings in a discussion with @galelis, @JamesAPetts, and @dannyrb

@galelis, if you transition off this issue, please provide insights/next-steps in a follow-up comment.

[dclunie]

There are 2D (image) relative coordinates in DICOM SR, and 3D (frame of reference e.g., patient, volume, slide).

All of Graphic Types specified for those that are co-planar with a referenced image orientation should be supported (3D types that are not in the same plane as the image would require volume "awareness", which may be out of scope for now):

2D: POINT, MULTIPOINT, POLYLINE, CIRCLE, ELLIPSE
3D: POINT, MULTIPOINT, POLYLINE, POLYGON, ELLIPSE

See:

http://dicom.nema.org/medical/dicom/current/output/chtml/part03/sect_C.18.6.html
http://dicom.nema.org/medical/dicom/current/output/chtml/part03/sect_C.18.9.html

In DICOM SR TID 1500 (which is the template to focus on), see examples of use in these templates:

http://dicom.nema.org/medical/dicom/current/output/chtml/part16/chapter_A.html#sect_TID_1410
http://dicom.nema.org/medical/dicom/current/output/chtml/part16/chapter_A.html#sect_TID_1411
http://dicom.nema.org/medical/dicom/current/output/chtml/part16/chapter_A.html#sect_TID_320

Examples at:

http://dicom.nema.org/medical/dicom/current/output/chtml/part17/sect_RRR.5.html

David

PS. Note that "generic" SR annotation rendering (as opposed to authoring) code may theoretically "ignore" what template is defined to be or appears to be used, and just search the tree for any SCOORD or SCOORD3D content item, and then explore the local neighborhood for children that are images, and parents that describe its context (i.e., any associated description, identification, measurements, etc.). That approach is robust and can support all sorts of SR templates and applications from the generic TID 1500 to Mammography CAD, as well as private templates.

[dclunie]

Agree "arrows" should be single point.

[JamesAPetts]

Thanks the input David, I thought as much for the arrow.

Yeah the inital scope for this issue would be the in plane annotations that cornerstone supports by default.

Out of plane annotations as well as TID 1411 (?, off the top of my head as I'm on a train, the volumetric measurements for ROIs) will come down the line as separate issues.

[dannyrb]

... "generic" SR annotation rendering (as opposed to authoring) code may theoretically "ignore" what template is defined to be or appears to be used, and just search the tree for any SCOORD or SCOORD3D content item, and then explore the local neighborhood for children that are images, and parents that describe its context (i.e., any associated description, identification, measurements, etc.). - @dclunie

This is a good note. When our application is not the author of the report, and without additional assurances/constraints, this may be the best we can do without introducing hazards.

Overview

Current approach

We parse and render the contents of a DICOM SR by doing the following:

import * as dcmjs from 'dcmjs';

const { DicomMetaDictionary, DicomMessage } = dcmjs.data;

  setContentFromByteArray(byteArray) {
    const arrayBuffer = byteArray.buffer;
    const dicomData = DicomMessage.readFile(arrayBuffer);
    const dataset = DicomMetaDictionary.naturalizeDataset(dicomData.dict);
    dataset._meta = DicomMetaDictionary.namifyDataset(dicomData.meta);

    const mainData = getMainData(dataset);
    const contentSequence = getContentSequence(dataset);
    const content = (
      <>
        {mainData}
        {contentSequence}
      </>
    );

    this.setState({
      content,
    });
  }

A sopClassHandler is registered that creates displaySets for DICOM SR SOP Class UIDs. The OHIFDicomHtmlViewport uses the DicomLoaderService to get the byteArray for a given DisplaySet. It then sets that byteArray as a prop for the Viewport. The Viewport uses DCMJS, as outlined above, to parse and display the content in a more or less raw format. When mapping a measurement to tool state, we use the parsed dataset with a DCMJS adapter.

Issues with current approach

• Extensions are forced to have a hard dependency on DCMJS and the DicomLoaderService
• Extensions are required to have an intimate knowledge of DICOM SR to extract/persist data
• Extensions that need the same information would dispatch multiple (duplicate) network requests
• Mapping SR measurements to tools can be done in a more interoperable way
• Mapping SR measurements to CSTools' State bypasses the platform and makes CSTools the source of truth
• We are unable to display measurements created by other sources

Goals

• Centralize DICOM SR Parsing / loading
• Broad support for rendering DICOM SR measurements
• Ability to create interoperable DICOM SRs that have edit support for primitives
• Ease burden of DICOM SR knowledge for extension authors
• Ease ability to create new extensions that can leverage information in DICOM SRs
• (Showcase) Best practices around consuming enhanced data services in a practical example (DICOM SR Extension)

Blockers

• MVP completion of the MeasurementService
• We should complete the decoupling of the "MeasurementsPanel" extension, so we can remove it for this feature showcase. This extension could reuse the components, but should provide them itself, and marry them with DICOM SR functionality in @ohif/core
• Current studyMetadataManager usage is convoluted

Phases

This task is too large to tackle in one go. Here are some reasonable milestones we can use to iteratively add value:

Shift where/how we query and store DICOM SR information

The MeasurementService works well for our critical path, but it doesn't encapsulate all of the information contained in an SR and it's not immutable. We likely want to extend the existing studyMetadataManager that contains instances of OHIFStudyMetadata. It can be used to hydrate the MeasurementService.

1. Add a sopClassHandler for DICOM SR
   • No longer show SR thumbnails
2. Register/add a copy of the SRs information to a "data service" like the studyMetadataManager
3. Replace the @ohif/extension-dicom-html logic with a query to the data service that fetches and displays content similar to what is already shown

Display DICOM SR measurements in a table and on a 2D canvas

The simplest approach here is likely what @dclunie outlined above. We may want to limit the (initial) supported measurements to SCOORD as they'll have a reference image and are easier to map to Cornerstone's imageIdSpecificToolState. While we can support SCOORD3D, how that impacts rendering and tabular data is a UX concern that requires more thought.

1. Display a list of SRs in a DICOM SR side panel
   • Order from most recent to oldest
2. Similar to DICOM Seg functionality, hydrate the measurements service with the most recent report. Selecting a previous report should swap out Measurements specific to one report with the newly selected reports and trigger a re-render. Some meta summary for each report should exist.
3. It is problematic/hazardous to assume, without further guarantees, that the parsed measurements can be safely mapped to "edit" tools. Until we have more information, we should use one of the following strategies for render-only behavior:
   • A single tool that, given some schema, can render POINT, MULTIPOINT, POLYLINE, CIRCLE, and ELLIPSE value types for a given imageId
     • If we're feeling ambitious, this could be the beginning of a tool state SVG renderer that allows interop between VTK and Cornerstone
   • OR we duplicate existing tools (length, freehand, circle, ellipse) with alternative tool names in the "enabled" (read-only) mode that we designate as renderers for mapped measurements

This may sound more straightforward than it is. It's worth looking over the various templates and how they group measurements. We'll need to account for related measurements that are linked by UID (and span measurement groups), as well as a few other scenarios. This may impact how we display information in our table. Some measurements will explicitly map to a POINT or POLYLINE, while others are implied or derived from neighboring information.

Notes:

• Do timepoints play a part here?
• Would we ever want to show multiple SRs measurements at the same time?

When a TID 1500 Report originated from a supported OHIF version/schema, map to edit tools

We currently accomplish this by keying off of a Measurement's "Code Meaning". For example, we encode a POLYLINE that has a CodeMeaning of Length, then assume if we filter/match on that string key, the measurement works with the cornerstone-tools Length Tool. (This currently takes place in a DCMJS "adapter", not within this monorepo).

The TID 1500 includes Author Observer Sequence. This would allow us to encode information about the application that generated the report and to make further assumptions about the encoded data. For example, if we see that there is an Author Observer (there may be a better field for this) that has:

• (0018,1020) Software Version(s) X
• (0008,0080) Institution Name Y
• etc.

We may be able to reasonably assert that because our application supports Application Y and Software Version X that the encoded measurements are no longer generic, but rather, map 1:1 to our application's tools.

The biggest change here is identifying what information we key off of, and how we might go about supporting schemas as we continue to version/grow what we do support. Ideally, this would be extensible as we grow this to support multi-step and composite tools, but I'm likely getting ahead of myself.

1. Spell out the information we set when persisting information and restoring it
2. Create new fields in the Data Service that help indicate the OHIF version/schema associated, if any, with the SR
3. If the application supports the defined version/schema, map tool state to edit tools instead of the generic render tools
4. Support round trip persistence

This may sound more straightforward than it is. There are several "gotchas" that may not be immediately obvious. For example:

There is no "rectangle" value type. A rectangle ROI would be encoded as a POLYLINE. Rendering a POLYLINE is simple enough, but if we need to know this should map to a "rectangle tool", we would need to look at the points more closely to identify that. Knowing when a square POLYLINE maps to a square tool, a rectangle tool, or a freehand roi tool is an interesting problem.

POLYLINEs are only "closed" if the first point is the same as the end point.

POLYLINEs do not guarantee a "winding" order. You cannot assume that the points are read from left to right, or right to left. This makes encoding an "arrow" with an end point an interesting problem. We may need to use a single point (end point), and a magnitude/direction

Support for persisting tools outside the scope of primitive value types (far future)

We will likely need to create a custom "Code Scheme Designator", "Code Meaning"s, and "Code Value"s when persisting tool state for tools that can't be encoded using a single value type. A GitHub repository (and npm package) can describe these concepts, and create an avenue for other systems that would like to consume the data beyond simply rendering/listing the values.

I am currently unsure of how we might provide a path to add these dynamically or ad-hoc through extensions. Interoperability beyond our application may only be possible with published definitions and a concerted effort.

Not Discussed

SCOORD3D Support

While we have a frame of reference tool state manager in cornerstone tools, it's not commonly used. For most measurements (created in OHIF), we'll want to tie them to a specific instance. As we broaden VTK annotation support, SCOORD3D support will becomes more pressing.

Pathway for rendering w/ VTK Viewports

...

Automatic mapping

While we could make optimistic guesses to map SCOORD and SCOORD3D to edit tools, the outcome is likely too hazardous to pursue. A future enhancement could be made to allow manually transferring the state to a tool that supports that same general schema, but with a very blatant warning.

If there is an obvious pattern where MANUFACTURER maps to these tools, we could build this in as a more robust feature where users can setup the automatic mappings.

Akin to using ImagerPixelspacing or calibrating an image's pixel spacing. "You can do this, but at your own risk."

I am doubtful we will have the bandwidth to tackle this anytime soon, but I believe it's worth listing as it may catch the right financier's eye <3

Parting Words

Please feel free to push on any/all of this. This is my best guess at our path forward with a few sparse implementation details peppered in. I'm fairly new to DICOM SR, and curious to hear how other implementers are tackling these same problems.

Note: I've created a Structured Reports :memo: label to tie together related issues.

[JamesAPetts]

No longer show SR thumbnails

Some meta summary for each report should exist.

Just so we are on the same page, the HTML render will still be available, just accessed from elsewhere?

A single tool that, given some schema, can render POINT, MULTIPOINT, POLYLINE, CIRCLE, and ELLIPSE value types for a given imageId

OR we duplicate existing tools (length, freehand, circle, ellipse) with alternative tool names in the “enabled” (read-only) mode that we designate as renderers for mapped measurements

Duplicating the tools is overkill I reckon? We could add a locked state for all toolState that makes them immutable. This may be too intrusive to CST (unless this a feature others would value?). I think the “SR measurement render only tool” you suggested is better.

If we’re feeling ambitious, this could be the beginning of a tool state SVG renderer that allows interop between VTK and Cornerstone

I feel like this is a mix of concerns that will only confuse intergation, and is better left as a separate ticket/project later.

Would we ever want to show multiple SRs measurements at the same time?

Ideally yes, if they contain different data and not just a “corrected” version, but how could we possibly work this out? :thinking_face: . I don’t think we can. Maybe we have to limit to only one SR per series (even that is a little complicated as SRs can reference lots of images from a study I believe). Of course other than a PWA, a more complicated backend could triage time versioned SR.

We will likely need to create a custom “Code Scheme Designator” …. I am currently unsure of how we might provide a path to add these dynamically or ad-hoc through extensions.

I think the OHIF standard we make must be consumed and added to a Code Scheme library. An extension could add its on CSD + definitions to this library (e.g. an extension adds a tool and how to store it?).

[dannyrb]

Just so we are on the same page, the HTML render will still be available, just accessed from elsewhere?

The HTML Viewport Extension would use the studyMetadataManager or similar service to query for the data it needs to render similar output. Setting it as a viewport, if we remove SR thumbnails, would require an action to be dispatched to Redux.

Improved programmtic control of the Viewport Grid and Viewports is something we have slated.

Duplicating the tools is overkill I reckon?

csTools.addTool(csTools.LengthTool, { name: 'DicomSrLength' });
csTools.setToolEnabled('DicomSrLength');

But I agree. I think a render only tool would be cleaner.

I feel like this is a mix of concerns that will only confuse integration, and is better left as a separate ticket/project later.

I agree. A pipe dream of mine.

I think the OHIF standard we make must be consumed and added to a Code Scheme library.

👍

[galelis]

Regarding the way we display the SRs and how to get old SRs

Just so we are on the same page, the HTML render will still be available, just accessed from elsewhere?

As SR is also used for more things other than just measurements, we could filter the DICOM SR with measurements from thumbnail, maybe using (0018,1020) Software Version(s) X or (0008,0080) Institution Name or other tag. And display the non-measurement ones as we use to do now.

Would we ever want to show multiple SRs measurements at the same time?

We could have a runtime load of a new SR, similar to what we have on series-list <-> viewports, maybe drag and drop into the table or a simple drop down with old SRs (this was suggested by Steve when I first add measurement table on Meteor version)

Maybe we have to limit to only one SR per series (even that is a little complicated as SRs can reference lots of images from a study I believe)

Now a days, we use dcmjs and store one SR per Study, doing one by series, I don't personally don't see any gain on having it by series.

Regarding the way we display the tools, I like the idea of render only tool.

Have you guys thought about tool versioning and synchronization with what we have on a DICOM SR? I mean, what if the data we stored in the past, new app does not know how to render anymore, like the structure has changed..

[fedorov]

Wow, that's a lot of stuff guys! I have to say I could not follow all of the points in the discussion due to lacking understanding of the OHIF internals perhaps. My overall suggestion would be to consider focusing on specific use cases, find a very specific customer that will be invested in using this functionality and provide feedback to guide development. Just one TID 1500 template has so many capabilities and usage flavors that even exhaustively covering those is a huge project!

On a positive side, I guess (considering the "I am doubtful we will have the bandwidth to tackle this anytime soon" comment!), planar annotations are not an immediate priority for the IDC project. Our current target is development of the MVP (see our plans in the IDC MVP document linked from here), which will provide static view of a subset of TCIA collections. None of those collections (with perhaps the exception of the annotations produced by CCC for some of them) have planar annotations. It is much more important for IDC to be able to handle measurements and SR objects derived from volumetric segmentations - plenty of datasets of that kind, and more on the way.

I promised to @pieper and @swederik that I will submit all issues that I encountered and also create issues for functionality we need for IDC MVP. We can discuss those after that in a call to hopefully bring everyone to the same page.