GSoC 2026 Final Report: Data Density Heatmap, PCDC Data Portal

This is my Google Summer of Code 2026 work product with Data for the Common Good (D4CG). It covers the project goals, what shipped into the PCDC Data Portal, where the code lives, the current state, and what remains.

Contents#

1. Project information
2. Goals
3. Approach
4. What shipped
5. Current state
6. What’s left
7. Code
8. Challenges and lessons
9. About the author

Project information#

Goals#

From D4CG’s GSoC ideas list, Project 5:

Develop a web-based heatmap visualization tool that represents the completeness and distribution of data across an entire dataset. Display data “density” by GraphQL node types and their specific attributes, so users can quickly identify areas with high or low data availability. Support researchers and data managers in assessing dataset quality and guiding curation.

Expected outcome: a configuration-driven heatmap generated from live GraphQL. Project size: 350 hours.

The PCDC Explorer already shows what is in a cohort. Summary charts, tables, and survival views do not show which fields are sparsely filled. Density here is completeness, not geography:

completeness = availableCount / totalCount

availableCount comes from Guppy histogram aggregations. totalCount is the filtered cohort size. Values are clipped at 1 so nested histograms cannot overshoot. Percentages use at most two decimal places, with trailing zeroes dropped.

In practice: out of every hundred subjects in the current cohort, how many have this field filled in? A data manager should be able to see that before designing a study around a sparse variable.

Approach#

During proposal season I built a standalone prototype, data-density-heatmap-application ( demo): Next.js, TypeScript, D3, a seeded GraphQL endpoint, and CSV/SVG export. It matches the original idea list: nodes on one axis, attributes on the other, cells colored by nonNullCount / totalRecords.

The product is not that sidecar. Researchers already work in the PCDC Data Portal. The useful path was a new Explorer view on the same cohort, the same filters, and the same portal config.

Downloading every subject row to count nulls would freeze the portal. The live view therefore uses Guppy _aggregation queries: histogram counts per field, grouped by GraphQL node, rendered as completeness strips using the portal’s rose / bee / lime tokens.

The prototype remained a design reference. The D3 cell matrix did not ship in Explorer.

What shipped#

Work landed in sequence so each step could go upstream on its own.

Tab and configuration. PR 709, merged 22 June 2026. A density heatmap view in ExplorerVisualization, a placeholder card, field utilities, tests, and:

"heatmapConfig": {
  "enabled": true
}

Enabled in pcdc.json. Disabled in default.json, so other commons do not get the tab unless they opt in.

User agreement. PR 716, merged 4 July 2026. The view uses the same agreement pattern as Survival Analysis before showing cohort completeness.

Functional heatmap. PR 722, merged 15 July 2026. The live view:

• Shows total records and fields tracked.
• Groups rows by the first segment of the GraphQL path (histologies.*, survival_characteristics.*, subject-level fields).
• Renders each field as a label, the raw path, a completeness strip, percentage, available count, and missing count.
• Defaults to curated fields from Object.keys(chartConfig).
• Adds a Filter-related fields section for active OPTION, RANGE, and ANCHORED filters via extractFieldsFromFilter.
• Offers Show all fields for the full Guppy mapping.

The aggregation query, simplified:

query ($filter_main: JSON) {
  _aggregation {
    main: subject(filter: $filter_main, accessibility: all) {
      sex { histogram { key count } }
      histologies {
        age_at_course_anc_500 { histogram { key count } }
      }
    }
  }
}

The client walks the response along each field path, sums histogram counts, divides by cohort size, and paints the strip.

Field labels. PR 729, merged 6 August 2026. Nested paths such as histologies.age_at_course_anc_500 were rendering as “Histologies Age At Course Anc 500”, repeating the node already used as the section header.

1. Prefer fieldMapping / filterConfig.info labels when present.
2. Otherwise strip the first path segment and capitalize the attribute.
3. Leave top-level fields such as file_type as File Type.
4. Keep the raw GraphQL path on the second line.

Before:

After:

Progressive loading. PR 735. Show all fields currently issues one GraphQL document for every mapped attribute. That query can take minutes and block Summary and Table while the response is in flight.

The follow-up splits fields by category, fetches with concurrency 2, stores the job in Redux as densityHeatmapResult so leaving the tab does not cancel work, reuses cache for the same filter and field set, starts a new job on filter change, paints skeletons then rows, and prioritizes visible sections. New files live on feat/density-heatmap-progressive-load:

• src/redux/explorer/densityHeatmapAPI.js
• src/redux/explorer/densityHeatmapThunks.js
• updates in slice.js and the heatmap component

This work is not on pcdc_dev yet.

Current state#

On pcdc_dev, Explorer exposes a density heatmap tab when heatmapConfig.enabled is true. PCDC config already has that on. A researcher can open the view, see completeness for curated fields on the current cohort, read labels that match the rest of the portal, and see filter-related fields in their own section. Nested fields no longer repeat the node name.

What it is not:

• Not the D3 cell matrix from the prototype. Explorer uses completeness strips grouped by node, which scans better for a nested clinical model. buildDensityHeatmapModel remains in utils.js from the first design and is unused by the live view.
• Not finished for Show all fields. That path still uses a single aggregation query.
• Not a row-level data download. Density is a map of missingness.

Clone chicagopcdc/data-portal on pcdc_dev for the live behavior. Check out feat/density-heatmap-progressive-load on the fork for the progressive loader.

What’s left#

1. Land progressive loading. Failed category fetches should not count as loaded. Retry up to three times, then show an error for that slice. Visible sections should take the next worker slot. Then merge to pcdc_dev.
2. Harden Show all fields. After chunking, slow categories still need timeouts, partial paint, and a retry control in the UI.
3. buildDensityHeatmapModel. Either wire a bucketed raw-data mode, or remove the unused helper.
4. Export. The prototype could write CSV and SVG. The portal view cannot yet.
5. Empty cohorts. Completeness is already guarded at totalCount === 0. The empty-state copy can be clearer than “No density data available for the configured fields.”
6. User documentation. A short note in the portal guide: what the colors mean, why curated fields are the default, and when to use Show all fields.

The idea list asked for a configuration-driven heatmap on live GraphQL. That view is in Explorer. The remaining work is keeping the rest of the explorer usable when the full field set is requested.

Code#

Merged upstream#

On a branch#

Commits#

• 73ef3c53 feat(explorer): add density heatmap placeholder tab
• b6d8468d feat(heatmapConfig): toggle visibility from config
• 65fbc6a6 feat(density-heatmap): user agreement, same pattern as Survival Analysis
• bcb72a1c feat(ExplorerDensityHeatmap): functional view
• 9f70dad1 fix: ESLint
• f54f716d fix(density-heatmap): strip node prefix from field labels
• 1286d9a2 feat(density-heatmap): load densities by category in the background

Files to extend#

• src/GuppyDataExplorer/ExplorerDensityHeatmap/index.jsx
• src/GuppyDataExplorer/ExplorerDensityHeatmap/utils.js
• src/GuppyDataExplorer/ExplorerDensityHeatmap/utils.test.js
• src/GuppyDataExplorer/ExplorerDensityHeatmap/ExplorerDensityHeatmap.css
• Wiring: ExplorerVisualization/index.jsx (heatmapConfig.enabled, primaryFields={Object.keys(chartConfig)}, expandByFilter)
• Config: data/config/pcdc.json, data/config/default.json
• Types: src/redux/explorer/types.d.ts, src/redux/explorer/utils.js

Prototype#

• Repository: devSuryansh/data-density-heatmap-application
• Demo: data-density-heatmap-application.vercel.app

Challenges and lessons#

Ship in the portal, not beside it. A standalone heatmap would have been easier to demo. It would not have been used. Density has to sit on the same cohort and filters as Summary and Table.

Aggregate, do not scan rows. Completeness for a commons this size belongs in histogram counts, not in a browser-side null tally.

One query does not scale to the full mapping. Curated fields are a reasonable default. Show all fields needs batched, cancellable loads from the start.

Reuse portal conventions. Colors from Gen3 tokens, curated fields from chart config, field lists from GuppyWrapper, agreement flow from Survival Analysis. Parallel pipes go stale when a data release lands.

Labels are part of the feature. histologies.age_at_course_anc_500 is a correct path. It is a poor title. Mapping first, then strip the node.

Test the pure helpers. Histogram walking in the component is messy. formatDensityPercentage, getDensityHeatmapFieldLabel, and extractFieldsFromFilter are not. Sixteen cases in utils.test.js are why the label change did not break the heatmap.

The original idea was a matrix. The lasting design problem is a queue: how to tell the truth about every field without locking Explorer.

What’s next#

I still intend to land progressive loading on pcdc_dev, add export for curation tickets, and keep the view usable when Show all fields is on. The heatmap is in the portal. The full-field path still needs a slower, safer engine.

About the author#

Suryansh Singh is a Computer Science student at the Noida Institute of Engineering and Technology. During GSoC 2026 he worked with Data for the Common Good on the PCDC Data Portal.

• Website: devsuryansh.in
• GitHub: @devSuryansh
• LinkedIn: suryansh–singh
• X: @0xSuryanshHere
• Email:

Acknowledgements#

Thank you to Brian Furner, Luca Graglia, and Tianyun Zhang for mentorship this summer, to the D4CG and PCDC teams for making room for this view in Explorer, and to Google Summer of Code. The researchers who use this tab now have a color for the fields that cannot yet answer their questions.

Google Summer of Code 2026 · Data for the Common Good · PEDS-1599 · chicagopcdc/data-portal