COMET — The CanDIAPL Project

What is CanDIAPL?

The Canadian Data-Intensive Astrophysics PLatform (CanDIAPL) is a national, production-grade ecosystem designed to help Canadian astronomers work effectively with petabyte-scale survey data.

CanDIAPL pairs on-site streaming compute at SKA pathfinders (MeerKAT in South Africa; MWA in Australia) with dedicated off-site storage and analysis at the Alliance/CANFAR Nibi cloud to turn raw throughput into science-ready products in near real time.

The goal is to abstract away infrastructure complexity, providing researchers with a unified interface for multi-messenger data access so they can focus on scientific analysis.

Three Software Pillars

On top of the hardware layer, three integrated software pillars serve both time-domain and "static-sky" use cases:

Dynamic Datasets(DDS)
Processes survey data from Rubin and radio facilities through automated pipelines, keeping catalogs and image products continuously up to date. Supports user-defined triggers that execute when specified science conditions are met, enabling researchers to act on evolving datasets as new observations arrive.
Real-Time Analysis(RTA)
Ingests alert streams from Rubin, SKA pathfinders, and other facilities into FASTDB, a high-throughput time-series database. Containerized classifiers score each alert; high-confidence events are routed to curated science datasets, while ambiguous cases surface in a Transient Marshal interface for human review. Machine learning models are refined over time based on these decisions, improving classification accuracy for future alerts.
Multi-Messenger Portal(MMP)
A unified web interface for browsing images, cubes, catalogs, and alerts across optical, radio, gravitational-wave, neutrino, and high-energy datasets. Users can launch notebooks and batch jobs on compute resources co-located with the data. Features include quick cutouts, instant previews of light curves and spectra, and cross-matching across messengers — enabling rapid assessment of regions and events.

Where COMET Fits In

COMET (CANFAR Orchestration for Managed Environment Transfers) is a practical tool developed under the Dynamic Datasets pillar. It streamlines the process of deploying code to CANFAR by automating SSH key setup, GitHub Actions configuration, secret management, and file synchronization.

Rather than manually configuring each component, COMET provides a guided, low-code wizard that:

Generates and uploads SSH keys to CANFAR
Connects to your GitHub account via OAuth
Creates or configures your repository with the correct secrets
Commits a production-ready CI/CD workflow that deploys on push
Offers curated templates and file previews (with rename/collision warnings) before committing

The result: push code to GitHub, and it automatically deploys to CANFAR. Less time on infrastructure, more time on science.

Science Drivers

Initial science drivers span two themes:

Cosmic Explosions & the Transient Universe — Fast radio bursts, supernovae, gravitational-wave counterparts, and other time-critical phenomena that demand rapid cross-wavelength follow-up.
Galaxies, Gas & Dark Matter — Deep surveys of neutral hydrogen (HI), polarization, continuum emission, and optical counterparts to probe galaxy evolution and large-scale structure.

Beyond accelerated discovery, CanDIAPL secures Canadian data rights, standardizes tooling, and delivers inclusive training in high-performance computing and data-intensive methods.

Collaboration & Community

CanDIAPL operates collaboratively with CANFAR, CADC, the Digital Research Alliance of Canada, and international partners. We align with community standards for open science:

We welcome feedback, bug reports, and contributions from the astronomy community. If you have ideas for improving COMET or CanDIAPL tooling, please get in touch.

Learn More

Visit the official CanDIAPL website for news, publications, and team information:

candiapl.ca →