Documentation

Tooling vs Container

Choose based on how you want your code to run on CANFAR.

Tooling Repository

Best for: scripts, notebooks, pipelines, or utilities that run directly on CANFAR nodes.

• How it works: On each push, GitHub Actions or GitLab CI syncs your repo to a CANFAR directory (e.g., /arc/projects/YourProject/your-tool).
• CI file: .github/workflows/deploy-canfar.yml (GitHub) or .gitlab-ci.yml (GitLab)
• Your CANFAR jobs reference: the folder path where your code lands.

Container Repository

Best for: packaged applications, services, or tools with complex dependencies that benefit from containerization.

• How it works: On each push, GitHub Actions or GitLab CI builds a container image and pushes it to CANFAR's Harbor registry (images.canfar.net).
• CI file: .github/workflows/build-docker-harbor.yml (GitHub) or .gitlab-ci.yml (GitLab)
• Your CANFAR jobs reference: the image tag (e.g., images.canfar.net/project/tool:<commit-sha>).

Harbor accepts OCI container images; Docker is the most common runtime, but other container formats (e.g., Singularity/Apptainer) can be used in Kubernetes-backed workflows.

Templates, AI & File Review

Step 2 is now the main staging area for templates, AI-assisted setup, test generation, and file review. Nothing is committed until you approve the final file list.

Template choices

• None — Deployment-only baseline. COMET prepares the workflow files without adding starter README or test files.
• CanDIAmazing | Python 3.10+ | pytest — Pulls the latest committed files from the public CanDIAmazing repository and is best for a batteries-included research package scaffold.
• Generic | Python 3.x | pytest / unittest — Adds a lightweight README + starter test path and is best when you want a minimal baseline.

Quick Filter and AI Fit

• Quick Filter is deterministic: choose desired features and COMET keeps only the matching template file bundles. It never rewrites template contents and has no daily limit.
• AI Fit uses the full template plus your current repo context to adapt the selected features to the repo's language, framework, naming, and tooling. Managed AI currently allows 3 AI Fits per day.
• Custom AI can be enabled in the popup for either feature. You can use COMET-managed AI or bring your own OpenAI, Anthropic, Google, or AWS Bedrock credentials for the current browser session.
• AI Fit and Quick Filter always start from the full template set each run, so rerunning them replaces the previous template result instead of narrowing an already-filtered set.

Coverage estimation and unit test generation

• Use the estimate icon to get a coverage estimate for the entire current repo test suite. The saved estimate stays with the current session until relevant files change.
• Use the unit test generation popup to set a target coverage percentage, then let COMET add only the tests needed to move toward that target. Managed AI currently allows 3 test generations per day.
• Existing tests are preserved. COMET only removes the default generic always-passing starter when it is the only test present and real generated tests are replacing it.
• The generator can infer suitable test paths, use the repo's existing framework when possible, create small fixtures or fixture scripts when needed, and update the committed workflow coverage threshold to match your selected target.
• Each generated test file includes a tooltip blurb describing what that file tests, rather than just a generic AI-generated label.

Reviewing files before commit

• File Overview is always available in Step 2 and can sort by filename or status. Use its open buttons to jump directly into file blocks below.
• New only / All files toggles whether the main editor shows just COMET-managed changes or also existing repo files. Existing files are editable, but they are only committed if you include or modify them.
• Flat / folder view lets you switch between individual file cards and a tree-style explorer. Folder rows never get edit/delete actions; file rows do.
• Resizable commit area lets you drag the lower edge to make the file list scroll internally, or use the expand/collapse arrow to jump between compact and full-height views.
• When you target a new branch in an existing repo, COMET shows the base/default branch files in All files because the new branch does not exist yet.
• On existing repositories, COMET warns about filename collisions before overwriting. There is no automatic merge: if a path collides and you proceed, the existing file will be replaced.

Why CI/CD?

CI/CD (Continuous Integration/Continuous Delivery) automates building, testing, and deploying your code. Here's why it matters for CANFAR work:

• Reproducibility: Every deployment is tied to a specific commit. You always know exactly what code is running.
• Fast iteration: Push code → see it deployed. No manual SSH, rsync commands, or file copying.
• Collaboration: Team members can review changes before they deploy. No "it works on my machine" problems.
• Auditability: GitHub Actions / GitLab CI logs every deployment with timestamps, making debugging easier.
• Security: Secrets/variables stay encrypted in your Git provider—never committed to your repo or exposed in logs.

The simple mental model:

1. You push code to GitHub or GitLab.
2. GitHub Actions or GitLab CI runs your pipeline (tests optional, deploy required).
3. Your code lands on CANFAR (as files or a container image).
4. You run CANFAR jobs that use the updated code.

Secrets Reference

COMET automatically adds these secrets/variables to your Git provider.

• GitHub: GitHub → Settings → Secrets and variables → Actions
• GitLab: GitLab → Settings → CI/CD → Variables (the SSH private key is stored as a File variable)

Tooling Mode

• CANFAR_SSH_PRIVATE_KEY — The PEM from Step 1
• CANFAR_USER — Your CADC username used for CANFAR SSH
• CANFAR_SSH_HOST — ws-uv.canfar.net
• CANFAR_SSH_PORT — 64022
• CANFAR_REMOTE_DIR — Your chosen destination folder

Container Mode

• HARBOR_URL — images.canfar.net
• HARBOR_PROJECT — Your Harbor project name
• HARBOR_REPOSITORY — Your image repository name
• HARBOR_USERNAME — Your Harbor/CADC username
• HARBOR_PASSWORD — For GitHub, a Harbor CLI secret COMET prepares from CADC OIDC; for GitLab, an existing Harbor CLI secret

Custom AI credentials

Custom AI credentials entered in the Custom AI panels are not written to GitHub secrets, GitLab variables, or committed files. They stay in the current browser session only and are sent only when you explicitly run AI Fit or unit test generation with a custom provider selected.

Security note: Rotate secrets if you suspect compromise. You can regenerate a new key in COMET anytime—just run through the wizard again for the same repo.

Path Conventions

For Tooling deployments, pick paths that are stable and descriptive.

• /arc/projects/YourProject/tool-name — Standard project path
• /arc/home/username/tool-name — Personal/development path

COMET's directory browser starts at /, which maps to the CANFAR /arc root (the API exposes everything inside /arc). You can navigate or create new folders directly in the browser.

Nested Folder Option

When configuring deployment, you can choose to deploy into:

• The selected folder directly — Files land in /arc/projects/YourProject/
• A nested subfolder named after your repo — Files land in /arc/projects/YourProject/my-repo/

The nested option is useful when deploying multiple repos into the same project directory.

Workflow Customization

COMET commits a minimal workflow. You can extend it after the initial setup.

Adding Tests

COMET includes a test step by default and can now estimate coverage or generate additional tests before you commit. The AI test flow uses the same coverage target you pick in the popup and updates the committed workflow or CI file to enforce that threshold. You can still expand or replace the generated checks entirely. Click a tab below for language-specific examples:

- name: Run Python tests
  run: |
    pip install -r requirements.txt
    pip install pytest pytest-cov astropy
    pytest --cov=. --cov-fail-under=75

# test_fits_processor.py
import pytest
from astropy.io import fits
from astropy.wcs import WCS
import numpy as np
from fits_processor import extract_coords, validate_header

@pytest.fixture
def sample_fits(tmp_path):
    """Create a minimal FITS file for testing."""
    hdu = fits.PrimaryHDU(np.zeros((100, 100)))
    hdu.header['CRPIX1'] = 50.0
    hdu.header['CRPIX2'] = 50.0
    hdu.header['CRVAL1'] = 180.0  # RA
    hdu.header['CRVAL2'] = 45.0   # Dec
    hdu.header['CDELT1'] = -0.001
    hdu.header['CDELT2'] = 0.001
    hdu.header['CTYPE1'] = 'RA---TAN'
    hdu.header['CTYPE2'] = 'DEC--TAN'
    path = tmp_path / "test.fits"
    hdu.writeto(path)
    return path

def test_extract_coords_returns_skycoord(sample_fits):
    """Verify coordinates are extracted as SkyCoord."""
    coords = extract_coords(sample_fits)
    assert coords.ra.deg == pytest.approx(180.0, rel=1e-3)
    assert coords.dec.deg == pytest.approx(45.0, rel=1e-3)

def test_validate_header_catches_missing_wcs(tmp_path):
    """Ensure missing WCS keys raise ValueError."""
    hdu = fits.PrimaryHDU(np.zeros((10, 10)))
    bad_path = tmp_path / "bad.fits"
    hdu.writeto(bad_path)
    with pytest.raises(ValueError, match="Missing WCS"):
        validate_header(bad_path)

- name: Run JavaScript tests
  run: |
    npm ci
    npm test

// tests/catalog.test.js
const { crossMatch, parseFITS, convertCoords } = require('../src/catalog');

describe('Catalog cross-matching', () => {
  const testSources = [
    { ra: 180.0, dec: 45.0, mag: 15.2 },
    { ra: 180.1, dec: 45.1, mag: 16.8 },
  ];

  test('crossMatch finds sources within tolerance', async () => {
    const matches = await crossMatch(testSources, {
      catalog: 'gaia_dr3',
      radius: 2.0,  // arcseconds
    });
    
    expect(matches).toHaveLength(2);
    expect(matches[0]).toHaveProperty('gaia_id');
    expect(matches[0].separation).toBeLessThan(2.0);
  });

  test('crossMatch returns empty for no matches', async () => {
    const farSource = [{ ra: 0.0, dec: -89.9, mag: 20.0 }];
    const matches = await crossMatch(farSource, {
      catalog: 'gaia_dr3',
      radius: 0.1,
    });
    
    expect(matches).toHaveLength(0);
  });
});

describe('Coordinate conversion', () => {
  test('converts equatorial to galactic', () => {
    const galactic = convertCoords(
      { ra: 266.405, dec: -28.936 },  // Near galactic center
      'equatorial',
      'galactic'
    );
    
    expect(galactic.l).toBeCloseTo(0.0, 0);
    expect(galactic.b).toBeCloseTo(0.0, 0);
  });
});

- name: Run Java tests
  run: |
    mvn -B -ntp test

// src/test/java/org/astro/VOTableParserTest.java
package org.astro;

import org.junit.jupiter.api.*;
import static org.junit.jupiter.api.Assertions.*;
import java.io.*;
import java.util.List;

class VOTableParserTest {

    private VOTableParser parser;

    @BeforeEach
    void setUp() {
        parser = new VOTableParser();
    }

    @Test
    @DisplayName("Parses VOTable with multiple rows")
    void testParseMultipleRows() throws Exception {
        InputStream stream = getClass()
            .getResourceAsStream("/test_catalog.vot");
        
        List<Source> sources = parser.parse(stream);
        
        assertEquals(100, sources.size());
        assertNotNull(sources.get(0).getRa());
        assertNotNull(sources.get(0).getDec());
    }

    @Test
    @DisplayName("Handles missing columns gracefully")
    void testMissingColumns() throws Exception {
        String minimalVOT = """
            <VOTABLE>
              <RESOURCE>
                <TABLE>
                  <FIELD name="ra" datatype="double"/>
                  <DATA><TABLEDATA>
                    <TR><TD>180.0</TD></TR>
                  </TABLEDATA></DATA>
                </TABLE>
              </RESOURCE>
            </VOTABLE>
            """;
        
        List<Source> sources = parser.parse(
            new ByteArrayInputStream(minimalVOT.getBytes())
        );
        
        assertEquals(1, sources.size());
        assertEquals(180.0, sources.get(0).getRa(), 0.001);
        assertNull(sources.get(0).getDec());  // Missing column
    }

    @Test
    @DisplayName("Throws on malformed VOTable")
    void testMalformedInput() {
        String badXML = "<not>valid</xml";
        
        assertThrows(VOTableException.class, () -> {
            parser.parse(new ByteArrayInputStream(badXML.getBytes()));
        });
    }
}

- name: Run Go tests
  run: |
    go test -v -race -coverprofile=coverage.out ./...
    go tool cover -func=coverage.out

// fits/reader_test.go
package fits

import (
    "os"
    "path/filepath"
    "testing"
)

func TestReadHeader(t *testing.T) {
    // Create a minimal FITS file for testing
    testFile := filepath.Join(t.TempDir(), "test.fits")
    createTestFITS(t, testFile)

    header, err := ReadHeader(testFile)
    if err != nil {
        t.Fatalf("ReadHeader failed: %v", err)
    }

    t.Run("extracts BITPIX", func(t *testing.T) {
        bitpix, ok := header.GetInt("BITPIX")
        if !ok {
            t.Error("BITPIX not found")
        }
        if bitpix != -32 {
            t.Errorf("BITPIX = %d, want -32", bitpix)
        }
    })

    t.Run("extracts NAXIS dimensions", func(t *testing.T) {
        naxis1, _ := header.GetInt("NAXIS1")
        naxis2, _ := header.GetInt("NAXIS2")
        
        if naxis1 != 1024 || naxis2 != 1024 {
            t.Errorf("dimensions = %dx%d, want 1024x1024", naxis1, naxis2)
        }
    })

    t.Run("extracts WCS keywords", func(t *testing.T) {
        crval1, ok := header.GetFloat("CRVAL1")
        if !ok {
            t.Error("CRVAL1 not found")
        }
        if crval1 < 0 || crval1 > 360 {
            t.Errorf("CRVAL1 = %f, want 0-360", crval1)
        }
    })
}

func TestReadHeader_InvalidFile(t *testing.T) {
    _, err := ReadHeader("/nonexistent/file.fits")
    if err == nil {
        t.Error("expected error for missing file")
    }
}

func TestReadHeader_NotFITS(t *testing.T) {
    // Create a non-FITS file
    badFile := filepath.Join(t.TempDir(), "bad.fits")
    os.WriteFile(badFile, []byte("not a fits file"), 0644)

    _, err := ReadHeader(badFile)
    if err == nil {
        t.Error("expected error for non-FITS file")
    }
}

- name: Run R tests
  run: |
    Rscript -e "install.packages(c('testthat', 'FITSio'), repos='https://cloud.r-project.org')"
    Rscript -e "testthat::test_dir('tests')"

# tests/test_photometry.R
library(testthat)
library(FITSio)

test_that("aperture_flux returns positive value for star", {
  # Create mock image with a Gaussian "star"
  img <- matrix(0, nrow = 50, ncol = 50)
  for (i in 1:50) {
    for (j in 1:50) {
      img[i, j] <- 1000 * exp(-((i-25)^2 + (j-25)^2) / 18)
    }
  }
  
  flux <- aperture_flux(img, cx = 25, cy = 25, radius = 5)
  expect_gt(flux, 0)
  expect_lt(flux, sum(img))  # Should be less than total
})

test_that("background_estimate excludes source region", {
  img <- matrix(100, nrow = 50, ncol = 50)  # Flat background
  img[23:27, 23:27] <- 5000  # Bright source
  
  bg <- background_estimate(img, exclude_center = TRUE, radius = 10)
  expect_equal(bg, 100, tolerance = 5)
})

test_that("read_fits_data handles missing file gracefully", {
  expect_error(read_fits_data("nonexistent.fits"), "File not found")
})

- name: Run Julia tests
  run: |
    julia -e 'using Pkg; Pkg.instantiate()'
    julia -e 'using Pkg; Pkg.test()'

# test/test_spectrum.jl
using Test
using FITSIO
using SpectrumTools  # Your module

@testset "Line fitting" begin
    # Create synthetic spectrum with a Gaussian emission line
    wavelength = 6500.0:0.1:6600.0
    flux = 100.0 .+ 500.0 .* exp.(-((wavelength .- 6563.0).^2) ./ 4.0)
    
    @testset "detect_lines finds H-alpha" begin
        lines = detect_lines(wavelength, flux, threshold=3.0)
        @test length(lines) == 1
        @test isapprox(lines[1].center, 6563.0, atol=0.5)
    end
    
    @testset "fit_gaussian returns correct parameters" begin
        fit = fit_gaussian(wavelength, flux, 6563.0)
        @test isapprox(fit.amplitude, 500.0, rtol=0.05)
        @test isapprox(fit.sigma, 2.0, rtol=0.1)
    end
end

@testset "FITS I/O" begin
    mktempdir() do dir
        path = joinpath(dir, "test_spec.fits")
        
        @testset "write and read roundtrip" begin
            wave = collect(4000.0:1.0:7000.0)
            flux = rand(length(wave)) .* 100
            write_spectrum(path, wave, flux)
            
            w2, f2 = read_spectrum(path)
            @test w2 ≈ wave
            @test f2 ≈ flux
        end
    end
end

- name: Run pipeline tests
  run: |
    chmod +x tests/run_tests.sh
    ./tests/run_tests.sh

#!/bin/bash
# tests/run_tests.sh
set -e  # Exit on first failure

echo "=== Testing calibration pipeline ==="

# Setup: create test data directory
TEST_DIR=$(mktemp -d)
trap "rm -rf $TEST_DIR" EXIT

# Test 1: Verify bias subtraction
echo "Test: bias subtraction..."
cp testdata/raw_frame.fits "$TEST_DIR/"
cp testdata/master_bias.fits "$TEST_DIR/"
./scripts/subtract_bias.sh "$TEST_DIR/raw_frame.fits" "$TEST_DIR/master_bias.fits"
# Check output exists and has correct dimensions
if [[ ! -f "$TEST_DIR/raw_frame_bias.fits" ]]; then
    echo "FAIL: Output file not created"
    exit 1
fi
echo "PASS"

# Test 2: Verify flat fielding
echo "Test: flat field correction..."
./scripts/apply_flat.sh "$TEST_DIR/raw_frame_bias.fits" testdata/master_flat.fits
MEAN=$(python3 -c "from astropy.io import fits; print(fits.getdata('$TEST_DIR/raw_frame_bias_flat.fits').mean())")
if (( $(echo "$MEAN < 0" | bc -l) )); then
    echo "FAIL: Negative flux after flat fielding"
    exit 1
fi
echo "PASS"

# Test 3: Verify WCS solution
echo "Test: astrometric calibration..."
./scripts/solve_wcs.sh "$TEST_DIR/raw_frame_bias_flat.fits"
HAS_WCS=$(python3 -c "from astropy.io import fits; h=fits.getheader('$TEST_DIR/raw_frame_bias_flat.fits'); print('CRVAL1' in h)")
if [[ "$HAS_WCS" != "True" ]]; then
    echo "FAIL: WCS not written to header"
    exit 1
fi
echo "PASS"

echo "=== All tests passed ==="

Branch Targeting

COMET asks which branch to deploy from. The workflow triggers on pushes to that branch. If you choose New… on an existing repository, COMET creates the branch from your default branch (main/master) before committing files, and Step 2 previews the base branch files in All files so you can still inspect what you are building from. For new repositories, COMET seeds main and then creates your selected branch if needed. To deploy from multiple branches to different destinations, duplicate the workflow file and adjust the branch and CANFAR_REMOTE_DIR.

Excluding Files from Rsync

The tooling workflow uses rsync with --delete, so files in the target directory that are not in your repo will be removed. Use a dedicated folder (or the nested option) and add excludes to skip files:

rsync -avz --delete \
  --exclude '.git' \
  --exclude '__pycache__' \
  --exclude '*.pyc' \
  --exclude 'node_modules' \
  -e "ssh -p ${{ secrets.CANFAR_SSH_PORT }}" \
  ./ ${{ secrets.CANFAR_USER }}@${{ secrets.CANFAR_SSH_HOST }}:${{ secrets.CANFAR_REMOTE_DIR }}/

More Workflow Patterns

Optional, production-friendly additions you can layer onto the minimal deploy workflow.

1) Schedule a nightly sync

# Add to your deploy-canfar.yml
on:
  schedule:
    - cron: '15 3 * * *' # 03:15 UTC nightly
# (keep push: to deploy on main pushes too)

2) Notify Slack/Teams on result

# Example Slack webhook (repository secret SLACK_WEBHOOK_URL)
- name: Notify (Slack)
  if: always()
  run: |
    STATUS="${{ job.status }}"
    COLOR="#10B981"
    if [ "$STATUS" != "success" ]; then COLOR="#EF4444"; fi
    curl -s -X POST -H 'Content-type: application/json' \
      --data "{\"text\":\"CANFAR deploy: *$STATUS* for $GITHUB_REPOSITORY@${GITHUB_REF##*/}\"}" \
      "$SLACK_WEBHOOK_URL"

3) Keep dev and prod separated

# Use environments or inputs to switch target path
env:
  DEPLOY_ENV: prod
  # Derive CANFAR_REMOTE_DIR from DEPLOY_ENV if you prefer
# Or define two jobs targeting /projects/.../dev and /projects/.../prod

4) Gate on coverage (Python)

- name: Run tests with coverage
  run: |
    pip install coverage pytest
    coverage run -m pytest -q
    coverage report --fail-under=75

5) Submit a job after deploy (optional)

# Only if your policy allows post-deploy jobs
- name: Submit CANFAR job
  if: success()
  env:
    CANFAR_USER: ${{ secrets.CANFAR_USER }}
    CANFAR_SSH_PRIVATE_KEY: ${{ secrets.CANFAR_SSH_PRIVATE_KEY }}
    CANFAR_SSH_HOST: ${{ secrets.CANFAR_SSH_HOST }}
    CANFAR_SSH_PORT: ${{ secrets.CANFAR_SSH_PORT }}
  run: |
    mkdir -p ~/.ssh
    printf '%s\n' "$CANFAR_SSH_PRIVATE_KEY" > ~/.ssh/id_canfar
    chmod 600 ~/.ssh/id_canfar
    ssh -i ~/.ssh/id_canfar -p "${CANFAR_SSH_PORT:-64022}" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
      "${CANFAR_USER}@${CANFAR_SSH_HOST}" "sbatch /projects/CanDIAPL/HIPS-main/jobs/hips_daily.sbatch"

Common Errors

Permission denied (publickey)

The SSH key isn't recognized. Check that:

• The PEM was copied correctly into CANFAR_SSH_PRIVATE_KEY (include the BEGIN/END lines)
• The public key was appended to ~/.ssh/authorized_keys on CANFAR

Fix: Re-run the COMET wizard to generate a fresh key pair.

Connection refused / timeout

CANFAR's SSH port is 64022, not the default 22. Ensure CANFAR_SSH_PORT is set correctly.

Rsync: No such file or directory

The remote directory doesn't exist. Either:

• Create it via the COMET directory browser (Step 3)
• Add mkdir -p to your workflow before rsync

Container: unauthorized

Harbor credentials are incorrect. For GitHub deployments, reconnect CADC OIDC so COMET can refresh the Harbor CLI secret; for GitLab, verify the Harbor CLI secret.

Workflow not triggering

• Push to the correct branch (the one you configured in COMET)
• Check that your CI file exists (.github/workflows/… on GitHub, or .gitlab-ci.yml on GitLab)
• Look at GitHub → Actions (or GitLab → CI/CD → Pipelines) for error details

AI Fit or test generation failed

Check the popup error message first. Common causes are daily managed-AI limits, invalid custom AI credentials, provider/model access issues, or a repository that could not be read from GitHub/GitLab. If you are using a custom provider, verify the selected model and region match what your account can access.