Prepare caches locally, over SSH and on Slurm#

The dataset TOML is the complete recipe for cache population. It defines the observations, source readers, cache directory and worker settings. No metric definitions or analysis script are required. The cache contains reduced source arrays; metrics, reductions, selections and weights are chosen later.

QUALITY and calibration deliberately remain uncached: they are fast to read. Populate the expensive PS and horizon-filtered variance sources, then compute their metrics alongside freshly read QUALITY/calibration values in the notebook.

Define the recipe once#

[blocks]
pspipe_config = "ps/l3.toml"

[sources.quality]
path = "results/{night}_{field}_{level}_{sw}/aoquality/{night}_{field}_{stage}.qs"
stage = "l3"

[sources.ps]
pspipe_config = "ps/l3.toml"
fmhz = [60.5, 72.5]
umin = 15
umax = 40
stokes = ["I", "V", "dt_V"]
ps_config = {ft_method = "nudft"}

[sources.variance]
pspipe_config = "ps/l3.toml"
fmhz = [60.5, 72.5]
umin = 15
umax = 40
stokes = ["I", "V", "dt_V"]
horizon_factor = 1.0

[cache]
dir = "cosmic_metrics_cache"
sources = ["ps", "variance"]
batch_size = 4
require = true

[worker]
nodes = "slurm"
max_concurrent = 8
slurm_nodes = 1
max_time = 3600
slurm_options = "--clusters=astro_thin -p def --account=fmertens --qos=astro_thin_def_long --exclude=node0168 --mem=0"

[worker.resources]
cache = {cpus = 1, mem = "4GB"}

This is an NT04 example: adapt paths, cluster, account and QoS to your site. The complete dataset example is also in the repository. Paths are resolved relative to the dataset TOML. Keep frequency coverage, Stokes products, UV cuts and filtering in the source sections; the population command never infers them from metrics. For example, several 1 MHz variance metrics can share the spectrum produced by the broad-band variance source above.

[cache].sources selects source aliases, not metric names. Omit it to populate all readers declaring cache=True. Selecting an uncached reader is an error. batch_size limits blocks per worker task, within one source and observing night; each block is nevertheless persisted immediately. require=true controls later metric computation, not cache preparation.

Populate and check#

cosmic-metrics cache populate dataset.toml --dry-run
cosmic-metrics cache populate dataset.toml
cosmic-metrics cache status dataset.toml

# Rebuild selected entries, including ones that are already valid.
cosmic-metrics cache populate dataset.toml --force

The dry run reports valid, missing, stale and corrupt entries plus the planned execution and pending batch count. It loads no source arrays, writes nothing and submits no jobs. status checks every expected current entry and exits nonzero when some are unavailable. Unlike cache.inspect(), it includes absent files.

Valid entries are skipped without rewriting their files. If every selected entry is valid, population returns without starting local workers or submitting a Slurm job. Use --force to rebuild all selected entries, locally or on Slurm; --force --dry-run previews the work without changing files or submitting jobs. In Python, use dataset.cache.populate(force=True). A failed forced rebuild retains the previous entry until a successful load can replace it atomically.

Older or invalid entries are recomputed using the same dependency checks and atomic writes as normal dataset computation. Previous cache formats without the current metadata must be rebuilt; they are not migrated by guessing their source configuration. Changing metric definitions does not change the raw cache. Changing a source configuration selects a different namespace.

Execution settings#

Population uses libpipe.worker.get_worker_pool(), following nenucal’s execution pattern. [worker].nodes is a libpipe host specification:

`nodes`	Execution
`"localhost"`	Local worker processes; no Slurm installation required
`"node01,node02"` or `"node[01-04]"`	Workers on SSH hosts
`"slurm"`	Slurm tasks; libpipe manages the allocation and job steps

Before execution, the command records the TOML hash, source identities, block bounds, cache path, effective worker settings and package implementation in a manifest under cache/runs/<run>/. All backends run the same source/block tasks and produce the same reports.

As with nenucal, the command runs in the foreground until all tasks finish. Keep the terminal open or use a persistent session such as tmux. With nodes="slurm", slurm_nodes requests an allocation through libpipe. Inside an existing Slurm allocation, steps use that allocation without creating a nested one or requesting cross-cluster database access from compute nodes. Slurm 20.11+ is needed for exact job steps.

Command-line settings override the corresponding TOML values for that run:

cosmic-metrics cache populate dataset.toml --hosts localhost --max-concurrent 4
cosmic-metrics cache populate dataset.toml --hosts 'node[01-04]' --env-file workers.sh
cosmic-metrics cache populate dataset.toml --hosts slurm --slurm-nodes 4
cosmic-metrics cache populate dataset.toml --hosts slurm --slurm-options='-p def --mem=0'

--slurm-nodes overrides [worker].slurm_nodes without editing the TOML or replacing the configured Slurm account, partition or other options. The dry run reports the requested node count. --slurm-options accepts raw Slurm flags; libpipe settings such as slurm_nodes=4 do not belong in that string. Inside an existing Slurm allocation, the command reuses that allocation regardless of the requested node count.

Workers use the same Python executable as the submitting command, retaining its virtual-environment path. Optional worker.env_file is sourced before every task. All source/configuration paths, the Python environment and the cache directory must be accessible from the allocated nodes. Worker replay verifies the configuration, implementation, source identities and cache path before reading data, and reuses the submission’s block list without rediscovering observations.

max_concurrent sets libpipe’s task concurrency per host. For managed Slurm allocations, libpipe sizes its worker pool for the allocated nodes and schedules steps according to available CPUs; worker.resources.cache supplies each step’s CPUs and memory. Start with one CPU per task and measure memory/throughput before increasing concurrency. BLAS is limited to one thread per population worker, avoiding nested process/thread oversubscription. max_time is in seconds and limits tasks and Slurm allocations; an explicit time in slurm_options takes precedence for Slurm. Source data must not change during the run.

Compute nodes may have no internet access. If populating an ionosphere source, download the required IONEX maps beforehand into the configured shared directory.

Progress, failures and resuming#

Each task logs completed block IDs with valid/written/missing/error status and elapsed time. Libpipe displays progress and reports CPU time and peak memory for local and Slurm workers. The run folder contains per-task logs/JSON results, manifest.json and a combined report.json. Missing inputs and worker failures produce a nonzero final job exit status; partial results remain available.

To resume, run the same populate command again. It schedules only entries that are not valid; each worker checks again before loading. Successfully written blocks survive interrupted steps. If the TOML or package changes while a job is queued, that run fails validation: submit a fresh run with the intended settings. Logs and manifests are preserved when clearing managed cache entries.

Use the cache in a notebook#

import cosmic_metrics as cm
from cosmic_metrics import metrics as m

dataset = cm.Dataset.from_toml('dataset.toml')
rfi = m.BandVarianceRatio((67, 68))
quality = m.StdOverDStd('XX')
table = dataset.compute_metrics([rfi, quality])
table.plot(rfi.name)

By default, compute_metrics() prints the absolute cache directory and a summary for each requested source alias, aggregated across nights and metrics:

Cache directory: /data/nt04/cosmic_metrics_cache
...
variance: 317 blocks loaded from cache; 0 blocks loaded from source
quality: 0 blocks loaded from cache; 317 blocks loaded from source (uncached source)

Counts describe actual reads: newly computed entries count as source loads on that call, then as cache loads on later calls. Missing blocks are reported as unavailable. verbose=False silences both progress and cache summaries.

With [cache].require=true, computation checks required entries before loading any source. Missing, stale or corrupt PS/variance entries raise CacheMissError instead of starting expensive calculations. QUALITY and calibration still read directly. A late cache miss also raises; it cannot silently fall back to loading cubes. Override explicitly with compute_metrics(..., require_cache=False) when local recomputation is intended.

For direct Python preparation, dataset.cache.populate() uses the same raw-cache implementation locally. It returns one report row per source/block and needs no metrics. The CLI adds TOML-driven worker scheduling around this operation.