Checkpoints and progress log

CheckpointSettings(name; directory=".", interval=1800, overwrite=false)

Checkpoints of a run called name, written to directory:

• <name>.checkpoint.h5, the last checkpoint, and <name>.checkpoint.prev.h5, the one before it, kept so that a failed write never loses the last good state;
• <name>.segment<k>.h5 (k with four digits), the output of segment k, holding the states the segment saved up to its last checkpoint.

A checkpoint is written after an accepted step when interval seconds of wall time have passed since the last one (0 writes one after every accepted step), and always at the end of the segment. Each checkpoint rewrites the output file of the segment. Files are written to a temporary file, synced and renamed.

A run refuses to replace the segment file of another run unless overwrite is true. A run continued with resume = true replaces the file with the number of its segment: the output is written before the checkpoint, so an attempt of the segment that was killed between the two leaves its file behind, and the checkpoint still names the segment before. A run refuses to start while segment files of its name with higher numbers than its segment exist, with overwrite = true as well: read_output would join those of an earlier run under the name, or of a continuation that the run replaces, to its output. The run deletes no file.

StopCondition(; end_time=nothing, wall_budget=nothing, stop_file=nothing, slack=300)

When a segment stops before the last time point. integrate asks after every accepted step. It stops when

• the wall time left until end_time is less than the margin: the longest accepted step of the segment so far, plus the longest write of a checkpoint with its output so far, plus slack seconds (the rule of SCR: step, write time and a slack; the default slack is the margin of GS2, 300 s);
• the file stop_file exists.

end_time is a UNIX time in seconds. By default it is SLURM_JOB_END_TIME of the environment when that is set, so a run in a Slurm job stops before its time limit without further settings. wall_budget gives the end as seconds from now. With both, the earlier end counts.

A stop file left by an earlier segment is removed when the segment starts. A segment always completes one step before it stops, so that every segment of a run makes progress.

StoredState

A state of a run read from a file, from which integrate resumes (resume): the time, the state vector in the layout it was stored with (rebuilt from the file's layout record, for matching the unknowns by name), the states of the interval updates by name, the settings record of the run that wrote it, the number of the segment that wrote it, the number of states the run had saved up to the time (saved_count), a summary of the run so far (attempted and accepted steps, Newton and Krylov iterations, wall time), the record of the time stepper at the time (its step length and what it carries between steps, FixedSteps; empty for a state read from an output file, whose time is then a time point of the run), the file it came from and the units that the file stores, by unknown. A file of the format 0.0.4 or older stores no unit, and its units are empty, while its layout holds the default units. Without units, the constructor takes those of layout. A caller who rebuilds a state read from a file passes the units of that state. read_checkpoint and read_state return one.

exit_status(solution)

Exit status for the script of a segment, which a batch script that chains the segments reads: 0 when the run finished, 85 when the segment stopped and wrote a checkpoint, so that another segment follows, and 86 when it stopped without checkpoints, where another segment would start from the beginning.

read_checkpoint(settings::CheckpointSettings)
read_checkpoint(filename)

Read a checkpoint as a StoredState. Reading with the settings falls back on the previous checkpoint when the last one is missing or cannot be read, and returns nothing when neither file exists. When checkpoint files exist and none of them can be read it throws an ArgumentError, so that a run with resume = true does not start again from its initial state over its own files. Defined by the package extension Runes2PHDF5Ext, which loads with HDF5.

read_output(filenames...)
read_output(settings::CheckpointSettings)

The TimeSolution of one output file, or of the segments of a run in order, joined. The layout is rebuilt from the file's layout record. The joined layout is that of the first segment, with the unit of each unknown from the latest segment that stores one. Where two segments store different nonempty units for one unknown, a warning names the unknown, the two units and the two files, and the values are joined unchanged. Of a segment that went on after its last checkpoint and was followed by another, the states and statistics after the start of the following segment are left out, since that segment repeats them. The settings are those of the last file. Segments in the wrong order, or whose joined times repeat one (two runs from the same start), are refused with an ArgumentError. So is a segment that did not resume from the one before it: its resumed_from_segment must be the index of that segment and its previous_setup_hash the setup hash of that segment, where both records hold them. The higher segments of an earlier run under the same name with the same setup pass this check; a run refuses to start while segment files with higher numbers than its own exist (CheckpointSettings). Defined by the package extension Runes2PHDF5Ext, which loads with HDF5.

read_state(filename; time_index=nothing)

The state saved at time_index (the last one by default) of an output file, or the state of a checkpoint file, as a StoredState to resume from. Defined by the package extension Runes2PHDF5Ext, which loads with HDF5.

run_status(solution)

:finished when the run reached its last time point, :stopped when the segment stopped before it (StopCondition). solution.settings["segment"] has the reason (stop_reason) and the time reached (end_time).

stop_reason(stop, longest_step, longest_write; now=time())

Why the segment stops now, "stop file" or "wall time", or nothing when it goes on. longest_step and longest_write are the wall times in seconds of the longest accepted step and of the longest checkpoint write of the segment so far.

write_checkpoint(settings::CheckpointSettings, stored::StoredState)

Write a checkpoint (write_atomically, keeping the previous one). Defined by the package extension Runes2PHDF5Ext, which loads with HDF5. See also read_checkpoint.

Exit status of a script whose run finished.

Exit status of a script whose segment stopped at a checkpoint, to be followed by another segment.

Exit status of a script whose segment stopped without a checkpoint to continue from.

File of the last checkpoint.

File of the checkpoint before the last.

Output file of segment k.

segment_files(settings::CheckpointSettings)

The segment output files of the run that exist, in the order of the segments, for read_output.

The number of a segment output file <name>.segment<k>.h5.

write_atomically(write_file, path; previous=nothing)

Write a file so that path holds either its old content or the new one, whatever happens during the write: write_file(temporary) writes to a temporary file in the directory of path, the file is synced and renamed onto path, and the directory is synced. With previous, the file at path is first renamed to previous. A failure of write_file removes the temporary file and leaves path and previous as they were. Between the two renames path does not exist and previous holds the last good file, which is why a reader of checkpoints falls back on it.

ProgressLog(; file=nothing, io=stdout, color=(io is a terminal), milestones, guard_minutes,
            events_shown=3, state=true, on_signal=true, request_file=nothing)

Progress log of a run, the keyword log of integrate. It writes to io, or appends to file, which the segments of a run share.

• A header when the segment starts: the commit of Runes2P and whether its tree was clean, the Julia version and the machine, the segment, then the solver settings, the time points, the layout and grids, the terms of every equation and what the caller described, from the record of the run (run_settings). Arrays are summarized, so the header identifies a run without the settings file and stays short.
• A progress entry at the milestones, fractions of the span of the run (0.01 %, 0.1 %, 1 % and then every 1 % by default), and at the start and the end of the segment: the simulated time and its fraction, the steps since the last entry with the halved ones, the time step, the wall time since the start and the last entry, per step and its split, the Newton and Krylov counts, the memory, and an estimate of the wall time left from the recent intervals. Below it, set off, the state: per kinetic unknown the density and the current density per radius, the total current and the negative cells, and the values of the radial and scalar unknowns (state = false leaves it out).
• A guard entry after a silence. The silence allowed grows through guard_minutes (1, 2, 5, 10, 20, 30, 60, 90, 120 by default) and stays at the last value. The sequence only advances: an entry at a milestone does not set it back, so a slow phase is reported often at first and then at a steady low rate.
• The events when they happen: a step that failed and is retried with half the time step, a step accepted at the residual floor, negative values, a checkpoint written, a stop request. Of each kind events_shown are written between two entries; the rest is counted in the next entry.
• An entry on demand, after the step in progress: when the file request_file appears (touch it; the log removes it), and with on_signal = true when the signal SIGUSR1 reaches the Julia process. The entry takes the place of Julia's profile report. Julia still prints its notice and a stack trace of about 45 lines to the error stream and profiles for a second, so the file is the quiet way.

Warnings of integrate (a resume with changed settings) go to the log as well, and a message with maxlog = n is written at most n times. With a terminal as io, warnings and errors are coloured.

Minutes of silence after which the guard writes an entry, in the order used.

Fractions of the span of a run at which a progress entry is written: 0.01 %, 0.1 %, then every 1 %.

log_event!(log, kind, level, message)

Write an event of the run to the log when it happens. Of each kind, events_shown are written between two progress entries and the rest is counted, which the next entry reports, so that a run with many events of a kind does not fill the log.

report_progress!(log, u, layout, t, times, interval, stats; errors=nothing)

Called by integrate after every accepted step: writes the entry of a milestone passed, of a request by signal or of the silence guard, whichever applies.

report_step!(log, u, layout, t, times, interval, stats; errors=nothing)

Called by integrate after every failed or rejected step: the guard and the request by signal are served with the state of the last accepted step, so that a stretch of failed steps is not silent.

start_log!(log, settings, u, layout, t, times, interval, stats)

Begin the log of a segment: the header and the entry of the segment start. With on_signal, SIGUSR1 asks for an entry from now on, until integrate returns and puts Julia's own report back.

write_entry(log, reason, u, layout, t, times, interval, stats; errors=nothing)

A progress entry: reason says why it is written (milestone, silence, segment start, segment end, on demand), u is the state at the time t, interval the number of intervals of times done, stats the statistics of the segment, errors the error estimate per unknown of the last accepted step when the stepper has one.

write_header(log, settings)

The header of a segment, from the record of the run: what identifies the run first (code, machine, segment), then the solver settings, then the physics and grid settings.