CapordiaDefine what migrates and what must wait
This guide is intended for the person coordinating compute, data, and operations. A batch here means an identified packet of work, distinct from a batch of GPU cards in the catalog. The method is suitable first for outputs that can be verified before acceptance. For a service receiving requests without interruption, plan where new inputs will wait, how long that wait is acceptable, and how they will join the target. A copied package does not by itself define this admission policy.
The proposed exercise works on a closed set of eight synthetic batches. It sends no network request and uses neither a GPU nor a Capordia account. It teaches a resumption logic on a single workstation and a single worker process. It does not validate a distributed queue, the continuity of an exposed service, or the live migration of a computation. Before a real migration, your team must qualify these elements with the chosen application and hardware.
Build an inventory that allows reconstruction
Gather the code and its version, the dependencies, the parameters, the models, the inputs, the scheduled tasks, and the output destinations. For each, state the owner, the reconstruction method, and the expected check on the target. Note the names of the secrets and how they are provided, without placing their values in the shared register. A manual installation or a task triggered at startup may be missing from a simple copy of the project directory.
Separate input files, accepted results, and the state needed to restart. For training, this state may go beyond the model weights: the PyTorch tutorial on saving a general checkpoint notably includes the optimizer state and the progress. The required content depends on your code. Restore it in a fresh process, then compare the outputs according to a tolerance decided before the test; do not infer a universal numerical identity from a single successful restart.
Assign the decision, the transfer and the acceptance
Assign roles before setting the window. The same person can hold several of them on a small team, but the decisions remain distinct. The cutover lead confirms that the old producer has stopped and authorizes the start of the target. The data lead checks the inventory, the fingerprints and the outputs. The workload owner judges functional quality. Operations keeps the change log and prepares the handover.
The cutover sheet must answer four concrete questions: who can stop the inputs, who confirms that the old workload is no longer publishing, who accepts the first results and who decides on the rollback. Also record a decision deadline and the ways to contact these people. "Roll back if necessary" is not enough once the target has already produced results that the old ledger does not know about.
Keep a ledger that distinguishes started from accepted
Assign a stable identifier to each batch and tie it to a precise version of the data. Keep its state, the number of attempts, the location of the result and the check that authorizes its acceptance. An in-progress batch is not finished just because an output file exists: that file may be partial or may come from a previous attempt. The procedure must spell out what will be kept, what will be recomputed and what will remain quarantined for review.
In the exercise, a local SQLite ledger keeps the state and the acceptance of the result together in a single transaction. Its states are termine, en_cours and a_reprendre; the latter also covers a batch that has not started yet. The target takes over the unvalidated batches instead of indiscriminately rerunning all eight inputs. The JSON and CSV exports make tracking readable. The blank CSV supports your preparation and human review; it does not replace the coordination mechanism of a real application.
| Batches | State at the boundary | Decision before resumption |
|---|---|---|
| LOT-001 to LOT-003 | termine | Keep the results and verify their integrity |
| LOT-004 | en_cours, result not accepted | Explicitly prepare a new attempt |
| LOT-005 to LOT-008 | a_reprendre, no attempt | Start on the target after authorization |
Download and run the cutover and resumption exercise
Have Python 3.10 or later with the sqlite3 module, and a local directory you can write to. Download the complete archive and extract it, keeping the script, the data folder and the manifest together. The README file details the available commands. No driver, compute package or model needs to be installed. Read the script before running it, then use a fresh output folder so that the evidence from several attempts does not get mixed up.
From the directory containing exercice.py, the command below produces a continuous baseline, an interrupted run, a cutover copy and a resumption compared against the baseline. The eight CSV files contain 48 fictional dimensional measurements of coupons, expressed in whole micrometers and created for this case. No real industrial inspection is represented. The CPU computation is meant to examine batch management, not to estimate GPU throughput or the time to migrate your data.
python exercice.py demonstration --dossier essai-capordiaRead the evidence of the resumption, batch by batch
Capordia example: a team prepares eight packets of files to process. The baseline processes them without interruption. On the other path, the stop leaves three batches finished, LOT-004 in progress and four batches to resume with no attempt. The cutover lead does not ask to "resume to the middle": they designate LOT-004. The resumption preparation returns it to the a_reprendre state: three batches are then finished and five are to be run. Batches already accepted must not be recomputed.
See etat.json for identifiers, states and attempts, registre-lots.csv for the review, resultats.json for the outputs, then rapport-verification.json for the checks. In the local run of September 24, 2026 under Windows AMD64, Python 3.14.6 and SQLite 3.50.4, the 16 checks from 37 script runs passed: the eight reused results match the reference, completed batches are not recomputed, and a new acceptance of the same batch is refused. Altered inputs, duplicate identifiers and corrupted outputs are also detected.
Copy a consistent state, then check its integrity
In a real switchover, stop admission or place new entries in a queue explicitly provided for that purpose. Wait for in-progress work to finish, or record interrupted batches according to your recovery rule. Then confirm that the old instance can no longer publish. Sealing the source in this exercise prevents the script from reusing it as a producer. It remains a local protection of the scenario, not mutual exclusion between several machines that would each hold a copy of the state.
To copy a SQLite database used by an application, its documentation offers a backup API that produces a consistent snapshot. An ordinary copy of a database file that is being written to is therefore not a procedure to improvise. Also check the manifest of the transferred files. SHA-256 hashes, available in hashlib, help here to detect a difference in content; they do not tell you whether the result is correct for your business, nor whether the reference manifest is trustworthy.
Decide when to resume and when to roll back
Allow the target only when the inputs, the register and the retained results agree, the old producer is stopped, and the software prerequisites are met. Start with a scope whose outputs can be reviewed. The process owner accepts the quality; the data owner checks identifiers, missing items and duplicates. A start with no errors is not enough if the useful result is incomplete.
If the defect appears before any new publication, keep the target stopped and restart from the checked source state according to the planned procedure. After a publication, first preserve the target's results and register. Reconcile the batches it completed with those of the source, then choose a single authorized recovery. Reactivating both sides to save time can create precisely the duplicates the migration was meant to avoid.
| Observation | Immediate decision | Condition for proceeding |
|---|---|---|
| Input missing or hash different | Suspend the transfer and isolate the copy | Retrieve an input matching the reference manifest |
| Duplicate identifier or unexpected output | Do not publish the batch concerned | Understand the conflict and retain a single accepted version |
| Recovery state unusable | Stop the target and preserve the evidence | Restore a known state or apply the prepared rollback |
| Result outside the business criterion | Stop expanding the recovery | Correct and then redo the functional validation |
| Decision deadline reached | Trigger the rollback or postponement decision | Explicit approval from the owner and capacity to finish the export |
Measure the entire window and prepare the handover
Repeat the installation, copy, verification, recovery and export in their real order. Record the time for each phase and the human interventions, using a sample whose transfer is authorized. To estimate your window, factor in the new entries that will wait and the capacity to clear that backlog. The best time observed on a small folder is not a commitment for a larger volume.
The chosen period must cover preparation, rehearsal, cutover, observation, and retrieval of results. Capordia plans of 3, 7, or 30 days remain periods to be organized; this guide does not infer any compute time from them. Before closing, a second person should locate the log, open the outputs from their destination, and know how to rerun the processing. Archive the rollback end decision, the accepted deviations, and the remaining actions before removing access that has become unnecessary.
What the exercise verifies and what remains to be qualified
The value of this file is a readable procedure and a reproducible recovery experience on synthetic files. The same batch must be identifiable before shutdown, during cutover, and after recovery. The log, the comparison against a reference, and the deliberately triggered rejections make this rule observable. The interruption is triggered by a controlled exception in the script: no power failure test was performed. The technical documentation cited explains the mechanisms used; it does not certify your future production environment.
Only one process works at a time in the scenario. Local sealing is not a distributed lock; a fingerprint is not a quality test; a CPU result is not a CUDA, ROCm, or multi-GPU qualification. An application that also writes to an external database, shared storage, or queue requires a consistent rule across all these destinations. Have this rule and your rollback plan validated before migrating the real processing.
To prepare for what comes next
A backup must make it possible to resume work.
Restore into an isolated target and verify that the copy really allows you to resume.
Hand over with a state, an action and an owner.
Hand over the actions, the limits and the responsibilities to another person.