CapordiaSort files by role and by owner
Separate the inputs needed for processing, the intermediate files, and the results to be kept. Inputs may come from a repository or an archive; temporary files can be recreated; results require a durable destination. Add a fourth family: the states that allow an execution to be resumed. A checkpoint is neither the original inputs nor the final result.
For each family, name an owner and a retention rule. Who can authorize the deletion of a cache? Who confirms that a result is usable? Which file is missing if the work stops? Record these answers in the project sheet, along with the chosen destination. The catalog describes the GPU; disk capacity, quotas, and retention terms must be verified for the scope actually provided.
Calculate peak storage, not just the initial archive
Estimate the maximum volume during execution. Decompressed images, working copies, or successive versions may take up more space than the archive received. The right calculation adds up the items that coexist at the same moment. You must neither add up mutually exclusive phases nor forget that an export may be prepared while processing is still producing its outputs.
Fictional example, using decimal GB: a team keeps a 180 GB archive during preparation of 420 GB of decompressed files. It plans for 120 GB of temporary files, 60 GB of outputs, and three checkpoints of 24 GB. If everything coexists, the peak is 180 + 420 + 120 + 60 + 72 = 852 GB. A chosen internal margin of 20% brings the requirement to 1,022.4 GB. This project assumption is not a capacity included in an offering.
| Category | Calculation | Volume |
|---|---|---|
| Input archive retained | 1 × 180 GB | 180 GB |
| Decompressed inputs | 1 × 420 GB | 420 GB |
| Temporary files at peak | Working assumption | 120 GB |
| Outputs in preparation | Working assumption | 60 GB |
| Checkpoints | 3 × 24 GB | 72 GB |
| Total before margin | Sum of the five categories | 852 GB |
Check the usable location before launching a batch
Check the available space on the file system that will actually host the project, then verify write permissions and the maximum expected file size. With a very large number of small files, also look at inode availability when the system uses them. GNU df distinguishes available blocks from inodes: a capacity indicator in bytes does not cover every reason a file creation can fail.
If Linux and the GNU tools are present, the commands below allow a read-only survey of the current directory. Note the time and the path examined. These surveys do not measure storage speed and do not constitute a reservation of space; additional quotas may also apply. Ask about their scope before considering the preparation complete.
df -B1 --output=source,size,used,avail,pcent,target .
df -i .Organize names and states that the team can review
Give outputs a name that identifies the batch and the processing version. Separate files in progress from validated deliverables; the delivery folder must not mix an accepted result with a partial copy bearing the same name. A registry can link batch identifier, code version, location, and validation status without containing the data itself.
For long computations, define the checkpoints that the application can produce and verify that they can be read. Keep the necessary parameters with the saved state. If a process is still writing to a file while it is being copied, do not consider that copy a consistent state without a mechanism provided by the application. Prefer a finalized and identified backup, then pass its name to the person preparing the next step.
Check the transfer and use of the result separately
Transfer an initial result to its final destination before the last day. Compare the list of expected files and their sizes, then the hashes where your procedure provides for them. GNU SHA-2 tools compute and verify file hashes. A manifest must correspond to finalized files and be kept with a known provenance; on its own it does not validate the business quality of the content.
After the integrity check, open the result from its destination with the application that will use it. Verify the number of items, the format and external dependencies. A perfectly copied archive may still contain a file that was missing from the start. The receiving owner must therefore confirm two distinct things: the copy matches the expected package, and that package does serve the next step.
sha256sum --check MANIFEST.sha256Decide on retention before the end of the period
In the 852 GB example, deleting the working archive after confirming an independent copy would bring the requirement down to 672 GB before margin. This gain is valid only if restoring the archive remains possible and no subsequent step requires it on site. The decision belongs to the data owner; it must not be improvised by the person noticing a lack of space.
Common mistakes are keeping every version without limit, waiting until the end to test destination permissions, and counting a copy located in the same place as an exit solution. Set the list to keep, the person responsible for the export and its final check. Over 3, 7 or 30 days, the duration of the computation and that of the retrieval are part of the same schedule.
Questions to decide
Should all temporary files be kept?
Not necessarily. Identify the ones the application can rebuild, then compare their volume with the cost and time of that rebuild. Keep the items needed for a validated restart. Deleting a file still in use or an essential state must be excluded from the cleanup rule.
Is a correct hash enough to accept an export?
It verifies the match with the expected hash. It remains to check that the manifest covers the right files and that the result opens with its dependencies. Acceptance therefore combines package integrity and a functional check from the destination.
Can GPU memory replace missing disk space?
No: VRAM holds the data used by the computation on the card. It does not replace the destination for files or retention after the rental period. Describe GPU memory, system memory and storage separately in your file.