Explain expectations before looking for a more powerful GPU.

Describe the problem with a result and a time window

Replace "the GPU isn't working" with a verifiable observation: the first result arrives late, the steps show pauses, or the number of accepted outputs decreases. Provide the input set, the application version, the settings, and the observed period. A first run may include phases that differ from an already prepared execution; don't mix them without saying so.

Break the path into file opening, CPU preparation, transfer, GPU computation, writing, and then validation. These phases can overlap: the diagram is a hypothesis to be checked against the traces. Record the boundaries with the same time zone and sufficient precision. A hardware graph taken after the pause cannot explain the pause, even if its value looks unusual.

Correctly read percentages and memory

In NVIDIA SMI, GPU utilization indicates the share of the sampling interval during which at least one kernel was running. It is not a performance grade for all components. Memory utilization describes its read or write activity; it must not be confused with the amount of memory occupied. The documentation also notes that some fields may be unavailable depending on the environment.

So keep the exact name of the metric and its unit. A high amount of memory may correspond to data kept waiting; low activity may appear during a normal preparation phase. Conversely, high activity does not establish that the program is doing useful work. You need to look at the outputs and application progress at the same time as the hardware.

Collect a correlated and sufficiently short report

Choose a representative sequence and limit collection to that sequence. Record the timestamps, the GPU identifier, the activity, the memory used and the application phases. At the same time, use the tools available on your system to observe the CPU process, its waits, its reads and writes. An average across all cores can mask a single saturated thread; also look at the processes involved.

The read command below is provided for an authorized NVIDIA target that already has NVIDIA SMI. It was not run for this guide. It keeps the units in the CSV and displays one observation per second until it is interrupted. Check the fields supported by your version and stop collection at the end of the scenario. An N/A field remains unknown: do not turn it into zero.

The minimum log ties together a job identifier, a timestamp with time zone, the current phase, an identifiable entry or batch, progress and any error. It must contain neither secrets nor business content that is not useful for diagnosis. If telemetry is missing, first check its collection mechanism and the application progress: an empty curve does not prove that a process has stopped. Mark the period with no observation instead of reconstructing it from memory.

nvidia-smi --version
nvidia-smi --query-gpu=timestamp,uuid,utilization.gpu,memory.used,memory.total --format=csv --loop=1

Telling a lead apart from a conclusion

If GPU activity drops while the application is waiting for its inputs, examine the file path. If it drops during a long CPU preparation, examine that phase and its actual parallelism. If the devices are waiting for the other participants in a distributed job, check load balance and communications. Several symptoms can occur together; their correlation is not yet proof of causation.

Write a short hypothesis and a change that will help decide between them. For example, comparing the same inputs already prepared against their usual preparation can isolate a step, provided the results stay identical. Do not change the batch size, the precision, the number of processes and the file locations all at once: an improvement would then be hard to attribute.

Fictional example: analyzing a one-hundred-second loop

The fictional team "Design review" describes a loop with no overlap: sixty seconds of CPU preparation, twenty of GPU computation, then twenty of export. The total is 100 seconds and the time fraction attributed to GPU computation is 20/100, or 20%. This teaching calculation does not predict the percentage of a sampling tool and does not correspond to any measurement of a GPU in the catalog.

Suppose the preparation could be reduced to thirty seconds, without changing the computation or the outputs. The loop would then take 30 + 20 + 20 = 70 seconds: thirty seconds less, with a GPU fraction of 20/70, or about 28.6%. Conversely, hypothetically halving the GPU computation alone would give 60 + 10 + 20 = 90 seconds. The scenario explains why addressing the longest wait can matter more than changing the card.

Going deeper into a phase with a profiler

If the general observation locates a slow area without explaining it, use the application-appropriate profiler over a short window. PyTorch Profiler can collect CPU and CUDA events and organize collection windows. The documentation notes overhead when recording shapes or stacks is enabled. A detailed trace is therefore a diagnostic instrument; its total time must not be presented without qualification as the program's normal time.

Keep phase names explicit enough to link a technical operation to your work: preparing a batch of files, transfer, application step, or output publication. Start with the level of detail needed for the hypothesis. After the change, repeat a comparable observation and then a run without heavy instrumentation. This second check verifies that the benefit also exists under normal conditions.

Avoid corrections that mask the problem

Increasing concurrency can fill the GPU while overloading the disk, host memory, or outputs. Increasing batch size can change latency and required memory. Reducing precision can change the result. These decisions are not forbidden, but they must be evaluated against the same business objective, along with their side effects, rather than adopted just because the displayed percentage goes up.

Do not reset a device or change its hardware settings to diagnose a simple drop in activity. Start by reading, then have explicit errors reviewed by the authorized person. Also avoid publishing raw traces containing private paths, confidential settings, or input data. Prepare an excerpt useful for diagnosis and keep the full evidence within the team's authorized scope.

Close the diagnosis with evidence useful for the decision

The final record must state the symptom, the hypothesis retained, the change tried, its conditions, and the observed result. Keep at least one before/after comparison on the same set of inputs, the same quality, and the same definition of completion. If the result is inconclusive, write down what was ruled out and the next observation needed. A documented rejected lead keeps someone else from starting over at random.

A low normal activity level does not always require a correction. If the task meets its deadline, quality, and budget, reducing complexity may be preferable to a fragile optimization. If a new configuration is being considered, pass the demonstrated constraints to the acceptance record. No isolated reading ranks the catalog, and no value in this example constitutes a promised performance.