Independent study 02 · Data center cooling

FAT Nonconformance Investigation — Cooling Skid Differential Pressure

A 1 MW cooling skid reached 195 kPa against a 250 kPa requirement at design flow, repeatably. This is the full 8D: containment, cause elimination, a validated root cause that predicts the measurement, corrective action, verification, prevention — and a shipment disposition that changes five times as the evidence accumulates.

Independent Engineering Study

This investigation is synthetic. The supplier, the equipment, the measurements and the records are constructed for this study, and it does not represent professional data-center project experience. The transferable content is the investigation structure and the disposition logic.

Project record

Independent Study

Project type: Independent engineering study
Domain: Data center cooling — factory acceptance nonconformance
Methods: 8D · Containment · Fishbone · 5-Why · Quantitative cause validation · CAPA · PFMEA update
Date: September 2026
Data: Synthetic. All measurements and records are constructed.
Confidentiality: No proprietary employer information used. Technical grounding from public sources, cited at the end.

Required

250 kPa

Minimum differential pressure at design flow

Observed

195 kPa

Repeatable across 3 tests

Achieved

78%

Of the acceptance requirement

Disposition history

The most useful thing about this page is not the root cause. It is that the shipment decision is a function of the evidence available at each stage, and it changes as that evidence changes. Containment does not make equipment acceptable. A repaired unit does not close a systemic cause. Both of those are shown below as HOLD, deliberately.

Disposition history — Skid serial 003

At discovery (D2)HOLD
Performance acceptance criterion not achieved at design flow — 195 kPa against a 250 kPa minimum, repeatable across three tests. No approved deviation.
After containment (D3)HOLD
Containment in place and the affected population identified, but no root cause established. Containment limits spread; it does not make equipment acceptable.
After corrective action (D5)HOLD
Equipment corrected and the immediate performance restored, but the systemic cause was still open — the same error could have been present on other units and undetected by the existing controls.
After verification (D6)CONDITIONAL RELEASE
Performance demonstrated on three consecutive units and the affected population re-inspected. Prevention actions were approved but not yet evidenced in the process documents, so release was conditional on that closure.
At closure (D8)RELEASED
Prevention actions implemented and evidenced, ITP revised to Rev B, and the full shipment release checklist satisfied with objective evidence.

Team

Investigation team and responsibilities

Role	Responsibility
Owner quality engineer (investigation lead)	Investigation structure, disposition recommendation, closure evidence standard.
Supplier quality manager	Containment execution, records quarantine, corrective action delivery.
Supplier mechanical engineer	Hydraulic analysis, pump configuration verification, retest execution.
Supplier controls engineer	Instrument scaling verification, control-mode confirmation.
Supplier production supervisor	Assembly record retrieval, affected-unit identification.
Owner mechanical engineer	Acceptance criterion interpretation, deviation assessment, retest witness.

Scroll table horizontally →

The investigation is led from the owner side rather than delegated to the supplier. The supplier owns the corrective action; the owner owns the evidence standard and the disposition. Collapsing those two roles is how an investigation ends at the first plausible cause.

Problem definition

During factory acceptance testing of Northstar Thermal Systems LC-1000, skid serial 003 at ITP step 150 — the performance test hold point — the skid achieved 195 kPa differential pressure on the secondary circuit at the design flow of 1,440 L/min (24 L/s), against a specified minimum of 250 kPa. That is 78% of the acceptance requirement, a shortfall of 55 kPa.

The result was repeatable across 3 consecutive tests in the same configuration, with flow confirmed at the design value and the pumps at rated speed each time. Repeatability matters: it rules out a transient and makes the failure a property of the equipment as built rather than of the test.

Is / is not. The shortfall is present on the secondary circuit at design flow on serial 003, at rated pump speed, in both single-pump and changeover configurations. It is not a thermal shortfall — heat transfer met the acceptance band — and is not present on the primary circuit, which met its own requirements.

Engineering Assumption

The 250 kPa requirement is a study assumption, derived from the assumed loop resistance in the equipment specification. On a production project it would come from the owner’s hydraulic calculation for the served technology loop, and the acceptance band would include the measurement uncertainty of the test instrumentation.

Risk

Skid delivers insufficient differential pressure at design flow, starving the served technology loop of coolant flow under full load.

Severity: 9
Occurrence: 3
Detection: 4
RPN · Priority: 108 · Medium

Severity is high because the consequence is a thermal event in service rather than a commercial one. Detection is rated on the state at the time of discovery: the condition was caught at FAT, which is late but before shipment — it was not detectable by any control upstream of the performance test.

Bands: RPN ≥ 150 high · 60–149 medium · < 60 low

Containment

Containment limits spread while the cause is unknown. It is not a fix, and on this page it explicitly does not change the disposition.

Containment actions taken

Shipment of skid serial 003 placed on HOLD; release blocked in the shipping release gate, not only noted on the test record.
FAT on skid serial 004 paused before the performance test to avoid generating a second ambiguous result.
Configuration frozen on all skids in build: no assembly, parameter or component changes without investigation approval.
Pump, control valve, sensor and assembly records for serials 003 and 004 quarantined as investigation evidence.
Affected population identified by purchase-order line rather than by build date: all pumps received against that line, including spares in stock.
Owner engineering notified within the same working day, with the measured values and the containment already in place.
Sub-tier supplier notified and asked to confirm what was shipped against the order line.

Two of those actions are the ones that get skipped. Pausing FAT on serial 004 before its performance test avoids generating a second ambiguous data point that would then have to be explained. And identifying the affected population by purchase-order line rather than by build date turned out to be decisive — the error originated in procurement, so a date-based recall would have missed stock spares entirely.

Root cause analysis

Fishbone to enumerate and eliminate, 5-Why to reach the systemic level, then a quantitative check to confirm the cause actually accounts for the measurement.

Cause and effect — effect under investigation

Secondary circuit differential pressure 195 kPa at design flow against a 250 kPa minimum

Machine

Pump impeller configuration
Confirmed — Physical inspection found an impeller trim smaller than the BOM required.
Control valve Cv undersized
Eliminated — Valve verified against the BOM; test performed with the valve at the design position.
VFD speed limit set below rated
Eliminated — Drive parameters read out; pump running at rated speed during the test.
Strainer or filter partially blocked
Eliminated — Differential across the filter within normal range; elements inspected clean.

Measurement

Differential-pressure sensor scaling error
Eliminated — Loop check against an applied reference at two points; indication within instrument accuracy.
Flowmeter calibration error
Eliminated — Calibration certificate current; flow cross-checked against pump curve and absorbed power.
Test instrumentation range or resolution inadequate
Eliminated — Instrument ranges reviewed; the 55 kPa shortfall is far outside combined measurement uncertainty.

Method

Test loop resistance not set to the design operating point
Eliminated — Flow confirmed at design value while differential pressure was recorded; the operating point was correct.
Valve lineup incorrect during the test
Eliminated — Lineup verified against the P&ID before each of the three test repeats.
Control mode not at constant differential-pressure setpoint
Eliminated — Control mode and setpoint read from the PLC before the test.

Material

Incorrect impeller supplied against the purchase order
Contributing — The received impeller matched the sub-tier packing list; the purchase order line itself carried the wrong trim.
Piping internal diameter or fitting mismatch
Eliminated — As-built piping verified against the drawing; no restriction found.

People / method of verification

No independent verification of impeller trim at assembly
Confirmed — Assembly work instruction required the pump part number but not confirmation of impeller trim identification.
Operator error during assembly
Eliminated — The pump was installed as received; the error was upstream of assembly.

The measurement branch is eliminated first, on principle. Until the instrumentation is trustworthy, every other branch is being investigated against an unknown. The differential-pressure transmitter was loop-checked against an applied reference at two points spanning the operating range, and the flow reading was cross-checked against pump absorbed power — two independent routes to the same conclusion.

5-Why

Question	Answer	Evidence
Why did the skid not achieve the required differential pressure at design flow?	The installed pumps could not produce the required head at design flow.	Measured 195 kPa at 1,440 L/min against a 250 kPa minimum, repeatable across three tests at rated pump speed.
Why could the pumps not produce the required head?	The installed impeller trim was smaller than the trim the design required.	Physical inspection: 190 mm impeller installed where the BOM specified a 216 mm trim.
Why was the smaller impeller installed?	The pumps were received configured with that trim, and the purchase-order line specified it.	Purchase order line and sub-tier packing list both show the 190 mm trim; goods-in records match.
Why did the purchase order specify the wrong trim?	The two trims differ by one character in the supplier part number and share an identical catalogue description, and the order was placed from the description.	Sub-tier catalogue extract showing both part numbers with the same description text.
Why did the error survive receipt and assembly?	Neither receiving inspection nor the assembly work instruction required the impeller trim to be confirmed against the BOM — only the pump part number, which was itself wrong.	Receiving inspection record and assembly work instruction, neither containing a trim verification step.

Scroll table horizontally →

Direct cause

Incorrect pump impeller trim installed — 190 mm supplied and fitted where the design required 216 mm.

Systemic cause

Insufficient differentiation between the two impeller part numbers in the sub-tier catalogue, combined with no independent verification of impeller configuration at receipt or at assembly. The ordering error was a single point of failure with no downstream detection.

Quantitative validation of the cause

Pump head varies approximately with the square of impeller diameter at constant speed. If the trim is the cause, the shortfall is predictable rather than coincidental.

(190 / 216)² × 250 kPa = 0.774 × 250 kPa ≈ 194 kPa

Observed: 195 kPa measured

The predicted and observed values agree within measurement uncertainty. The impeller trim accounts for the entire shortfall, which also means no second cause needs to be postulated.

This step is the difference between a root cause and a plausible story. A cause that cannot predict the magnitude of the observed effect might be a cause, and leaves open the possibility of a second one still hiding. Here the predicted 194 kPa against an observed 195 kPa closes that question: the impeller trim accounts for the entire shortfall, so no second cause needs to be postulated and the corrective action can be scoped confidently.

Corrective action

Immediate and permanent corrective actions

Type	Action	Owner	Verification
immediate	Replace the impellers on serial 003 with the specified 216 mm trim and repeat the performance test in full.	Supplier mechanical engineering	Performance test record at design flow, witnessed by the owner.
immediate	Correct the purchase-order line and re-inspect all pumps received against it, including stock spares.	Supplier procurement and quality	Inspection record per pump showing the as-received impeller trim.
permanent	Assign unique internal part numbers to the two impeller trims and break the shared catalogue description in the ordering system.	Supplier engineering	Item master extract showing distinct part numbers and distinguishing descriptions.
permanent	Add impeller trim identification to receiving inspection for all pumps, verified against the BOM rather than the packing list.	Supplier quality	Revised receiving inspection instruction plus completed records on the next three receipts.
permanent	Add an independent pump configuration verification step to the assembly traveller, signed by someone other than the installer.	Supplier production	Revised traveller plus completed records on the next three skids.
permanent	Add a component configuration verification witness point to the ITP at material receipt for pumps and heat exchanger.	Owner quality engineering	ITP revised to Rev B with the new witness point at step 020.

Scroll table horizontally →

The two immediate actions restore the equipment. The four permanent actions attack the systemic cause in the order it failed: make the parts distinguishable, then verify at receipt, then verify independently at assembly, then place an owner witness point at receipt so the control is not purely internal to the supplier. Any one of those alone would have caught this instance; the set is what stops the class.

Verification of effectiveness

Performance verification after corrective action

Unit	Flow	ΔP (kPa)	Margin	Result
Serial 003 (reworked)	1,440 L/min	263	+13	Pass
Serial 004	1,440 L/min	258	+8	Pass
Serial 005	1,440 L/min	261	+11	Pass

Scroll table horizontally →

Three consecutive units met the acceptance criterion at design flow after corrective action. Synthetic data, generated for this study — on a real programme this is the evidence that would be required, not evidence that exists.

Three consecutive units, not one. A single passing unit after a repair demonstrates that the repair worked on that unit; it says nothing about whether the process now reliably produces the correct configuration. The margin column is there for the same reason — results that clear the limit by a handful of kilopascals would raise a capability question even while passing.

Prevention — systemic changes

This is where an investigation either becomes a system improvement or stays a repaired skid. Each row is a controlled document that changed.

Documents updated as a result of this investigation

Document	Change
PFMEA	Added the failure mode "incorrect pump impeller configuration installed" with the detection rating re-scored against the new receipt and assembly verification steps rather than against factory acceptance testing.
Control plan	Added component configuration verification at receipt and at assembly, with the record as the control evidence.
Inspection & test plan	Revised to Rev B: impeller trim verification added as an owner witness point at material receipt (step 020).
Assembly work instruction	Independent pump configuration confirmation added as a signed step before the pump is coupled.
Receiving inspection instruction	Pump configuration verified against the BOM, explicitly not against the supplier packing list.
Supplier audit checklist	Added a check on part-number differentiation for functionally distinct components sharing a catalogue description — the systemic condition, not the specific part.
Sub-tier purchase order template	Configuration-critical attributes stated on the order line rather than relied upon from the catalogue description.

Scroll table horizontally →

The loop closes

The ITP change is the visible one: step 020 of the inspection and test plan now carries an owner witness point for component configuration, including pump impeller trim, verified against the BOM. That witness point exists because of this investigation. The supplier audit checklist change is the one I would argue matters more: it checks for the systemic condition — functionally distinct components sharing a catalogue description — rather than for this specific impeller.

Closure

Closure evidence accepted

Three consecutive conforming units at design flow, witnessed and recorded
Affected population re-inspected, including stock spares, with a record per pump
Configuration-control update implemented: distinct part numbers and distinguishing descriptions in the item master
ITP revised to Rev B with the receipt witness point in place
PFMEA and control plan updated, with the detection rating re-scored against the new controls
Assembly traveller and receiving inspection instruction revised, with completed records on three subsequent units
Effectiveness verification completed under the revised nonconformance procedure before closure

Engineering Decision

Shipment status — skid serial 003

RELEASED

Basis

The acceptance criterion is now met with margin on the affected unit and on two subsequent units. The direct cause is corrected, the systemic cause is addressed in controlled documents, the affected population including stock spares has been re-inspected, and effectiveness verification has been completed under the revised nonconformance procedure.

The nonconformance MQA-NCR-014 is closed on evidence, not on explanation — which is the distinction that the supplier’s original process could not make, and the reason it was a major qualification finding in the first place.

What would have kept this on hold

Fewer than three consecutive conforming units, or any unit clearing the limit only within measurement uncertainty
Prevention actions approved but not evidenced in the controlled documents
Affected population scoped by build date rather than by purchase-order line, leaving stock spares unverified
Root cause unable to account for the full magnitude of the shortfall, implying a second cause still present
Effectiveness verification skipped — the exact gap this supplier was cited for at qualification

MQA-8D-001 Rev A · September 2026 · Closed — verified

Engineering judgment

What decision had to be made?

Can this skid ship? Asked five times, at five different states of evidence — and answered differently each time. The interesting decision is not the final release; it is the refusal to release after the equipment had been repaired and was demonstrably performing, because the systemic cause was still open.

What evidence mattered?

At discovery: a repeatable measurement against a stated criterion, with the instrumentation independently verified. That is enough to hold and not enough to do anything else.

At the root-cause stage: the quantitative agreement between the predicted and observed shortfall. That single calculation is what allowed the corrective action to be scoped to the impeller with confidence rather than hedged across several candidate causes.

At release: three consecutive conforming units with margin, the affected population re-inspected by the correct scoping rule, and the prevention actions evidenced in controlled documents rather than promised.

What would cause me to change the decision?

Deviation instead of hold, in one specific circumstance. If the served technology loop’s actual resistance had been re-calculated by the owner’s engineer and shown that 195 kPa still delivered required flow with margin at full load, then the criterion — not the equipment — was wrong, and the correct action is an engineering deviation with a recorded technical basis, plus a correction to the specification for future units. That is a real possibility here, because the 250 kPa figure is itself an assumption.

What would not change it: schedule pressure, a supplier assurance that the next unit will be correct, or a single passing retest. And if the root cause had accounted for only part of the shortfall — say 20 kPa of the 55 — the investigation would not have been closable at all, because an unexplained residual means an unidentified cause is still in the process.

Engineering documents

MQA-8D-001 · Rev A

PDF

8D Report — Cooling Skid Differential Pressure Nonconformance

Full D1–D8 investigation record including containment, root cause, corrective action, verification and prevention.

Synthetic data — not for production use

Download PDF

MQA-ITP-001 · Rev A

XLSX

Inspection & Test Plan — LC-1000 Liquid Cooling Skid

Operation-level inspection and test requirements with owner quality points and required records.

Synthetic data — not for production use

Download XLSX

MQA-FAT-001 · Rev A

PDF

Factory Acceptance Test Protocol — LC-1000

Preconditions, test steps and acceptance criteria across visual, mechanical, controls, performance and documentation scope.

Synthetic data — not for production use

Download PDF

References & technical basis

ASHRAE TC 9.9 — Thermal Guidelines for Data Processing Environments (liquid cooling chapter, 5th edition)
Informed: The facility water temperature classes (W17 / W27 / W32 / W40 / W45 / W+), each named for its maximum allowable supply fluid temperature, and the existence of companion water-quality and surface-temperature classifications. Used to place the study’s assumed supply temperatures on a recognised scale rather than inventing them.
Open Compute Project — Cooling Environments project and Cold Plate sub-project
Informed: The functional definition of a coolant distribution unit — heat transfer between dedicated liquid loops with control of pressure, flow, temperature, dew point, cleanliness and leak detection — and the requirement to validate material compatibility between wetted materials and the coolant. Shaped the equipment scope and the controls section of the FAT.
Open Compute Project — Data Center Liquid Distribution Guidance & Reference Designs
Informed: Typical filtration sizing: 25–50 µm on the technology cooling loop, set by the most particle-sensitive component, and 150–250 µm on the facility side. Used directly for the two filtration parameters in the specification.
Open Compute Project — Liquid-to-Liquid CDU Test Methodology and Performance Rating
Informed: That a published, structured test methodology exists for rating liquid-to-liquid CDU performance. Informed the decision to define the operating point, the hold period and the repeat condition in the performance test rather than recording a single instantaneous reading.
ASME B31.3 — Process Piping
Informed: The hydrostatic leak test requirement of not less than 1.5 × design pressure, subject to not exceeding the rating of any non-isolated component, and the allowable-stress correction when design temperature exceeds test temperature. Used to derive the test pressure rather than assert it.
ASME Boiler and Pressure Vessel Code, Section IX — Welding, Brazing and Fusing Qualifications
Informed: The WPS / PQR / welder-qualification framework and the role of essential variables. Used to define what "approved welding procedure" has to mean as an auditable acceptance criterion in the ITP and in the supplier assessment.
ISO 9001 — Quality management systems
Informed: The structure of the quality-management-system, document-control and nonconformance areas of the supplier qualification assessment, including the distinction between a defined control and a control demonstrated to be effective.

Standards listed here were reviewed to structure this study’s quality approach. They are referenced, not claimed as professional application experience, and any parameter not traceable to a source above is marked as a study assumption.

Limitations

Every measurement, record and result on this page is synthetic. The internal consistency is deliberate — the affinity-law check works because the numbers were constructed to be coherent — and should be read as a demonstration of how a cause is validated, not as test data.

The 250 kPa acceptance requirement is a study assumption. A real investigation would begin by confirming the criterion against the owner’s hydraulic calculation and the approved specification, because “the requirement is wrong” is a legitimate and not uncommon outcome of a performance nonconformance.

RPN values use a generic 1–10 scale for illustration. Rating scales, thresholds and the actions they trigger would come from the applicable PFMEA procedure on a real programme.

Professional foundation and independent study

Professional foundation

Industrial equipment and rotating equipment, including industrial pumps
Manufacturing engineering on high-volume and engineered-to-order production
Equipment acceptance: acceptance criteria, vendor run-off, punch-list disposition, buy-off
Process validation and issue-closure governance across multiple sites
RCA, SPC, FMEA, control plans and nonconformance management
Mechanical drawings, GD&T, P&IDs and build-to-print verification

Independent study

Data-center cooling architectures and liquid cooling equipment
Coolant distribution unit scope, instrumentation and control functions
Facility water temperature and quality classifications
Application of supplier-quality methods to mission-critical cooling equipment
FAT methodology for equipment where failure consequences are operational rather than commercial
Referenced piping and welding qualification codes as acceptance-criteria sources

Methods

8D
Containment
Fishbone
5-Why
Affinity-law validation
CAPA
Effectiveness verification
PFMEA
Control plan
ITP revision

The prevention side of this story — qualification, readiness, the inspection and test plan the failure improved, the FAT protocol and the release gate — is in the supplier qualification and FAT study. The professional equivalent of this disposition decision, on production equipment, is in Production Equipment Run-Off & Acceptance.