From Ticking Boxes to Real-World Commissioning
Commissioning is no longer just about ticking boxes. As the complexity and interdependence of data center systems grow, so does the need to challenge assumptions. The final IST (Integrated Systems Testing) is the ultimate proving ground - but it's often too clean, too rehearsed, and too optimistic.
We are looking into a future with more data centers, and the criticality of these systems are increasing. That calls for data center commissioning that is more than just ticking boxes.
So, now enter the Red Team mindset 🚷. Borrowed from cybersecurity and military strategy, Red Teaming involves actively probing for weaknesses. In commissioning, that means injecting controlled failures, and surfacing hidden vulnerabilities before the facility is live, but with a dynamic and realistic approach.
What Red Team Testing Looks Like in Practice
You have the prepared IST test plan with detailed sequences and line items - that will form the core of your IST. As a commissioning professional we like to have structure and a plan, but with these IST's we must try to deviate and follow other paths. For example, during one of the logic sequences testing the switch from medium to high load, then triggering an unannounced utility outage or disrupting sensor signals to expose system logic flaws.
These tactics aren't about pointing fingers - they're about finding gaps when it still matters. We've seen well-designed systems fail under trivial fault conditions simply because those scenarios weren't imagined during design or tested during commissioning.
Lessons from the Field
These two examples are from a real IST we did for a client. The first example, with the UPS transfer, was actually not planned - it was an accident! But it really showed that the not-planned and odd-cases can be very revealing.
A UPS transfer test is underway during an Integrated Systems Test (IST). Unexpectedly, another team starts multiple generators and energizes several pieces of equipment, creating a sudden load imbalance. The imbalance prevents a clean transfer, resulting in a system-wide failure and unexpected downtime.
Consider another case where a building operator misses a critical local alarm due to a temporary disruption in the Building Management System (BMS) network. In this example, the only fallback alert was a blinking LED hidden in a panel—no buzzer, no escalation. A brief two-minute simulated network outage uncovered a two-hour delay in response.
These aren't far-fetched edge cases - they're representative of the types of issues that could hapen - but shouldn't happen.
Designing Smarter IST Plans
A good IST plan covers expected transitions - load transfers, generator starts, chilled water balancing. A great one goes a bit further and introduces friction. What if a technician is out of position? What if a PLC drops a signal for 30 seconds? What if a chilled water valve sticks mid-swing?
This doesn't mean rewriting the whole test plan. It means layering real-world stress on top of your core IST sequence. Add optional triggers with a specific objective: "a new tech meets in, sees the switch is off, and turns it on and walks in to grab a cup of coffee".
Top 10 IST Test Edge Cases to Consider
Here are ten challenges that you can use as inspiration to push your test a bit further. Use them as inspiration. If you have any good experiences or examples, please let us know and we can add them to the list (and provide credit!).
| ⚠️ Edge Case | 🔍 Stress Focus | 🎯 Objective |
|---|---|---|
| Simultaneous ATS stuck open and breaker fail | Switching logic under dual fault | Reveal sequence failure when multiple paths break |
| Partial UPS battery failure during generator warm-up | Layered power resilience | Test UPS autonomy in degraded state |
| Chiller restart under low ambient conditions | Thermal recovery control | Ensure restart logic handles cold climate quirks |
| BMS communication loss across multiple devices | System decentralization | See how local logic holds when central logic fails |
| Conflicting setpoints from BMS and field controllers | Coordination between control layers | Detect signal prioritization and override logic |
| Blocked fire damper without override signal | Fire safety automation | Expose gaps in override paths or alarm triggers |
| Elevated room temp with sensor failure masking it | Sensor dependency | Trigger conditions when redundant sensing fails |
| Missed alarm escalation when email server is offline | Alarm routing robustness | Validate alternate comms and escalation paths |
| Emergency stop during active switchgear load | Emergency logic under load | Test controlled failure in high-energy scenario |
| Power restored with one system still in failover mode | Recovery coordination | Reveal auto-recovery flaws or desync behavior |
Safety and Control
The "Red Team" mindset in commissioning is high-impact - but it must be controlled. Every injected failure should have a rollback plan and pre-approval; but that's of course quite difficult if "an idea" happens during a test.
A good idea before starting an integrated systems test is to have a list of "things that can go wrong" and a plan for each of them. The same goes for having a risk and acceptable risk list.
Conclusion: Commissioning for the Real World
Red Team testing isn't about being reckless - it's about being thorough. By adopting this approach, you're commissioning for the real world, not just the design spec. You're empowering your client with confidence and protecting the facility's reputation before it even goes live.
At CxPlanner, we help teams document and manage complex commissioning activities and testing with built-in test logging, fault tracking, and team communication tools in commissioning software. If you're ready to go beyond the standard IST template, we're ready to support you.