Issue
If an APC Switched Rack PDU's Network Management Interface is rebooted (manually by the user, or automatically due to network watchdog timer as explained in knowledge base ID FA156063), any pending outlet commands will be lost. This is especially a concern when the network device (switch, router, hub, etc) powering the network connection to the Rack PDU is powered by one of the outlets on the Switched Rack PDU and that outlet is turned off for 9.5-10 minutes and the watchdog timer expires.
Product Line
- Rack Power Distribution
- AP79XX Switched Rack PDU
- AP89XX Switched Rack PDU 2G
- AP86XX Switched Rack with Metering By Outlet PDU
Environment
- Users with their network connection to a Rack PDU powered by an outlet
- Users turning the outlet off for more than 10 minutes
- Users rebooting their Rack PDU with outlet commands pending
Cause
Pending outlet commands are stored by the Rack PDU in Random Access Memory (RAM), as opposed to Non-Volatile RAM, by design. Because of this, anytime the PDU Network Interface is rebooted, either on purpose, or because of a failsafe such as the network watchdog timer explained in knowledge base ID FA156063, pending outlet commands may be lost and outlets will not perform as expected.
Additional information on the network interface watchdog timer:
The default gateway can be any valid node's IP on the network management card's network. The Management Card implements internal watchdog mechanisms to protect itself from becoming inaccessible over the network. For example, if the Management Card does not receive any network traffic for 9.5 minutes (either direct traffic, such as SNMP,or broadcast traffic, such as an Address Resolution Protocol [ARP] request), it assumes that there is a problem with its network interface and restarts to prevent further problems. To ensure that the Management Card does not restart if the network is quiet for 9.5 minutes, the Management Card attempts to contact the default gateway every 4.5 minutes. If the gateway is present, it responds to the Management Card, and that response restarts the 9.5-minute timer. If your application does not require or have a gateway, specify the IP address of a computer that is running on the network most of the time and is on the same subnet. The network traffic of that computer will restart the 9.5-minute timer frequently enough to prevent the Management Card from restarting.
Furthermore, executing certain commands may take 3-5 seconds each depending on other tasks present in the system or how many outlets you are applying it to.
Resolution
Here is some feedback and recommendations:
- Do not power the Rack PDU's network connection by the same Rack PDU's outlets (and turn it off for longer than 10 minutes) - If the Rack PDU's network connection will be unaffected when controlling outlets on a particular PDU, the network watchdog timer won't be activated unless the network traffic is too little or too much, or if the PDU encounters an isolated internal problem that prevents it from seeing network traffic. Essentially, the watchdog timer is only activated when the Rack PDU discovers it is connected to a network. If the PDU is never connected to the network on start up, this timer will never start.
- Do not power cycle or restart the PDU's Network Interface while any outlet commands are pending - Any pending outlet command will be indicated by a * next to the outlet in the web interface control page or in the Command Line Interface via local, telnet, or SSH access. The commands will be lost so if you need to reboot the Rack PDU to apply configuration changes or reboot manually for another reason, do so after all commands have completed successfully. Note: This also means making sure that any on/off delays have expired.
- Application dependent - if your Rack PDU's network connection must be powered by an outlet you're controlling, set the delay to less than 9 minutes to be safe - Since the watchdog timer expires in about 9.5-10 minutes, make sure the outlet on/off/reboot time for the specific outlet powering the Rack PDU's network connection is not off for more than 9 minutes.
Released for:APC Canada
