Horizen Node Tracking System Maintenance

The Super Node and Secure Node Tracking and Payment Systems will be undergoing a system-wide update to their database servers.

The web servers and the node servers will be put into ‘service mode’ while the database servers are being updated. The systems will be unavailable for an extended period.

Important Information

Node operators: Node operators do not have to take any action. Nodes will connect to a temporary server during the service window and then reconnect to their home server when the servers are back online. There is no need to move homes during the service period.

Earnings: There will be no impact on reward earnings. Any new downtime or exception due to the service will be excluded from earning calculations.

Duration: The systems may be in service mode from 8 to 24 hours.

When: The service period will take place on the weekend of May 25th.

New Nodes: Registering a new node will not be available during the service period.

For nodes running automated scripts: Any request from a node while in service mode will be ignored. The only response to the node is an acknowledgment for a statCheck. If possible, please suspend automated requests during the service period and allow the servers to generate tasks such as challenges.

Tracking Server changes:

In conjunction with the maintenance, the tracking servers will be updated to a new version.

The release date of these changes is to be determined.

Some of the significant updates will be:

Fix

• Some challenge Exceptions were not closing if more than one existing was left open due to timing issues.
• Ensure credit is paid to the proper stake address (from previous stake change restriction updates).
• Handle bursts of DNS errors from misconfigured nodes that were causing a memory leak.

Change

• If the cause for the same type of Exception changed, the existing Exception will now be closed and a new one created with the new error description to help with troubleshooting.
• There are some query optimizations converting single to bulk updates to help with database load.

New

• Cert Exceptions now have a minimum threshold of 3 minutes like Downtimes before counting against earnings. Unlike Downtimes, short duration Cert Exceptions are created and flagged with an ‘x’ status if under the threshold. An email alert is sent if over the threshold.
• Cross system check to compare IP and Stake addresses across Secure and Super node system and create Exceptions if duplicates are found.
• API Subkeys allow hosting providers to generate an API key for their individual customers. This is based on the optional category nodetracker setting.
• Old IP address re-use. Dead nodes are no longer checked when a new node registers. This applies to IP4, IP6, node t-address, stake address, and FQDN. There is a new interim period with a status of ‘zombie’ with a 20 day period after a node goes ‘inactive’ during which the node may be revived. The total time before a node is set to ‘dead’ is 30 days (5 days down, 5 days inactive, 20 days zombie).  This frees up many IP4 addresses for re-use.
• If multiple stat checks are missed, a block height (blk) Exception is created and the node forced to reconnect. If stat checks continue to be missed and 10 or more Exceptions are created in one earning period, the node will be excluded from rewards.
• Service mode to allow for system-wide downtime.