Horizen Node Tracking System Changes And Enhancements

In the continuing push to secure and improve the Node Tracking Systems, the following changes are being deployed.  Some of these changes were previously deployed and reverted due to issues outside the tracking system. Additional updates and fixes were incorporated into this current build resulting in a more extensive update.  We ask the node operators to monitor and report any issues.

NODE TRACKING SERVER  UPDATE

All Secure and Super Node Servers will be updated on the weekend of September 21/22. Short periods of unavailability of individual tracking servers are possible. Nodes will fail over to different servers and no impact on earnings is expected. No action is required by node operators and any unjustified downtimes/exceptions caused by the maintenance will be excluded.

Some of the significant fixes and changes are:

Fixes

• Some challenge Exceptions were not closing if more than one existing was left open due to timing issues.

• Ensure credit is paid to the proper stake address (from previous stake change restriction updates)

• Handle bursts of DNS errors from misconfigured nodes that were causing a memory leak

• In some rare cases multiple overlapping down times would not be calculated properly resulting in some nodes having extra up time.

New Features/Functionality

• Cert Exceptions now have a minimum threshold of 3 minutes (like Downtimes) before counting against earnings. Unlike Downtimes, short duration Exceptions are created and flagged with an ‘x’ status (excluded from up time calculations) if under the threshold to help with trouble-shooting.  The email alert is sent if over the threshold.

• Enhanced node connection checking.  There have been some recent node tracking system exploits from dishonest node operators that these enhancements should counteract. 

• Cross system check to compare IP and Stake addresses across Secure and Super node system and create Exceptions if duplicates are found. For IP addresses an Exception is created in both systems.  For Stake addresses the node that used the stake address first will not have an Exception.

• API Subkeys allow hosting providers to generate an API key for their individual customers. This is based on the optional category in the Nodetracker configuration. The subkeys work the same way as a parent key with a few data restrictions. These would also work for node operators who have a significant number of nodes and would like group them.

• Old IP address re-use. Dead nodes are no longer checked when a new node registers. This applies to IP4, IP6, node t-address, stake address, and FQDN.  There is a new interim period with a status of ‘zombie’ with a 20 day period after a node goes ‘inactive’ during which the node may be revived. The total time before a node is set to ‘dead’ is 30 days (5 days down, 5 days inactive, 20 days zombie).  This frees up many IP4 addresses issued by hosting providers for re-use in new nodes.

• A ‘stat’ Exception is created if a node misses multiple consecutive stat checks.  The tracking server will ask the node to reconnect its socket. If no stat check is received after the reset, the tracking server will disconnect the node in order to try to re-establish a connection.

• API call for Payment Masters  with data regarding a summary of an earning period.

• Aggregated Monthly Earnings display on the My Earnings page (based on API key) and a corresponding API call.

• Close and open a new Exception of the same type if the detail changes to help with troubleshooting

• Enhanced validation of data from nodes, web pages and API calls. 

• Return a JSON object instead of the Not Found page on a bad API path in the URL

• Invalid API calls will return an http status and message along with details of the issue.

• Check a stake balance even when a node is in the ‘down’ state. A clever node operator found a potential exploit and reported it. 

• Display estimated start of next earning period on tracking server home page

Administration and Optimizations

• Service mode for system-wide server updates.  This allows nodes to be parked during maintenance. They connect but are not managed and won’t failover to other servers.

• Some query optimizations converting single queries to bulk queries to help with database and server load.

• Update to node.js version 12 and current node modules along with some code clean up to reduce memory footprint.

• Various administrative features added to help with node management and identify potential exploits.

Please let us know if you have any questions. Thank you for your continued support!