Horizen Node Challenge Time Reductions and Nodetracker Changes

Dear Horizen community,

With the goal of enhancing our node network and make it stronger, we would like to announce two important set of changes:

• Nodes challenge times
• Nodetracker v0.4.0

Node challenge time changes

Starting on  February 18th, the challenge times will be reduced both on Secure Nodes and Super Nodes. The new challenge times will be:

• Secure Nodes: 200 seconds
• Super Nodes: 100 seconds

The new split circuit design (Groth16) implemented with ZEN 2.0.16, as a result of the latest mandatory network upgrade, reduced the time needed to execute a shielded transaction – and, therefore, a node challenge – by around 50%.

The new implementation is also able to scale with multi-core CPUs much better, meaning that some node configurations will see even greater improvements.

A strong node network represents an essential part in supporting the ZEN ecosystem. In order to keep the hardware configurations with the same level of robustness as before, we decided to reduce the time allowed for Secure and Super Nodes to complete the challenge execution.

Nodetracker v0.4.0 changes

The latest Nodetracker version 0.4.0 has several changes and improvements which help with connectivity and uptime.

All Secure and Super Nodes should be updated to this new version. Version 0.4.0 will be enforced (earning eligibility) starting March 1st.

General instructions for updating a node can be found in the github readme

Detailed instructions are in the Maintenance Guide: Upgrading Tracker Software (systemd)

Summary list:

• Add randomize delay before reconnect
• Remove socket reinit on reconnect
• Add drop socket and connect on server request
• Include region in move home
• Remove tls peers on stat check
• Add tls peers on server request
• Add stat ack timeout to reset socket after random retry
• Add reset timer and other settings from tracking server
• Add saving timer and other settings
• Add multiple zaddr check to use highest balance
• Add latency check based on socket level ping/pong
• Add periodic ‘checkIn’ (ping/pong) at application level
• Add error check and retry timer on failure to get zaddress balance
• Save application settings to local and allow updates from server
• Add ipv6 lookup family to ipv6 dns workaround (credit to emminer – thanks!)

Details:

Randomize delay before reconnect-  on a disconnect, the tracker randomizes the amount of time before attempting reconnects.  This spread reconnects of the nodes across a much wider time period (default of about a one minute window) which is helpful if a tracking server is restarted. This uses native socket.io features.

Remove socket reinit on reconnect- re-initializing the internal objects was found not to be needed.  This now just logs the event.

Add drop socket and connect on server request- a command from the tracking server can be sent to reset the socket.io connection.

Include region in move home- if a node was moved between regions it may have moved back to its original region if the setup was run again.

Remove tls peers on stat check- no longer used for secure nodes

Add tls peers on server request- server may request the list of peers from the node for troubleshooting

Add stat ack timeout to reset socket after random retry- if the server does not reply to the periodic stat check (6min), the tracker will resend the stat check based on random wait time. If there is still no response from the server after 3 retries, the connection is reset. If after the reset there is still no replies, the retry interval is increased to  the max (default 2min)

Add reset timers and other settings from tracking server-  certain settings can be managed from the tracking server.

Add saving timer and other settings- certain settings are saved to the local storage of the nodetracker.  The timestamps for each section are sent in each statCheck and a comparison is done on the server.  If the server values are newer they are sent to the nodetracker. Running timers are reset (conCheck and statCheck)

Add multiple zaddr check to use the highest balance- if more than one zaddress is available for challenges, the one with the highest balance will be used.

Add periodic ‘checkIn’ (ping/pong) at the application level- the existing connection check (every 30s) which starts the failover timer if the server is disconnected has been updated to also send a request over the socket to the server.  It counts the replies missed within a 5-second response window and if more than 2 are missed it will reset the connection. This feature is off by default and will be enabled from the tracking system at a later time.

Add error check on failure to get zaddress balance- reported issue which is difficult to reproduce. If there is a ‘ERROR zend get addrwithbalance’ a timer is now set to retry until successful. 

Add ipv6 lookup family to ipv6 dns workaround- PR from community (emminer) to include the ‘family: 6’ in the parameters for the dns.lookup when using only ipv6.