Node Tracking System Servers Update – Coming Soon!

Servers are targeted to be updated on the weekend of December 7th.

In the continuing push to secure and improve the Node Tracking Systems, the following changes are being deployed. There are a couple of significant changes for both Secure Node and Super Node Tracking Servers in this update.  We ask the node operators to monitor and report any issues.

• Weekly Rollup of Reward Payouts: Rather than seven individual payments per week for each node, the payments will be rolled up into one payment. This change, along with reducing the number of payments in each transaction (from 1200 to 300) in a block, should help reduce some of the issues seen managing payments in software and hardware wallets.
• API Rate Limiting: API usage has increased and has started to strain some of the tracking servers as the number of nodes and third party applications continue to grow.  Rate limiting will be enabled to help smooth out the load. Please see details below.
  • API Rate Limiting will be implemented gradually, and will not take effect immediately when the tracking server is updated. You can expect full API rate limiting starting December 10th or shortly after.
  • The rate limiting predominantly affects hosting providers, monitoring applications, aggregators, etc. No effect is anticipated for applications such as wallets.

Other Fixes/Changes

Tighter Node Authentication and Validation: The identification of a node is now more tightly linked to its WebSocket. The data from the nodes also now has additional validation. Both of these changes help close potential exploits.

Tracking Nodes Across Servers: The last server update included additional management of a node when not on its home server.  This update includes some more checks to ensure the tracking servers properly track nodes and close Downtimes.

Reconnection Limits: Some potential exploits were identified that included nodes manipulating their status by reconnecting often. Reconnection limits will be imposed and Exceptions created for node connecting too many times in a short period.

Ensure Uptime Does Not Exceed 100%: A few instances recently occurred where the calculations during payment generation resulted in uptime over 100%.  This change is an additional check during the calculations as an override in case of an unknown cause. The known cause has been fixed.

API Rate Limiting

Due to increased use and its effect on some of the tracking servers, rate limiting is being enabled on all the tracking servers (Secure and Super Nodes). The intent is to spread out the API calls as there have been large spikes causing increased memory and CPU usage.  The limits will be adjusted based on usage patterns and feedback from the community over time.

The rate limiting predominantly affects hosting providers, monitoring applications, aggregators, etc. No effect is anticipated for applications such as wallets.

General Guidelines

• Static Data: Much of the data, such as payments, does not change often. There is very little need to constantly make calls to update this data. Please review your application data needs and adjust how often API calls are made where possible.
• API vs. ‘/grid’: The tracking server web pages retrieve server data using the ‘/grid’ path and were never meant to be used for API calls for third party applications.  Please refrain from using ‘/grid. NOTE: rate limiting also applies to ‘/grid’ and will be more restrictive.
• Initial Rates: The target for API calls is a maximum of one call per second. The initial rates allow for bursting over short periods, but the maximum within the period will apply. For example: a path may allow a maximum of 30 calls in 30 seconds. If 30 calls are made in 10 seconds, the requestor will have to wait 20 seconds for a reset of the count.
• Rates by Path: some API paths will allow for higher or lower rates. Those rates will be published in the API guide on the tracking servers.
• Headers: rate limiting data is available in the response headers for your application to consume if needed. The following HTTP headers can be used to determine Rate Limiting budgets:
  
  

  HTTP/1.1 429 Too Many Requests
  X-RateLimit-Limit: 30
  X-RateLimit-Remaining: 0
  X-RateLimit-Reset: 1575391883
  Retry-After: 60

• Exceeding the Limit: When the limit is exceeded, a JSON object is returned which includes the status code of 429, information about the limits (max/secondes) and how long it will be until the count resets. Example:
  
  
  {\”status\”:429,\”error\”:\”rate limit\”,\”message\”:\”Too many requests. Limit to max 30 per 30 seconds. Try again in 12 seconds.\”}
   
• Target Servers: Each of the tracking servers are capable of returning API calls.  The tracking server infrastructure cannot be configured to round-robin incoming calls to servers without some extensive changes.  For now, it would be beneficial if your applications could point to a specific server or randomly select a server rather than use the URL without a number. E.g. securenodes3.na.zensystem.io instead of securenodes.na.zensystem.io.

If the API does not supply data needed for a specific purpose, let the team know. The request will be reviewed and incorporated into the next server build if possible.

For technical details visit Confluence. Please let us know if you have any questions. Thank you for your continued support!