Internet Computer Dashboard 10 hours behind

It appears the IC Dashboard (dashboard.internetcomputer.org) is 10 hours behind on data (as of right now). Would love to know more details on why (maybe related to ic0 spam issue?) and how long until a fix.

I’m specifically referring to ICP transactions in the dashboard and up-to-date wallet address balance information.

Thanks!

3 Likes

Hi Bob, thanks for flagging this.

3 Likes

This may not get resolved until tomorrow, so you can always use ICP Explorer in the meantime. It’s not as fully featured, but it uses Rosetta directly for transaction and account info (and currently has up-to-date data), as opposed to the IC dashboard which stores transactions and account info in a database (and is apparently stuck right now).

2 Likes

Yeah the reason I’m bringing it up is we used to use Rosetta API directly, but were running into some issues (5-6 months ago), so in a few products we switched to using IC dashboard to pull data.

But seems like perhaps Rosetta API is the better source of truth here (at least in this case)?

1 Like

The problem should be fixed now.

That’s hard to answer. Rosetta is somewhat limited in what you can do with it. It can’t be used for more advanced operations, for example sorting a list of accounts by account balance or calculating the total amount of minted tokens, but it’s good for basic operations like listing transactions. And perhaps it’s less prone to falling behind as the IC dashboard ledger API did today.

3 Likes

Hi. Thank you for pointing this out. The syncing mechanism fell behind obviously. We will work out the reason why and put better alerting to notify if this happens again. A restart of the syncing process is usually the solution to this problem.

I would suggest using rosetta-api.internetcomputer.org or running your own Rosetta node so you have a backup solution available.

The ledger API can run into similar problems as the Rosetta node when it comes to syncing, but it does have its own particular challenges. The last time the ledger API had a syncing problem was back in October 2022. There was an issue with a boundary node that was routing to a replica that had fallen behind. That effected the other dashboard processes as like proposal syncing.

5 Likes

Are there any independent (non-DFINITY associated) members that are currently running their own Rosetta Node? Just curious what the experience is like…

Dashboard is down again. 9 hours ago as of right now. :slight_smile:

Thanks for the report @bob11, the dashboard team rebooted a stuck service. It should be fixed now

1 Like

It has fallen behind again. The dashboard team is investigating the issue.

UPDATE 1: We have restarted the service. It is syncing properly again. I suspect there is an issue in our database connection pool. We will debug further. In the meantime, we will be releasing an update for our health check and status alerting.

UPDATE 2: We have pushed out a few updates to address potential problem areas. We will be monitoring over the weekend.

4 Likes