Rosetta-CLI check:data memory requirement

Hi!

We have tried to run the rosetta-cli check:data and it resulted in an out of memory error. Do you have any recommendations?

[STATS] Blocks: 362000 (Orphaned: 0) Transactions: 116801 Operations: 329732 Reconciliations: 1061473 (Inactive: 766686)
2020/08/05 08:35:16 Syncing 362000-362999[STATS] Blocks: 362000 (Orphaned: 0) Transactions: 116801 Operations: 329732 Reconciliations: 1061635 (Inactive: 766840)[STATS] Blocks: 362000 (Orphaned: 0) Transactions: 116801 Operations: 329732 Reconciliations: 1061757 (Inactive: 766962)
[STATS] Blocks: 362000 (Orphaned: 0) Transactions: 116801 Operations: 329732 Reconciliations: 1061887 (Inactive: 767092)
[STATS] Blocks: 362000 (Orphaned: 0) Transactions: 116801 Operations: 329732 Reconciliations: 1061969 (Inactive: 767174)2020/08/05 08:36:13 block 362922 fetch error: Post http://localhost:8080/block: EOF
2020/08/05 08:36:13 retrying fetch for block 362922 after 0.337160s2020/08/05 08:36:13 block 362921 fetch error: Post http://localhost:8080/block: EOF2020/08/05 08:36:13 retrying fetch for block 362921 after 0.345595s
[STATS] Blocks: 362000 (Orphaned: 0) Transactions: 116801 Operations: 329732 Reconciliations: 1061991 (Inactive: 767196)
fatal error: runtime: out of memory

Interesting :eyes: !! This is the first time I’ve seen this error and have run this tool on many different implementations (my computer has 16 GB of RAM). We aggressively write to disk to minimize how much we keep in RAM but there are some places where bloat could occur under certain circumstances.

Initial Questions

  • How much RAM does your computer have?
  • Did this occur while at tip or behind?

Theories

One is that your implementation has significantly more accounts than other blockchains I’ve tested and it is causing the seenAccounts cache and inactiveQueue to get quite large (the only real memory user in check:data).

Another is that there is a large increase in the number of active accounts at a particular account and it causes the backlog channel to get overwhelmed (could be using more memory than I initially realized).

I’ll need to do some of my own research/profiling and get back to you on this.

I created an issue on rosetta-cli.

We just merged some substantial optimizations based on this issue. They will be in a release soon!

1 Like