Syscoin Rosetta integration

[sidhujag]

Hi there,

Syscoin is a fork of Bitcoin with some token platform support on top. We plan to integrate token support in a later version but to start we can integrate Syscoin itself which should be fairly easy since it leverages all of Bitcoin’s structure in a backward compliant way. There are no changes to the transaction structure or block structure but there are some consensus updates with token transfers, however I do not think its a problem for rosetta sdk afaik. I did an initial implementation here: https://github.com/syscoin/rosetta-syscoin

The unit and security checks pass. Is there anything else I need to do for integration?

Thanks,
Jag

[patrick.ogrady]

@sidhujag

There are no changes to the transaction structure or block structure but there are some consensus updates with token transfers, however I do not think its a problem for rosetta sdk afaik. I did an initial implementation here: https://github.com/syscoin/rosetta-syscoin

Glad to see you could make use of rosetta-bitcoin

@sidhujag

The unit and security checks pass. Is there anything else I need to do for integration?

Have you tried running rosetta-cli check:data and rosetta-cli check:construction on your implementation yet? The “Testing” section of following post provides some details on the rosetta-cli settings we will use to test your implementation:

Rosetta End-to-End Testing Prerequisites

Rosetta End-to-End Testing Requirements Before we begin testing a Rosetta implementation, we usually need to collect important information about deployment nuances and/or communicate the testing settings we use (for the rosetta-cli). In an effort to expedite testing, we’ve shared a list of details you should provide when submitting your implementation and a list of end conditions we use when testing. Please reach out if you have any questions! Deployment Details Please include the following in…

[sidhujag]

Hi Patrick,

Please see issue https://github.com/syscoin/rosetta-syscoin/issues/1

I haven’t been able to get it to sync (mainnet) and stay stable, I am not sure where its dying or what the cause of it is. Any help would be appreciated.

Jag

[patrick.ogrady]

@sidhujag

Hi Patrick,

Please see issue https://github.com/syscoin/rosetta-syscoin/issues/1

I’ll take a look and respond there!

[sidhujag]

@patrick.ogrady

rosetta-cli check:data

Hi Patrick,

Now that we have resolved the performance bottleneck (https://github.com/coinbase/rosetta-sdk-go/pull/247#pullrequestreview-536776455) we have run rosetta-cli check:data and it looks good. I think it is ready. I am keen on upgrading it as soon as you update the base SDK with these or like improvements. I can also help you if needed on the designs and implementation on other industry standard performance tricks to further improve the ecosystem. We will look to add our token layer soon after.

Thanks,
Jag

[patrick.ogrady]

@sidhujag

I am keen on upgrading it as soon as you update the base SDK with these or like improvements.

Thanks again for digging into this! With everything happening in Rosetta right now, optimization work doesn’t always get priority.

@sidhujag

I can also help you if needed on the designs and implementation on other industry standard performance tricks to further improve the ecosystem. We will look to add our token layer soon after.

Appreciate the offer. I’m planning to get the SDK update out in the next week or so (work grinds to a halt around Thanksgiving) but am always interested in hearing any suggestions folks have about improvements.

[sidhujag]

No problem Patrick!

Enjoy your thanksgiving, for now I have pointed rosetta-syscoin to the SDK with my fix and its happy. Will point it to official SDK once you have some update out and I can verify it works the same or better

In the short term, is there anything else you need me to look at regarding our integration?

[patrick.ogrady]

@sidhujag

In the short term, is there anything else you need me to look at regarding our integration?

We published our expectations for testing a week or so ago (linked previously):

Rosetta End-to-End Testing Prerequisites

Rosetta End-to-End Testing Requirements Before we begin testing a Rosetta implementation, we usually need to collect important information about deployment nuances and/or communicate the testing settings we use (for the rosetta-cli). In an effort to expedite testing, we’ve shared a list of details you should provide when submitting your implementation and a list of end conditions we use when testing. Please reach out if you have any questions! Deployment Details Please include the following in…

If it meets these expectations, then you should be good! We just need to hook things up on our side to start testing (we have a service that automatically tests implementations using the rosetta-cli).

If you’re looking for something else to do related to Rosetta, however, I have a few ideas (obviously no pressure to do any of these):

• /mempool/transaction support in rosetta-bitcoin (could be ported to your implementation)
• /account/coins mempool support in rosetta-bitcoin (could be ported to your implementation)
• SQL-based Rosetta Indexer implementation
• Generic Rosetta Wallet (like Metamask for any Rosetta implementation…maybe powered by Rosetta DSL?)

[sidhujag]

Finally got through check:data on mainnet after all of our fixes I think you have to merge in https://github.com/coinbase/rosetta-cli/pull/198

Success: Reconciliation Coverage End Condition [Coverage: 96.125190%]

±-------------------±-------------------------------±-------+
| CHECK:DATA TESTS | DESCRIPTION | STATUS |
±-------------------±-------------------------------±-------+
| Request/Response | Rosetta implementation | PASSED |
| | serviced all requests | |
±-------------------±-------------------------------±-------+
| Response Assertion | All responses are correctly | PASSED |
| | formatted | |
±-------------------±-------------------------------±-------+
| Block Syncing | Blocks are connected into a | PASSED |
| | single canonical chain | |
±-------------------±-------------------------------±-------+
| Balance Tracking | Account balances did not go | PASSED |
| | negative | |
±-------------------±-------------------------------±-------+
| Reconciliation | No balance discrepencies were | PASSED |
| | found between computed and | |
| | live balances | |
±-------------------±-------------------------------±-------+

±-------------------------±-------------------------------±-----------+
| CHECK:DATA STATS | DESCRIPTION | VALUE |
±-------------------------±-------------------------------±-----------+
| Blocks | # of blocks synced | 792777 |
±-------------------------±-------------------------------±-----------+
| Orphans | # of blocks orphaned | 0 |
±-------------------------±-------------------------------±-----------+
| Transactions | # of transaction processed | 1622077 |
±-------------------------±-------------------------------±-----------+
| Operations | # of operations processed | 6853262 |
±-------------------------±-------------------------------±-----------+
| Active Reconciliations | # of reconciliations performed | 2976346 |
| | after seeing an account in a | |
| | block | |
±-------------------------±-------------------------------±-----------+
| Inactive Reconciliations | # of reconciliations performed | 10035865 |
| | on randomly selected accounts | |
±-------------------------±-------------------------------±-----------+
| Exempt Reconciliations | # of reconciliation failures | 0 |
| | considered exempt | |
±-------------------------±-------------------------------±-----------+
| Failed Reconciliations | # of reconciliation failures | 0 |
±-------------------------±-------------------------------±-----------+
| Skipped Reconciliations | # of reconciliations skipped | 682085 |
±-------------------------±-------------------------------±-----------+
| Reconciliation Coverage | % of accounts that have been | 76.648184% |
| | reconciled | |
±-------------------------±-------------------------------±-----------+

badger 2020/12/03 19:16:31 INFO: Storing value log head: {Fid:62 Len:36 Offset:3159392}
badger 2020/12/03 19:16:32 INFO: [Compactor: 173] Running compaction: {level:0 score:1.73 dropPrefixes:} for level: 0
badger 2020/12/03 19:16:42 INFO: LOG Compact 0->1, del 2 tables, add 1 tables, took 10.175999381s
badger 2020/12/03 19:16:42 INFO: [Compactor: 173] Compaction for level: 0 DONE
badger 2020/12/03 19:16:42 INFO: Force compaction on level 0 done

[sidhujag]

check:configuration depends on https://github.com/coinbase/rosetta-bitcoin/pull/50

[patrick.ogrady]

Glad we finally got this working! Appreciate the debug help!

[sidhujag]

Thanks on your continued focus on excellence towards optimization and making it just “work”… I think we are going to test your new optimizations before tagging our release and then pass it off to you guys. Once we have verified your recent changes we will be in a good spot to make it public.

[brqgoo]

Hi @sidhujag very interesting. Did you modified the base Bitcoin scripting layer to determine the validity of a token transaction with smt like OP_GROUP? Or is it something similar to other “colored coin” protocols which does not enforce miner validation?

[sidhujag]

@brqgoo Hi thanks for the great question.

We didn’t modify the scripting layer we certainly could have gone in that direction but found an easier path that didn’t need to change that to do the same thing and more compatible with bitcoin parsers like explorers and SDK’s.

The commitment of the asset is stored in OP_RETURN (with added benefits of being prune able). So for example the asset commitment is to output #1 of a transaction the value is stored like [1234 (asset guid), {1, 10000 (value)}] a map of entries per asset representing an array of outputs indexing into the outputs. The UTXO database was extended to have asset guid info namely the tuple {1234, 10000} in this case. Dust satoshi is needed on any output so it follows the same security policies as bitcoin. This is done on Syscoin the only connection to Bitcoin physically is the merged-mining concept which allows Syscoin to share security with Bitcoin via common nonce generation.

The commitment values are compressed with a lossless compression algorithm and then varint stored to reduce network footprint of the 8 byte values representing the satoshi values.

Miner’s must adhere to policies coded in consensus namely that the input map of asset values must be equal to the output. There may be many inputs of many different assets sending to outputs but the in must equal out because there are no fees applied to the asset layer.

[brqgoo]

Got it, thanks. Looks similar to Simple Ledger Protocol, but the consensus-enforced one. I think it would be beneficial to introduce new scripting capabilities like OP_CHECKDATASIG or OP_PUSHTXSTATE to allow token-native covenants for dApps like AMM-swaps, hedge, long etc. Also this can potentially be a Bitcoin SPV side chain since BIP-Tapscript proposes to make splice opcodes OP_SUCCESSx.

[sidhujag]

It should be compatible with any existing op_codes or new introduced ones. These UTXO’s are coded to be spendable with the same opcode structure as bitcoin and so if taproot adds a few new ones it will work with those, the UTXO looks like dust but it really has assets on it that spend as soon as you spend that dust.

About adding new opcodes I think it depends on the security of adding those. If we do add new opcodes specific to assets then I think at that point we would probably just make script system assets aware aware at that point anyway.