On the universality and uniqueness of the tx_hash identifier

Quick context: Decred has an on-chain mechanic where some transactions of a (so-called disapproved) block n can get reversed (by a disapproving block n+1) and appear again on a future block n+x (where x > 0). This surfaces some interesting points for discussion. See issue #53 on rosetta-cli for a slightly more detailed background.

dcrros (Decred’s Rosetta middleware) currently represents the reversed transactions on a disapproving block as a transaction where all operations have status == "reversed" and negative balance change and once the transaction is mined again on a future block, it’s represented in the standard way (status == "success", positive balance change) but even with the fix to issue 53 in rosetta-cli, we still have an outstanding problem when x == 1(i.e. the reversed transaction is included in the disapproving, n+1 block).

During some internal discussions on this matter, a few possible solutions for the problem of representing the same transaction appearing twice have been proposed:

  1. Using block_hash:tx_hash as the actual transaction_identifier.hash, thus uniquely identifying the transaction as belonging to a certain block.
  2. Have dcrros “trail” the underlying blockchain by one block and simply discard reversed transactions in their dcrros operation.
  3. Use a “tag” suffix on the tx_hash to indicate it’s actually a reversed transaction (so a transaction_identifier.hash of reversed transactions would be something like tx_hash:"rev".
  4. Include both the reversed operations and the new operations in the same Rosetta tx if the underlying blockchain transaction is both reversed and mined again in the same block (i.e. the situation where x == 1).

Now, for the point of discussion for this specific community thread:

Solution 1 doesn’t handle the case where x == 1 given that you’d still have two transactions with the same hash on the same block and both solution 1 and solution 3 break what I’m calling the “universality of the transaction.hash identifier”: namely, that a transaction hash identifier always has the same format and that it corresponds to the underlying blockchain’s transaction identifier.

The specific point for discussion in this thread is:

Should Rosetta implementations be constrained to always respecting tx_hash universality?

Another way of phrasing this would be to ask: Is it ok for different transactions (in the same blockchain) to have different “styles” of identifiers (if the underlying block chain does not have them) and should these identifiers precisely correspond to the ones of the underlying blockchain.

This is a great question @matheusd! Thanks for posting it to the community. I’ve shared my thoughts below and welcome you to follow-up if anything is unclear.

Transaction Hash as a Unique Identifier

For context, most Rosetta API users (including Coinbase) assume that network_identifier:block_identifier:transaction_identifier is a globally unique identifier of any transaction. Many systems above the “blockchain” layer will often rely on this assumption (ex: database IDs) and it is unlikely that these clients would or could loosen this assumption without significant work.

In #57, I put up a change that loosened “duplicate hash checking” to ensure there were no duplicates of network_identifier:block_identifier:transaction_identifier instead of just transaction_identifier (this was too strict). This replaced restriction even failed for Bitcoin block 91842 and 91812 for transaction hash d5d27987d2a3dfc724e359870c6644b40e497bdc0589a033220fe15429d88599 (although BIP-34 now prevents this from happening).

I noted that making “duplicate hash checking” optional would allow an implementation to pass rosetta-cli validation but it would likely prevent most clients from integrating (the whole point of the rosetta-cli to begin with). So, I decided we should not add this parameter.

Abstract/Uncorrelated Identifiers

As you mentioned above, it is possible to “hallucinate” identifiers that do not correspond to anything in the underlying blockchain (ex: tx_hash:"rev"). Following such an approach may allow for passing rosetta-cli validation but may make transaction parsing very fragile or complex. This is particularly concerning across updates where this uncorrelated identifier may change (it is far more likely for this to occur than for a canonical blockchain identifier to completely change form).

That being said, I think it is a bad idea to unilaterally say that using “abstract” identifiers is bad practice as there may be very good reasons to do this in some blockchains (ex: representing state changes that are attributable to an internal state machine instead of any transaction…like staking payouts). However, I would encourage anyone considering this path to explore alternative abstractions that do not introduce “uncorrelated” identifiers. If this approach is taken, I recommend including the “raw” identifier in the “metadata” of any affected objects so that interested clients can correlate the “abstract” identifier with the canonical representation.

Proposed Solution: Trailing + Native-Value “Reversed” Transactions in “Disapproved” Blocks

Given all the information you have provided (both here and in various Github issues), I recommend modifying your implementation to trail tip by 1, include reversed transactions in the disapproved block they occurred in with status reversed where reversed = not successful, and use the actual amounts (not opposites).

Trail by 1 + Reversed Transactions in Disapproved Blocks

By trailing by 1 block (this should still work even with re-orgs), it is possible to know if a block is disapproved before indicating it is queryable (clients will never query blocks past current_block_identifier and implementations must error if they do). If a block is disapproved, transactions in the block should be kept (instead of discarded, for visibility) and all Operation.Status should be set to reversed. This will prevent an identifier collision and clearly indicate which blocks were disapproved (if the block has no transactions, it will not be possible to determine it was disapproved unless Block.Metadata is parsed). This is very similar to how invalidated transactions are displayed on the Decred block explorer (ex: 88068).

Use reversed = not successful

Reversing the value of all operations and using an Operation.Status that is successful can be confusing for clients. In the modeling approach you took, this was required to reverse the effect of the reversed transactions. When reversed transactions are in disapproved blocks, all operations can more or less be made to be “no ops” using an Operation.Status that is not successful. Because not successful operations do not affect balance, you can use the native value representation (instead of the opposite).

Hello Patrick,

Thank you for the detailed feedback!

I agree that using abstract identifiers is a bad practice in general and should be avoided if other solutions are possible.

However, I’m still unsure clients will behave in a safe way when using the “trailing solution” for Decred’s disapproved blocks problem. Let me try to provide a more concrete example of where I see the problem:

Let’s say you have the underlying blockchain in the following situation:

b1 <- b2 (approves b1) <- b3 (approves b2)

So, using the trailing solution, a dcrros client is now considers b2 is the tip block and assumes it has processed all transactions contained in b1, and all of those have status == success.

We now reorg to the following longer chain:

b1 <- b2 <- b3
   \- b2a (disapproves b1) <- b3a (approves b2) <- b4 (approves b3)

If we store the individual transaction disapproval tx status in b1, then once reorged to b4, the status of transactions in b1 itself have changed to status == reversed. This means the client would need to reprocess b1, which was not part of the reorg itself (the reorg only happened starting at b2).

Given that the only new blocks seen by a client during a reorg were b2a, b3a and b4, how would the client know it needs to request b1 again (because the transactions in b1 have now been reversed)?

I meant to write more about this but ended up skipping it, but my main concern can be summarized as:

Should individual blocks returned by the data api be inalterable?

In other words, should /block requests be idempotent (for a given block hash)? After seeing block b, can clients be sure they’ll never have to request block b again (because its contents won’t ever change)?

I’ve been operating under that assumption, so rewriting the status on block b1 seemed to me to not be an option.

Should individual blocks returned by the Data API be inalterable ?

You bring up a VERY good question. Yes, any block data returned at a particular BlockIdentifier is ABSOLUTELY considered inalterable. If you want to put up a PR in the rosetta-specifications elaborating on this point, I’d be happy to approve it!

Possible Paths Forward

[1] Configurable trailing depth

When writing my original response, I considered elaborating on this side effect but opted not to for some reason (don’t remember why, maybe before I had my coffee :upside_down_face:).

It would be possible to allow anyone running your implementation to provide a configurable trail-depth parameter behind some safe reorg depth (it looks like past 6 blocks is considered pretty unlikely according to your docs) where they should never observe inalterable data.

There are some annoying repercussions with this approach:

  1. If a user does not think carefully about this trail-depth parameter, they may create some pretty bad data inaccuracies in their systems (as this inalterable assertion will be violated).
  2. Having to trail tip by X blocks will delay people from building “realtime” applications on top of your implementation (likely people will trail by at least 6 blocks to be safe).

[2] Change the model of transaction execution to be in n + 1 instead of n

Another approach (although a little more out there, I admit) is to consider a transaction only executed in block n+1. This would mean that all transactions would only appear if they weren’t reversed and that you would never represent reversed transactions.

This approach has some very annoying side effects:

  1. All your code must be altered to reflect this off by 1 offset (like /account/balance responses). This may also make it much more confusing to access/spend UTXOs.
  2. On another note, this is also not how transaction execution is canonically understood in Decred (executed then reversed as opposed to delayed execution).

[3] Use block_hash:tx_hash for all reversed transactions where block_hash is the disapproved block

Unlike your suggestion in your initial message, I propose using the block_hash of the disapproved block in any reversed transactions. I also suggest including the “unmodified” transaction hash in Transaction.Metadata.

Solution 1 doesn’t handle the case where x == 1 given that you’d still have two transactions with the same hash on the same block

This approach avoids the issue you described earlier for transactions that show up in block n+1 and avoids some of the annoying side effects discussed in the other solutions (with the great exception being that these transaction hashes do not correspond to the underlying blockchain’s transaction identifier). I think this path is particularly interesting because it most accurately reflects that block 2a is the cause of the “reversal” operations (as it includes the disapproving votes) not the block that was actually disapproved.

Example

I wrote up this example to clearly explain this approach. I think it covers the edge cases you’ve previously called out but welcome feedback if you can think of an exception!

b1 <- b2 <- b3
   \- b2a (disapproves b1) <- b3a (approves b2) <- b4 (approves b3)

b1 contents

tx1
tx2

b2 contents

tx3
tx4

b2a contents

tx2 (transaction originally included in disapproved block included again here, no uniqueness conflict)
tx3 (transaction originally from b2)
tx5
b1:tx1 metadata: {"original_tx_hash":tx1}
b1:tx2 metadata: {"original_tx_hash":tx2}

[4] Virtual Re-org for Disapproved Blocks

Another concept I was playing around with when examining this dilemma was using a “virtual re-org” model. In this approach, a disapproved block would have an “abstract” identifier of block_hash:disapproved and the block that did the disapproving would point to this abstract hash (containing no-op reversed transactions) instead of the actual parent block hash. As a side effect, disapproved transactions would not need to be injected into the disapproving block. Clients would treat this scenario like a normal re-org and “disapproval” would be abstracted away.

Using “abstract” identifiers for transactions is one thing (see approach 3), however, I believe using “abstract” identifiers for blocks is a more substantial leap (in the wrong direction). Modifying block identifiers to give the impression of a re-organization that did not occur feels like too significant of a departure from what actually occurred on-chain (regardless of whether it would pass rosetta-cli check).

Example
b0 <- b1 <- b2 <- b3
       \- b1d <- b2a (disapproves b1) <- b3a (approves b2) <- b4 (approves b3)

Final Thoughts

All this being said (and given the counterpoint you brought up in your last message), I think approach 3 (use block_hash:tx_hash for all reversed transactions where block_hash is the disapproved block) seems like the simplest/best approach here (this is why I didn’t really want to unilaterally say that “abstract hashes were bad” in my earlier message).

Given the tradeoffs of the other possible approaches, I think a lightweight “abstract” identifier for “reversed” transactions is the best modeling strategy. This is a pretty simple change to your current implementation and doesn’t introduce any unnecessarily complex mechanisms for the happy path (and the additions for the happy path are simple and well understood).

This is indeed annoying and while for mainnet it’s reasonably safe to assume a 6-block reorg won’t occur (in fact, IIRC the longest reorg we had was either 2 or 3) that’s not a consensus rule but rather a policy that stakeholders (PoS participants) usually uphold. We do occasionally see large reorgs on testnet for example.

This also has all the other downsides you noted.

In the past this has led to bugs, even on consensus code (see here for a writeup on one such occasion) so I’d be weary of using this option.

Agree completely that rewriting the chain to look like an “extra” block happened between the disapproved and the disapproving block goes in the wrong direction. Crucially, it also breaks the correspondence between a BlockIdentifier.index and the block height in an unrecoverable way.

Seems like a reasonable compromise. My only concern (apart from the one you noted in the previous message) is that this means there are now two different “styles” for the transaction identifier: one that prepends a block hash and one that doesn’t, which means clients must be aware of this if they intend to deal with the identifiers in a non-stringy fashion (that is, if they’ll try and convert the identifiers to the native, raw, byte-based types).

Any thoughts on this? Am I being too defensive here?

In the past this has led to bugs, even on consensus code (see here for a writeup on one such occasion) so I’d be weary of using this option.

Thanks for sharing this! Great context!

Seems like a reasonable compromise. My only concern (apart from the one you noted in the previous message) is that this means there are now two different “styles” for the transaction identifier: one that prepends a block hash and one that doesn’t, which means clients must be aware of this if they intend to deal with the identifiers in a non-stringy fashion (that is, if they’ll try and convert the identifiers to the native, raw, byte-based types).

In Rosetta, Identifiers are considered “opaque” blobs (realizing now that we don’t really elaborate on this in the docs). As long as the Rosetta implementation is consistent with its usage of Identifiers and automatically handles any unwrapping that may need to occur because of the use of “abstract” identifiers, Rosetta clients and automated testing should just work.

To be honest, I’m not too concerned with people trying to convert these “abstract” transaction hashes to the native, raw, byte-based types as people doing these Decred-specific things would probably just be using the node directly, not Rosetta (which was created to remove the need to do these things to begin with). I’m more concerned with people attempting to take the transaction hash they see on a “reversed” transaction and using it to perform a lookup in a Decred block explorer (that isn’t Rosetta-based). This is why I think including the “raw” transaction hash in the Transaction.Metadata is a good idea (so no introspection of the TransactionIdentifier must be performed).

Any thoughts on this? Am I being too defensive here?

@matheusd do you have an example of where you think it would be important to perform these native type conversions or were you just postulating?

Just postulating since we usually use native types in our code.

I guess the overall direction to go is using the abstract identifiers on reversed transactions then. This should be an easy change to perform.

Thank you for the detailed discussion!

Appreciate the back and forth here. I’m sure many others will find this dialogue useful!