Implementing the Construction API for UTXO-model coins

In blockchains like Bitcoin, the values of the transaction inputs are not given in the transaction itself. Conversely, the Rosetta transaction type specifies the value (and associated address) of both inputs and outputs. This means that, in order to convert to the Rosetta format, implementations need to look up the value of each input in their UTXO set.

In the Construction API, however, the server is running in an offline context, so it doesn’t have access to the UTXO set. Instead, this information comes from the /account/balance endpoint, which includes (as metadata) the outpoint and value of each UTXO controlled by the address. The client can then include this information (as metadata) along with the Operations in its subsequent /construction/payloads request.

The values are also needed in /construction/parse, which converts a “native” transaction to a Rosetta transaction. Unfortunately, /construction/parse does not have a metadata field, so the client cannot provide the input values. This means that /construction/parse would have to run in an online context, with access to the UTXO set. Is that correct? Or is the intent for /construction/parse to run offline as well?

1 Like

Another question, relating to fees:

For blocks that have already been mined, it’s easy enough to represent the fee as an Operation that credits the address of the miner. But what about transactions being constructed by the client? The client represents their desired transaction as a list of Operations, but they can’t represent the miner fee as an Operation, because they don’t know which address will receive the fee. Should the fee be specified in a metadata field instead?

1 Like

The Rosetta Construction API purposely does not handle coin selection. The caller must provide the UTXOs they wish to spend in the Operation.CoinChange field when communicating with the Construction API. This means the actual transaction construction from the provided UTXOs can be done offline. In our Bitcoin implementation, we use the /construction/metadata call to fetch the ScriptPubKey for each UTXO we are planning to spend. It would also be possible to assert each of the UTXOs we are trying to spend is still unspent.

The best way to get around this is for /construction/payloads to return additional metadata about the transaction which you may need later on for combining & parsing and wrapping the raw unsigned/signed transaction with this metadata. For example, in the UnsignedTransaction field of ConstructionPayloadsResponse, you can return a “rich” Sia transaction, which has additional info you need in /construction/combine.

/construction/parse must run offline. Check out this list of endpoints that must run offline: https://www.rosetta-api.org/docs/node_deployment.html#offline-mode-endpoints. The solution here is to wrap the “native” transactions with extra information in Rosetta, to include all the metadata you need to parse it later. For example, your “rich” Sia-Rosetta unsigned transaction can include information about the UTXOs being spent, as well as all the info in a “native” transaction (as mentioned above).

There is no expectation that the transactions which are constructed in Rosetta can be parsed by network-specific tools or broadcast on a non-Rosetta node. All parsing and broadcast of these transactions will occur exclusively over the Rosetta API.

The operations that are passed into the construction API do not need to be fully specified, but instead only capture the “intent” of the transaction. The intent is always a strict subset of the actual transaction operations that happen on-chain, because there may be other operations that happen on-chain due to your transaction, which you don’t know at tx construction time.

For example, if you are creating a Uniswap transaction, the “intent” may be captured by operations which represent a transfer of x tokens from your account to the Uniswap contract. However, when the tx happens on-chain, the account gets back some number of tokens (you don’t know this number beforehand), and when the Data API reads this transaction it should create these additional operations on top of the original “intent” operations.

To follow-up on @juliankoh’s point, here is an example of a Bitcoin “intent” we would provide to the Construction API:

[
  {
    "operation_identifier": {
      "index": 0
    },
    "type": "Vin",
    "status": "",
    "account": {
      "address": "{{ SENDER }}"
    },
    "amount": {
      "value": "{{ SENDER_VALUE/UTXO_VALUE }}",
      "currency": {
        "symbol": "BTC",
        "decimals": 8
      }
    },
    "coin_change": {
      "coin_action": "coin_spent",
      "coin_identifier": {
        "identifier": "{{ UTXO_IDENTIFIER }}"
      }
    }
  },
  {
    "operation_identifier": {
      "index": 1
    },
    "type": "Vout",
    "status": "",
    "account": {
      "address": "{{ RECIPIENT }}"
    },
    "amount": {
      "value": "{{ RECIPIENT_VALUE }}",
      "currency": {
        "symbol": "BTC",
        "decimals": 8
      }
    }
  }
]

This was taken from the Bitcoin configuration file we use in automated Construction API testing.

/construction/parse must run offline… The solution here is to wrap the “native” transactions with extra information in Rosetta, to include all the metadata you need to parse it later.

There is no expectation that the transactions which are constructed in Rosetta can be parsed by network-specific tools or broadcast on a non-Rosetta node.

I was confused by this at first, but I think I understand now. Mosts requests have a Metadata field, but /construction/parse does not. So the only way to add metadata is to add it to the encoded transaction, i.e. the string returned by /construction/payloads and /construction/combine. I don’t quite see why this is necessary though; couldn’t /construction/parse have a Metadata field?

Some follow-up questions:

  • I notice that types.Coin does not include timelock information. How should I provide the timelock to clients of /account/balance ? Should I populate a separate timelocks map in the metadata?
  • If I’m populating CoinChange for my Operations, should I not populate Account ?
  • Relatedly, when a coin is spent, should the Amount of the operation be negative?
  • Should the response from /construction/parse specify the CoinIdentifier for each UTXO that was created? The response is supposed to match the operations specified by the client, and the client probably shouldn’t need to compute these identifiers, so I’m guessing the identifiers should be left blank?

This is a GREAT question @lukechampine!

We very purposely avoided the use of Metadata for requests to /construction/combine, /construction/parse, /construction/hash, and /construction/submit. Unlike the Data API endpoints and other Construction API, these endpoints endpoints operate on a very network-specific understanding of data (using encoded payloads instead of some observable “Rosetta-based type”). We did not think it was a good idea to make it possible to change how these endpoints treat this opaque data based on the population of some Metadata field or that the output of these endpoints could be somehow effected by accidentally manipulating this Metadata (we thought it would be much harder to accidentally manipulate an encoded string).

You are in no way incorrect that we could do less “wrapping” of these payloads and instead use a Metadata field, but made this decision based on these design opinions. Happy to continue discussing and invite you take a look at the rosetta-cli for an example of what it looks like to walk through the construction flow:

I notice that types.Coin does not include timelock information. How should I provide the timelock to clients of /account/balance ? Should I populate a separate timelocks map in the metadata?

This would be a great PR/issue to open against rosetta-specifications. I think a metadata field for types.Coin makes a lot of sense! Here are some PRs folks have opened (I think this would be the first to modify the spec data itself!):


If I’m populating CoinChange for my Operations, should I not populate Account ?

You must still populate Account. Regardless of whether your implementation uses “UTXO-based” accounting or “Account-based” accounting, we will still run “address reconciliation” using the address provided in the AccountIdentifier. If you do not populate Account, this process will fail.

We do not currently perform any reconciliation between what is returned in the Operation.CoinChange field and what is returned in the AccountBalanceResponse.Coins field. This is something we hope to add but don’t view it as a replacement for address-based balance tracking (more as a supplement). That being said, this reconciliation occurs implicitly when running automated Construction API testing. If Operation.CoinChange is populated incorrectly, we likely won’t be able to get any transactions to land on-chain.

Relatedly, when a coin is spent, should the Amount of the operation be negative?

As mentioned above, regardless of whether your implementation uses “UTXO-based” accounting or “Account-based” accounting, we will still run “address reconciliation”. If balance decreases are not represented with negative values, this process will fail.

To be specific, we do not interpret a negative Amount coin spend as a coin creation (although, I could see how that conclusion could be drawn).

Should the response from /construction/parse specify the CoinIdentifier for each UTXO that was created? The response is supposed to match the operations specified by the client, and the client probably shouldn’t need to compute these identifiers, so I’m guessing the identifiers should be left blank?

At construction time (when we pass Operations to /construction/preprocess or /construction/payloads), we don’t expect to know the CoinIdentifier of any created Coins. In these Operations, CoinChange will not be populated for any Operations that are expected to create Coins. However, you are encouraged to populate it on the response from /construction/parse if it is possible to compute (I believe it should be).

When verifying “intent” (AKA do Operations passed at construction math Operations returned in /construction/parse), we check that the information initially provided is some subset of what is returned. So, /construction/parse can return information not known during construction. In the case of ETH, for example, we return an additional Operation to represent the maximum possible fee payment in the /construction/parse response.

Hi !

We have the same question 1 about fees and I’m not sure I got the answer from the ones posted above. How is the fee going to be received when /construction/payloads is invoked?

We are actively working on better ways to surface the suggested fee (based on provided intent and network conditions) and to provide a specific fee in the Construction API. We think of these as 2 different problems.

Surfacing Suggested Fee

We have discussed adding a SuggestedFee field to ConstructionMetadataResponse that would allow the caller to adjust the intent (i.e. alter the amount to send to a change address)/abort the transaction based on too high of a suggested fee.

We think it is important to avoid is some network-specific understanding of fee (as this would derail a lot of the underway automation efforts), so this would be returned as an Amount (any network-specific fee information would be returned in the existing Metadata field for use in /construction/payloads, e.g. gas_limit or gas_price). For context, most account-based Rosetta implementations already determine the suggested fee automatically and pass it to /construction/payloads as Metadata but don’t expose it to the caller in a Rosetta type.

We are planning to put something up in the next day or so on this point. If you have any strong preference here, would :heart: to see an issue or PR on rosetta-specifications.

Specifying Custom Fee

UTXO-based Blockchains

In UTXO-based chains, the caller provides the fee (as I’m sure you are well aware) by setting the difference between inputs and outputs. Rosetta implementations should not expect any other fee parameter in calls to /construction/payload. Based on the result of the SuggestedFee field (if added to the ConstructionMetadataResponse), the change amount could be altered or new UTXOs could be selected prior to calling /construction/payload (the flow would start over if this was done, AKA /construction/preprocess and /construction/metadata would be called again with any modified intent to make sure the new SuggestedFee didn’t change dramatically).

Account-based Blockchains

In account-based chains, the fee (along with other fee-related params like gas_limit or gas_price) is determined in the call to /construction/metadata and passed to /construction/payloads in ConstructionPayloadsRequest.Metadata. It is much tougher, in our opinion, to provide an abstraction for altering some of these fee-related parameters. It is likely that using anything other than the SuggestedFee would require network-specific modification of the ConstructionMetadataResponse.Metadata object.

In UTXO-based chains, the caller provides the fee (as I’m sure you are well aware) by setting the difference between inputs and outputs.

Nitpick: This is how fees are handled in Bitcoin and many other UTXO chains, but is not inherent to the UTXO model. Sia transactions specify their fees explicitly.

Given that the miner’s address is not known until the block is mined, how should transactions returned from the Mempool service represent transaction fees? Is it acceptable for fees in mempool transactions to not have an associated Operation?

Could you elaborate on how this looks in Sia, @lukechampine? Does it look like an extra fee op type with a value equal to sum(inputs) - sum(outputs) or is it different abstraction?

We don’t do much validation on mempool transactions because we assume they are “best effort” representations of what could occur on-chain. Our primary concern is observing what transaction hashes are in the mempool. I wrote about this a little here:

In terms of representing fees, do you mean representing fees without a related_operation? If so, that is fine! A word of warning, assertion will fail if you populate an Operation with an Amount but no AccountIdentifier.

Could you elaborate on how this looks in Sia

In addition to inputs and outputs, Sia transactions have a MinerFees field that contains bare Amounts. To be considered valid, the transactions inputs must equal its outputs plus its MinerFees.

(To be clear, I don’t think this changes anything wrt Rosetta – it just means you can fetch fee amounts directly instead of computing inputs minus outputs.)

In terms of representing fees, do you mean representing fees without a related_operation ?

Sorry, I meant representing a fee as a Metadata field rather than as an Operation. AFAICT it would not be possible to represent a fee as an Operation in the transactions reported by the mempool, for aforementioned reasons. They can be represented as Operations in the actual block once it’s been mined, though.

Given that you’re treating mempool transactions as “best effort,” I think my mempool implementation will just omit fees for now. Thanks!

Alrighty! And, will the input be fully populated, i.e., operations will have the (optional) amount field or do we need to populate it from the DB fetching using coin_change.coin_identifier.identifier?

We have the same model and I agree it doesn’t require a change on Rosetta but that’s why we asked on the first place.

Got it! That’s what I figured. I’m assuming the coinbase transaction doesn’t include any miner fees then (if this is the case, all makes sense to me)? If not, I’m wondering how you avoid double counting fees (showing in each transaction and in the block reward)?

If you do want to expose this fee info in a more “network-specific form”, you could use the string {MINER} as the recipient of the fee operation in the mempool (although I admit this isn’t great). Your conclusion about not including fee ops here also makes sense for now. We can revisit this if we end up iterating on the mempool endpoints.

The Amount will be provided! As mentioned earlier (around "verifying where possible), you may want to check that it matches what you have stored in your DB.

1 Like