Construction API will not collect UTXOs, the UTXOs should be passed in by the caller of the Construction API. For normal transfer, the caller puts UTXO infors in the Operation.CoinChange field as inputs, in /construction/payloads, generate outputs according to Operations and finally produces transaction. Is my understanding correct? @patrick.ogrady
suppose Alice wants to transfer 100 BTC to Bob, is the transaction generation process like this?
step 1: call /construction/preprocess to generate options which use to collect UTXO set.
step 2: call /construction/metadata use preprocess generated options to fetch UTXO set. through the UTXO set indexer and put these UTXO into metadata.
step 3: call /construction/payloads use UTXO set from metadata to create unsigned transaction.
step 4: call /construction/combine/ to create signed transaction.
step 5: call /construction/submit/ to broadcast Signed Transaction.
hi, @alan.verbner I noticed that you may also be developing a rosetta implementation of UTXO mode. Can we talk about how you assembled the transaction? Can you help me see if my above ideas are suitable?
I think we are doing what you have just mentioned:
We haven’t implemented /construction/preprocess as we have nothing to do here
We invoke /construction/metadata which needs to calculate a value based on the tip’s block number
/construction/payloadscreates a set of operations that will be transformed into the actual transaction. To do so we use /account/balance to retrieve unspents
Then the workflow is as you mentioned.
We have this example in our repo (still private, sorry) but might be easier to follow than the steps above:
It’s important to mention that you can use rosetta-cli check:construction` that implements almost the same steps as the ones I shared and probably is better to stick to it.
The UTXO set will be explicitly provided to /construction/preprocess in the form of Operation.CoinChange. You should not determine which UTXOs to use in your Rosetta implementation.
This call should only ever fetch information about the UTXOs provided to /construction/preprocess. It should not attempt to fetch other UTXOs that weren’t specified in the operations provided to /construction/preprocess.
These are correct!
A few things:
/construction/metadata should never take caller-provided metadata. The only place that the caller can specify “metadata” is in the call to /construction/preprocess.
/construction/payloads will be run in an offline environment and will not have access to /account/balance. All UTXOs to be used in the transaction must be provided in /construction/preprocess as Operation.CoinChange.
I recommend avoiding creating your own tool to perform the checks in check:construction because you may assume a different flow than what is expected.
I don’t get what’s the difference between getting the utxo from the account balance or getting it from the block changes. I mean, let’s suppose that invocation will happen sometime before starting the transaction process.
Ahh gotcha. Just make sure this is provided to /construction/preprocess not /construction/metadata by the caller. In you example above, it appears that this is provided in /construction/metadata.
You are correct that either approach will work for testing. I thought you meant your /construction/payloads implementation called /account/balance.
We use block data because it serves as a nice consistency check that the CoinIdentifiers showing up in blocks can be used to specify correct transactions. In our testing tool update, we will allow for providing preloaded addresses for CI testing and their coins will be fetched by /account/balance, for context.
I’m not sure I follow. It’s provided to /construction/metadata as it need online information (best block slot number is required to calculate the ttl)
I have one question:
Can only part of the input’s CoinChange be consumed in the actual transaction? e.g., input needs 500 CKB, but the sum of CoinChange is 1000 CKB. The transaction fee in ckb is related to the transaction’s size. The reasonable transaction fee cannot be known before the transaction is constructed, so the transaction fee cannot be provided in advance by an operation. Therefore, it may be necessary to provide moreCoinChange. @patrick.ogrady
In other words, before the transaction is constructed, I don’t know what the reasonable fee is and how much the change output is. How can this be expressed in the intent?
As far as I undestand fee is the inputs - outputs operation values.
Re: calculating the fee based on transaction size, we haven’t implemented that yet. I would say that basically you need to provide tx size or a way to get transaction size from /construction/metadata endpoint as it returns it as suggested fee. To do so, as /construction/preprocess is the one that receives the operations, there is where the magic should happen. I’m not quite sure where to return an error in case the fee is insufficient.
the fee is indeed equal to inputs - outputs. When we generate operations, we need to put all the UTXO used to construct the transaction into Operation.CoinChange, which means that the caller must calculate the fee before calling the construction API. However, the fee in CKB needs to be calculated based on the transaction size and the transaction size is not known before the transaction is constructed. Therefore, the input part of the fee cannot be put into Operation.CoinChange and There is no way to get transaction size through /construction/metadata.
The solution I currently think of is that the operations when requesting the /construction/preprocess endpoint for the first time only fill in the transfer intent, without considering the fee and then when requesting the /construction/metadata endpoint, construct an unsigned tx based on the intent, then calculate the size and give the suggested fee in response, and then caller adjusts the intent and request /construction/preprocess again until the cost is right.
Well, isn’t that iteration what users usually need to do even if they don’t use Rosetta?
I think that might break check:construction workflow. Also, it seems that there are several ways to tackle this implementation so I think it’s better to come up with something we all follow. @patrick.ogrady what do you think? how should we approach this?
Thanks for raising this question, @kingstone. I don’t think there is much guidance on the website of how to approach fee suggestion in UTXO-based blockchains (where the caller may need to change the “intent” after learning the suggested_fee). For other readers, Account-based blockchain fee estimation is a little more straightforward as the caller makes a go/no-go decision based on the suggested_fee but typically doesn’t need to alter the intent).
Rosetta-Bitcoin
I think the best way to explain our thinking here is to discuss a concrete example, rosetta-bitcoin (hopefully released soon ).
Caller Flow
Create “intent” (note that we set the value of our “change” OUTPUT to be of value 1) here before we know the suggested_fee. This ensures we get an accurate size estimate (which is usually a function of the number of inputs, outputs, and their address types). Note: We must suggest a collection of inputs with a value >= sum(OUTPUT) + max_acceptable_fee.
Call /construction/metadata -> returns suggested_fee (for this example we assume this is 500)
If the suggested_fee > max_acceptable_fee, we must abort the construction process with these INPUT and adjust our max_acceptable_fee parameter.
Update “intent” (setting the “change” OUTPUT to be sum(INPUT) - sum(OUTPUT) - suggested_fee) Note: there is minor network-specific understanding required here to know how to adjust the change output.
Our implementation only supports sending from “Native SegWit” addresses, so the input calculation is pretty straightforward. Estimating the output size is where things are a little more complicated. Fortunately, there are some existing packages that help with this (like https://godoc.org/github.com/btcsuite/btcd/txscript).
Fee Estimation
We return the estimated size in ConstructionPreprocessResponse.Options and calculate the suggested fee in /construction/metadata using the response from the estimatesmartfee method in bitcoin-core (returned in BTC/vKB). Unless fees are extremely volatile, we only expect to do perform this “dry run” phase once. If the suggested_fee is greater than max_acceptable_fee, we have to start over with new coins.
Miscellaneous Thoughts
In the example you’ve shared above, I assume you mean that the coin_spent associated with the INPUT has 1000 CKB on it. Typically, I would represent this as:
A Coin should always be spent in its entirety and the fee is inferred as sum(INPUT)-sum(OUTPUT) (this doesn’t break reconciliation as the coinbase contains the block reward + fees). I elaborate a few times on how to we provide CoinChange in this response, however, I’ll reiterate here as well. The gist of it is we select coins where sum(INPUT) >= sum(OUTPUT) + max_acceptable_fee and then set the CoinChange using the suggested_fee returned by /construction/metadata where CoinChange=sum(INPUT)-sum(OUTPUT)-suggested_fee.
As I explained in the rosetta-bitcoin case, we provide an intent with the inputs and outputs we wish to create plus a change output of 1. We then amend this intent based on the value of suggested_fee returned by /construction/metadata (replacing this change output amount with sum(INPUT)-sum(OUTPUT)-suggested_fee.
I highly encourage defensive programming here. If you are able to determine that a transaction would not land on-chain, you should throw an error to prevent the caller from silently broadcasting an “under-fee’d” transaction (which could get stuck in the mempool on some blockchains). For example, we return an error in the /construction/preprocess method in rosetta-bitcoin if the inferred fee given the estimated size would yield a fee rate lower than the minimum allowed.
The recently released Construction API testing framework (in rosetta-cli >= v0.5.0) has support for a lot more complex flows. I’ve been testing it in tandem with our rosetta-bitcoin implementation. Coincidentally, I’m adding built-in support for the “dry run” flow I described above this week. I’m adding support for “fund return” (suggested by @alan.verbner) next week!
Progress has been a little delayed because I’ve been bogged down by some performance improvements I’ve been making to improve check:data.
This is usually only ever at most 2 calls for us (create intent with value 1 change, re-run intent with populated change). With a simple wallet package (hoping someone steps up to help on this soon!), this would be hidden from the user and probably add milliseconds, at most, to the construction process (we aren’t doing any signing or parsing during this “dry run” phase).
Hey, is this still relevant for Rosetta-Bitcoin? I’m struggling submitting a transaction, and was curious if there was a complete walkthrough anywhere for this?
For step 1, calling /construction/preprocess, what should be provided as operations? Should it just be the current transaction (Alice sends Bob 1 BTC)? Or should we query data from /account/balance, then make new operation objects to pass up from that response, with the current transaction as the last in the array?
For step 2, when calling /construction/metadata using the options param from the preprocess response, I’m getting the error “coin amount does not match expected with difference 223”, with the message “Missing ScriptPubKeys”, though I’m sure it has something to do with my incorrect preprocess request.
You can run the automated Construction API tests for rosetta-bitcoin by following the instructions here:
This will print out all request/responses to/from rosetta-bitcoin during transaction construction using the bitcoin Rosetta Constructor DSL config:
Example Request/Responses
To make it easier to follow (so you don’t need to sync to tip), I’ve done this for you and pasted the logs for this tx below (including both dry run + final run):