Construction API will not collect UTXOs, the UTXOs should be passed in by the caller of the Construction API. For normal transfer, the caller puts UTXO infors in the Operation.CoinChange field as inputs, in /construction/payloads, generate outputs according to Operations and finally produces transaction. Is my understanding correct? @patrick.ogrady
reference:
suppose Alice wants to transfer 100 BTC to Bob, is the transaction generation process like this?
step 1: call /construction/preprocess to generate options which use to collect UTXO set.
step 2: call /construction/metadata use preprocess generated options to fetch UTXO set. through the UTXO set indexer and put these UTXO into metadata.
step 3: call /construction/payloads use UTXO set from metadata to create unsigned transaction.
step 4: call /construction/combine/ to create signed transaction.
step 5: call /construction/submit/ to broadcast Signed Transaction.
hi, @alan.verbner I noticed that you may also be developing a rosetta implementation of UTXO mode. Can we talk about how you assembled the transaction? Can you help me see if my above ideas are suitable?
Hi @kingstone,
I think we are doing what you have just mentioned:
/construction/preprocess as we have nothing to do here/construction/metadata which needs to calculate a value based on the tip’s block number/construction/payloadscreates a set of operations that will be transformed into the actual transaction. To do so we use /account/balance to retrieve unspentsWe have this example in our repo (still private, sorry) but might be easier to follow than the steps above:
const doRun = async (): Promise<void> => {
const keys = generateKeys(
'41d9523b87b9bd89a4d07c9b957ae68a7472d8145d7956a692df1a8ad91957a2c117d9dd874447f47306f50a650f1e08bf4bec2cfcb2af91660f23f2db912977'
);
logger.info(`[doRun] secretKey ${Buffer.from(keys.secretKey).toString('hex')}`);
const address = await constructionDerive(Buffer.from(keys.publicKey).toString('hex'));
const unspents = await accountBalance(address);
const metadata = await constructionMetadata(1000);
const operations = buildOperation(unspents, metadata, DESTINATION);
const payloads = await constructionPayloads(operations);
// TODO: PARSE
const signatures = signPayloads(payloads.payloads, keys);
const combined = await constructionCombine(payloads.unsigned_transaction, signatures);
logger.info(`[doRun] signed transaction is ${combined.signed_transaction}`);
// TODO: PARSE
const hashResponse = await constructionSubmit(combined.signed_transaction);
logger.info(`[doRun] transaction with hash ${hashResponse.transaction_identifier.hash} sent`);
};
It’s important to mention that you can use rosetta-cli check:construction` that implements almost the same steps as the ones I shared and probably is better to stick to it.
great help, thank you. 
The UTXO set will be explicitly provided to /construction/preprocess in the form of Operation.CoinChange. You should not determine which UTXOs to use in your Rosetta implementation.
This call should only ever fetch information about the UTXOs provided to /construction/preprocess. It should not attempt to fetch other UTXOs that weren’t specified in the operations provided to /construction/preprocess.
These are correct!
A few things:
/construction/metadata should never take caller-provided metadata. The only place that the caller can specify “metadata” is in the call to /construction/preprocess./construction/payloads will be run in an offline environment and will not have access to /account/balance. All UTXOs to be used in the transaction must be provided in /construction/preprocess as Operation.CoinChange.
I recommend avoiding creating your own tool to perform the checks in check:construction because you may assume a different flow than what is expected.
Ahh gotcha. Just make sure this is provided to /construction/preprocess not /construction/metadata by the caller. In you example above, it appears that this is provided in /construction/metadata.
You are correct that either approach will work for testing. I thought you meant your /construction/payloads implementation called /account/balance. 
We use block data because it serves as a nice consistency check that the CoinIdentifiers showing up in blocks can be used to specify correct transactions. In our testing tool update, we will allow for providing preloaded addresses for CI testing and their coins will be fetched by /account/balance, for context.
I’m not sure I follow. It’s provided to /construction/metadata as it need online information (best block slot number is required to calculate the ttl)
Take Alice transferring 500 CKB to Bob as an example.
[
{
"operation_identifier":{
"index":0
},
"type":"INPUT",
"account":{
"address": "AliceAddress"
},
"amount":{
"value":"-600",
"currency":{
"symbol": "CKB",
"decimal": 18
}
},
"coin_change":{
"coin_action":"coin_spent",
"coin_identifier":{}
}
},
{
"operation_identifier":{
"index":1
},
"type":"OUTPUT",
"account":{"address":"BobAddress"},
"amount":{
"value":"500",
"currency":{
"symbol": "CKB",
"decimal": 18
}
}
},
{
"operation_identifier": {
"index": 2
},
"type": "OUTPUT",
"account": {
"address": "AliceAddress"
},
"amount": {
"value": "100",
"currency": {
"symbol": "CKB",
"decimal": 18
}
}
}
]
I have one question:
Can only part of the input’s CoinChange be consumed in the actual transaction? e.g., input needs 500 CKB, but the sum of CoinChange is 1000 CKB. The transaction fee in ckb is related to the transaction’s size. The reasonable transaction fee cannot be known before the transaction is constructed, so the transaction fee cannot be provided in advance by an operation. Therefore, it may be necessary to provide moreCoinChange. @patrick.ogrady
In other words, before the transaction is constructed, I don’t know what the reasonable fee is and how much the change output is. How can this be expressed in the intent?
As far as I undestand fee is the inputs - outputs operation values.
Re: calculating the fee based on transaction size, we haven’t implemented that yet. I would say that basically you need to provide tx size or a way to get transaction size from /construction/metadata endpoint as it returns it as suggested fee. To do so, as /construction/preprocess is the one that receives the operations, there is where the magic should happen. I’m not quite sure where to return an error in case the fee is insufficient.
I think @patrick.ogrady is the one to answer that question.
thank you for your reply 
the fee is indeed equal to inputs - outputs. When we generate operations, we need to put all the UTXO used to construct the transaction into Operation.CoinChange, which means that the caller must calculate the fee before calling the construction API. However, the fee in CKB needs to be calculated based on the transaction size and the transaction size is not known before the transaction is constructed. Therefore, the input part of the fee cannot be put into Operation.CoinChange and There is no way to get transaction size through /construction/metadata.
The solution I currently think of is that the operations when requesting the /construction/preprocess endpoint for the first time only fill in the transfer intent, without considering the fee and then when requesting the /construction/metadata endpoint, construct an unsigned tx based on the intent, then calculate the size and give the suggested fee in response, and then caller adjusts the intent and request /construction/preprocess again until the cost is right.
Well, isn’t that iteration what users usually need to do even if they don’t use Rosetta?
I think that might break check:construction workflow. Also, it seems that there are several ways to tackle this implementation so I think it’s better to come up with something we all follow. @patrick.ogrady what do you think? how should we approach this?
Well, isn’t that iteration what users usually need to do even if they don’t use Rosetta?
Yes, But if the user does not use Rosetta, he can do it in one method call, and it may take several times when using Rosetta.
Thanks for raising this question, @kingstone. I don’t think there is much guidance on the website of how to approach fee suggestion in UTXO-based blockchains (where the caller may need to change the “intent” after learning the suggested_fee). For other readers, Account-based blockchain fee estimation is a little more straightforward as the caller makes a go/no-go decision based on the suggested_fee but typically doesn’t need to alter the intent).
I think the best way to explain our thinking here is to discuss a concrete example, rosetta-bitcoin (hopefully released soon ).
OUTPUT to be of value 1) here before we know the suggested_fee. This ensures we get an accurate size estimate (which is usually a function of the number of inputs, outputs, and their address types). Note: We must suggest a collection of inputs with a value >= sum(OUTPUT) + max_acceptable_fee.
[
{
"operation_identifier": {
"index": 0
},
"type": "INPUT",
"account": {
"address": "A"
},
"amount": {
"value": "-2000",
"currency": {
"symbol": "BTC",
"decimal": 8
}
}
},
{
"operation_identifier": {
"index": 1
},
"type": "OUTPUT",
"account": {
"address": "B"
},
"amount": {
"value": "1000",
"currency": {
"symbol": "BTC",
"decimal": 8
}
}
},
{
"operation_identifier": {
"index": 2
},
"type": "OUTPUT",
"account": {
"address": "A"
},
"amount": {
"value": "1",
"currency": {
"symbol": "BTC",
"decimal": 8
}
}
}
]
/construction/preprocess
/construction/metadata -> returns suggested_fee (for this example we assume this is 500)suggested_fee > max_acceptable_fee, we must abort the construction process with these INPUT and adjust our max_acceptable_fee parameter.OUTPUT to be sum(INPUT) - sum(OUTPUT) - suggested_fee) Note: there is minor network-specific understanding required here to know how to adjust the change output.
[
{
"operation_identifier": {
"index": 0
},
"type": "INPUT",
"account": {
"address": "A"
},
"amount": {
"value": "-2000",
"currency": {
"symbol": "BTC",
"decimal": 8
}
}
},
{
"operation_identifier": {
"index": 1
},
"type": "OUTPUT",
"account": {
"address": "B"
},
"amount": {
"value": "1000",
"currency": {
"symbol": "BTC",
"decimal": 8
}
}
},
{
"operation_identifier": {
"index": 2
},
"type": "OUTPUT",
"account": {
"address": "A"
},
"amount": {
"value": "500",
"currency": {
"symbol": "BTC",
"decimal": 8
}
}
}
]
Construction API flow (only re-run if the suggested fee has changed > threshold, usually this is unnecessary)In rosetta-bitcoin, we estimate the size of the transaction in /construction/preprocess using the following logic:
// Fee estimate constants
// Source: https://bitcoinops.org/en/tools/calc-size/
const (
MinFeeRate = float64(0.00001) // nolint:gomnd
TransactionOverhead = 12 // 4 version, 2 segwit flag, 1 vin, 1 vout, 4 lock time
InputSize = 68 // 4 prev index, 32 prev hash, 4 sequence, 1 script size, ~27 script witness
OutputOverhead = 9 // 8 value, 1 script size
P2PKHScriptPubkeySize = 25 // P2PKH size
)
// estimateSize returns the estimated size of a transaction in vBytes.
func (s *ConstructionAPIService) estimateSize(operations []*types.Operation) float64 {
size := bitcoin.TransactionOverhead
for _, operation := range operations {
switch operation.Type {
case bitcoin.InputOpType:
size += bitcoin.InputSize
case bitcoin.OutputOpType:
size += bitcoin.OutputOverhead
addr, err := btcutil.DecodeAddress(operation.Account.Address, s.config.Params)
if err != nil {
size += bitcoin.P2PKHScriptPubkeySize
continue
}
script, err := txscript.PayToAddrScript(addr)
if err != nil {
size += bitcoin.P2PKHScriptPubkeySize
continue
}
size += len(script)
}
}
return float64(size)
}
Our implementation only supports sending from “Native SegWit” addresses, so the input calculation is pretty straightforward. Estimating the output size is where things are a little more complicated. Fortunately, there are some existing packages that help with this (like https://godoc.org/github.com/btcsuite/btcd/txscript).
We return the estimated size in ConstructionPreprocessResponse.Options and calculate the suggested fee in /construction/metadata using the response from the estimatesmartfee method in bitcoin-core (returned in BTC/vKB). Unless fees are extremely volatile, we only expect to do perform this “dry run” phase once. If the suggested_fee is greater than max_acceptable_fee, we have to start over with new coins. 
In the example you’ve shared above, I assume you mean that the coin_spent associated with the INPUT has 1000 CKB on it. Typically, I would represent this as:
[
{
"operation_identifier": {
"index": 0
},
"type": "INPUT",
"account": {
"address": "AliceAddress"
},
"amount": {
"value": "-1000",
"currency": {
"symbol": "CKB",
"decimal": 18
}
},
"coin_change": {
"coin_action": "coin_spent",
"coin_identifier": {}
}
},
{
"operation_identifier": {
"index": 1
},
"type": "OUTPUT",
"account": {
"address": "BobAddress"
},
"amount": {
"value": "500",
"currency": {
"symbol": "CKB",
"decimal": 18
}
}
},
{
"operation_identifier": {
"index": 2
},
"type": "OUTPUT",
"account": {
"address": "AliceAddress"
},
"amount": {
"value": "100",
"currency": {
"symbol": "CKB",
"decimal": 18
}
}
}
]
A Coin should always be spent in its entirety and the fee is inferred as sum(INPUT)-sum(OUTPUT) (this doesn’t break reconciliation as the coinbase contains the block reward + fees). I elaborate a few times on how to we provide CoinChange in this response, however, I’ll reiterate here as well. The gist of it is we select coins where sum(INPUT) >= sum(OUTPUT) + max_acceptable_fee and then set the CoinChange using the suggested_fee returned by /construction/metadata where CoinChange=sum(INPUT)-sum(OUTPUT)-suggested_fee.
As I explained in the rosetta-bitcoin case, we provide an intent with the inputs and outputs we wish to create plus a change output of 1. We then amend this intent based on the value of suggested_fee returned by /construction/metadata (replacing this change output amount with sum(INPUT)-sum(OUTPUT)-suggested_fee.
I highly encourage defensive programming here. If you are able to determine that a transaction would not land on-chain, you should throw an error to prevent the caller from silently broadcasting an “under-fee’d” transaction (which could get stuck in the mempool on some blockchains). For example, we return an error in the /construction/preprocess method in rosetta-bitcoin if the inferred fee given the estimated size would yield a fee rate lower than the minimum allowed.
The recently released Construction API testing framework (in rosetta-cli >= v0.5.0) has support for a lot more complex flows. I’ve been testing it in tandem with our rosetta-bitcoin implementation. Coincidentally, I’m adding built-in support for the “dry run” flow I described above this week. I’m adding support for “fund return” (suggested by @alan.verbner) next week!
Progress has been a little delayed because I’ve been bogged down by some performance improvements I’ve been making to improve check:data.
This is usually only ever at most 2 calls for us (create intent with value 1 change, re-run intent with populated change). With a simple wallet package (hoping someone steps up to help on this soon!), this would be hidden from the user and probably add milliseconds, at most, to the construction process (we aren’t doing any signing or parsing during this “dry run” phase).
Let me know if you have any other questions @kingstone or @alan.verbner!