[Feedback Request] Automated Construction API Testing Improvements

Before beginning work on Construction API testing improvements, we wanted to gather feedback from the community on our proposed approach (attached below). We spent a lot of time thinking through scenarios we wanted to test when prototyping but want to get a second opinion in case there are any missing primitives.

Current Status

The testing tool only supports account-based and UTXO-based transfers between accounts with derived addresses. You can read more about it in the release post.

Shortcomings

The testing tool does not yet support multi-currency sends, on-chain address generation, testing non-transfer transactions, or testing account/utxo transfers on the same network.

Goal of Improvements

Allow an integrator to automatically test any transaction they would ever create in production.

Proposed Solution

Questions

  • Do you feel you could automate all supported transaction types with this proposed solution? Are there any primitives missing?
  • What is your preferred format for writing these tests? JSON, YAML, or a DSL (domain-specific language)?
1 Like

Thank you @patrick.ogrady. From the perspective of a human test author, I wonder about the feasibility of using a more friendly format than JSON. JSON is great for machines, but not ideal for a human.

A suggestion would be to explore integrating a BDD library such as https://github.com/cucumber/godog which gives Cucumber-style tests. Not sure how much work it would be to integrate this particular library, but I do think this DSL approach has the potential to be a lot more human-friendly.

Interested in thoughts from the community here.

1 Like

Let me start by saying that we haven’t spent much time on our construction API implementation yet (though this will change this week), so take my feedback with a grain of salt.

Overall, I like the design and I think it will work for us!

Yes, this seems sufficient – on our network, we have delegation, staking, payments, and a handful of extra transactions for custom tokens. It seems like all of them should work.

I think a DSL would be best, but the nice thing about using JSON is you let the implementor build whatever DSL they want that can compile to the required JSON.

I think this is a great callout, @bkase. If we think of a DSL as a layer that sits on top of this lower-level interface, we could iterate on them independently/support community-created DSLs for certain test cases (not to get too far ahead of things).

1 Like

I believe the improved Construction API could help automatically check most of the transaction types. But I suggest the Construction API only focus on some specific test fields, including multi-asset exchange, custody and so on. Because it seems very hard to cover all transaction types, at least for a generalized UTXO chain like Nervos CKB.

  1. A UTXO transaction actually consists of both spent and unspent outputs, it also require an off-chain execution environment to generate the outputs. Without the environment, you cannot figure out the right output data. Take Bitcoin Omin layer as an example, you need to calculate the data in the output OP_RETURN filed off-chain. It’s basically not a problem, because there is some SDK for you to do that, and the balance calculation is simple. But for some Turing complete UTXO chain, like Nervos CKB, You need a complete VM environment off-chain to meet the general requirement of various kinds of transactions. You need to write generation script (including input gathering and selection script) for every kind of transaction, like DEX deposit, withdraw, gamble bid, … But these scripts rely on blockchain VM to execute, which should be absent in the Rosetta test framework.

  2. Another characteristic of Nervos CKB is we allow user-defined crypto primitives to serve as the low level signature algorithm. Which means you cannot simply mark “secp256k1” as the signature algorithm, or leave a possible selection group to the transactions. A transaction on CKB may use P-256, ed255619, SM2, RSA, or even hash revealing algorithm to guard the UTXOs, it seems impossible to prepare them ahead.

So my suggestion is to focus on the token balance, transfer, deposit/withdraw (POs), multi asset operations. It will cover 90% or more user scenarios, and leave the other tests to a case by case integration or test design.

We have also just started working on this endpoints.

I do agree with

One thing that we are facing, not sure if others do as well, is that sometimes when creating a transaction, custom things or values need to be defined or set. For example, we need to receive a value that defines when the transaction should expire. It requires some online data and some processing based on a client defined parameter.

My point is that It seems that there might be a lot of small differences that make to create a single and unified testing tool for construction.

One thing that I can say without putting too much brain into it is that, maybe, some sort of plugin system / hooks were people could transform / add custom logic could be a good idea although:

  1. Not sure how many people will actually use it
  2. I assume it’s not straight forward nor easy to develop
  3. Will it be easy to use? Will people need to know how to code in go?

Anyways, sorry for not being specific but those are my two cents.

In our organization, we’re interested in using the automated test in our CI tests. For that, we desire a way to run it for a finite amount of time.

1 Like

We just merged a PR into the rosetta-cli that allows for setting “enhanced end conditions” for stopping a check:data run (i.e. stop when reaching tip or after some amount of time instead of at an end block). We plan on using this feature extensively in our own testing.

In the near future, we plan to extend this to check:construction as part of this work (as well as add more end conditions to check:data)!

This is a great point @cipherw. I’ve very actively tried to avoid getting carried away by writing some crazy generic testing framework that can test every possible thing any blockchain could do (likely impossible haha). I’d much iteratively improve the tool by adding support for new types of transactions one at a time (if this new testing format doesn’t support one).

What I mean by this “goal”, to clarify, is that we (Coinbase) would like to test any transaction we would ever create in production. This is likely limited to the “focuses” you mentioned here in the near term (i.e. transfers, staking, account origination, governance). If there is some operation you would like an integrator/exchange to support that is more network-specific (like a one-off smart contract invocation), you should plan on making it possible to test automatically.

For other readers, I provided more guidance on using “live” metadata in /construction/preprocess metadata on a different topic.

I also put up a PR to improve the wording on the spec around this point.

This testing tool improvement will allow for providing any “non-live” metadata to ConstructionPreprocessRequest.Metadata that may be relevant for transaction construction (like this “ttl” you mentioned).

I’m thinking most of the gains here could be accomplished with a custom DSL on top of these low-level operations. If we find this isn’t the case, having a “custom” op that just calls out to an HTTP endpoint may be a good compromise.

Do you feel you could automate all supported transaction types with this proposed solution? Are there any primitives missing?

We wanted to highlight the following transaction primitives that Kadena would find useful:

  • (A) We’re a sharded blockchain so the ability to test different types of transactions across different shards would be important. Will there be a way to specify which shard (sub-network) a transaction is expected to execute on in the testing framework? Or does testing transactions in different shards mean creating different configuration files for each shard?

  • (B) Will it be possible to connect related transactions? For example, could we specify that transaction A occurs after transaction B? Expanding on the previous ask (A), could we specify the sub-network of related transactions? For example, can we state that transaction A must occur after transaction B; and also state that transaction A must occur in chain (shard/sub-network) 1, while transaction B must occur in chain (shard/sub-network) 2?

Motivation for ask (B):

  • Kadena’s public blockchain has two transaction workflows: executions and continuations. Execution transactions are a set of operations that all execute at the same time (i.e. what’s generally considered a regular transaction in other blockchains). An example of this is a transfer transaction that will debit funds from one account and credit another account at the same time. Continuation transactions, on the other hand, are partitioned into operations that occur sequentially but at different times (i.e. multi-step transactions). For example, one transaction can begin a continuation by debiting funds from one account at block height 1, while another transaction occurs at block height 4 that finishes the continuation by crediting the funds to another account if certain on/off-chain conditions are met. And some steps in continuations might be programmed to take place in specified chain(s) only.

  • Continuations are a vital component of what we call “cross-chain transfers”. This is the process of burning coins on one chain (shard/sub-network) and redeeming them in another chain. The coins can only be redeemed if an spv proof is provided that shows that the coins were burned in the previous chain. We implement this workflow by using continuations: the first transaction (the first step) debits the funds in the source chain; an spv proof is generated proving that the funds were burned in the source chain; and the last transaction credits the funds to the specified account in the target chain. The last step can only be executed in the target chain and with an spv proof that shows that the first step successfully occurred in the source chain.

Thanks for chiming in @linda! Getting some perspective from a sharded blockchain is great!

Right now, testing is scoped to a single NetworkIdentifier (i.e. you need to create different configuration files for each shard) :frowning_face:. This is more of a result of how the rosetta-cli stores data and our current logging implementation (which doesn’t provide any network context), less so any technical limitation.

In the near future, we plan to support concurrent syncing across multiple shards and believe this should “just work” when that is completed (scenario execution is currently isolated to NetworkIdentifier). To make sure this sort of test is fully supported, I’m going to add NetworkIdentifier to scenario. This is a GREAT callout @linda!

When designing the new testing interface, we thought a lot about transaction linking (we allow for accessing variables from linked scenarios…we call linked scenarios a flow). That being said, we did not consider linking scenarios across multiple NetworkIdentifiers. I do not believe this should cause any issues, however (especially with the NetworkIdentifier in scenario change mentioned above)!

Thanks for providing all this detail about your use case. Helps a lot when thinking about improvements here. I have a specific follow-up question…is it possible to query the unclaimed balance for an account on a particular chain? In other words, the amount that can be claimed? Or would you need to read the output of the on-chain transaction on chain A to know how what to claim on chain B?

Thanks for providing all this detail about your use case. Helps a lot when thinking about improvements here. I have a specific follow-up question…is it possible to query the unclaimed balance for an account on a particular chain? In other words, the amount that can be claimed? Or would you need to read the output of the on-chain transaction on chain A to know how what to claim on chain B ?

At the moment we don’t have a mechanism for querying unclaimed cross-chain transfer balances because that requires information from two separate chains. This breaks the chain abstraction that we have established. How important is this? Our first reaction is that this should be handled externally by tracking all cross-chain transfers, setting them as initially unfinished and then marking them as finished when the finishing transaction comes through on the target chain.

From the perspective of automated testing, not important! I was more-so curious about the abstraction you were using (and its limits) so I could consider it when working on this testing improvement. Thanks for the info!