[Feature-Request] Pass results between actionwords


#1

Hello everyone.

I am looking for a way to pass some result from one Actionword to another. And as I understand the current Hiptest-design this is not yet possible.

Use-Case: i would like to use Hiptest for end-to-end testing of a distributed service-landscape (no ui). The idea is to model commands for a service A with the usual parameter way. Since my command will trigger several other services (decoupled via event-messaging) I need to extract some identifier which was returned form the first service-call and pass this as a parameter to the next actionword which will assert that the following steps have occured.

Given CreateSomethingServiceA (name=A) => result would be id=1
Then ValidateServiceB (id=1) <= passed from previous actionword

My current understanding of Hiptest concludes that adding a simple return-value to actionwords wont work for several reasons

  1. actionwords can be composed freely by scenarios and other actionwords
  2. for Spock this gets even trickier because under certain conditions the generated Test uses an Expect-clause where actionwords must return a boolean-value.
  3. some more I guess

With the current design, I need to use more coarse-grained actionwords which, in my case, handle the validation as well. Modeling failure cases makes those actionwords even more complex because you need to add parameters which change the actionword-behaviour (e.g. skipValidation = true).

A workaround would be to use some shared state between all actionwords. The problem is, that shared mutable state can cause nasty sideeffects.
With this solution, you need to add technical “cleanup”-actionwords which also makes composition harder because it is easy to forget when you need to cleanup your state.

So, after giving this some though, how about using a shared state only within the generated test, so each test gets a new context-object.

This could be modelled in the hiptest-ui by setting a simple checkbox within a actionword meaning that this actionword makes use of shared state. Now the hiptest-publisher generates an additional parameter (a simple map with string->any probably suffices), for each actionword requiring it. Furthermore, the actual instance of the state is generated into the actual test reducing the side-effects to that test-execution.

With this in place, my use-case actionword-implementation would now set the id=1 within the shared state-instance, and the second actionword can read the id=1 from the state. I know this affects composition of actionwords but its a start.

What to you think? What other design considerations play a role here?


#2

Hello Marc,

Actionwords can’t return values. As you suggested, if there is some value to be returned, then it must be stored in some shared state object.

Why is it like this? Well the description of the behavior of your app has to be decoupled from its implementation. Storing id after an object creation so this id can be reused later is glue code and should be hidden because showing this level of detail would make understanding the high-level feature harder.

That being said, if you write you tests like an example and want identifiers anyway, you can use your own identifiers like so:

Given a service A creates "Hello"
Then service B should validate "Hello"

You still have a shared state to store a map between Hello and the actual service id created by service A but it’s easier to manipulate.

You can also do as you suggested and store the id of the last created object in the share state instance. We have some scenarios in Hiptest that acts like this. It’s convenient and is pretty legible.

About the cleanup phase, it’s tests and they should always run in isolation from each other. Often the framework provides hooks so we can make the necessary cleanup before running the feature. For example, in Cucumber, there is a Before hook that runs before the first step of each scenario. It’s the perfect place to clear shared state.

What do you think of it?


#3

Hi Christophe

Thanks for confirming my thoughts about shared state. I like the idea of being more clear in my test-specification by giving it a context “hello” at design time and mapping this as implementation detail as you suggested.

Still, I think you have to be very careful which actionwords are being reused across scenarios.

The initial idea was to make this dependency at least somewhat visible during test-design, but sure, you have to be very careful not to leak implementation details.

With regards to the cleanup: Hiptest could use a “cleanup” section for scenarios (as it does with “setup”).

A short story: we had some trouble with our UI tests which used multiple data-rows. Some tests used a “setup” step to move to a certain part in the application where the actual test started. The problem was that each test-iteration needed to end in a dedicated actionword which moved the ui back to the screen where the test started (after setup), otherwise the next iteration could not execute.
In our case we used Spock, and the real problem was that when one data-set caused an error, all following iterations failed as well (because the cleanup-actionword was not reached). With Spock, the real benefit from the Unroll clause stems from the fact that you can easily figure out which test-data caused a failure.

So…it would be great if Hiptest could add a “cleanup” section to the szenario :slight_smile:

regards
Marc


#4

Yes, I understand the need of a cleanup section but I find it brittle, especially in cases as you described. That’s why I prefer having the setup doing the necessary cleanup if needed. That’s a personal preference, not a golden rule.

That being said, we have talked multiple times about adding a cleanup section similar to the setup. The major difficulty for us is that is does not map really well to Cucumber exports. See this related stack overflow question for more background about it. Anyway, I’ll add it to our product board for consideration at our next product meeting.