Update action and event data filters

[tsurdilo]

What would you like to be added:
Action data filter: add additional property which defines which state data element to add the result of the action to (default ".")
Event data filter: add additional property which defines which state data element to add the event payload to (default ".')
Why is this needed:
currently we assume the action results / event payload is merged into top-level state data.

[tsurdilo]

open for suggestions on the naming of the new param in both filters

[jorgenj]

outputPath? outputMergePath? mergePath?

Also, wondering if we should rename the existing dataOuputPath to more clearly indicate that it transforms the output? Perhap outputTransform? outputExpression?

[jorgenj]

I think another thing that's missing is a detailed description of the expectations for how a merge should be done. Is it a shallow or a deep merge? ie;

Given a state input of:

{ data: { foo: bar }}

And an action just completed which returned the following:

{ data: { bar: foo} }

What would the merged result be?

If it's a shallow merge, I'd expect the output to be:

{ data: { bar: foo }}

For a deep merge, I think I'd expect:

{ 
  data: {
    foo: bar,
    bar: foo
  }
}

If we're thinking deep merges, then I think there's some complex scenarios to consider that I don't think there's a sane way to handle other than to throw a runtime exception, like...

• merging arrays {data: [foo]} + {data: [bar]} (this is probably not an error condition, my concern is the order of operations needs to be explicitly called out or we might get different results each time this is done)
• merging dis-similar data-types {data: "foo"} + {data: ["foo"]} + {data: {foo: bar}} + {data: null} + etc.

[tsurdilo]

The adding/merging should be done same as with the jq "+=" operator, meaning merge if possible and right overwrites left. data types should be taken into account

[singhkays]

I debated whether to create a new issue but since we're on this topic, I'd like to propose a simplification to data filters based on @jorgenj's comment above

outputPath? outputMergePath? mergePath?

Also, wondering if we should rename the existing dataOuputPath to more clearly indicate that it transforms the output? Perhap outputTransform? outputExpression?

Problems

1. The description for these "paths" says it should be an expression but the name doesn't convey that semantic.
2. Why do we have three types of filters (State + Action + Event data filter) with similar properties. For a new user it increases the breadth of things to understand especially when they are very similar

Proposal

1. Combine State + Action + Event data filter into a generic "data filter" object. This can then be used wherever data filtering is required - State data, action data, event data
2. Simplify property names

Examples

Merge path supplied

{
  "dataFilters": {
    "inputFilter": "${ {vegetables: .vegetables} }",
    "outputFilter": {
        "expression": "${ {vegetables: .vegetables[] | select(.veggieLike == true)} }",
        "mergePath": "path where the data will be merged"
  }
}

When merge path not needed

{
  "dataFilters": {
    "inputFilter": "${ {vegetables: .vegetables} }",
    "outputFilter": "expression": "${ {vegetables: .vegetables[] | select(.veggieLike == true)} }",
}

[tsurdilo]

@singhkays I like your idea. will try to work it in.
there are two things I'd like your input on:

1. not all data filters have the same properties:

• state data filter has input and output filtering. there is no adding/merging needed
• event data filter has only output filtering (payload), and can include adding/merging
• action data filter has input and output filtering and includes adding/merging
  It's really hard to describe which properties would be required based on where this definition is placed (using json schema). so we would have to make all properties always available for users and hope they read the documentation to know which ones to used for a specific data filtering scenario.
  We could make this easier by still having 3 different base definitions of filters (state/event/action) but call them the same thing on the property level, but that may rise the confusion for users as in their mind they are using the same thing all the time.

2. Does using the same property name for all filters look ok or is it confusing. Here is a single state with 3 filters. wdyt?

        {
            "name": "WaitForCustomerToArrive",
            "type": "event",
            "dataFilter": {
                "inputFilter": "${ .languageGreetings } ",
                "outputFilter": {
                              "mergePath": "${ .finalCustomerGreeting }"
                  }
            },
            "onEvents": [{
                "eventRefs": ["CustomerArrivesEvent"],
                "dataFilter": {
                    "outputFilter": {
                              "expression": "${ .payload.customer }",
                              "mergePath": "${ .customer }"
                  }
                },
                "actions":[
                    {
                        "functionRef": {
                            "refName": "greetingFunction",
                            "arguments": {
                                "greeting": "${ .languageGreetings.spanish } ",
                                "customerName": "${ .customer.name } "
                            }
                        },
                        "dataFilter": {
                             "outputFilter": {
                              "mergePath": "${ .finalCustomerGreeting }"
                          }
                    }
                ]
            }],
            "end": true
        }
   

to me, honestly, having specific names for filters catches the eye better (would make this state more readable). what is your opinion on this?

[tsurdilo]

@singhkays the way I started working on this today, the example above would look like;

        {
            "name": "WaitForCustomerToArrive",
            "type": "event",
            "onEvents": [{
                "eventRefs": ["CustomerArrivesEvent"],
                "eventFilter": {
                    "data": "${ .customer }",
                    "toStateData": "${ .customerInfo }"
                },
                "actions":[
                    {
                        "functionRef": {
                            "refName": "greetingFunction",
                            "arguments": {
                                "greeting": "${ .spanish } ",
                                "customerName": "${ .customerInfo.name } "
                            }
                        },
                        "actionFilter": {
                            "fromStateData": "${ greetings.hello }",
                            "results": "${ .greetingMessageResult }",
                            "toStateData": "${ .finalCustomerGreeting }"
                        }
                    }
                ]
            }],
            "stateFilter": {
                "input": "${ .greetings } ",
                "output": "${ .finalCustomerGreeting }"
            },
            "end": true
        }
  

what do you think?
I wanted to make the filter property names make sense to user.
So for example a CE event has "data" (payload) so we call the prop "data". We add/merge data to state data so prop is called
"toStateData", with actions they filter data from state data that should be available to them, so "fromStateData", actions have results so prop called "results", and finally can add/merge back to state data so "toStateData"
With states we define that states have a data input and produce data output, so props called "input" and "output"

Would this be ok?

[tsurdilo]

@jorgenj @singhkays - #278