common formulas without coding (via helpers)

[djay]

user problem

A form designer wants to be able to specify simple validation rules without knowing code. It should be possible for them. Several out of the box rules should be available. Some would need to be configured such as ranges. It should be possible to use many at the same time. It should be possible for someone else in the same app to define custom rules which form designers can apply to certain fields.

This also applies to hidewhens, and save formulas too. A very common form of hidewhen is to hide a section until a certain option in a radio or checkbox is picked.

options

1. some kind of subobject of a field? ie "Create email field"
   • con: can't combine
   • con: can't change easily after creation
2. another rule object added to a form which you specify which fields it applies to? Makes it possible to specify validation involving multiple fields. Such as one date much be greater than another date. Would it be extra clicks?
3. a kind of behaviour/helper which when enabled on a field or form or hidewhen that will insert generated code into the formula. e.g. "show when option picked" or "email form to" or "validate phone number".
   • These have their own form UI to configure them.
   • one helper could write code into multiple formulas
   • We could use plomino itself to define these helpers.
   • Will need a way to read config back out of generated forumulas and combine formulas between behaviours
   • con: what happens when your helper is updated? have a way to update all the generated code? or always ensure your generated code calls shared code?
   • pro: doesn't require anymore config to be saved.
4. allow formula inheritance from base forms. The UI could include a list of events or formulas in explcit base forms (or from subforms) and you pick from that. (@jean)
5. same as above, except formulas are separate objects and they are applied to a field/form by a naming convention based on field_id/form_id and event type. (@jean)

Plan

Do helpers as follows

• create a widget to be added to the DX schema for forms/fields etc. Lists titles or currently installed helpers to edit/remove and has an autocomplete to add a new one.
• list of helpers is list of forms with a certain id e.g. helper_form_emailaspdf or helper_field_number_rangevalidation. If another database is in the site with an id helper_... then it's form names will also be searched
• Adding a helper is similar to popup for datagrid. Previous data is loaded into a temp document, from an internal store in yaml or json. (Might be stored in a special formula comment).
• Formulas in the helper form get access to context about the form and other helpers in use. e.g. can create a dropdown to select another field in the current form.
• Hidden computed fields are used to determine which formulas (or other data) to override using a convention for id e.g help_onSave will be calculated to python code which emails to a certain address the contents of the form.
• Other parts of a field/form can be modified other than formulas. For example html attributes.
• On field/form save all the helpers computed formulas are used to work out the final code. Special comments around the code will allow it to be replaced.
• Certain formulas will have default ways to combine multiple helpers trying to write to the same field. e.g. validation and hidewhens will default to OR. (or should there be a way to run helpers inside helpers to combine code?)

Specific example helpers to implement

• email form data to recepient
• show if item in list selected
• show if value within range
• validate phone number
• validate email address
• validate number in range
• validate date in range
• validate less than X chars or words
• display total from adding fields
• show if total from adding fields is within range
• POST form data via web service call

Future

• could extend helper concept to be able to insert section of html + fields etc. Then edit will update those fields. Useful for summary pages, table of contents, etc.

[ebrehault]

We have always tried to keep the amount of field properties as low as possible, considering most of the config settings a user can need can be implemented through formula. That's the Plomino approach.
Plomino is designed for people able to write formula.
Nevertheless, I understand your concern, and I think there are probably 2 ways:

• rules (as you mentionned) implemented by someone else, and applied wherever the user wants => it would probably be useful to implement that in Plomino.
• HTML5 attributes: HTML5 can do a lot regarding validation (required, range, etc.) => it already exists (but for now it is a formula returning a dict, we might also allow a simple string)

[djay]

I have an idea might combine with the plomino approach. What if there was a way to create a code generator for a formula?
In terms of UI it could be done via JS:

1. looks up for a given formula if there are any predone code generators.
2. presents them as a list. if in designer mode, thats all you see
3. if you pick one you get a form with options
4. result is a generated formula saved in the formula field. Perhaps with the options also saved inside comments so parsing is easier.

This way no more options need added and this could apply to lots of different kinds of formulas, not just validation for instance like action adaptors in PFG.

In addition this could be done TTW using plomino itself. You could create a special plomino form which asks the options and has a calculated field to generate the code (and some libraries to make this easy). This means coders could build things that make it easier for designers to help build applications.

Not sure how to combine it with HTML5 attributes. A validator generator would have to change both the template and the attributes at the same time. Maybe a given generator could control multiple fields on a given object not just a single formula?

Not sure how to handle combining formulas togeather. Both action adaptors and validators in PFG have the ability to have multiple selected. It's very helpful to be able to both email the form results and save to the database at the same time. or email multiple people. or validate if a field is both valid postcode and between 2000 and 2999.

[djay]

oh and @jean also suggest you could have a mode where you edit the generated code. so it would be a way into coding more perhaps if you had that level of permission.

[ebrehault]

It sounds like an excellent idea.
Any formula field would provide 2 modes (editor or generator), and the generated code would contain some markers, just like archgenxml used to do. Something like:

## GENERATED CODE FOR <the_feature_name> START
# you can change this code, but it will break the generator compliancy
<our generated code>
## GENERATED CODE FOR <the_feature_name> END

That way we can easily make sure we do not override manually changed code, we can easily concatenate several generated sections, etc.

[djay]

I was thinking some kind of naming convention for forms that are meant to generate code instead of be used in the application. This way we don't need an extra kind of registration. Not sure what the convention would be.

Still not sure on how to combine a validator rule that injects html attributes as well as code to test the value. If we injected values into the HTML attributes injection field then how can we remove it again? Thats harder to combine with others. Perhaps the on save event can manipulate other fields so its done on a case by case basis.

[jean]

One refinement that I thought of is that instead of result is a generated formula saved in the formula field, the default result is a reference to the rule instance. Only when customized is the generated formula copied to a rule object for the current field.

(I'm imagining that instead of event and validation formula properties on forms and fields, we have event/validation/rule objects and lookups. Look at rules listed in order in a rules property that should work similar to a collection (i.e. can show contained rules here or elsewhere, or pick individual rules).)
By default this lists before_save (in form then globally), after_save, etc.)

(This idea should work for fields also --- instead of forms only seeing contained fields, make available referenced fields for placing on the form.)

[djay]

not sure about the references thing yet.

But in terms of code generation I think it makes sense in terms of the UI that a "formula helper" is picked for a whole form or field or action rather than on a specific fomula field. Users might not know which formula they are meant to look at. This way a given helper can target multiple formulars/settings at once if enabled (such as html attribute injection). It would kind of be like DX behaviours.

[jean]

Some clarification and expansions.

allow formula inheritance from base forms. The UI could include a list of
events or formulas in explcit base forms (or from subforms) and you pick from
that. (@jean)

== Clarifications ==

• It's not inheritance in a strict sense. That's a loaded term with probably
  unintended associations. I mean that a form references other forms to use as
  source of fields/formulas. They aren't "base" forms --- they may be libraries
  grouping sets of related formulas, or they may be template forms, or they may
  be the standard form and we're creating an exception, ...
• I'd like to get rid of subforms, as they're untidy. If you're using a subform
  to pull in existing fields, instead of saying "these forms are sources for
  fields/formulas", you type something into the layout that isn't meant to add
  layout, and doesn't (currently) make explicit what you're getting --- you'd
  just have to go and look at all the mentioned subforms (and their subforms,
  if any).
• I'm leaning away from picking individual fields/formulas, that seems like an
  easy way to create spaghetti. Rather pick forms as sources, and get all the
  fields/formulas provided by that form.

== Expansions ==

Currently the content model of Plomino is (from a design perspective):

• Forms that contain Fields
• Views that contain Columns
• Agents

I want to look mainly at Forms / Fields.

Forms supply layout, fields, validation, and events. (Fields also supply
validation.) Of these, validation and events are handled differently to fields.
They are not visible as content, and there is a fixed set of properties where
all possibilities need to be shoehorned into. E.g. if A and B need to happen
beforeSave, then the event property on the form needs to mush them all into one
formula. If A is something that needs to happen on a group of forms, then
either the code needs to be repeated, or a #Plomino include something line
needs to be added in the right place everywhere.

This makes it very hard to see what's happening, to make sure the right checks
or events are being executed everywhere they should be, and necessitates
repetitive changes across forms that risk breaking other unrelated things in
the same formula.

What I would like to do is to hoist these formulas out into objects as well, so
that instead of a formula in a validation property, we have a formula object.

Now, how are the relevant formulas found and applied?

At the moment, a form mainly operates on contained fields (ignoring subforms for now).
I would like to change this so that a form can designate a list of other forms
to use as source. The other forms may contain fields and/or formulas.

I would like to add a summary view to a form so that it lists not only
contained objects, but also those found on source forms, in order of
precedence, with overrides marked.

I propose that formulas are differentiated based on naming. When you add
validation to thisfield, a formula object named validate_thisfield is
created. When you copy & paste a form, the new form does not contain copies of
the original form's contents (fields, formulas). Instead it references the
original form as source. If we override thisfield in the new form, e.g. to
display only the year portion of a date, we will get a new thisform/thisfield
object, but the validation formula in the original form will still be found and
executed.

This should not be mysterious: when looking at the form summary, we should see
something like:

• thisfield (here) / validate_thisfield (from origform),
  validate_thisfield_special (here)
• thatfield (from origform)
• otherfield/validate_otherfield (from otherform)

When looking at the field, we should see the validate_thisfield and
validate_thisfield_special formulas. (That "special" formula is to illustrate
that there may be any number of validation rules that apply to this field, that
may be contained in the current form or found in a referenced form. The same
applies to events, e.g. onsave_notifysomeone, onsave_updaterelateddocs, ...)
There should be an action on the formula to create a local copy for customisation.

The same goes for event formulas: all found formulas starting with
beforecreate_... are executed during the before-create event.

I think that forms should be able to store a local ordering of found formulas,
so that they can be manually re-ordered if necessary.

If you looked at the summary view of a form, it would list:

• the contained formulas
• the referenced formulas
• categorised in terms of event/validation
• in the order that they execute (local ordering if any, then original
  ordering)
• noting any overrides (e.g. local formulas override referenced formulas)

You would start a new form for this category of documents, or this type of
functionality, by copy/paste an existing form, or template form.

That's similar to what you do now, but instead of ending up with many copies of
every field, in this case the copies would just reference the original form as
field source (and if the original form were referencing another form as field
source, it would get those references too).

Form validation formulas would be named validate_....
Field validation formulas: validate_fieldid_...

At the moment every field has a property for a validation formula.
I'd prefer them to find their validation formula by looking for a formula with
their name in it, in the parent form and in referenced forms.

That way they can share validation formulas.
E.g. a 'country' field could find a local validate_country_isEU formula and a
referenced validate_country_isISOcode (silly example).

There's no need to add and correctly name everything by hand .. there should be UI
to "add a beforecreate formula", which adds a formula named
beforecreate_whatevernext (where 'whatevernext' is how the user named it).

[djay]

#705 Contains a potential mockup of option 3, the helper idea.

1. Each Form or Field or HideWhen contains a add helper button on its edit page.
2. Adding a helper displays its configuration form.
3. Once saved, it will modify formulas and/or html attributes and save its settings as comments in a single formula.
4. User can update or delete existing helpers and the generated code will be updated.
5. Helpers can be combined. For example you might had a "between two numbers" validation helper and a "must be an integer" validation helper to the same field.
6. You can create your own helpers yourself either using z3cform in an egg, or using plomino and marking the form as a helper (how? using a special helper to turn a form into a helper?)

[ebrehault]

I don't know if z3c.form is the right approach here.
If the UI is built as a single page app, generating a form on the backend and insert it in our UI sounds like a bad idea.
It will be probably easier to implement the helpers screens in pure JS (we will be able to use advanced widgets like sliders, or make client side validation, etc.), and just provide an endpoint implemented in Python that will received the parameters submitted from the helper UI and do the appropriate changes in the various formulas.

[djay]

@ebrehault In that case the simplest option is using plomino itself. It's an optimisation to allow installing via eggs and one that perhaps we won't need. So perhaps how you create a helper is

1. Create a seperate db to contain the helpers.
2. Create a form and somehow register it as a helper. The form is the configuration screen for the helper.
3. In the forms save formula use a helper api to specify the fields and formulas you want to change.
4. In the DB where you want to use the set of helpers there is something in the DB settings to register the helper DB as being used.

Might be nice if the helper DB could optionally be in the control panel somewhere as its not neccerily end user facing.
Also maybe @jean suggests the generated code might be better off using utilities or libraries which would come from the helper DB. There would have to be an easy way for the generated code to call these without knowing where the code is located.

[djay]

@ebrehault Seems like implementation would depend on #726 and #618. Waiting to see if there is movement on those.

[djay]

@ebrehault added a plan

[djay]

@ebrehault progressing well. I;m going to have a setting in the db object to specify which db's to import helpers from. Seems more secure than id conventions. The helpers themselves will still use an id convention.