WIP: Specification of Timecourses

[fbergmann]

This draft PR adds the descriptions from the Time course specification to the release/2.0.0 branch.

preview site is: https://petab--581.org.readthedocs.build/en/581/documentation_data_format.html#timecourses-table

[dweindl]

Thx Frank.

Maybe #541 and #542 can also be addressed here.

[dweindl]

Resolved merge conflicts.

This needs to be updated to match the new mostly-agreed-upon format in #586

• changed conditions table
• changed experiments table
• add examples from google doc - to be updated to the new format
• update v2/gfx/petab_scope_and_files.svg: changed names, changed columns, match color of conditions+experiments table
• v2/gfx/petab_scope_and_files.svg / v2/gfx/petab_files.svg : smaller SBML logo / indicate alternatives

[FFroehlich]

It seems natural to have an addToCurrentValue as well

[dweindl]

Fine to have it for coherence. For simplicity, we could leave it out, since in this case, the target value for setCurrentValue and X could easily be written as X + 10 instead of addToCurrentValue, X, 10.

Another question is what other operations we would need. multiply{CurrentValue,Rate,...}? exp? log?

It would be nice if we could always just refer to the old value in targetValue, but that would require introducing additional functions there (rateOf, rateExpressionOf, ...). That would reduce the list of operations, and one could also use the old values of other entities, not just that of the current target.

[FFroehlich]

Yeah, that sounds like it would be helpful to introduce targetValue in favour of having addTo, multiply, etc.

Why would we need operators like rateOf, rateExpressionOf? I don't see why one would ever transfer information from rate to value and everything else should be achievable one way or another.

Also, what about using setValue in favor of setCurrentValue?

[dweindl]

One use case where one might want to use both the current value and the rate expression could be some piecewise definition of a rate expression where the condition depends on the current state. I'd don't consider super important.

[FFroehlich]

While I agree that we can drop the name in the conditions table, I think it would be nice to have an optional name column here.

[dweindl]

Fine to add experimentName here if we drop the condition names. However, if we want to keep condition names, which then would have to be specified in some other place (yaml, annotations table, ...?), I would collect all of that in the same place.

[FFroehlich]

In that case I would propose to move all names (including observables etc) to such a table. Would anyways be nice as it would introduce the possibility of, e.g. naming states.

[dilpath]

Thanks, I'm happy to review this again after the current feedback round

[dilpath]

I think we usually refer to this as the "conditions table"

Suggested change

	- (optional) A condition file specifying model inputs and condition-specific parameters
	- (optional) A conditions file specifying model inputs and condition-specific parameters

[dweindl]

Right. Makes sense to use coherent terminology. Also for all other tables/files...

[dilpath]

Should list everything (explicitly or by some grouping)

Suggested change

	(:ref:`math_expressions`, overriding values in the condition table, etc.)
	(:ref:`math_expressions`, overriding values in the condition table)

[dweindl]

To be updated once all changes here have been finalized.

[dilpath]

Suggested change

	Each line specifies a change of one specific property of a specific entity.
	Each line specifies the change in the numeric value of one specific non-estimated entity.

[dweindl]

This hides the fact that a single entity has different types of associated values (e.g., current value vs. rate) and excludes "symbolic assignment" like setAssignment, doesn't it?

[dilpath]

Ah right, make sense. Then to avoid the word property, which isn't defined yet:

Suggested change

	Each line specifies a change of one specific property of a specific entity.
	Each line specifies one change in one target entity.

[dilpath]

Agreed, in the measurements table it is NUMERIC\|inf instead

[dweindl]

Thanks for the feedback.

[sebapersson]

conditionName I think can be kept as optional as it can help with nicer plotting.

[dweindl]

The problem is that now there may be several rows per condition, which makes it not so ideal to specify any condition names there. Would you still keep it that table? Would all rows for that condition have to have the same value, or should there only be one non-empty entry per condition?

[sebapersson]

Maybe I have missed this discussion, but, is setRate, setAssignment and other operationType that alter in a sense the dynamic model (e.g. if setRate = 0.0 for one condition and setRate = 1.0 for another, the structural model differs between conditions, so I imagine that tools need to basically build multiple model structures during import), something that is going to be supported and encouraged to be supported by most PEtab tools?

Instead if this is a feature that is reserved to mean this, maybe it should be stated somewhere (so that users and potential developers of importers does not think this is a key important feature to support)

[FFroehlich]

No, this is discussed after Differential/Algebraic/Constant targets are introduced. To my understanding, structural alterations are not allowed.

[dweindl]

We might have different understandings of "structural alterations". Changing some algebraic expression e := x to e := y would be allowed, but changing some differential state to some algebraic expression would not be allowed. In how much that requires building different models is probably very much implementation dependent.

[sebapersson]

We might have different understandings of "structural alterations". Changing some algebraic expression e := x to e := y would be allowed, but changing some differential state to some algebraic expression would not be allowed. In how much that requires building different models is probably very much implementation dependent.

Thanks for clarification, just a last question to fully understand. So if we have a parameter given via an assignment rule; k = k1 * k2 would it then be allowed to change k via setAssignment?

[dweindl]

Yes, that was to my understanding the (only) use case for setAssignment.

[sebapersson]

This is not how SBML work right?

If I remember correctly if both A and B are assigned something in an event, basically the pre-event A and B values are used unless priority is specified.

But regardless, I agree compartment should have priority.

[dilpath]

This is not how SBML work right?

If I remember correctly if both A and B are assigned something in an event, basically the pre-event A and B values are used unless priority is specified.

I'm not sure, all I see in the SBML spec is (regarding simultaneously-occurring events)

A given simulator may use any algorithm to choose an order as long as every event is executed exactly once.

This choice of compartment priority (and more generally to respect the dependency graph) is partly based on discussions with @matthiaskoenig last year -- perhaps @matthiaskoenig can say more here?

[FFroehlich]

There is a difference between simultaneously occuring events and simultaneous assignments during the same event. I would treat assignments via conditions the same as a single event, i.e. not subject to priority or useValuesFromTriggerTime attributes. I would propose to not follow the dependency graph (and rather simply use pre-assignment values) with the exception of compartments. As done with SBML Event Assignments (even though this isn't crystal clear in the sbml spec).

For example it might make sense to average two species A = B = (A + B)/2. Due to circularity, this assignment is not possible when following the dependency graph, but straightforward with pre-change values. Following of dependencies can always be reproduces through substitution.

[matthiaskoenig]

We want 2 different behaviors. These are managed in SBML events via the useValuesFromTriggerTime flag. So we need probably this flag on the timecourse to ensure both scenarios are possible (see also comment below)

[dweindl]

I'd vote for @FFroehlich's suggestion here.

[matthiaskoenig]

In the timecourses, what we essentially have are event assignments triggered at t=t_trigger​. All event assignments must be executed simultaneously—this is not a step-by-step process. Instead, the model state must transition to the new state all at once. Any mathematical expressions that depend on updated parameters, compartments, or species must be re-evaluated accordingly. In other words, all changes are applied in a single step, and the AST (Abstract Syntax Tree) affected by these changes is fully re-evaluated. There is no predefined execution order—it is a single transformation of the model state.

Therefore, the claim that "compartment has priority" is incorrect and does not make sense. The underlying issue is that PETab changes are not clearly defined and have been incorrectly implemented by simulators. As seen in cases where changes affect both compartment volumes and concentrations, many simulators apply these changes sequentially, which is incorrect. Instead, all triggered changes must be applied in a single step. This clarification should also be explicitly stated for changes applied outside of timecourses.

Additionally, in SBML, there is a flag called useValuesFromTriggerTime, which allows the use of values before the assignments. This is likely what you are referring to:

The time of assignment is not affected by the Event attribute useValuesFromTriggerTime—that attribute affects the time at which the EventAssignment’s math expression is evaluated. In other words, SBML allows decoupling the time at which the variable is assigned from the time at which its value expression is calculated.

[matthiaskoenig]

In SBML, all ListOf constructs are unordered lists. This means an SBML model can be modified by shuffling any ListOf elements, such as ListOfEventAssignments, without affecting the numerical results of a simulation. In other words, there is no inherent order in these lists, nor are there execution priorities—except in cases where ordering is necessary, in which attributes like priority have been introduced to dictate the execution of multiple Events.

The same principle should apply when multiple changes are made in PETab: all changes should be applied simultaneously. If ordering is required, an explicit order or priority attribute must be introduced. If mathematical expressions need to reference values from the end of the previous time span, a useValuesFromTriggerTime mechanism is necessary. For cases where a mix of behaviors is needed, an Event-like construct could be introduced, allowing assignments to be grouped and prioritized.

There is a reason SBML Events are complex. Either we adopt a full event-based approach where changes can be triggered after each time span, or we adhere to a simpler default approach where all changes are applied simultaneously.

One concern is that specifications often end up reflecting existing implementations rather than defining a general, correct solution. This appears to be happening here, as most PETab implementations currently apply changes sequentially. This approach is likely incorrect, but since the specification does not clearly define the intended behavior, there is uncertainty. Moreover, it does not align with what users of SBML models would expect.

[dweindl]

since the specification does not clearly define the intended behavior, there is uncertainty

Right. That definitely needs to be addressed.

what users of SBML models would expect

To be honest, I am not sure for how many SBML users the users' expectations match the SBML standard when it comes to event assignments. I am still regularly confused when it comes to simultaneous changes in compartment sizes and concentrations.
That was one reason why we considered different options, such as the compartment-change-first approach.
I'd find some explicit priority the most intuitive and least error-prone. This would in my opinion also cause the least issues with non-SBML models, where the interpretation as dependent/independent quantity is maybe less clear.

[matthiaskoenig]

The main issue with establishing a fixed order for updating multiple variables (such as compartments, species, or parameters) is that mathematical calculations dependent on these variables could produce unexpected results depending on the update sequence.

For example, consider a compartment V and a concentration C. Any calculation that relies on both V and C may yield different outcomes depending on the update order. Suppose we start with V=1 and C=1, then update both to V=10 and C=10. If there is a calculation dependent on the relationship between V and C, the result will vary based on the order in which they are updated. For instance, an event triggered when V>C will only occur if V is updated before C. If C is updated first or both are updated simultaneously, the event may not trigger, leading to inconsistencies.

In other words, updating variables sequentially in a fixed order can lead to intermediate steps that place the model in an undefined state. This issue persists regardless of the specific order. Therefore, it is essential to either update all variables synchronously or clearly define and follow a predetermined update sequence for each variable to maintain a consistent model state (simply updating compartments before species does not resolve the issue).

The issue with the concentrations is just one of such a problem because species concentrations depend on their compartment volumes, so there is a math dependency linking C and V which depends on the order. But there can be many such dependencies in the math of the model.

[dweindl]

Okay, fair point. The intermediate model states would be a problem.