Prediction for Cox time varying covariates

[SimplyOm]

Hi All,

I have seen a few CrossValidated posts (here and here for example), which mention why prediction on time varying covariates is paradoxical and leads to survivorship bias. While I do somewhat agree with the paradox, I still believe there is quite some use case of allowing prediction on time-varying covariates.

As described by @GCBallesteros in #530, we can use the time varying covariates for prediction in cases where we can predict the underlying covariates. This is also useful in running what-if analyses. As such, does it make sense to allow this prediction in a future release, potentially with warning for avoiding misuse (rather than not allow it completely)? I believe the underlying baseline hazard and partial hazards are already available and it's the fundamental paradox which is blocking the feature, rather than implementational complications.

Currently, without any other alternative, a lot of us are falling back to the survival package on R.

[SimplyOm]

Hi, any thoughts on this?

Would assume there is no interest/plan for time varying Cox predictions, if no response.

[benslack19]

I'm interested in more features for time-varying fitter as well. For example, plot_partial_effects_on_outcome would be helpful. It seemed like this example from Stack Overflow made it appear that it was once available? Is there any workaround we can do to at least make the plots, for example by using the coefficients and passing in values sampled from a distribution?

[AhmetZamanis]

Hi,

I have recently used lifelines' time-varying Cox model in a project, and came across the same question. I am working with a historical, observational panel dataset where many subjects' event statuses are observed daily, along with a set of covariates.

• The event of interest is pretty rare by nature.
• Occurrence is not "death", the subjects continue to exist & be observed. Multiple occurrences per subject exist.
• My goal is not to predict future events, but to quantify the event risk after observing a new day, regardless if the event actually occurs or not.

Here's how I planned to do that:

1. Fit a Cox time-varying model on the historical dataset.
2. Observe a new, future day T, record the event status, survival time & covariate values.
3. Using the model, predict the survival function for this new observation.
4. Use (1-p) to quantify the "current event risk, as of T / at the end of T", regardless of actual event occurrence observed at T.

The goal here isn't to predict the event status at T before it occurs, but to use the model for something like "immediate historical analysis" after T is observed. I think it's a reasonable way to assess current event risk for a given subject, provided the covariates are any good.

If I am not missing anything that invalidates my reasoning, I think having a predict_survival_function for this purpose would be useful. It could come with a warning about making predictions into the future.

In the meantime, I referenced the code in #530 to work back from predict_partial_hazard & baseline_cumulative_hazard_ to get the survival function values for every fitted observation. I can't verify the maths fully, but the results seem correct compared to baseline_survival_. I also skipped the interpolation steps, so I only have the survival function values at survival times associated with actual events in the training data.

From there, I plotted the survival times & survival function values for each observation, colored by the covariate of interest, something akin to plot_partial_effects_on_outcome.

If it would be useful, I can share some boilerplate code & information about my dataset structure, but not the data itself.