Add Initialized and ComponentsCreated conditions to TrainJob API #2464
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@ tenzen-y |
| // TrainJobCreated means that the actual jobs creation has succeeded. | ||
| TrainJobCreated string = "Created" | ||
| // TrainJobInitialized means the TrainJob has been initialized. | ||
| TrainJobInitialized string = "Initialized" |
There was a problem hiding this comment.
Thank you for doing this @dineshkolhe1!
Let's discuss whether these conditions make sense given that we also have custom dataset and model Initializers in the TrainJob.
/hold
There was a problem hiding this comment.
obviously ! it make sense
the condition status -
Created – The job has been submitted but not yet started. (its make difficult to understand)
but ,
thise condition bit easily understandable -
Initialized – Indicates that the TrainJob request has been accepted and validated. &
ComponentsCreated – Confirms that all necessary components have been successfully created. (e.g., pods, PVCs, ).
There was a problem hiding this comment.
The current Created status may be too misleading.
Initialized and ComponentsCreated would offer better clarity on the state of the training job.
There was a problem hiding this comment.
My only concern is that we also support custom Initializers to fetch data and pre-trained model before running distributed training: https://github.com/kubeflow/trainer/tree/master/pkg/initializer
@kubeflow/wg-training-leads @Electronic-Waste @astefanutti @seanlaii @saileshd1402 @akshaychitneni @shravan-achar Any thoughts on how to make TrainJob conditions less confusing for users ?
There was a problem hiding this comment.
how about instead of "Initialized" and "ComponentsCreated".
We use "Created" and "Initialized"/"Active"?
I think this is more clear to users
There was a problem hiding this comment.
My recommendation is to replace Created with ComponentsCreated. For Active, this really should not be introduced since the active state is not a stable state.
This PR adds
InitializedandComponentsCreatedconditions to theTrainJobAPI to better track the state of job creation and initialization.Initializedcondition to indicate when the TrainJob has been initialized.ComponentsCreatedcondition to indicate when the components (e.g., JobSet, Jobs) have been created.