[Might Need Disscusion] Refine Time Limit

[FANGAreNotGnu]

Time Limit Implementation Strategies

issue #130

Core Principle

The core principle is that when a user specifies a time_limit, we should complete both training and prediction with the best available model, even if the time limit is reached. But as prediction time is hard to estimate, we have several options:

Option 1: Excluding Prediction Time from Time Limit (Current)

• Approach: time_limit applies only to training (end_of_fit - start_of_init), excluding prediction time
• Advantages:
  • Simple to implement
  • Maximizes training time utilization
• Disadvantages:
  • Will always exceed specified time_limit by the prediction duration
  • Unpredictable total execution time from user perspective

Option 2: Percentage-Based Allocation

• Approach: Reserve a percentage (e.g., 10%) of time_limit for prediction
• Advantages:
  • Scales proportionally with total time
  • Least likely to exceed time_limit
• Disadvantages:
  • May unnecessarily restrict training time for long time_limits
  • Example: With 12-hour time_limit, reserving 1.2 hours for prediction is excessive

Option 3: Fixed Time Reservation

• Approach: Reserve a fixed duration (e.g., 5 minutes) for prediction
• Advantages:
  • Predictable prediction time allocation
  • Works well for medium-sized datasets
• Disadvantages:
  • Requires minimum time_limit threshold
  • Inefficient for small datasets
  • May be insufficient for large datasets

Option 4: Hybrid Approach (Percentage with Cap)

• Approach: Reserve percentage of time_limit (e.g., 10%) with maximum cap (e.g., 20 minutes)
• Advantages:
  • Balances scaling and practicality
  • Prevents excessive prediction time reservation
• Disadvantages:
  • Introduces multiple hardcoded parameters
  • More complex to maintain and explain

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[tonyhoo]

/platform_tests
Platform Tests Output