Flat File Export and Daily Update Task

[MartinHinz]

Overview

This pull request introduces a new feature to enhance data export and streamline daily updates. The key additions are:

1. Flat File Export: If no query parameters are provided, the system will export data to a flat JSON file.
2. Scheduled Task: A scheduled task is set to update the JSON file daily at midnight.

Key Changes

1. Schedule Task Implementation:

   • File: schedule.rb
   • Code:

     every 1.day, at: '12:00 am' do
       runner "Data.store_data_as_json"
     end

   • Description: This task runs every day at midnight, invoking the store_data_as_json method to refresh the JSON data file.

2. Data Controller Enhancements:

   • File: data_controller.rb
   • Code:

     # Specify the file name you want to check in the public directory
     file_path = Rails.root.join("public", "all_data.json")
     
     # Check if none of the params keys start with "query_" and if the file exists
     if params.keys.none? { |key| key.to_s.start_with?('query_') } && File.exist?(file_path)
         send_file(file_path)
         return
     end

   • Description: This logic checks for the absence of query parameters and the existence of the JSON file. If both conditions are met, it sends the JSON file directly, significantly speeding up the export process.

3. Data Storage Method:

   • File: data.rb
   • Code:

     def self.store_data_as_json
         query = <<-SQL
           SELECT json_agg(json_build_object('measurement', subquery)) AS measurements
           FROM (SELECT "data_views".* FROM "data_views") subquery
         SQL
     
         result = C14.connection.exec_query(query)
         measurements = result[0]['measurements']
     
         File.open(Rails.root.join('public', 'all_data.json'), 'w') do |file|
           file.write(measurements)
         end
     end

   • Description: This method queries the database, aggregates the data into JSON format, and writes it to public/all_data.json. This file is then used for the flat file export.

Benefits

• Performance Improvement: By checking for existing files and using them when no query parameters are present, we avoid unnecessary database queries, thus speeding up the data export process.
• Automated Updates: The daily scheduled task ensures that the JSON file is always up-to-date with the latest data, reducing manual intervention and ensuring data consistency.

Note

The current bottleneck in the processing pipeline is identified as being within the xronos.r script. Further optimizations might be necessary in that area to fully leverage the improvements made in this pull request.

---

This enhancement not only optimizes the performance of the data export functionality but also ensures that the data is kept fresh with automated daily updates.