Feature Request: NULL value handling

[arriflex]

It would be pretty cool if there were an option to replace NULL values with a predefined one such as 0 in the config file. One of the csv files I have to deal with is full of nulls instead of zeros and I have no control over the generator.

[sankerust]

@arriflex Hello! Thank you for the suggestion!
Could you please specify what do you mean,
Do you have the values set to NULL in the table?
Or the table cells are empty and their value is considered as NULL?

[arriflex]

Good question! I should have specified. The csv I am being provided has no value between the commas rather than any value (including NULL) or zeroes. It has other issues as well, so I'm already having to massage it a fair amount. I've put the first raw 24 hours into a code snippet below for reference. Note also that there's also an extra line between each legitimate line.

Very unique to this csv is that it represents a year of hourly data from historically appropriate months so January might be from 1985 whereas March might be from 2003. Because I'm using this data to compare performance I use a regex find replace all the years with the current one. This isn't really something I would expect csv2influx to handle for me although it would be cool if it did.

Also, it looks like there's a potential feature request elsewhere for timezone support so I'll just mention here that I'd love the ability to set the timezone in the config file because this csv is in a local tz, but the timestamp isn't labeled as such.

`
hour_index,timestamp,global_horizontal_irradiance,direct_normal_irradiance,diffuse_horizontal_irradiance,dry_bulb_temperature,windspeed,albedo,solar_altitude_angle,solar_azimuth_angle,solar_incident_angle,horizon_elevation_angle,adjusted_ghi,poa_direct_irradiance,poa_diffuse_irradiance,poa_reflected_irradiance,shaded_direct_irradiance,shaded_diffuse_irradiance,shaded_reflected_irradiance,effective_direct_irradiance,effective_diffuse_irradiance,effective_reflected_irradiance,soiled_irradiance,total_irradiance,nameplate_power,avg_cell_temp,module_irradiance_derated_power,module_mpp_power,module_power,optimizer_input_power,optimizer_output_power,optimal_dc_power,optimal_dc_voltage,inverter_overpower_loss,inverter_underpower_loss,inverter_overvoltage_loss,inverter_undervoltage_loss,actual_dc_power,actual_dc_voltage,ac_power,grid_power

1,1991-01-01 00:00:00,0.0,0.0,0.0,1.1,0.0,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

2,1991-01-01 01:00:00,0.0,0.0,0.0,2.8,2.6,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

3,1991-01-01 02:00:00,0.0,0.0,0.0,2.2,2.6,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

4,1991-01-01 03:00:00,0.0,0.0,0.0,1.1,2.1,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

5,1991-01-01 04:00:00,0.0,0.0,0.0,0.0,3.1,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

6,1991-01-01 05:00:00,0.0,0.0,0.0,0.6,3.1,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

7,1991-01-01 06:00:00,0.0,0.0,0.0,-0.6,3.1,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

8,1991-01-01 07:00:00,28.0,10.0,27.0,0.0,0.0,0.19,3.51481,121.626,81.3,0.0,28.0,1.73864,26.5608,0.0425383,1.46448,26.4061,0.0425383,1.05361,25.2367,0.0135541,25.7778,25.7742,15072.0,0.81031,13027.5,14658.2,14192.4,0.0,0.0,14191.4,583.929,0.0,0.0,0.0,0.0,14191.4,583.929,13143.2,13142.3095703125

9,1991-01-01 08:00:00,153.0,112.0,127.0,3.3,2.1,0.19,13.3392,131.431,70.183,0.0,153.0,38.2048,139.307,0.232441,38.2048,138.495,0.232441,34.4804,132.362,0.0740635,163.578,163.572,95653.0,7.76465,93967.4,101795.0,97948.4,0.0,0.0,97905.2,609.16,0.0,0.0,0.0,0.0,97905.2,609.16,95125.1,95076.609375

10,1991-01-01 09:00:00,245.0,197.0,173.0,5.0,0.0,0.19,21.6531,142.991,60.5107,0.0,245.0,96.0543,185.184,0.37221,96.0543,184.105,0.37221,91.1005,175.952,0.118598,261.828,261.819,153104.0,13.2313,152874.0,161382.0,155995.0,0.0,0.0,155883.0,602.776,0.0,0.0,0.0,0.0,155883.0,602.776,152424.0,152299.734375

11,1991-01-01 10:00:00,300.0,177.0,217.0,7.2,2.6,0.19,27.8205,156.596,53.0904,0.0,300.0,106.805,231.528,0.455767,106.805,230.179,0.455767,103.253,219.986,0.145222,316.916,316.904,185314.0,15.5664,185934.0,194162.0,187695.0,0.0,0.0,187531.0,598.911,0.0,0.0,0.0,0.0,187531.0,598.911,183590.0,183409.265625

12,1991-01-01 11:00:00,462.0,466.0,221.0,10.0,2.1,0.19,31.1452,171.994,48.9649,0.0,462.0,305.914,246.128,0.701882,305.914,244.695,0.701882,297.912,233.859,0.223643,521.355,521.342,304868.0,24.2298,308051.0,309098.0,298573.0,0.0,0.0,298132.0,581.303,0.0,0.0,0.0,0.0,298132.0,581.303,292580.0,292120.71875

13,1991-01-01 12:00:00,465.0,464.0,226.0,11.7,2.1,0.19,31.1354,188.13,48.9781,0.0,465.0,303.381,251.4,0.706439,303.381,249.936,0.706439,295.439,238.868,0.225095,523.841,523.829,306318.0,25.9977,309525.0,308044.0,298668.0,0.0,0.0,298218.0,575.099,0.0,0.0,0.0,0.0,298218.0,575.099,292624.0,292164.65625

14,1991-01-01 13:00:00,400.0,446.0,192.0,12.8,4.1,0.19,27.7927,203.521,53.127,0.0,400.0,267.673,214.346,0.60769,267.673,213.097,0.60769,258.752,203.661,0.19363,453.354,453.344,265102.0,23.6397,267587.0,269252.0,260554.0,0.0,0.0,260220.0,582.146,0.0,0.0,0.0,0.0,260220.0,582.146,255257.0,254908.4375

15,1991-01-01 14:00:00,263.0,264.0,166.0,12.2,4.1,0.19,21.611,217.116,60.5645,0.0,263.0,129.431,181.628,0.399556,129.431,180.57,0.399556,122.734,172.574,0.127312,289.527,289.517,169300.0,19.1225,169501.0,174193.0,168375.0,0.0,0.0,168239.0,589.816,0.0,0.0,0.0,0.0,168239.0,589.816,164542.0,164396.484375

16,1991-01-01 15:00:00,132.0,208.0,84.0,11.7,3.1,0.19,13.2866,228.667,70.2484,0.0,132.0,70.5819,92.9706,0.200538,70.5819,92.4291,0.200538,63.6681,88.3361,0.0638979,149.027,149.022,87143.3,15.5059,85272.4,89212.2,86292.0,0.0,0.0,86255.9,588.028,0.0,0.0,0.0,0.0,86255.9,588.028,83284.0,83246.8125

17,1991-01-01 16:00:00,28.0,15.0,27.0,10.0,2.6,0.19,3.455,238.467,81.3731,0.0,28.0,1.72413,26.627,0.0425383,1.45242,26.4719,0.0425383,1.0409,25.2996,0.0135541,25.827,25.8236,15101.0,10.6818,13055.3,14027.5,13582.0,0.0,0.0,13581.0,557.457,0.0,0.0,0.0,0.0,13581.0,557.457,12547.2,12546.3203125

18,1991-01-01 17:00:00,0.0,0.0,0.0,6.7,2.1,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

19,1991-01-01 18:00:00,0.0,0.0,0.0,4.4,2.6,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

20,1991-01-01 19:00:00,0.0,0.0,0.0,3.9,2.6,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

21,1991-01-01 20:00:00,0.0,0.0,0.0,2.8,3.1,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

22,1991-01-01 21:00:00,0.0,0.0,0.0,2.2,2.6,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

23,1991-01-01 22:00:00,0.0,0.0,0.0,3.9,2.6,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

24,1991-01-01 23:00:00,0.0,0.0,0.0,2.8,3.1,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

25,1991-01-02 00:00:00,0.0,0.0,0.0,1.1,0.0,0.19,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

`