v4.1.8 pre-release

[BJReplay]

@autoSteve has done his magic so this is no longer a placeholder. Amazing work on this massive release.

This has been released as a pre-release (beta) for early adopters to test first.

Details: #176 (comment)

[gcoan]

Placeholder Announcement for @autoSteve to announce the amazing work he has done on this massive release.

Sounds like a great release already, nothing to disruptive to have to cater for ;-)

[BJReplay]

Sounds like a great release already, nothing to disruptive to have to cater for ;-)

🤣

No, it shouldn't be disruptive.

If you have interesting timing for your automations for polling Solcast, you'll probably want to stick with them.

If not (or for new users), there's a lot to like.

[autoSteve]

Being "feature complete" is a bold statement to make, so I am not going to make that statement.

However this release will carry some hefty feature-completing stuff.

I can't wait. Final testing is happening.

There's quite a few things that have been improved.

Automated forecast updates

All users of this integration will have Home Assistant automation(s) set up to fetch forecast updates at desirable times of day. And that's great for the power users, who want an update just after midnight, then another strategic update pre-dawn, and then spread other updates up to API limit throughout the day (or not...).

The reason that all users have to do an automation now is because our forebears removed automatic updating when Solcast dropped their API limit maximum for hobbyist users to just ten per day.

It got too hard I guess.

It is not too hard. And it is done.

If you don't need to do anything special, then head into the integration config options and turn on 'Auto-update'. Disable or delete your automation. You can specify whether you want automation over twenty-four hours, or from sunrise to sunset. Too easy.

"But hang on! I want an automation to get one update just after midnight for PredBat, then I wouldn't mind taking advantage of this new auto-update thing. Can I?" Absolutely. Supposing you have two sites, and an API limit of 10, then set the limit in the integration config options to 8, turn on auto-update, then have your post-midnight automation do its thing (with a twist).

The twist is that there is now a new service call that must be used instead. This is force_update_forecasts, which basically ignores all configured limits. So change your midnight call to use this service. (As an aside, when auto-update is enabled the service call update_forecasts is ignored, and logged as such, so it must be changed.)

Full spread of attributes breakdown by site

Absolutely all sensor attributes may now be broken down by site, including the detailed half-hourly and hourly forecasts.

For why? I dunno. They're there for the taking, and for you to decide what you want to do with the data. How about build an Apex chart representation of each site forecast? Or build an automated per-site dampening automation? Your install, your things. (And tell us if you can't do what you want because the data is missing. We may have missed exposing something.)

All attributes sent to Home Assistant may be configured in config options. You choose the clutter. Turn them on or off.

The big attributes, like per-site detailed forecasts are not sent to recorder, so your Home Assistant database size will not suffer even with everything turned on.

Per-site dampening

Do you need to dampen just one site output in the forecast calculation? You can now. Check out the extended service call for set_dampening.

It is now possible to add an optional site variable to the call with the ID of the Solcast site, which will disable overall dampening and switch to per-site. Don't want per-site any more? Set the total dampening either in config options, or use the service call without specifying a site and it will reset to the total dampening settings.

Hard limit

Most of you don't know what this is. But it has been a thing, only configurable by service call. It is now in the config options dialogue. Don't know what it does? Ask the readme. You probably don't need it, but for those that do they will appreciate this.

Suppression of integration re-load on options change

Previously, should any config option be changed then the integration would simply re-load, because whatever. It does not now.

It is judicious.

It's a small thing, but I like it.

Basically, if the integration does not need to, then it will not re-load. Are selected sensor attributes changed? Fine. Just update the sensors and move on. Has dampening been adjusted? Fine, re-calculate forecast, splines, and then update the sensors. Change to auto-update forecasts? Re-load.

Legacy improvements

(Nerd alert 🚨) Some Home Assistant timers seemed to not be culled at times on re-load. This was usually an annoyance and waste of system resources. However, with auto-update of forecasts now implemented this has become super-important to avoid.

So it is avoided.

All coroutines that may be pending/running at shutdown are now gracefully cancelled.

In-code documentation

(Final nerd alert 🚨)

If you want to contribute to this brilliant integration, then you will be thankful for the extensive in-code documentation that has been added, describing functions, their input and return values, as well as the exceptions that they may raise. VSCode shows this stuff brilliantly, and I love it. (Your taste may vary, but please, bend to our taste.)

Any PRs in future will be expected to adhere to essential coding standards, along with a clean pylint bill of health and use of accepted Python coding standards, naming conventions and internal documentation. Community code shoud not just be about it "working", it must be about it being working and maintainable into the future, beyond any one individual's needs.

Set up the Pylint, Error Lens and Pylance extensions in VSCode for a substantial win.

[gcoan]

Well I thought it was a logical further extension to your enhancements. Should be possible to do in a further future release as another enhancement?

[BJReplay]

@BJReplay, have I done something stupid here?

No, I don't think so, @autoSteve.

Automation On or Off. One logical choice.
Sunrise to Sunset or 24 Hours. Another choice.

Putting it in a combo combines two different choices - whether or not to turn on automation, and the type of automation - into a single control. It hides options.

If you were to default automation to Sunrise to Sunset, then the question on "how do I turn off automation?" is hidden - you must actually click on and drop down the combo to find the hidden option to turn it off. That's bad design.

Even if you don't default to on, someone might come back a week, month or year later, and forget how to turn it off. You can't see how to turn it off, without dropping down the combo.

As @autoSteve has said, it's possible. And, as it is (almost certainly) he who would be writing the code, he could work out clever ways of doing it.

I, on the other hand, look at something like that, and say to myself "yeah, but how many people would really use this". Raymond Chen says "every feature starts off with -100 points" (I'm paraphrasing) - this is his way of saying that nothing gets built unless there is a compelling reason, ahead of things that add real value.

Harder things - like parsing date or time strings - that are notoriously hard - before you take into account localisation - that add little value over the fact that Home Assistant comes with it's own automation system built in that has it's own parsing and error reporting system built in for parsing dates and times - so why re-invent the wheel, when home assistant has this beautiful UI?

You can use helpers - as I have to control other things - like inputs

Or you can key in hard code times - and the UI validates them for you

It really couldn't get much easier.

And if you use up your API calls, the automation in the integration will cope with that gracefully.

[autoSteve]

Sure, we could implement a drop-down list for the auto-magic updates, and I actually like that.

I ended up liking it a lot.

This is done.

It results in a further options schema bump, which will see a couple of upgrades on upgrade in debug logging, but it's worth it.

It paves the way for future enhancement opportunities like you suggest @gcoan, with specifying some specific "extra" update times in options. (Parsing issues to deal with at a future date, as BJ points out, plus the "100 point rule"...) Implementing this also held up beta, but that hold also uncovered other issues that we don't want testers to hit, so that was a good call.

Testing has been extensive, and exhausing. Much API calls have been killed in its deliverance, including countless re-starts, re-config of option variants, waiting for UTC roll-over much times, plus quite a few fresh dev-instance spin-ups / roll-backs.

Like a fine wine needs time ... yeah, we need less than that, but you get the idea.

Quite a few days after the teaser, I reckon we'll be ready to release the beta real soon.

[gcoan]

Great news, looking forward to trying it out

[autoSteve]

But do please feel free to propose a PR for the "extra" time updates in options flow and coordinator.py once this is released (including Urdu translation)... I'm done for this week. 😂

[autoSteve]

As has become tradition, notifying the usual early-adopting-and-reporting folk. @dtswk, @EddieSpaghettie, @ffm777, @gcoan, @hwb45, @iainfogg, @isorin, @jeffb1974, @ladaan, @Markyt77, @Nilogax, @Noodleyman, @ProphetOfDoom, @rholligan, @sh00t2kill, @Unlik and @wimdebruyn.

[autoSteve]

Sunrise is kind of global pseudo-random when you think about it in the context of ~4000 known folks globally running this code, all with subtly different latitude and longitude, with forecast fetch at to-the-second precision...

(Except for @BJReplay. His scheduling seems off by eighteen seconds. 😂)

[ProphetOfDoom]

@autoSteve Just wanted to make doubly sure I was unique ;)

[autoSteve]

Lols. You're probably not now, conflicting with Sheryl in the next town!

We have pseudo-random back-off retries now for API busy times, so don't sweat the seven minutes.

[ProphetOfDoom]

@autoSteve I think I know Sheryl, I got the idea of using sunrise - 7 minutes from her, lol

[BJReplay]

Except for @BJReplay. His scheduling seems off by eighteen seconds. 😂

That's just on the occasion that you're aware of.

183 seconds yesterday for UTC reset.

The weird thing is that HA knows the time - it just doesn't fire events at the time.

[EddieSpaghettie]

All looking good to me!

[dtswk]

I disabled my scheduled agent and enabled the update in the config and like magic it updated. :)

[dtswk]

Sooooo boring... everthing is working. Back to my ESP32 project so I can deal with some broken stuff ;-)

[autoSteve]

Anyone else seeing things like this? Or am I just special?

[autoSteve]

Spook

I have.

Looks like it is a wider issue with 2024.10... home-assistant/core#127363

[gcoan]

I deliberately never install any of the HA .0 releases, it's too important for things to go wrong so I usually wait until the .2 patch release. Upgraded from 2024.8.3 to 2024.9.2 for example.

Life on the bleeding edge = too exciting. Let someone else debug it

[autoSteve]

Well, it's not. But there will be one heck of a lot of false-positive additions to that HA issue report in the days to come.

I am a dev, and I do dev things, like try out state_class variations for @gcoan, then revert. So yes, my state class had changed for these sensors, but that was a while back, and then I forgot all about it.

What has changed in 2024.10 is in-your-face visibility of the fact that the state class has changed for sensors at some point in their lifetime.

What does look broken in 2024.10 is pressing the DELETE button for the repairs does nothing. This was apparently fixed in a beta, but that would appear to not have stuck.

Developer tools fix for the win for me.

[BJReplay]

For anyone playing along at home, 2024.10.1 is out: https://github.com/home-assistant/core/releases/tag/2024.10.1

What's new, you ask?

If you have a Powerwall, with the Tesla Fleet integration (core), you can now control your Powerwall from within HA.

Imagine using Solcast and Tesla Fleet (and, maybe soon, predbat) to charge your Powerwall from cheap, free, or paid to charge power, depending on your tariff and retailer and pricing.

The five controls (top centre in the image above) are all controllable from within HA).

[gcoan]

If you have a Powerwall, with the Tesla Fleet integration (core), you can now control your Powerwall from within HA.

Imagine using Solcast and Tesla Fleet (and, maybe soon, predbat)

Excellent news for PW owners.

Predbat started off with GivEnergy inverters and has been extended to a number of different types (Solis, FoxESS, Ginlong, etc https://springfall2008.github.io/batpred/inverter-setup/). There have been people enquiring about Tesla integration but I'm not aware of anyone getting it working with 3rd party integrations.

This should make it a lot easier. There doesn't look to be a great deal of controls - no concept of charge/discharge start/end times, charge limit, charge/discharge rate - but simple set mode, charge/discharge now should be possible in Predbat 👍

[autoSteve]

I believe I have a concrete-ish retort for a lack of consistency of state_class settings in sensors, @gcoan.

My latest research indicates that if a sensor does not feaure a state class, i.e. state_class=None then its values do not flow to long-term statistics. I think. Therefore I think this is why they are set as they are.

[gcoan]

I believe I have a concrete-ish retort for a lack of consistency of state_class settings in sensors, @gcoan.

My latest research indicates that if a sensor does not feaure a state class, i.e. state_class=None then its values do not flow to long-term statistics. I think. Therefore I think this is why they are set as they are.

I think I'd counter argue that that is a happy coincidence of the state_class not being set rather than a deliberate design decision. I would argue that it would be better development practice to explicitly exclude a sensor from LTS if that's what's required rather than let it be excluded as a result of a side effect that may or may not continue in future HA releases.

Defensive coding and all that.

But who knows the mind of the original developer, but it gives us something to ponder over.

[autoSteve]

I actually agree, hence the -ish.

Because these sensors have never flowed to recorder I see nothing to be damaged by setting the state class appropriately and consistently.

It would warrant mention in readme regarding recorder settings to limit bloat, despite the sensor values being small.

I excluded about fifty sensors from mine for various integrations and the DB size dropped from 1.2GB to 350MB.

[gcoan]

I excluded about fifty sensors from mine for various integrations and the DB size dropped from 1.2GB to 350MB.

I've been doing the same over the last few weeks, investigating and reducing my sensor storage. I have reduced the HA backups down from 900+Mb to 500Mb and the HA db from 2.7 to 1.7Gb.

A lot of my reduction has come from purging sensor history more frequently and relying on LTS more. Also replacing noisy power monitor sensors with time based filter sensors and swapping template sensors to time triggered template sensors. All of which has dramatically reduced the sensor volume.

[gcoan]

Feedback on 4.1.8 beta:

A day after the announcement HACS advised me of the beta version being available, no action required on my part to toggle the beta switch on or off [prior releases I think I had to manually force an update, this time all OK]

Installed fine, rebooted and the system log shows a clean run, upgrading to configuration options 10, 11 and then 12.

Sensors as far as I can see are all working OK.

Looking at the new configuration options I can see that several have had the text expanded to make them clearer what they are and do (e.g. preferred forecast). All looks good. My existing config options have all transferred over OK and at the moment I don't see any particular need to change them (sorry Steve, all that new configurability and I'm not using it 😞)

Some questions/comments/observations:

Automatic updates over 24 hours I understand the principle, it's an auto update every 24//. What time is the first auto-update, is that midnight or just completely random (or horrors, potential enhancement for user configurable start time)?

Automatic updates over sunrise-sunset needs more explanation I suggest. And yes, I did read the readme. Again, how is this spread out and when does it start?
Say I have 5 polls and sunrise is 8am, sunset 6pm, so 10 hours between sunrise and sunset.
Is the first poll at 8am sunrise, then a poll every 2 hours so the last one is 4pm?
Or is first poll at 8am sunrise then a poll every 10/(5+1) hours so last poll is at sunset 6pm?
Or something different?

Enable site breakdown, my understanding is it creates site breakdown of hourly and/or half hourly detail depending upon what the prior configuration options are set. So if I have half hourly on but not hourly and I turn site breakdown on, I'll get only the half hourly site breakdown. Likewise if I don't turn on either of the detail toggles then turning the site breakdown on will do nothing. Possible extra logic checking in the configure dialogue to enforce what the rules are.

Further enhancement idea, enable configurability as to which site breakdowns are produced. Because of the way my sites and arrays are split I am only really interested in the detail for one of my sites/arrays which has the potential to clip. The other, not so interested in.
Happy this remains on the backlog as an idea, it's NTH because at the moment I'm not drilling into this (and with winter coming, is less of an issue)

I see mention in the dialogue that site breakdowns for half-hourly/hourly detail is explicitly not saved to LTS statistics.
I have to admit I thought LTS only stored the actual sensor value (and max, min and average) not any of the sensor attributes?
Am wondering whether this should be referring to not saving the site specific detail in the state_attributes table (which is subject to the normal 10 days HA purge frequency) not LTS?

And this brings me to a further request, I think I'd like more control over what attributes are saved to the state_attributes table. TBH I am not interested in storing the (non site specific) state attributes for the half/hourly forecast. The total daily forecast is useful and interesting (and maybe the peak time) but the site specific breakdown, nah, that was an estimate and it was the past, so not interested.
Having said that, maybe if I was graphing it (which I am not) then I might be interested up to a few days back, but certainly not my default 14 days of state history. Ugh, too much data noise for no benefit.

A great new release, thanks guys

[autoSteve]

new options

Glad you're not using them. We have a full spread of attributes now, and hardly anyone should be using all of them. But everyone should be able to choose what they need.

Auto-update times re. 24-hour / sunrise->sunset

The first poll of the day for 24-hour is 00:00. The first poll for sunrise->sunset is sunrise. Yes, 00:00 and on-the-hour subsequently has stuff-up written all over it, but I am tipping that the 24-hour option will be unpopular because so many retries encountered.

Could I have randomised it a bit? Yep. Did I? Nope. I considered it, but we have randomised back-off retries to cope. I will re-visit this if there is a need to.

For all auto-updates, time intervals are rounded down. i.e. If sunrise is 8am, sunset 6pm then there will be a call at 8am, but no call at 6pm. If you turn on DEBUG level logging then the scheduled calls are laid out at midnight for you in detail. Maybe these should be logged at INFO level, which is not a bad thought.

Site detail breakdown option

Correct. It would be foolhardy to expect a breakdown of something that one had not asked for in the first place, but I get the point. I had considered this but Home Assistant options flow is not smart enough to restrict a tick box if another tick box is not ticked, so I moved on.

enable configurability as to which site breakdowns are produced

This is possible, as is everything, but umm. That's a 'no' from me. Union rules. One out, all out. One in, all in. I am not entering my site IDs as a comma separated list anywhere, and the storage use is inconsequential because recorder suppression or tiny. And I am not translating that into Urdu. 😂

Recorder: not any of the sensor attributes?

LTS actually does store the attribute values, unless they are specifically excluded. I specifically exclude them. Recall your PredBat warning messages about recorder being overloaded with big fat stuff? Attributes be the answer there.

state_attributes table

I did not consider it because short-term, and pretty small in the scheme of things.

[gcoan]

Thanks for your detailed replies.

Auto-update times re. 24-hour / sunrise->sunset

The first poll of the day for 24-hour is 00:00. The first poll for sunrise->sunset is sunrise. Yes, 00:00 and on-the-hour subsequently has stuff-up written all over it, but I am tipping that the 24-hour option will be unpopular because so many retries encountered.

Could I have randomised it a bit? Yep. Did I? Nope. I considered it, but we have randomised back-off retries to cope. I will re-visit this if there is a need to.

I wasn't bothered about the randomisation at midnight, as you say there is code to cater for that, and if it gets such that the midnight call keeps on timing out due to too many retries, then agreed that'll be evidence for adding more randomisation.

For all auto-updates, time intervals are rounded down. i.e. If sunrise is 8am, sunset 6pm then there will be a call at 8am, but no call at 6pm. If you turn on DEBUG level logging then the scheduled calls are laid out at midnight for you in detail. Maybe these should be logged at INFO level, which is not a bad thought.

Suggest that a bit extra explanation of how these automatic schedules actually work in practice would be worthwhile to add to the readme.

Site detail breakdown option

Correct. It would be foolhardy to expect a breakdown of something that one had not asked for in the first place, but I get the point. I had considered this but Home Assistant options flow is not smart enough to restrict a tick box if another tick box is not ticked, so I moved on.

Cheers

enable configurability as to which site breakdowns are produced

This is possible, as is everything, but umm. That's a 'no' from me.

:-(

Recorder: not any of the sensor attributes?

LTS actually does store the attribute values, unless they are specifically excluded. I specifically exclude them. Recall your PredBat warning messages about recorder being overloaded with big fat stuff? Attributes be the answer there.

I'm sorry to say but I am pretty confident that you are wrong on this. vis, structure of the main statistics table:

CREATE TABLE statistics  (
  id INTEGER NOT NULL,
  created CHAR(0),
  created_ts FLOAT,
  metadata_id INTEGER,
  start CHAR(0),
  start_ts FLOAT,
  mean FLOAT,
  min FLOAT,
  max FLOAT,
  last_reset CHAR(0),
  last_reset_ts FLOAT,
  state FLOAT,
  sum FLOAT,
  PRIMARY KEY (id),
  FOREIGN KEY(metadata_id) REFERENCES statistics_meta (id) ON DELETE CASCADE
)

(and statistics_short_term is the same). All that is saved in LTS is the entity state and the min, mean and max of the state over the time period (1 hour for the main state table).

There is no statistics_attributes table and storing of state attributes in LTS, nor I add, no mechanism to even view/manipulate them in HA if they were squirrelled away.

The predbat warnings I had to deal with were about storing attributes in the database, not LTS:

State attributes for predbat.base10_export_energy exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored

Going back to the original comment, the labelling on the configuration dialogue refers to not saving things in LTS, it should refer to not saving things in state attributes which the developer is in control of.

state_attributes table

I did not consider it because short-term, and pretty small in the scheme of things.

For info, here's the number of rows in state_attributes for the solcast sensors:

(includes a few of my own template sensors)

None of them are particularly massive in terms of row count (my largest by comparison is the inverter battery power with 15,000 rows and as the Predbat sensors are refreshed every 5 minutes, each predbat sensor has ~6000 rows in state_attributes), but still, solcast is adding 16,000 records.
Is interesting that the most populous are next_x_hours, power_next_30_mins, power_next_hour. etc, the interpolated ones with site breakdown, many of which sensors I don't use so having seen this I will disable them.

It doesn't appear that the sensors with half hourly site breakdowns (pv_forecast_today, etc) are massively contributing to the number of rows in the state_attributes table, so I guess they are already suitably supressed.

Thanks

[autoSteve]

I'm sorry to say but I am pretty confident that you are wrong on this. vis, structure of the main statistics table

Yep. I'm wrong.

Readme updated. Translations updated.

I don't see the problem with 16,000 small records though. Percentage count shown vs. overall is a rounding error. I guess because bank where I work 16,000 small records are a drop in the ocean. 16 billion? Then I might pay attention to storage, especially if much larger records than these.

Those high count sensors update every five minutes, so that would explain the 'high' count. The storage needs would be tiny for each, though.

[gcoan]

Yep. I'm wrong.

👍

Those high count sensors update every five minutes, so that would explain the 'high' count. The storage needs would be tiny for each, though.

In the grand scheme of things I'm not worried about the solcast sensors, they're not storing massive numbers of rows in the database which is what I was first concerned about.

I've disabled the sensors that I am not interested in, because I might as well remove them, than keep them adding unnecessary storage. They weren't being retained in LTS only normal state history for 14 days, so I'm only losing short term history I never looked at anyway.

For comparison some of my sensors were holding 120,000 state change records (before I trimmed them down). I have a CT clamp over the AC feed from a non-HA integrated Solar inverter which was recording a new state record for every single watt of power change from the panels. Stupid amount of unnecessary data. Solcast not the problem child here.

Cheers

[BJReplay]

72 downloads so far. No bug reports. That's a good sign, @autoSteve.

[autoSteve]

Full of good bodings with so many lines of code touched.

Just wait until the code touchings in v4.1.9 or v4.2.0/1 see the light of day. 😂 I've not managed to detonate 🧨 anything yet. (There is no emoji for a cliche dynamite plunger, which is disappointing...)

(There is consistency of the day 1/2/3/4... sensors state_class in v.next, @gcoan, which might make a little squeal escape your lips having that off the to-do list. Other sensors remain devoid of state_class for now.)

[BJReplay]

101. Will do that PR review now. Have a feeling that it might be pre-release v4.1.9

[autoSteve]

v4.1.9 is set free. Closing.