Possible issue with File Editor in web config

[DirkO4YB]

I'm not sure when a problem can be entered as "formal" issue, so let's first start a discussion.
I have a functioning set of pages on my plate, and even have done a custum 'extra' by reporting an extra sensor value (the temperature sensor on the Lanbon8 plate (mine has it). Every minute the plate sends an extra mqtt message to the broker with the present value. That all works, and stays working after a long period (12 hours and counting).
My problem arises when i try to start the File Editor in the web config system. After i hit the button on the form, the console shows:
�[1000D�[0KPrompt > �[1000D�[9C[08:45:30.680][ 3060/ 4912 37][44940/45172 1] HTTP: Sending 200 /edit.htm to client connected from: 192.168.0.149 �[1000D�[0KPrompt > �[1000D�[9C[08:45:30.871][ 1332/ 3056 56][44940/45172 1] HTTP: Sending 200 /static/script.js to client connected from: 192.168.0.149 �[1000D�[0KPrompt > �[1000D�[9C[08:45:47.205][ 1396/ 5272 73][44940/45172 1] MQTT: Disconnected �[1000D�[0KPrompt > �[1000D�[9C[08:45:47.218][ 1396/ 5272 73][44940/45172 1] MSGR: File not found: L:/mqtt_off.cmd �[1000D�[0KPrompt > �[1000D�[9C[08:46:02.231][ 1396/ 5272 73][44940/45172 1] MQTT: Transport error �[1000D�[0KPrompt > �[1000D�[9C[08:46:02.240][ 1396/ 5272 73][44940/45172 1] MQTT: Disconnected
So right after the plate sends (or tries to) the /static/script.js file, the connection with the mqtt broker is gone, and the connection with the web config pages/forms is also gone, right after an empty File Editor page (after waiting for 10 seconds) is presented (empty as in: the page is there, but no files are presented).

My suspicion is that the script.js file is somehow broken. I tried (via ftp) to upload the old File Editor (the standard Arduino one, from version 6.3.0; as edit2.htm), and that edit page is working.
(Other parts/forms of the web config are also working.)

I'm using v0.7.0-rc7_74d0f5b for building in PlatformIO, not sure if that is the latest, because i started with this version two week ago...

[fvanroie]

Your device sRAM is running on fumes: [ 3060/ 4912 37] is very low, so the question is what the configuration looks like.
Are you using TTF fonts or dynamic/PNG images? Maybe the custom sensor code has a memory leak....

Please post the Memory section from the Information webpage right after a reboot.
Then, monitor the sRAM in the log, it's the values in the second square brackets.
Does something eat up a lot of memory or are you seeing a gradual decline in available sRAM?
Since you are custom compiling, also make sure PSram is ebabled.

[DirkO4YB]

So memory is the problem... After a lot of trial and error...
To answer your questions first:
-I'm not using (TTF) fonts, just the standard ones.
-I -was- using a PNG image on the pages, i was aiming for a logo-banner on the plate by using a 240x60 pixel image, that has a size of 11.5 kb. But in my trials i have now removed the image, and use the standard pages.jsonl that comes with the Lanbon8 build.
After reboot the memory section from the information page says:

Device Memory
Free Heap 19.33 KiB
Free Block 17.98 KiB
Fragmentation 6%
PSRam Free 3.98 MiB
PSRam Size 3.99 MiB

which is allready lower than the log states:
[0KPrompt > �[1000D�[9C[15:58:58.472][24564/26108 5][45016/45036 1] MSGR: page=1
I suspect that is because producing the info page allso takes memory..

Then, when i am asking the pages.jsonl file in the file editor, the memory in the log drops:
[0KPrompt > �[1000D�[9C[16:01:57.056][21492/23604 8][45016/45144 1] HTTP: Sent /pages.jsonl page to 192.168.0.150
[0KPrompt > �[1000D�[9C[16:01:57.154][ 9716/14432 32][45016/45144 1] HTTP: If-None-Match: 0
[0KPrompt > �[1000D�[9C[16:01:57.164][21492/23380 8][45016/45144 1] HTTP: Sending 200 /pages.jsonl to client connected from: 192.168.0.150

Then, when i have set the mqqt server (a local ip adress) and the mqqt publishing starts (after reboot), the memory according to info page drops to:

Device Memory
Free Heap | 5.69 KiB
Free Block | 4.23 KiB
Fragmentation | 25%
PSRam Free | 3.98 MiB
PSRam Size | 3.99 MiB

At this moment the log states:
[0KPrompt > �[1000D�[9C[16:06:23.236][ 9204/10272 10][41340/44592 8] MQTT PUB: discovery => {"node":"plate01","mdl":"Lanbon L8","mf":"openHASP","hwid":"ac67b2f67b88","pages":12,"sw":"0.7.0-rc7","uri":"http://192.168.0.147","input":{},"power":[],"light":[12,14,27],"dim":[]}

From this moment i can still call the info page, but the file editor- pages.jsonl page causes the system to collapse: no wifi connection anymore.

My custum code is very modest, only this was added, meant to publish the sensor temperature once every minute:

void custom_every_5seconds() { //LOG_VERBOSE(TAG_CUSTOM, "5 seconds have passsed..."); //dispatch_state_subtopic("my_sensor","{\"test\":123}"); static byte minutecount = 0; if (minutecount == 20) { minutecount = 0; char buffer [6]; // send temp to controller itoa ( analogRead(39), buffer, 10 ); mqttPublish("tKamer", buffer, 6, false); } minutecount ++; }

What can i do to avoid running out of ram?
You said : "make sure PSram is enabled"
I thought this would be enabled in the Lanbon8 build, but to make sure i added -DBOARD_HAS_PSRAM in build_flags in platformio_override.ini.
Never worked with PSram before, so i don't know what else to do.
How can i put the image for my pages in this PSram?
Or even: how can i put it as a bin file (after converting it) in the pages.jsonl file?

And even without the image, is a combination of publishing to a mqtt server and use of the file editor to much asked from the Lanbon plate?

[nagyrobi]

Any free heap under 20k will lead to instability on ESP32. With any firmware binary.

[nagyrobi]

Also it would be worth dropping arduino, and go esp-idf instead. On ESPHome the switch made a huge difference, in heap free, responsiveness and stability.

Both options are still available there, as not all components are ported - but comparison can be made. Several new components are idf-only these days...

[fvanroie]

Likewise, you can't just drop Arduino and replace it with ESP-IDF and expect everything to work. Some libraries are Arduino only...

[nagyrobi]

Sure sure, was just my 2 cents...

[DirkO4YB]

Thanks for your replies and suggestions. Always good to start from scratch when your stuck. Here is what i did. I started at the beginning by taking a now clone from GitHub. Other than 4 weeks ago, this time RC8 came up. First i built the bin without alterations, to get a comparison, as you suggested. Indeed the free ram was over 50k. Then step for step added bits of custom parts, like my custom pages, custom sensor publishing and finally also the image banner. At this stage i am only seeing a little drop:

Device Memory

Free Heap | 47.33 KiB
Free Block | 43.98 KiB
Fragmentation | 7%
PSRam Free | 3.93 MiB
PSRam Size | 3.99 MiB

And (as it is way above 20k) the File Editor and other webinterface functions are working properly, without the wifi connection dropping.
Discussion can be closed now, although i am wondering why i got stuck before, with the same bits of custom code. The only thing that changed as far as i can think of, is going from RC7 to RC8. Perhaps some magic happened in between. I observe that the building is going a lot quicker now; before, it took more than 4 minutes, now it only takes less than 80 seconds.

Anyhow, i'm happy, everything works as i would like it to work . Perhaps my next trial will be to hook up a Dallas DS18B20 onewire sensor to IO39 but first i want to see how accurate the present sensor is (i suspect it is a thermistor, but it is not known which one).

Thanks again, Dirk

[DirkO4YB]

Final my 2 cents on environment, i prefer to stick with Arduino; i can understand the arduino libraries (like json..) a little, but i have none experience in ESP-IDF.