AVR folder removal appears to have caused issues with libraries

[Tim-ONT]

Hello,

Firstly, thanks for providing this core module! It works really well and has enabled me to get the MKR WiFi 1010 board working quickly!

In the 1.8.10 release, it appears the "avr" directory, which includes a number of basic files such as pgmspace.h, was moved:

$ git checkout 1.8.10
Previous HEAD position was b9b84b6 Release 1.8.9
HEAD is now at d5d336c Merge pull request #574 from giulcioffi/APIintegration
$ find . -name "pgmspace.h"
$ git checkout 1.8.9
Previous HEAD position was d5d336c Merge pull request #574 from giulcioffi/APIintegration
HEAD is now at b9b84b6 Release 1.8.9
$ find . -name "pgmspace.h"
./cores/arduino/avr/pgmspace.h

This has caused an issue in my project where I am using Arduino libraries which make the assumption that these files will exist.

For example, I have a simple program which uses the AceTime library:

$ arduino-cli core install arduino:[email protected]
Tool arduino:arm-none-eabi-gcc@7-2017q4 already installed
Tool arduino:[email protected] already installed
Tool arduino:[email protected] already installed
Tool arduino:[email protected] already installed
Tool arduino:[email protected] already installed
Tool arduino:[email protected] already installed
Downloading packages...
arduino:[email protected] already downloaded
Updating arduino:[email protected] with arduino:[email protected]...
arduino:[email protected] installed
$ arduino-cli compile rtc_example -b arduino:samd:mkrwifi1010
Sketch uses 29812 bytes (11%) of program storage space. Maximum is 262144 bytes.
Global variables use 4552 bytes (13%) of dynamic memory, leaving 28216 bytes for local variables. Maximum is 32768 bytes.
$ arduino-cli core install arduino:[email protected]
Tool arduino:arm-none-eabi-gcc@7-2017q4 already installed
Tool arduino:[email protected] already installed
Tool arduino:[email protected] already installed
Tool arduino:[email protected] already installed
Tool arduino:[email protected] already installed
Tool arduino:[email protected] already installed
Downloading packages...
arduino:[email protected] already downloaded
Updating arduino:[email protected] with arduino:[email protected]...
arduino:[email protected] installed
$ arduino-cli compile rtc_example -b arduino:samd:mkrwifi1010
In file included from /Documents/Arduino/libraries/AceTime/src/AceTime.h:20:0,
                 from /Documents/arduino/rtc_example/rtc_example.ino:16:
/Documents/Arduino/libraries/AceTime/src/ace_time/common/compat.h:38:12: fatal error: avr/pgmspace.h: No such file or directory
   #include <avr/pgmspace.h>
            ^~~~~~~~~~~~~~~~
compilation terminated.
Error during build: exit status 1

The offending lines in that library are:

#elif defined(ARDUINO_ARCH_SAMD)
  #include <avr/pgmspace.h>

which makes it very clear that this path is expected to exist. I have similar issues with other libraries and have seen this mentioned on forums as a reasonable workaround for Arduino libraries which need to work over a wide range of boards/platforms.

So my questions are:

• why was this removed?
• can it be reinstated as it appears to break numerous libraries?
• if not, what is the recommended upgrade path?

I hope that the information I have provided makes sense, please let me know if further diagnostics or information would be of use to you. I have obscured some of the file paths slightly for privacy purposes.

Thanks,

Tim

[facchinm]

Hi Tim,
thanks for reporting!
<avr/pgmspace.h> is now part of ArduinoCore-API repository, which gets merged with this samd during core release.
The issue comes from platform.txt which does not include deprecated-avr-comp (we were a bit too optimistic and didn't add it 🙂 ).
Next SAMD release will fix all these issues

[Tim-ONT]

Hi Martino,

Thanks for responding so quickly!

As with some of the other api tickets on that link, downgrading to 1.8.9 is working for me at the moment.

I don't know enough about platform.txt to work on a PR right now, but is there anything else I can do to help?
And should I leave this ticket open?

[matthijskooijman]

Note that you could make an argument that all these avr-specific header files are not part of the official, documented Arduino API, so any sketches that use it limit themselves to being AVR-only.

Some elements of these header files are part of the official API (i.e. PSTR(), PROGMEM, etc.), but those are exposed by Arduino.h (at least usually, maybe not on all platforms). From that perspective, one could argue that any portable code that needs these things should just include Arduino.h and not the avr/* headers directly.

However, the name deprecated-avr-comp suggests that maybe this directory is intended to be made available now, with some deprecation warning maybe, to be removed from the include path later?

[Tim-ONT]

Hi Matthijs,

I'm not including these files in my sketch directly, they are included from within libraries I am using.

From an initial search, this error is not uncommon and the pattern of achitecture/platform specific imports appears to be seen to be a reasonable workaround (see here and here).

I will look into changing the header in the libraries I am using to Arduino.h though - are there any resources/best practices/guidelines avaliable for portable Arduino libraries?

Thanks,

Tim

[davetcc]

If you want to move this to a new issue, please feel free, but it's still very related to this one.

To be honest, I think there's a lot more to it than just this one change, the new core API appears to be incompatible with many libraries in the wild, this will hit many beginners of Arduino using libraries, and I'm sure that's not what's intended here - especially with this being the core for MKR and Zero that are very commonly used. I can see what you're trying to do and agree with the idea, but the changes are probably going to break a lot of libraries.

Three examples, my libraries IoAbstraction and TaskManagerIO did not compile because of the PinStatus change on attachInterrupt - this one is easily fixed with a check for the API version define. However, AUnit no longer compiles either for many reasons. This is just from my locally installed libraries.

To name a few incompatibilities I found this morning (and this is not exhaustive):

• __FlashStringHelper and Print have moved to arduino namespace, meaning anyone who had previously forward referenced it in a library now does not work - at least Aunit.
• WString not included by default although many, many libraries still use it. probable platformio problem not picking up deprecated directory, this in itself is an issue as many Arduino users use platformio.
• attachInterrupt mode parameter moved from uint32_t to PinStatus, and the enum is not of type uint32_t (why not enum PinStatus: uint32_t { ... } instead?
• avr/pgmspace.h , at least AUnit from what I have locally, probably many, many more.

If this is now the default, is this good timing when so many newcomers will be using these boards at Christmas?

[davetcc]

I will make sure TaskManagerIO and IoAbstraction are updated today so that they compile again in a backwards compatible way, for the benefit of other library developers hitting this, I assume that the following is the way to determine if a board's core has moved to the core API: #if (ARDUINO_API_VERSION >= 10200)

If not could someone correct me please with a better approach?

[JAndrassy]

I will make sure TaskManagerIO and IoAbstraction are updated today so that they compile again in a backwards compatible way, for the benefit of other library developers hitting this, I assume that the following is the way to determine if a board's core has moved to the core API: #if (ARDUINO_API_VERSION >= 10200)

If not could someone correct me please with a better approach?

I don't have problems with my libraries after this change, maybe because I made them compatible with megaavr and mbed cores, which use the core API from the beginning.

always use #include <Arduino.h> and not partial includes from the core
this helps with 3th party cores too

[davetcc]

I don't have problems with my libraries after this change, maybe because I made them compatible with megaavr and mbed cores, which use the core API from the beginning.

Task manager and IoAbstraction are compatible with Uno, Mighty, and Mega. Fully tested with unit tests for each release - even with extra tests for AVR! Unfortunately, Aunit (a library I use for unit testing) seems to be very badly hit by these changes.

Some APIs are not the same on different boards, if we take a look at attachInterrupt, in this case the mode parameter is a mix of values depending on what board you pick, some cores used uint8_t, some used uint32_t, some used InterruptMode, now some use PinStatus too. It's a bit of a minefield, and hopefully longer term, the API will make it better.

The changes in my libraries were very small and will be released today, I pretty much stick to the rules where they are set, but there are still some edge cases, like attachInterrupt tricky to deal with.

[facchinm]

First of all, thanks everyone for the discussion and for bringing this up.
To clarify:

• we are sorry for the inconvenience we caused to many third party developers; maybe the timing was wrong and surely the "compile examples" CI is missing a lot of libraries. Anyway, we need this transition to finally enable the ecosystem consistently for all platforms, getting rid of years of dust.
• Most of the issues are due to undocumented "features" that have been exploited during the years. For example, all the issues with @bxparks' libraries are due to forward declarations (which clash with the new arduino namespace). About this, I'm attaching the patches that I'll forward to the proper repos (AUnit, AceTime and so on)
  patches.zip
  You can see that the patches simply replace the forward declarations with "Arduino.h" inclusion. This has always been the cross-platform compatibility layer; in fact the fix doesn't even need to be protected by ifdefs because it was the intended way to expose the platform declarations.
• About partial includes, we aknowledged that they are quite frequent, particularly in older libraries, so we kept the compatibility with headers that, in fact, had the same name just by a chance (eg. WString.h is not supposed to exist in a core, as opposed to Arduino.h which is compulsory). In this case, we add the proper include path in platform.txt, so if platformio is not using it the bug should be filed on them.

@davetcc to accomodate different signatures you can just check for #ifdef ARDUINO_API_VERSION

[Tim-ONT]

Thanks for the help @facchinm!

[davetcc]

Many thanks for picking this up so quickly. I have released both IoAbstraction and TaskManagerIO libraries now, tcMenu was fine. As far as I can tell, all the libraries are working on all supported boards and should be safe if other cores move to the new API.

[bxparks]

Hi,
I would like to address some of the issues raised by this ticket, which I became aware of only when I was directly referenced by @facchinm. This post became a lot longer than I had planned, so I reorganized it into 3 parts:

• Part A contains specific answers to why my libraries are written the way they are, and why I hesitate to apply the patches sent to me by @facchinm.
• Part B contains more general comments about the overall Arduino-API migration.
• Part C contains my plan of action related to this ticket, and other bugs like this.

Part A: Forward Declarations

The patch provided by @facchinm replaces forward declarations like:

class Print;
class __FlashStringHelper;

in my various *.h header files with:

#include <Arduino.h>

There are 2 reasons I explicitly use forward declarations in my header files.

1) Working Around Bugs in the Arduino API

There are a number of cases where bugs of a particular platform's <Arduino.h> must be clobbered and fixed for my libraries to work properly. To do that, I must make sure that <Arduino.h> is included before my patch that fixes the problem, like this:

#include <Arduino.h>
#include "compat.h"

If in the chain of header file includes, the order is reversed, the compiler generates an error, such as a #defined being defined multiple times.

There are 2 examples that I can think of the top of my head (there may have been more):

a) For an Arduino Zero, that platform defines SERIAL_PORT_MONITOR incorrectly as Serial instead of SerialUSB, even when the "Arduino Zero (Native USB Port)" variant is selected. The fix is to #undef and redefine it like this

#if defined(ARDUINO_SAMD_ZERO)
  #undef SERIAL_PORT_MONITOR
  #define SERIAL_PORT_MONITOR SerialUSB
#endif

b) For an ESP32, until recently, the FPSTR() macro was broken, returning the wrong type. The solution was to #undef and clobber it:

#if defined(ESP32)
  #undef FPSTR
  #define FPSTR(p) (reinterpret_cast<const __FlashStringHelper *>(p))
#endif

I found that if I included <Arduino.h> in my *.h header files, it seemed impossible to guarantee that the <Arduino.h> was included first before my "compat.h". (The order of inclusion is too complex to track manually, I have 55 header files in the AceTime library, some of them programmatically generated by code-generators, with inter-related dependencies. Far too complex to keep in my head all at once most of the time.) I solved the problem by following the simple rule that none of my *.h header contain the <Arduino.h> unless absolutely necessary, and only the *.cpp files should include the <Arduino.h>.

It might be possible to solve this inclusion order problem by following the rule that every instance of #include <Arduino.h> should be followed by a #include "compat.h", but this leads to the following problem that I was also trying to solve...

2) Avoiding Unnecessary Dependency and Reducing Compile Time

The validation test suite for the AceTime library currently takes about 40 minutes to run on a quad-core i7 machine with 32GB of RAM. The removal of all of the <Arduino.h> from my header files was an attempt to reduce the compile times. If I add them back in, I don't know how much it will increase my compile times.

It seems wasteful to include the entire <Arduino.h> when I need just a declaration to __FlashStringHelper for example. I mean, __FlashStringHelper is not even a complete type; it exists only as a forward reference. My programming style is that the *.h header file should contain only the bare minimum necessary for something else to use the corresponding *.cpp file. The vast majority of my header files need just 1 or 2 forward declarations.

3) (Appended) Avoid Leaking "compat.h" Externally

Edit: I just remembered a 3rd reason why I had to take out the <Arduino.h> from my header files

My "compat.h" fiddles around or clobbers existing definitions inside <Arduino.h>, by its very nature and purpose. But compat.h should be an implementation detail, I did not want it to leak outside of my libraries, where it would (and did) interfere with the other people's library or code. So if I included <Arduino.h> in my header file, I would have had to include "compat.h" immediately after it. But my uber header files (e.g. <AceTime.h>, <AUnit.h>, <AceButton.h>, etc) aggregate and export my implementation header files, and would cause my "compat.h" file to be exported to the outside world. The solution was the same: Never include <Arduino.h> in my own header files, unless absolutely necessary; and of course, never include "compath.h" in my own header files, only in .cpp files.

I spent dozens of hours of trial and error, to work around these various bugs on various platforms. This all worked fine, until now.

Part B: Arduino API Migration

I often read comments from some people, saying that the problems experienced by 3rd party libraries are caused by violation of some "official Arduino API" or the use of some "undocumented" features. I find these comments to be unhelpful.

As far as I know, there is no official Arduino API that is sufficiently detailed and technical enough for 3rd party library developers to write non-trivial libraries. The Arduino Reference (https://www.arduino.cc/reference/en/) is essentially useless. For one thing, none of the function/method parameters have any type information (int, long, uint16_t, etc). For another, the word "Arduino.h" does not appear anywhere in that Reference.

The only way for 3rd party library developers to figure out how to write Arduino libraries is to learn from the examples of other 3rd party libraries, then dive into the source code of Arduino.h, then try to reverse engineer the intended purpose of all those thousands of lines of code. Most of that code is undocumented and uncommented. To make a library work across multiple platforms (e.g. AVR, MegaAVR, ESP8266, ESP32, SAMD, Teensyduino, STM32duino, ...), 3rd party developers must dive into the implementation details of each of those other platforms, then try to extract common patterns of features, API signatures, and functionalities among those platforms, most of which are undocumented.

I feel like 3rd party library developers should be given more respect and credit for being able to navigate the numerous variants of the Arduino API across different hardware platforms, finding creative workarounds for various bugs, and then producing libraries which are useful and reasonably robust. Instead, we are constantly admonished for not using the "official Arduino API" or using "undocumented features", whatever that means, since most technical information can be found only by reading the implementation code. Perhaps it would be more helpful to improve the technical documentation of the Arduino API to define exactly what those features are.

So maybe the Arduino Core API project is attempting to provide that "official API". But I don't understand this project at all. It does not document the current Arduino API. It makes sweeping backwards incompatible breaking changes. Why is digitalRead() and digitalWrite() using new PinStatus and PinMode enums? This breaks every Arduino program ever written using those functions. What is the purpose for moving all the C++ classes into the arduino namespace, since this will break huge amounts of existing code? It breaks forward declarations like mine that work in every other Arduino platform that I am aware of. There are probably other breakages, but I don't know what I don't know. All I get are drip-drip-drip bug reports filed against my libraries telling that some random thing is broken on a MegaAVR, or on a SAMD in this case.

How can we improve this situation? Unless the following document already exists, I think it would be immensely helpful if there was an authoritative design document that contained the following information:

• What changes to the API are being made? I know a few breaking changes:PinStatus, PinMode, digitalRead() and digitalWrite(), and moving classes into namespace arduino {}. But there must be more.
• Why are these changes being made? I'm sure the Arduino devs have valid reasons, but those reasons are mysterious to me.
• Will other platform maintainers (e.g. ESP8266, ESP32, Teensyduino (@PaulStoffregen), STM32duino, MegaCoreX (@MCUdude), etc) also migrate to the new API? If so, when? If not, why not? What is the implication of having 2 fragmented ecosystems of Arduino platforms, one based on the old API and one based on the new API?
• There seems to be at least a couple of 3rd party Cores which have dependencies to one of the Arduino Cores. For example, I believe the SparkFun SAMD Core depends on Arduino SAMD Core, and the SparkFun AVR Core depends on Arduino AVR Core. Will these downstream Cores break when the corresponding Arduino Core is migrated to the new Arduino API?
• What guidelines do you have for 3rd party library developers to support the new Arduino API? To be more blunt, what's in it for 3rd party library developers? Why should we spend time and effort to support this new API?

Part C: My Plans for the New Arduino API

Before this issue, I had decided to blacklist the MegaAVR platform for my libraries. In addition to using the new Arduino API, the MegaAVR breaks the F() macro, whose main purpose is to return the __FlashStringHelper* pointer type, but on that platform, it returns the wrong type. How am I supposed to work around that? I don't even have a MegaAVR board, so I can't test anything.

It looks like the SAMD is migrating to the new API, so my current inclination is to blacklist that too. I don't use the SAMD. I have only a single hardware board that I can test it on, and I would be very happy not to deal with the broken SERIAL_PORT_MONITOR on that platform.

I don't know if the AVR architecture will move to the new API, but if it does, I may end up dropping support for that too. I have only a limited amount of time and energy. Currently, I am mostly invested in the ESP8266, ESP32, and potentially the STM32duino. If those remain on the old API, then that's where I will spend my time. At the end of the day, it's all about whether I enjoy working on this. And fixing obscure backwards compatibility breaking changes on hardware that I cannot test on is not something that I enjoy.

And finally, I would like to ask for some lenience if I have misunderstood something fundamental, or am lacking some historical background of the Arduino project. I started playing with Arduino less than 3 years ago, so anything that happened before then is not familiar to me.

[JAndrassy]

only some comment from a 3rd library developer:

• https://www.arduino.cc/en/Hacking/LibraryTutorial
• I recommend Nano Every for megavr
• I started 3 years ago with Arduino too
• I welcome the common API event with its problems

[davetcc]

This is maybe not the best place for such an open letter, but it is public and easily available for anyone who wants to read it.

There are many users of the Arduino eco-system from beginner right through to seasoned developers like myself (background in assembly, C, C++, Software design, C#, and Java).

Regardless of skill level, we all want the same thing, a stable, well-documented eco-system where libraries work on all boards, and I think the API has the ability to help get there, but more care is needed around backward compatibility to avoid making things worse in the interim period. To have a chance of gaining traction, the API needs to be the easiest option for forks, otherwise, they will not take it and we will all end up in an even worse position with another variant to support, with library writers taking the worst of it.

You need to understand that a lot of the good-will that you have, and the size of the developer community is down to the wide range of generally high-quality libraries written by many people such as @bxparks, @ladyada et al, and even my humble offerings. It's then further widened by the other boards offered by SparkFun, AdaFruit, Seeed, Espressif, Teensy, etc. If there's a serious effort to make that API viable, then someone needs to reach out to some of the most popular forks and ask them to take the API too. Is there a need for a new forum category for the discussion of libraries and the API? I don't really know how third-party developers are seen internally, so It is hard for me to determine.

I agree with a lot of what @bxparks has said, traditionally, it was impossible to work out what the Arduino API was, many things were found either by searching google or looking at the code. I think even the new API is lacking documentation quite honestly. Also, I think that deprecation should be done in a more reliable way with at least a few months' notice. Personally, for my projects, I document all classes and files and use Doxygen to generate a series of HTML pages.

I'm trying my best not to "tell people how to drive on the public highway here" but this breaking change was made in a patch release version, this should have been 1.9.0 at the very least if not 2.0.0 quite honestly. It was released a few days before Christmas when many. many new MKR and Zero users would be expected, and quite honestly on my MKR1300 even an AdaFruit ST7735 SPI display didn't work properly on 1.8.10; which would certainly be the starting point after "blink" for many users.

If things are genuinely deprecated a comment would not get through to the end-user:

// including Server.h is deprecated, for all future projects use Arduino.h instead

I think the following would be better:

#warning("DEPRECATED: You have included server.h that is now deprecated, please see the Arduino guide")

Then when it's finally no longer supported, for a release leave it in place and move to #error with instructions on how to remediate.

For functions, methods, and classes that are deprecated, if you used Doxygen, it has a deprecation tag that many IDEs pick up. In CLion/platformio (and I think VSCode with C++ support enabled) anything deprecated is highlighted in the IDE.

As a company, we're fully committed to supporting as many Arduino boards as we can with our libraries, even though it's sometimes hard to do, from Uno to dual-core RTOS boards. Further, I'll always be grateful to the Arduino framework for getting me back into embedded development; which started for me about 2013/4, when I first started other options still needed special hardware, lacked any abstraction, and had no supporting libraries.

Dave Cherry - thecoderscorner.com

[bxparks]

So, I was going to start blacklisting arduino:megaavr and arduino:samd (>= 1.8.10) for my libraries. Then I received a note from @ubidefeo that he used one of my libraries (AceButton) in a recent Arduino CLI tutorial on YouTube . I have mixed feelings: I'm glad that Ubi liked my library enough to use it as an example. But I now feel obligated to make sure that my library continues to work for those users who watched the video. The new samd core version >=1.8.10 does not break AceButton directly, but it breaks AUnit which is how I unit test and validate my library across the supported hardware platforms.

My plan now is to tweak things with the least amount of change, which I think has the highest chance of success: I'm going to replace my forward declarations with something like this:

#if defined(ARDUINO_API_VERSION)
namespace arduino {
class Print;
class __FlashStringHelper;
}
#else
class Print;
class __FlashStringHelper;
#endif

(If I recall, C++11 does not allow forward declarations with just class arduino::Print).

The trouble with replacing the forward declaration with #include <Arduino.h> is that it changes the order of header file inclusions, and I have seen too many problems occur when the header file ordering was changed. I want to do the smallest change that will make my libraries work under the new Arduino API.

Apparently, I have 236 *.ino files which need to be verified with these changes. Maybe some are trivial to verify, maybe some are harder. I don't know, and it will take time to go through them. I currently have no hardware where I can actually run these programs using the new Arduino API. Sometimes (though not often), it is not enough to just verify that things compile, sometimes, the program must be run on actual hardware. I guess I will have to spend $40-80 buying a few of the newer boards that use the new Arduino API. Many of the less expensive boards seem out of stock or hard to get right now. Given my current schedule and what's on my plate right now, I'm going to guess this whole process will take about 6-8 weeks.

Hopefully, this will take care of the samd >=1.8.10 boards. The megaAVR seems hopeless. I have too much code that depend on the F() macro to return a const __FlashStringHelper*, which it does not on the megaAVR. Which is too bad, because the megaAVR seems like a really fun microcontroller on paper.

[ubidefeo]

@bxparks
I absolutely didn't think about the implications of using your lib in a demo of our CLI. My bad.
I really thought you were developing AceButton for the Arduino community.
I think @per1234 and @facchinm could be of assistance here giving you some directions,
especially regarding this one

Hopefully, this will take care of the samd >=1.8.10 boards. The megaAVR seems hopeless. I have too much code that depend on the F() macro to return a const __FlashStringHelper*, which it does not on the megaAVR. Which is too bad, because the megaAVR seems like a really fun microcontroller on paper.

Also, @bxparks , you should take a look at our new tool for project/library/platform validation
https://github.com/arduino/arduino-lint which might be useful in your tests and CI

[davetcc]

@bxparks I've extensively used your excellent aunit library for many of my open-source projects. I would try my best to help out in any way needed if it would help you to support the core API, and not drop support for platforms. Honestly, my only other option would be to change unit testing frameworks. Please feel free to reach out to me on github, or using the help button on the lower right of my site (https://www.thecoderscorner.com) if you think an additional tester / contributor would help matters for you. I work with Mega, Mighty, SAMD, Nano BLE, ESP8266, ESP32 on Arduino, and STM32 using mbed.

It sounds like the mighty core is in bad shape and may not be supportable as it is, but last time I checked I don't think it's maintained by Arduino directly. Maybe step 1 for that is to raise an issue in their repo. That comes back to my original point - unless a majority of board maintainers are brought into this effort, it will not result in much improvement for 3rd party library developers. sorry I'd misread mighty for megaAVR Ignore this paragraph..

[ianfixes]

Subscribing for interest, I have a unit testing library in which it's unclear how far the library mocks should extend.

[bxparks]

@ubidefeo: Hi, when you wrote, "I really thought you were developing AceButton for the Arduino community", I think you meant to write something different than what you actually wrote, but I'm not exactly sure what you intended to write. I'm sure it was benign though. I mean, I created AceButton mostly for myself, but I published it for the benefit of the Arduino community. I think I can say honestly that the documentation of the library, the testing, and quality of the library is far above average, and a lot of that was for the benefit of the community. I do this on my free time, I don't get paid for this.

Oh, btw, the https://github.com/arduino/arduino-lint link is a 404. Maybe it's still private? I tried searching for it on GitHub and on search-engines, but couldn't find it.

I think we would all agree that having a consistent Arduino API is beneficial to everyone. Ubi's video is a good example. He uses my AceButton library on a Nano 33 IoT. I don't own that board and I have never validated AceButton on it. However, last year, I decided that I should officially support the SAMD21, so I got a couple of cheap SAMD21 M0+ clones on eBay, which claimed to be "Arduino Zero compatible", but of course, I could not get arduino:samd:zero to work on it (nor arduino:samd:mkrzero). But the SparkFun:samd:samd21_mini core works for it. And because the Nano 33 IoT board (prior to arduino:samd version 1.8.10) used the same API as the SparkFun:samd:samd21_mini board, my AceButton library was compatible. SparkFun gets a library that's been tested on their Core. Arduino gets a library that works on the Nano 33 IoT. Everyone is happy.

But the new Arduino API breaks that compatibility, for reasons which aren't clear to me. Yes, there is some attention to backwards compatibility to older Cores, e.g. in the "Compat.h" file for PinStatus and PinMode. But I don't see how we get forwards compatibility, for lack of a better term. In other words, new Arduino sketches using PinStatus or arduino::Print will not work on boards using 3rd party Cores which are based on the older Arduino API. It seems like we have bifurcated the Arduino Core ecosystem. For 3rd party library developers like myself, I'm not sure that I would ever use the new Arduino API, because my code will be incompatible with most of the other 3rd party Cores. So I have to keep using the old Arduino API, and hope to hack enough things together to be compatible with the new Arduino API.

I realize that the Arduino devs are all busy, overloaded, with competing priorities and deadlines. But I think it would help 3rd party developers significantly if we got more technical guidance on this new Arduino API. I think this is a reasonable request. I'll make a concrete suggestion: There is no longer a publicly editable wiki on the https://www.arduino.cc/reference website. But GitHub has a wiki for each project, so the wiki at https://github.com/arduino/ArduinoCore-API would be a convenient place to capture information about the new Arduino API.

@davetcc: Thanks for your offer to help, I may take you up on it for validation testing on boards that I don't have access to. Two quick questions for you:

1. I want to avoid spending more money on new boards than I have to, especially since I'm not going to use them for anything other than testing my libraries against the new Arduino API. Which SAMD boards are the most popular? I get the impression that the Nano 33 IoT is one of them. The Arduino MKR product line is very confusing to me. If I validate on a Nano 33 IoT, do I need to test on an MKR board?
2. You mentioned STM32 on mbed. I just received a batch of STM32 boards, and I was going to validate my libraries for the STM32duino (which I just verified is on the old Arduino API). Is there some reason that you use mbed instead of STM32duino? (I know very little about mbed or the STM32.) Thx.

[ubidefeo]

hey @bxparks

Hi, when you wrote, "I really thought you were developing AceButton for the Arduino community", I think you meant to write something different than what you actually wrote, but I'm not exactly sure what you intended to write.

let me rephrase that: also making sure that it would be compatible with the needs of the Arduino community.
I have used the library and it is certainly not immediately accessible to novices

void handleEvent(AceButton*, uint8_t, uint8_t); the Pointer is scary to them
case AceButton::kEventPressed confuuuuusiiiiing :D

I really meant you were putting effort into making it compile successfully on Arduino

Oh, btw, the https://github.com/arduino/arduino-lint link is a 404. Maybe it's still private? I tried searching for it on GitHub and on search-engines, but couldn't find it.

oops! I might have given that away 24-48hrs before time.
It's coming, hope people will appreciate it :)

For 3rd party library developers like myself, I'm not sure that I would ever use the new Arduino API, because my code will be incompatible with most of the other 3rd party Cores. So I have to keep using the old Arduino API, and hope to hack enough things together to be compatible with the new Arduino API.

You have a fair point there, Brian.
I would invite @facchinm to get back in the conversation because API is not my realm inside Arduino.
I make sure the tools help people develop, but when it comes to the framework and its APIs I don't have a saying :)

Which SAMD boards are the most popular? I get the impression that the Nano 33 IoT is one of them. The Arduino MKR product line is very confusing to me. If I validate on a Nano 33 IoT, do I need to test on an MKR board?

I'll chime in on this, although the question is to @davetcc
SAMD21 and SAMD51 are fairly common microcontrollers used by Arduino, Sparkfun, Adafruit, Seeed and many more.
The MKR family is all based on SAMD21, same as the Arduino Zero and Nano 33 IoT

cheers.ubi

[davetcc]

Following on from @ubidefeo, I think most official Arduino SAMD boards use pretty much the same core, although someone with more knowledge please correct me if I am wrong. I have an MKR1300 board with an MKR ethernet shield attached, it should be good enough for testing. Along with this I also have a Seeed SAMD21 device with BLE connectivity - because it's used by one of my clients and therefore I need like for like.

Sorry to bring mbed into the picture, it's there because all of my libraries need to work on mbed too, but I don't run the unit tests there as I can't presently, so I have at least a smoke test project instead.

I would invite @facchinm to get back in the conversation because API is not my realm inside Arduino.

It would be very good if someone from the Arduino team could give us more details on the plans. It would help at least with expectations around where this API is heading. IE: is there a plan to work with the various third party cores such as ESP, Seeed, Mighty etc to get them to adopt it too?

[bxparks]

@ubidefeo:

let me rephrase that: also making sure that it would be compatible with the needs of the Arduino community.
I have used the library and it is certainly not immediately accessible to novices

void handleEvent(AceButton*, uint8_t, uint8_t); the Pointer is scary to them
case AceButton::kEventPressed confuuuuusiiiiing :D

I really meant you were putting effort into making it compile successfully on Arduino

I think you are trying to say that you assumed that I was targeting my libraries to be accessible to Arduino novices? The answer is: Definitely Not, I'm targeting Advanced users. Just reading my README.md file should have told you that. :-) It seems to me that the "Community" is composed of people of varying levels of interest, skills and knowledge. Some are beginners, just getting started with connecting a button. Some are trying to make COVID-19 ventilators using Arduino (as scary as that sounds). I lean towards the more advanced users, but I am willing to support as many hardware platforms and 3rd party Cores as reasonably possible. This has become more difficult with the spread of the new Arduino API. Which is why it would be awesome to have more technical guidance about the new API from @facchinm, or whoever is responsible for such things (I am not familiar with the internal division of labor within the Arduino team). Information like the following:

1. Here are the things that changed...
2. If you were doing this before, do this instead...
3. Here's how to write code and libraries so that they run both on Cores using the old API and Cores using the new API...

With regards to the MKR boards, thanks for the info @davetcc and @ubidefeo. What I'm hearing is that all the MKRs use a SAMD21, just like the Nano 33 IoT. So if I validate on one of those, I should be ok with the others. In theory, of course.

[ianfixes]

[arduino-lint] might be useful in your tests and CI

Most definitely, can't wait.

[JAndrassy]

@bxparks, for Sparkfun M0 try Arduino M0 in SAMD core. M0 has a different bootloader then Zero and MKRs. (or replace the bootloader)