Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Standard library proposals #104

Open
milancurcic opened this issue Nov 29, 2019 · 46 comments
Open

Standard library proposals #104

milancurcic opened this issue Nov 29, 2019 · 46 comments
Labels
meta Related to this repository

Comments

@milancurcic
Copy link
Member

milancurcic commented Nov 29, 2019 •
edited
Loading

Edit: Moved to https://github.com/fortran-lang/stdlib

I'm opening this meta issue with the goal of a broad and open-ended discussion around specific proposals that aim to add intrinsic procedures to the standard. Recent examples of such proposals are: #96, #100, #101, #103.

While I'm personally in support of most such proposals, I often see them as putting the cart before the horse. Why so? A common argument for such additions is that they are commonly used and often re-implemented over and over again because they're not available as intrinsics. I get this. However, if such intrinsics are really needed, I imagine there'd be an obvious choice of a 3rd party library available. Reusable and open libraries emerge when the pain is high enough in the community. That's how datetime-fortran came about. Lack of open source libraries may be a signal that there just isn't great need for such procedures.

Should we, as the Fortran community, consider working on a non-standard library that would include such procedures that we aim to the propose to the committee for inclusion in the standard? This library, or a subset of it, could over time become a candidate for the Fortran standard library, in addition to built-in intrinsics.

I foresee multiple benefits:

  • Near-immediate quality of life improvement for many application developers like myself.
  • In the process we'd flesh out what's really useful and what's not so important; we could troubleshoot caveats in the implementations before they're committed in writing in the standard.
  • We'd have concrete examples to show to the committee, which would make it easier to argue for their inclusion in the language;
  • A maintained general purpose library could ease and increase adoption of Fortran by newcomers to the language.
  • Implementations of new procedures could be used to solicit use cases.

There are quite a few projects in this spirit, such as Arjen's flibs and Ondrej's fortran-utils. Such libraries could serve as design examples, and perhaps even contribute implementations toward a dedicated non-stdlib.
@milancurcic milancurcic added the meta Related to this repository label Nov 29, 2019
@certik
Copy link
Member

certik commented Nov 29, 2019

I think this is a great idea. The fortran-utils project began as a necessity, where I collaborated with as many people as I could, but that was about 10 years ago, and we now have a much better community and a will to collaborate on a "standard library", with the idea that that what is in there are strong candidates for inclusion into the standard. (Sort of like Boost in C++.) I am imagining that we would carefully discuss a design of each function in there. With many eyes and experiences that we have, I think it can be really well designed.

Indeed, currently the committee always puts the carriage in front of the horse. The fix for language features is to first implement them in a compiler. That will come in a few years, hopefully even sooner. But for library additions (this feature), the fix is to first implement them in the "standard library", get it used in projects, gain some experience, and then write up a proposal and get it into the standard.

What should we call this library?

@jacobwilliams
Copy link

FYI: @apthorpe had some good thoughts about this a couple years ago:

@marshallward
Copy link
Contributor

An ancillary issue is how much of such a library should be written in Fortran itself. Most libc implementations that I've seen are predominantly written in C, with occasional assembly code (most notably crt0.s). The CPython Standard Library, on the other hand, seems to be a blend of Python and C.

At the risk of repeating myself, I think the answer to this question would help to more strongly define what can and cannot (or ought and ought not) be done within the constraints of the language itself, and perhaps reveal more precisely when an extension of the core language is required.

Another issue is whether particular intrinsics ought to be part of such a library, rather than part of the language itself. I know that my own group would like more control over the behavior of many transcendental functions like exp() for the sake of bit reproducibility, for example.

Overall, I am very supportive of something like this and would love to contribute to its development.

@milancurcic
Copy link
Member Author

@certik I don't know about the name, for the time being we can refer to it as just standard library, and see if it meets adversity. My guess is that it'd be welcome and embraced by most of the community. If a better name candidate comes up, we'll know. More important are the processes that we adopt to elect candidates for stdlib, and how to implement them. Your strong experience with SymPy will come in useful here as well.

I think a good model in the beginning would be for stdlib candidates to evolve from certain proposals in this repo. If it's a request for adding an intrinsic procedure or module to the language, we tag is as stdlib-candidate, and issues that gather strong support by the community can "level up" to experimental implementation in the stdlib. The stdlib could be another repo under github.com/j3-fortran. This repo would also be a place to include unit and regression tests, benchmarks, and of course the stdlib docs.

@marshallward Good points, and I don't see a good reason to limit this to pure Fortran implementation. For example, sorting can be done with pure Fortran, but file system procedures would need quite a bit of C. I think this is OK.

Whether some intrinsics should be part of a 3rd party library or part of the language, we'll learn that along the way. If we find that a procedure or module should be standardized, we'd use this as a stepping stone in that process.

@certik
Copy link
Member

certik commented Nov 29, 2019 •
edited
Loading

@milancurcic I agree. Let's call it the "Fortran Standard Library", or stdlib in short. I would like to try get (informal) support / blessing from the committee for this effort. In SymPy or fortran-utils the bar to get a new feature in is lower --- one just has to send a PR and convince the core developers to merge it. In this stdlib the bar to get a feature in should be much higher --- there needs to be a proposal in this repository, it needs to have support from the wider community, and then also ideally several members of the committee would "approve" it. And the goal of stdlib would be that if the Fortran Standard Committee decides that Fortran should have a standard library specified in the Fortran Standard itself, this stdlib will be it. In particular, we want to prevent a situation that in the future the committee decides to have a standard library, and they would not like some API of stdlib and decide to change it in backwards incompatible way. So for that reason I want to have a buy in of the committee from the very beginning.

@sblionel what do you think?

@FortranFan
Copy link
Member

FortranFan commented Nov 30, 2019 •
edited
Loading

@certik wrote:

.. Let's call it the "Fortran Standard Library", or stdlib in short. ..

@milancurcic , @certik, @marshallward ,

It's great to see your enthusiasm toward this effort, the value of this to the Fortran community is immeasurable and priceless. I really hope the ISO IEC working group WG5 will wholeheartedly support and cooperate with such an initiative.

It will be truly remarkable if stdlib can cover the 4 broad categories noticed with 'standard libraries' of other languages:

  1. Containers
    a. string type
    b. bitsets
    c. Enhanced 'array' types such as vectors, singly-linked and doubly-linked lists, etc,
    d. Adapters such as stacks, queues, etc.
    e Associative ones such as dictionaries (maps), hash_sets, etc.

  2. Algorithms
    a. Generic methods for sort, findloc, etc. that can work with any type, intrinsic and derived,
    b. Operations and permutations on a range of elements such as merge/union, difference,
    etc.

  3. Utilities
    a. Iterator-like facilities which make it easy to work with Containers,
    b. Operator (<, >, ==, etc. ) and assignment(=) overload abstracttions that perhaps make
    the use of standard algorithms more efficient?
    c. Miscellaneous other functions, subroutines (like generic swap), datetime, named
    constants, etc,

  4. Special aspects: a placeholder for any fundamental needs that get identified as this initiative progresses:
    a. Any basic facilities (extensions perhaps to ABSTRACT INTERFACE block?) needed toward
    "special" functions such as Variadic ones in the language e.g., MAX, MIN
    b. Ability to "overload" array subsection notation facility with Containers that standard Fortran
    provides with its 2 built-in containers: arrays and CHARACTER intrinsic type.
    c. Any special mechanisms that can help aid with improved constructors of arrays/containers
    and derived types ('classes'). I envision certain fundamental 'computer engineering' aspects
    being pursued here that can enable, say, efficient operation on the diagonal of a matrix or
    initialization to an identity matrix; or efficient 'dynamic' construction of 'classes' in Fortran similar
    to that is achieved near universally using new keyword in other languages.

@FortranFan
Copy link
Member

FortranFan commented Nov 30, 2019 •
edited
Loading

@jacobwilliams wrote:

FYI: @apthorpe had some good thoughts about this a couple years ago:

Agree.

Thoughts by @apthorpe under the "Release and Distribution Model" e.g.

I have a background in nuclear safety and writing code to the ASME NQA-1 standard for
safety-related quality assurance where strict configuration management is the norm.
Having worked in both fast-paced startups and in the glacially-paced nuclear industry,
I have to say I prefer the more thoughtful release model. I feel it’s a better fit for Fortran
culture and it offers a more consistent, predictable, and reliable experience for end users

are particularly important for all the users I work with.

@ivan-pi
Copy link

ivan-pi commented Dec 1, 2019 •
edited
Loading

For the past five years I've been programming in Fortran, I have always felt there is a lack of a central library for common things such containers, string handling, IO, etc., but also general numerical routines for interpolation, LAPACK interfaces, ODE solvers, evaluating polynomials and more. It seems like each of us has (re-)implemented these tools at some point in time, wasting lots of human hours. Ironically, many wrapped Fortran libraries are much easier to use from Python (via SciPy) than from Fortran itself.

Would a Fortran stdlib also cover such routines for numerical work or should there be a second library for scientific Fortran routines? I imagine the scientific library would more or less reimplement what the NAG library and the former IMSL library (now RogueWave) already have. I wonder if perhaps these two libraries might be the reason why we have no open-source scientific Fortran library, as many Fortran users (or there employers) in the past simply bought a license.

Finally, I wanted to mention the Flibcpp project which provides Fortran bindings to the C++ standard library using auto-generated code from SWIG-Fortran. Would we allow the Fortran standard library to have different backends (e.g. via submodules)? For example one with "hand-written" Fortran/C routines, and another one which is just a thin wrapper of the C++ one. It would be interesting to see if this library effort gained momentum, would the compiler vendors slowly shift to this standard library interface and provide there own specialized implementations (kind of like the Intel distribution of Python which has linked up Numpy/SciPy with the Intel Math Kernel Library).

@certik certik mentioned this issue Dec 1, 2019
Open
@apthorpe
Copy link

apthorpe commented Dec 2, 2019 via email

@rweed
Copy link

rweed commented Dec 7, 2019

As a first step towards a standard library, i think it would be useful if we had a complete list (as possible) of currently available open source projects/libraries that address the things that most people would agree are of immediate need (for me thats enhanced string handling, sorting, and ADT containers and algorithms (lists, dictionaries/unordered maps, stacks, queues, etc.). If the authors of the projects are ameanable, their code could be forked into a standard repostitory that aggregates the code into one place. Then we could begin a community effort to evaluate the various ways people implemented the needed functionality we envision for a standard fortran library and write reveiws on what we liked about their approach, what we didnt' like and how appropriate the code would be as a candidate for inclusion into the standard library. I think this would also benefit the original authors because it would provide useful feedback for their individual projects. We could also encourage people to consider placing their code on the repository first prior to standing up their own github site.

@cmacmackin
Copy link

This is just a small contribution, but I did come up with a way of storing arbitrary data, with run-time type-checking: https://github.com/cmacmackin/PolyCon. At one point I'd planned to use this to build a bunch of ADT containers, but I never got around to it. The code is currently GPL licensed, but I'd be willing to release it under something more permissive if people could find a use for it.

@jacobwilliams
Copy link

  • I'm all for this!
  • I would prefer this activity to be 100% community led. It would be great to have some support or blessing from the Fortran committee. But, I don't wish to have a situation where we have to wait for approval from the committee or some number of members to get code merged in. That would be fatal mistake, I think.

@certik
Copy link
Member

certik commented Dec 8, 2019

@sblionel what do you think of this effort? What would be the best way to get an approval / blessing by the committee for this, per the discussion above?

@sblionel
Copy link
Member

sblionel commented Dec 9, 2019

@certik I don't see that any committee approval or blessing is needed for this. The 202X work list includes some features that such a library might find useful, though that of course means one can't implement things until compilers have those features. Early implementations might get revised once the features become available.

The idea of a "stdlib for Fortran" has a lot of appeal, and what I have suggested before is that proponents put together a list of what they think such a library should contain. This might influence future standardization direction, especially if prototype implementations highlight a need for certain features.

It would be great if some version of this library were used in a production application, with observations of how it helped and how general its application was. The standard has many features that were first implemented outside the standard.

@milancurcic
Copy link
Member Author

@sblionel I think this is exactly the kind of "blessing" we were looking for, at least IMO. Fortran stdlib can (and should) exist in parallel to the language standard, and shouldn't be confused for it. Parts of it may eventually become part of the language.

@apthorpe
Copy link

apthorpe commented Dec 10, 2019 via email

@certik
Copy link
Member

certik commented Dec 10, 2019 •
edited
Loading

Thank you @sblionel for the blessing.

@milancurcic do you want to take the lead on this effort? We can call it stdlib, and create some GitHub organization to host it under, and I think a key to its success is to get all the people who have created similar "utility" libraries in the past on the same page (I think we mostly already are) and then ensure that the API of the stdlib is "approved" by those people as well as the wider community.

For example, speaking for myself, if the API is good enough, and if the library can be both installed on all platforms using cmake, as well as if it works if I simply copy relevant modules to larger production codes (I think this is aligned with @apthorpe's recommendations), I'll be happy to use it instead of https://github.com/certik/fortran-utils/. If authors of similar projects will do the same, then I think this will be successful.

@gronki
Copy link

gronki commented Dec 10, 2019 via email

@certik
Copy link
Member

certik commented Dec 10, 2019 via email

@milancurcic
Copy link
Member Author

@milancurcic do you want to take the lead on this effort? We can call it stdlib, and create some GitHub organization to host it under, and I think a key to its success is to get all the people who have created similar "utility" libraries in the past on the same page (I think we mostly already are) and then ensure that the API of the stdlib is "approved" by those people as well as the wider community.

Happy to! Will follow up with everybody later this week.

@jacobwilliams
Copy link

On CMake: keep in mind that, depending on what goes into this standard library, it could be hundreds of modules with complex interdependencies. It might not be so easy to just pick and choose what you want to pull out. So, a good and modern build system will be critical. Like it or not, CMake is an option that a lot of people use. Once you get somebody that knows what they are doing to set it up, it is quite easy to use and add new stuff to.

@gronki
Copy link

gronki commented Dec 10, 2019 via email

@milancurcic
Copy link
Member Author

I don't like CMake either, but it will be important for cross-platform builds, especially as the project grows. Local CMake experts (ahem @jacobwilliams, @zbeekman?) will need to step up to the task to develop and maintain it. Another likely candidate is autotools. Many projects have 2 or more build systems in parallel.

I agree overall that we should only add build capabilities as needed, rather than "maybe we'll need this someday".

@apthorpe
Copy link

I see CMake as the best of a number of unpleasant alternatives. I use it because it allows me to support multiple platforms (Linux, Windows, OSX) with a single system, handle documentation generation, dependency management, testing, and packaging. Those may not be important in everyone's environment but they are in mine. CMake doesn't force me to alter my source code nor have I found it to affect resource consumption of the compiled artifacts (speed, size, memory consumption).

I understand CMake is polarizing; maybe someone with experience with other systems can explain how one can repeatably and automatically build and test arbitrary code including dependency detection (and potentially retrieval and build) across multiple platforms without using a system like CMake. CMake is widely used, actively maintained, and has good support for Fortran. The syntax is weird and there's a learning curve, but I've found that to be true of every build system.

Finally, as a community effort I'd hope that those of us with experience configuring build, documentation, and test infrastructure (I include myself in this) could assist in setting up these facilities for projects. I understand people not wanting to be bothered with CMake but I don't understand the resistance to having it available for those of us who need it.

@gronki
Copy link

gronki commented Dec 10, 2019 via email

@milancurcic
Copy link
Member Author

@gronki Yes, I believe that is true of any CMake project.

@zbeekman
Copy link

I don't like CMake either, but it will be important for cross-platform builds, especially as the project grows. Local CMake experts (ahem @jacobwilliams, @zbeekman?) will need to step up to the task to develop and maintain it. Another likely candidate is autotools. Many projects have 2 or more build systems in parallel.

If the goals are to support all major OSes, then I would recommend CMake. Meson is coming along, but doesn't have as much buy in, and I don't know much about it. Also, CMake has pretty first class modern fortran support with automatic dependency resolution, and the ability to create robust install and export logic for package maintainers (i.e., debian, fedora, Homebrew, etc.) and can even output package-config pc files so that auto-tools packages can find it.

A lot of the pain and hate with CMake stems from "bad" old-school CMake doing things in a weird/bad way, and just the fact that build systems & meta build systems are always a PITA. (I don't like CMake either, but I like it more than anything else I've tried to use in its place.)

I have experience using CMake for Fortran mixed language (C and C++ too) projects that are required to build and run on all 3 major OSes, and use Microsoft Visual Studio for the main Fortran source on Windows.

In addition, CMake has buy in at DOE, and many/most other national labs.

I'd be happy to help out with CMake stuff.

I agree overall that we should only add build capabilities as needed, rather than "maybe we'll need this someday".

Yes, don't build it before you need it is an important mantra to follow.

Off topic ⚠️

FYI, I haven't had time to go through this thread in any detail, but wanted to mention that I've been particularly frustrated by repeated implementing stuff in 3 main areas, and as a result, I've started working on my own "missing" standard library: https://github.com/zbeekman/ZstdFortranLib

The best place with somewhat coherent explanation of the rationale and how to build it is the README.md on the develop branch and the ReadTheDocs documentation.

I am going to pull in community libraries where they have compatible licenses (.e.g., functional-fortran) and focus on the following main areas (for now):

  • Strings
    • Conversion to/from integer/real/logical (all kinds of each)
    • Conversion on string concatenation
    • raw string processing functions inspired by Ruby & Python
    • string class to make using all the machinery easier via TBPs
  • Files
    • For now just name manipulations like dirname, basename, etc.
  • OS/Environment integration
    • is_a_tty(), OS%env("HOME"), .envExists. "USER", etc.
  • Unit testing & assertions stuff
    • Subtest summaries w/ color
    • File and line number triggering failures
  • Error Stack class/object
    • Maintain a call-stack
    • Raise errors, but optionally trap them later with good call stack
      including line number and file

@sblionel
Copy link
Member

I found the following thread to be apropos of discussions here: https://twitter.com/MichaelCaisse/status/1204207152237367297?s=20

@gronki
Copy link

gronki commented Dec 10, 2019 via email

@cmacmackin
Copy link

cmacmackin commented Dec 10, 2019 •
edited
Loading

The real advantage of CMake is its ability to automatically resolve dependencies in modern Fortran. If we are going to have an even remotely complicated interdependencies within this library then I think CMake is a necessity and raw make files simply are not a viable option.

As others have said, once someone has set up the CMake files, it is really not difficult to use. At work I was tasked with writing a sort of template/skeleton for new Fortran projects to use, part of which included a CMake build system. I think that would provide a more than adequate starting point and it would then be very easy to adapt for the needs of this project.

@certik
Copy link
Member

certik commented Dec 10, 2019

I suggest let @milancurcic take the lead, setup a repository and we can discuss CMake and Makefiles in there. Let's use this issue to discuss what should actually be in the library.

@apthorpe
Copy link

@zbeekman noted:

... (I don't like CMake either, but I like it more than anything else I've tried to use in its place.)

This is my feeling as well.

FYI, I haven't had time to go through this thread in any detail, but wanted to mention that I've been particularly frustrated by repeated implementing stuff in 3 main areas, and as a result, I've started working on my own "missing" standard library: https://github.com/zbeekman/ZstdFortranLib ...

I'd add logging to that; I extended FLIBS' m_logger module with syslog-like priority but the stock module is quite usable as it is. There's a trend towards 'observability' vs unstructured logs in some sectors which might be interesting to look into, but simple logging is better than WRITE() scattered through the code. syslog and Windows Event Log support would be nice but it's probably better to achieve something simple like m_logger first.

@zbeekman
Copy link

@apthorpe Thanks for the feedback. Yeah, logging is kind of the next feature on the list. I’m just so tired of reading variables and messing around with strings.

FYI, the way I get overloading for all kinds of each intrinsic type is through the very awesome Jin2For package. A thin extension of Jinja2 that interrogates the compiler and provides nice convenience. Templating has its advantages for generic programming when used wisely and in moderation until there is something better in the standard.

@milancurcic milancurcic mentioned this issue Dec 14, 2019
Open
@milancurcic
Copy link
Member Author

I created a repo here:

https://github.com/fortran-lang/stdlib

and bootstrapped some documents and issues.

I will reach out directly to many of you over the next few days with some ideas and questions about getting more closely involved.

@certik
Copy link
Member

certik commented Dec 16, 2019

Thanks @milancurcic for taking the lead on this.

Let's start with something simple that we can all agree on, to learn and figure out how the process should work, and then we can try to tackle some of the more difficult proposals.

It seems the least controversial will be a subset of NumPy and Matlab (both of which use very similar API):

And we just need to discuss the API (name, arguments, return value) and ensure it is familiar (identical or close to NumPy / Matlab / Julia, ...) and consistent with Fortran.

@certik
Copy link
Member

certik commented Dec 16, 2019 •
edited
Loading

I created fortran-lang/stdlib#10 for stdlib_linalg and eig in particular. We should be able to converge to an agreement on the API pretty quickly for eig.

@aamaricci
Copy link

Sorry guys I lost this precious discussion and I read it only by now.
Let me say that I find the proposal of building a standard library wonderful and I thank you all and Milan in particular for taking the lead.

I am willing to contribute to standard library.
First of all, as some of you may know, I devoted quite some effort to develop SciFortran. A project which aims to "port" what SciPy is for python into Fortran. The motivation is similar to what has been discussed here, although with a more scientific/math oriented goal.
Essentially: to provide the average fortran user a "simple" interface to the most common task that might be required when developing on scientific project. With "simple" I mean something that does not make explicit use of derived data type as much as possible so that the user keep using implicit data types and arrays. In some cases this is unavoidable of course but in most cases it can be avoided.

The project is open source and heavily influenced by my research activity, by older version of SciPy and by other similar projects (for which I got "inspired", e.g. some LAPACK interface from Certik's project :-))

Please feel free to use, copy, change, update, modify and whatever other action you may have in mind Scifor for the development of standard library. Scifor compiles with Cmake and try to make use of available lapack/blas/scalapack libraries present in the system (e.g. veclib, mkl, etc.): this is very important for optimisation and it is currently done also in the installation of scipy.
[to make concrete example scipy eigh take 0.9s to work on a 1000x1000 matrix on OSX using veclib blas. Scifor used local installation of blas/lapack and took up to 5 times more for the same task. compiling against veclib blas we reached the same or better performance as python].

Unfortunately there is essentially no documentation. (I am lazy I know).
Scifor contains already:

  • Common variables and physical units, colors, fonts to modify standard output
  • Linear algebra: a big part of LAPACK interface, many of the utilities used in python (diag, eye, diagonal, trace, etc...). This part should also contains BLAS interface, for now it is only limited to Xgemm (as function and as operator .x.). It also includes interface to Scalapack for few methods (eigh, matrix-matrix multiplication, matrix inversion, ...). It contains also kronecker products.
  • Random number generators, (KISS, Mersenne-Twister, etc..)
  • Optimization and Root finding (interface to MinPack plus other procedures such as broyden, fzero, and other few algorithms)
  • Sparse Linear algebra (very basic): simple interface to P/Arpack and some Lanczos methods
  • Integration (generic interface to QUADPACK into quad, simpson, trapezoidal rule, etc..)
  • Interpolation (PPPACK, CUBSPL, plus other including Numerical Recipes) 1d and 2d, linear, cubic, polynomial
  • Special functions
  • Derivatives (evaluates derivatives using increasing order finite difference methods, jacobian ,hessian, etc.)
  • Fast Fourier Transforms (interface to FFTPACK)
  • Input/Output tools (simple save,read procedures for arrays up to 7dimensions). This is not very general indeed at least not at the level of savetxt, loadtxt. It contains some additional procedure that come useful as str which converts any standard data type into a character(len=*) of proper length. This I use a lot and is very similar to str in python.
  • Input Parser: an internal data structure which parse input variables from a file or command line. all standard data type are supported.

I do have some additional comments, mostly based on my experience:
. It is important and useful to keep a modular structure so that a single part can be loaded independently of the rest.
. There is a non-trivial issue with the automatic inclusion of the .mod files. While in c/c++ the header can be placed in suitable path of the OS, in fortran the situation is more complicated.
I solved this by generating in Cmake a script very similar to what is used to load Ifort/MKL. Alternatively it generates an environment module file. I did not find any better way to have automatic use of the library.
. I think it is important to distinguish between development of intrinsic type and methods, which can import some functional programming in fortran, from the algorithms (used in different contexts). Having all in the same place is ok but in my mind these are two different problems.
From my point of view, I would inspire more to Python than C/C++ in this respect.
I am convinced that Fortran can easily reach the simplicity of python while keeping its lower level qualities. Perhaps I can elaborate more on this.

Sorry for the very long message.
Again: thanks a lot for proposing this. I really really appreciate it.

@certik
Copy link
Member

certik commented Dec 21, 2019

@aamaricci thanks for the input! Let's keep the discussion going.

I discovered your code and linked it at fortran-lang/stdlib#1 yesterday.

I am I think in the same camp as you are --- for example with the modern Fortran Python like interface to Lapack implemented in fortran-utils, that you ended up using. I am glad you liked it! And I would like the whole of SciPy ported to Fortran, in a natural, non OO way. I have no doubts it would go a long way towards much wider adoption of Fortran. Furthermore, I want to ship this stdlib with LFortran, that would allow interactive usage in Jupyter, thus providing almost equivalent experience as SciPy (in fact I hope much better experience!).

Others like @cmacmackin like OO interfaces, and I think we can have them too, in addition to non OO interfaces.

The other aspect is that we pretty much know how to implement all of the above --- either we have already done that, or know how to do it quickly. The biggest contribution of stdlib is to agree on an interface that we all like and approve as a community and commit to support for the long run. For example, the linalg routines, that you also ended up using or got inspired by, I designed them with a few other Fortran users about 10 years ago; but that is different and not enough --- what we are doing now is to have a much wider discussion about the API, and I would love if you could also be part of this discussion, you can start here:

fortran-lang/stdlib#23

fortran-lang/stdlib#10

then, once we mostly agree on the API, we merge into our "experimental" modules in master. You can see the PR 23 above for an example. Then much later, once we gain experience with the API, we will propose to move from experimental to main. The bar for that will be much higher. We are still figuring out all this workflow. But I am positive we can figure out a high quality process that would allow us to deliver on a Fortran Standard Library that we all eventually agree upon.

This was referenced Dec 23, 2019
Open
Open
Open
Open
@gronki
Copy link

gronki commented Dec 23, 2019

As a counter argument to "make everything stdlib rather than part of the standard", here's an article: https://www.davidhaney.io/npm-left-pad-have-we-forgotten-how-to-program/

I think there's important quality in standarizing the behavior of the most essential utilities (such as string split) over leaving them to the community.

@certik
Copy link
Member

certik commented Dec 23, 2019 •
edited
Loading

@gronki the goal of stdlib is to make it the standard. What does that mean? Here are a few minimal requirements that we are striving for stdlib to satisfy (in the future):

  • All API must be agreed upon by the wide community
  • Must be widely used in Fortran projects
  • The stdlib github repository itself must end up a successful open source project (lots of contributors, lots of stars at GitHub, ...)
  • Most community will agree that stdlib is the de-facto standard including most J3 and WG5 committee members

We are obviously not there yet, as we have just started. These goals will take years to achieve. But if we achieve these (and probably there are more goals that we need to achieve beyond these) we might start calling stdlib the standard.

The example you posted with leftpad in Javascript did not satisfy any of the above goals: it was quite widely used, but it had little github stars, one (?) contributor, the community did not seem to agree this is the Javascript standard library, etc. So I want to make it clear, we are not trying to achieve what leftpad did. We have much bigger goals as outlined above. In fact I would argue leftpad is an example what happens if you do not have a standard library --- people start re-implementing such basic features themselves, in an unorganized fashion and then codes start depending on such not well supported libraries and it causes all kinds of problems at outlined in the blog post you linked. If Javascript included leftpad either directly in the language, or in the standard library, there wouldn't be such a problem.

Assuming we do achieve the above goals in the future, then it will be a good time to see if some of the stdlib's more basic functionality (such as string manipulation) could be included in the standard itself, or if the Fortran standard could standardize the API of stdlib as the Fortran Standard Library. (In fact, we do not need to wait until stdlib becomes popular, we should push good ideas in this fortran_proposal repository in parallel with stdlib.) Also hopefully, stdlib will spur discussions about Fortran language improvements that would make some usage simpler.

@gronki
Copy link

gronki commented Dec 23, 2019 •
edited by certik
Loading

@certik ok that made me a bit calmer. I posted a thread in the other repo to address the possible solution for packages that are too big/specific for an stdlib. I am curious what you think.

@milancurcic
Copy link
Member Author

I can't think of any procedure that should be considered for the standard and that shouldn't be staged and implemented in stdlib first.

@certik certik mentioned this issue Jan 3, 2020
Merged
@leonfoks
Copy link

leonfoks commented Jan 7, 2020 •
edited
Loading

Just to add my two cents, my package Coretran contains some of the things discussed at the beginning of this issue. Its interfaced, object oriented, and uses submodules.

There are things outside of scope of a standard library like KdTree and Shewchuks arbitrary precision geometry stuff (are these outside scope though?), but I do have a relatively fast introspection sort and dynamic arrays.

I do use Cmake, but as @cmacmackin mentioned, I like how it handles the modern dependencies.

I called my package Coretran because I did not want to suggest I was good enough to create a standard library! But that was the original intention I guess.

I think contributing to a centralized 'approved' standard library is great! Im not saying use Coretran, im willing to migrate anything over if it makes sense.

@certik
Copy link
Member

certik commented Jan 7, 2020 •
edited
Loading

@leonfoks thank you for finding this project and getting in touch with us. I put a link to it into fortran-lang/stdlib#1, where we keep a list of similar and related projects.

Indeed, many of us (myself included) have written similar libraries that were intended as a "standard library", see the full list at fortran-lang/stdlib#1. And none of us are good enough to create a standard library on our own. But the difference how stdlib is developed is that we managed to create an initial community that is large enough and representative enough that we can refine the API as a community so that we are all ok with it to use it in all of our projects and with a high chance that people who were not part of the approval process for a particular feature will still like the result. We just merged in an initial workflow how new features should get into stdlib: https://github.com/fortran-lang/stdlib/blob/006bedafc0d40ff381da2bd4455f61b5e11fc2ee/WORKFLOW.md.

(Update: we added the motivation into README: https://github.com/fortran-lang/stdlib#goals-and-motivation)

Thank you for offering to migrate over things from Coretran. Would you mind following the workflow and get your things discussed and eventually implemented in stdlib? I am looking forward to collaborating with you.

@leonfoks
Copy link

leonfoks commented Jan 8, 2020

Thanks @certik Looking forward to this project! I saw the update to fortran-lang/stdlib#1 . Its cool to see a list of the other projects too.

I took a look at the workflow, I will start adding things and seeing what gets traction!

@ivan-pi ivan-pi mentioned this issue Jan 8, 2020
Open
@certik
Copy link
Member

certik commented Jan 8, 2020

@leonfoks do you want to start an issue for Coretran like I just did for my old library (fortran-lang/stdlib#103)? I think there are lots of open issues that parts of Coretran would fit into, and then you can open new issues for the rest. And let's get it discussed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
meta Related to this repository
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests