vscode/cfmleditor extension dependent on cfdocs JSON schema

[pfreitag]

Hello, I'm starting a discussion here to discuss this item from Cage:

I’ve been talking with Gareth about his fork of KamasamaK/vscode-cfml + vadim-sibiryanskiy-yumasoft/vscode-cfml called cfmleditor (GitHub - cfmleditor/cfmleditor: An extension for Visual Studio Code to assist in development with CFML.).
One item he wants to focus on is to provide additional completion suggestions for member functions for native types / enumerated values for global functions. In order to do this, the resource files in cfdocs need to be updated with some additional information.
Apparently your current schema already accounts for most of the needed information, but most of the files don’t have these sections filled out.
I have agreed to help Gareth out with this aspect. Since updating the files will be a fair amount of work, I wanted to do it in a way consistent with your overall long term strategy and that allows for future enhancement by others.
What are some particular issues / pain points / obstacles you’ve encountered in achieving your goals with cftags? And any specific long term direction you’d like to see cftags go in?

[pfreitag]

I think the only problem / challenge we have discovered with this is that it can be possible for different engines to have different method signature argument names. Otherwise if you can be more specific about what you want to add, what is missing I might be able to provide better guidance.

The long term goal is to have as accurate as possible representation of the CFML language in the json docs without making them too overly complex to maintain / edit.

[ghedwards]

Keep in mind we aren't the only ones using this data, additional keys I reckon, but keep the structure the same

[psarin]

Definitely, hoping that will be the case. Maybe Pete has a list of people / projects that depend on the data. Even if we're just adding keys, would be good to get their feedback and see what their needs are. For now, just trying to provide something concrete people can comment on, before spending a fair amount of time doing it a way that may not meet most users' needs.

[pfreitag]

I don't have a list, but I think the two most popular projects using it are the cfdocs.org website, and the vscode plugins. Ideally I think you should focus on non breaking additions to the schema, these projects are all volunteer run and we don't have tons of time to take on major refactoring.

@ghedwards I took a brief look at #1634 I think the addition looks ok, I'll take a closer look at it. For serializing / parsing the json we are just using serializeJSON and deserializeJSON functions in CFML at the moment. A lot of the edits to the JSON are actually done by hand (as people find a missing attribute they add it, etc) but having a standard pretty json format would be nice. There is a utility to create the JSON for examples: /utilities/json/ but it doesn't really do any formatting of the JSON

[ghedwards]

I wonder if it would be worth creating a CF function to spit out the JSON as close to the existing format as possible, nothing fancy, and doesn't have to be super dynamic, I might take a look at this if no one else has already got something that might be suitable

[psarin]

@pfreitag Sounds good, will focus on compatibility based on that. See new thread below for updates on schema.

[psarin]

Based on suggestions, have made updates to the schema to maximize compatibility. Essentially, have added properties to support the needs:

• "versions" array to keep track of headless/member functions, tags, components, lists
• add "group" property to allow differentiation between headless & member functions but also allow grouping by other options if needed
• allow params, examples, versions to have an optional "engines" object to denote differences between engines
• remove "member_details" since 99.9% of data files did not have data for it and is now supported by "versions"

I also modified the views/doc.cfm file to support new schema as well as support the old schema. Samples data files using the schema are: cfcatch.json, arraycontains.json, arrayfilter.json, application-cfc.json, array-functions.json. Also left some sample files in the old version. Please see commit log for more details.

The proposed changes are at https://github.com/psarin/cfdocs/tree/feature/schema_changes. HINT: you can visualize changes for the example files via http://localhost:8080/cfdocs/doc.cfm?name=arrayfindnocase or your equivalent running CFML engine.

After your feedback (and my doing lots more testing!) and additional changes, if everything seems acceptable, I will restore all the original data files in preparation for conversion. I have a utility that will allow conversion from old schema to new schema.

A few additional notes:

• the schema I uploaded deleted the keys that wouldn't be used anymore (e.g., moved) to keep it clean for your review. We can decide to add them back in with deprecated tag
• there is potential to remove the syntax property and determine it based on name / params (less documentation for contributers). Option could be to auto-determine unless syntax property is present (i.e., presence of key overrides)

[ghedwards]

Sorry mate, I really wasn't looking to change the schema so drastically, small steps not giant leaps

[ghedwards]

I'm curious, which member functions are different between engines ? How different are they, could this be solved by defining engines for each parameter?

[ghedwards]

Sorry, will have a better review and chat in the new year

[ghedwards]

Having a bit of a review during the break, can we go back a few steps and talk about the differences between engines we're trying to cater for in not using "member_details" ? what are they ? is it just the return types of some are different, or are there other things ?

[psarin]

Hi @pfreitag . Hoping you've had a chance to review the changes at https://github.com/psarin/cfdocs/tree/feature/schema_changes? Just keep in mind you'll want to compare the 2nd commit onward since the 1st commit is just spacing / formatting changes but comparing 1st to last inaccurately indicates major changes when there are only a few. Changes are listed in the commit log and in comment here. If any additional remediation needs to happen for cfdocs due to changes, I'm more than happy to work on those after addressing your feedback.

[pfreitag]

Sorry for the delay in getting back to you. This is quite a big change to wrap my head around without spending tons of time reviewing it. Would it be possible for you to summarize each change and why you think it is necessary? A few things I'm wondering:

• For position, is that necessary to account for differences between engines? Otherwise I think we could just use the position of the element right?
• For versions - can you point to some examples of why that is necessary, and how you are using it. I don't know if we need examples again inside this element.

thanks

[psarin]

@pfreitag Thanks for reviewing. Apologies for the confusing commits / changes as I think the initial commit standardizing the spacing makes review a bit difficult.

I think it would be best to review the diffs for the following two (sequential) commits:

• psarin@6691bbc

1. Removal of syntax, script, member, member_details, returns from top level properties key.
2. syntax, script, returns get moved to the versions block in next commit and no longer needed here.
3. member and member_details are not needed here because they are represented in the versions block.
   • These probably could be marked "deprecated" rather than simply deleted.

• psarin@69ea074

1. versions block added
   • syntax, script, returns are moved to versions after removal in previous commit
   • group added to versions - allows for differentiation between headless & member functions (as well as tags, components)
2. examples schema definition moved from top level to definitions

To answer your specific questions:

For position, is that necessary to account for differences between engines? Otherwise I think we could just use the position of the element right?

• I didn't think it was necessary but I left it in because the original schema had member_details.base_param_index. base_param_index was just renamed position.

For versions - can you point to some examples of why that is necessary, and how you are using it. I don't know if we need examples again inside this element.

• Versions essentially replaces member_details in original schema. By specifying group=headless or group=member, it supports defining both types.
• Headless and member functions can have different return types for the same engine as well as across engine and this schema supports that.
• Rationale for (optional) examples inside versions is to allow defining examples that are engine specific. An example is when return type is different between engines.
• FYI, the cfdocs application code would merge the examples blocks as needed so examples can be as generic or specific as needed without double documenting examples.
• Please see JSON files with commit message 'add examples using new schema' in https://github.com/psarin/cfdocs/tree/feature/schema_changes/data/en for specific examples.

Let me know if additional questions / concerns. I'll have a bit more time in April to work on this and merge in any new upstream commits that may have happened.

Thanks.

[psarin]

@pfreitag Hoping you have had a chance to review? Any questions I can answer or changes to be made to proposal?

[psarin]

One example of where versions block helps. Also includes use of params.examples for Lucee specific issue. Will upload code and some more examples later.

[ghedwards]

I think you have a proposed solution for something where you haven't properly defined the problem or the steps to get from a to b. Let's start with examples of member functions where their interface or return type are different between CF engines and how your proposals seek to solve this, as well as how you imagine the transition to a new format for cfdocs and for those consuming the existing data.

[psarin]

I was addressing the issue from Pete: "I think the only problem / challenge we have discovered with this is that it can be possible for different engines to have different method signature argument names." Several of the data files included in the PR are examples where return value is different between CF engines (e.g., arrayContains) or where the type of an input param differs (e.g., arrayfilter, cfdirectory). By using the versions block (array), it allows for 1 or more method signatures, supporting both member and headless functions. This also allows for support of different different method signatures for different engines without having to create new keys in the schema (in fact, it allows for removal of some keys). The schema is backward compatible (except for key member_details which was only being used in one data file) since new elements are optional. I have also included code changes in the PR that supports the new schema while still able to fall back to old schema. Hope that helps clarify a bit.