#1945 Make FeatureGrid attributes visibility selector counted when user exports grid data #7845

alexander-fedorenko · 2022-02-07T20:30:06Z

Description

This PR suggests changes needed to make FeatureGrid attributes visibility selector counted when user exports grid data.

Please check if the PR fulfills these requirements

The commit message follows our guidelines: https://github.com/geosolutions-it/MapStore2/blob/master/CONTRIBUTING.md
Tests for the changes have been added (for bug fixes / features)
Docs have been added / updated (for bug fixes / features)

What kind of change does this PR introduce? (check one with "x", remove the others)

Issue

What is the current behavior?

#1945

What is the new behavior?
WFS export will properly use list of hidden attributes to load data only for visible columns.

Breaking change

Does this PR introduce a breaking change? (check one with "x", remove the other)

Yes, and I documented them in migration notes
No

Other useful information

…n attributes.

- Keep geometry in exported file when custom propertyName value is passed to the request.

…m attributes list is passed to the action. New observables to use for custom attributes list.

alexander-fedorenko · 2022-02-13T20:54:48Z

Right now this feature is blocked by the limitation on GeoServer side. There are two WPS processes that can be used to achieve desired workflow when resulting layer data is filtered in a way that part of the attributes are removed from the output: "gs:Query" and "vec:Query". Both of them doing same things but first one is marked as deprecated.

"vec:Query" has following description: "Queries a feature collection using an optional filter and an optional list of attributes to include. Can also be used to convert feature collection format." The issue is that it doesn't accept list of attributes in any possible format.

Comma-separated list results into exception on process execution:

Process failed during execution java.lang.NullPointerException: PropertyDescriptor is null - did you request a property that does not exist? PropertyDescriptor is null - did you request a property that does not exist?

Passing each attribute name in a separate wps:Input tag makes process finish successfully, but in such case it take only one last attribute passed with the wps:Input tag. Resulting file contains only one attribute in this case.

...
    <wps:Input>
      <ows:Identifier>attribute</ows:Identifier>
      <wps:Data>
        <wps:LiteralData>STATE_ABBR</wps:LiteralData>
      </wps:Data>
    </wps:Input>
    <wps:Input>
      <ows:Identifier>attribute</ows:Identifier>
      <wps:Data>
        <wps:LiteralData>LAND_KM</wps:LiteralData>
      </wps:Data>
    </wps:Input>
...

tdipisa · 2022-02-14T11:34:46Z

@taba90 Can we have some assistance from you on this?

MV88 · 2022-02-24T13:39:54Z

@tdipisa i think @offtherailz is a more suitable person to review this
let me know your thoughts

tdipisa · 2022-02-25T14:24:24Z

@tdipisa i think @offtherailz is a more suitable person to review this let me know your thoughts

@MV88 this is in blocked for now, but ok.

offtherailz · 2022-02-25T15:34:14Z

I did a few attempts and I guessed that the correct format for multi-attributes is this:

<?xml version="1.0" encoding="UTF-8"?><wps:Execute version="1.0.0" service="WPS" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.opengis.net/wps/1.0.0" xmlns:wfs="http://www.opengis.net/wfs" xmlns:wps="http://www.opengis.net/wps/1.0.0" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:gml="http://www.opengis.net/gml" xmlns:ogc="http://www.opengis.net/ogc" xmlns:wcs="http://www.opengis.net/wcs/1.1.1" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsAll.xsd">
  <ows:Identifier>vec:Query</ows:Identifier>
  <wps:DataInputs>
    <wps:Input>
      <ows:Identifier>features</ows:Identifier>
      <wps:Reference mimeType="text/xml" xlink:href="http://geoserver/wfs" method="POST">
        <wps:Body>
          <wfs:GetFeature service="WFS" version="1.0.0" outputFormat="GML2" xmlns:gs="https://gs-stable.geo-solutions.it/geoserver/geoserver">
            <wfs:Query typeName="gs:us_states"/>
          </wfs:GetFeature>
        </wps:Body>
      </wps:Reference>
    </wps:Input>
    <wps:Input>
      <ows:Identifier>attribute</ows:Identifier>
      <wps:Data>
        <wps:LiteralData>STATE_NAME</wps:LiteralData>
      </wps:Data>
    </wps:Input>
    <wps:Input>
      <ows:Identifier>attribute</ows:Identifier>
      <wps:Data>
        <wps:LiteralData>STATE_FIPS</wps:LiteralData>
      </wps:Data>
    </wps:Input>
    <wps:Input>
      <ows:Identifier>attribute</ows:Identifier>
      <wps:Data>
        <wps:LiteralData>WATER_KM</wps:LiteralData>
      </wps:Data>
    </wps:Input>
    <wps:Input>
      <ows:Identifier>attribute</ows:Identifier>
      <wps:Data>
        <wps:LiteralData>the_geom</wps:LiteralData>
      </wps:Data>
    </wps:Input>
  </wps:DataInputs>
  <wps:ResponseForm>
    <wps:RawDataOutput mimeType="text/xml; subtype=wfs-collection/1.0">
      <ows:Identifier>result</ows:Identifier>
    </wps:RawDataOutput>
  </wps:ResponseForm>
</wps:Execute>

In this case I receive (with default format).

 <gml:featureMember>
        <feature:us_states fid="us_states.1">
            <feature:STATE_NAME>Illinois</feature:STATE_NAME>
            <feature:STATE_FIPS>17</feature:STATE_FIPS>
            <feature:WATER_KM>1993.335</feature:WATER_KM>
<!-- geometry, not included because is too big, but it was present too -->
        </feature:us_states>
    </gml:featureMember>

This is the CURL you can use for test:

curl --location --request POST 'http://gs-stable.geo-solutions.it/geoserver/wfs' \
--header 'Content-Type: application/xml' \
--data-raw '<?xml version="1.0" encoding="UTF-8"?><wps:Execute version="1.0.0" service="WPS" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.opengis.net/wps/1.0.0" xmlns:wfs="http://www.opengis.net/wfs" xmlns:wps="http://www.opengis.net/wps/1.0.0" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:gml="http://www.opengis.net/gml" xmlns:ogc="http://www.opengis.net/ogc" xmlns:wcs="http://www.opengis.net/wcs/1.1.1" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsAll.xsd">
  <ows:Identifier>vec:Query</ows:Identifier>
  <wps:DataInputs>
    <wps:Input>
      <ows:Identifier>features</ows:Identifier>
      <wps:Reference mimeType="text/xml" xlink:href="http://geoserver/wfs" method="POST">
        <wps:Body>
          <wfs:GetFeature service="WFS" version="1.0.0" outputFormat="GML2" xmlns:gs="https://gs-stable.geo-solutions.it/geoserver/geoserver">
            <wfs:Query typeName="gs:us_states"/>
          </wfs:GetFeature>
        </wps:Body>
      </wps:Reference>
    </wps:Input>
    <wps:Input>
      <ows:Identifier>attribute</ows:Identifier>
      <wps:Data>
        <wps:LiteralData>STATE_NAME</wps:LiteralData>
      </wps:Data>
    </wps:Input>
    <wps:Input>
      <ows:Identifier>attribute</ows:Identifier>
      <wps:Data>
        <wps:LiteralData>STATE_FIPS</wps:LiteralData>
      </wps:Data>
    </wps:Input>
    <wps:Input>
      <ows:Identifier>attribute</ows:Identifier>
      <wps:Data>
        <wps:LiteralData>WATER_KM</wps:LiteralData>
      </wps:Data>
    </wps:Input>
    <wps:Input>
      <ows:Identifier>attribute</ows:Identifier>
      <wps:Data>
        <wps:LiteralData>the_geom</wps:LiteralData>
      </wps:Data>
    </wps:Input>
  </wps:DataInputs>
  <wps:ResponseForm>
    <wps:RawDataOutput mimeType="text/xml; subtype=wfs-collection/1.0">
      <ows:Identifier>result</ows:Identifier>
    </wps:RawDataOutput>
  </wps:ResponseForm>
</wps:Execute>'

The result you said @alexander-fedorenko is probably the result of my first attempt, passing attributes this way

<wps:Data>
        <wps:LiteralData>STATE_NAME</wps:LiteralData>
        <wps:LiteralData>STATE_FIPS</wps:LiteralData>
      </wps:Data>
    </wps:Input>

In this case I was receiving only STATE_FIPS.

Correct me if I'm wrong.

It doesn't seems to be possible to create this request in demo -> WPS request builder, I'm guessing this is a limitation or a bug of the current builder. Maybe @aaime knows more about it. Not sure.

Anyway generating the request like this should work.

offtherailz · 2022-02-25T15:35:20Z

I should put this PR in draft, because it is not ready to be delivered, as far as I understood (and because it is blocked)

…ncluded into attributes list.

offtherailz · 2022-02-25T16:33:05Z

Thank you very much . Il giorno ven 25 feb 2022 alle ore 17:04 Alexander ***@***.***> ha scritto:

…

@offtherailz <https://github.com/offtherailz> Thanks for reviewing. You're right, your request (as well as mine in the last sample of the code) is correct. The problem was that I didn't notice initially that gs:Download method transform geometry attribute name to "coordinates" somewhere internally when geojson format is selected. I was using "extractGeometryAttributeName" method to include geometry attribute into output, which caused a problem because such attribute was not available in the data that is feed to the vec:Query process. I will do minor changes and then it can be tested. — Reply to this email directly, view it on GitHub <#7845 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJYMFRYMDFM3WJ363O3POLU46SB3ANCNFSM5NYQAKTQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

alexander-fedorenko · 2022-02-25T16:36:14Z

@offtherailz Thanks for reviewing.
You're right, your request (as well as mine in the last sample of the code) is correct.
The problem was in usage of geoJson format in the intermediate gs:Download process. In this case output is a FeatureCollection, coordinates are not the part of attributes list anymore, so vec:Query process cannot get them from the passed list and they are always ignored. I changed intermediate format to WFS collection and it works flawlessly. Please review :)

offtherailz · 2022-02-25T17:02:59Z

I tested it and I think that we should do it better. Download process has been created to manage big sets of data. With this solution I think you are downloading the data, then resending them to the service.

This is not optimal and for big data sets can be problematic. Anyway I think there is a way to optimize
by using process chaining.
I never used it but I think it can be the solution. Downloading and re-uploading the features to implement this features is not doable for the kind of dataset we need to manage with this tool.
I don't know if you can pass attributes directly in the first request or if you have to use chaining. I didn't studied the WPS Download process yet.
Take a look if you can find a solution without getting and sending the features to the process. If you need help, please ask

… vec:Query

alexander-fedorenko · 2022-02-25T20:13:43Z

Switched vec:Query process to use process reference as an input instead of the WMS collection.
Chain will work in the following mode:
gs:DownloadEstimator -> result -> gs:Download -> reference -> vec:Query -> reference. Result can be downloaded by user on demand via widget.
@offtherailz

MV88

Please check a few comments

testing this locally with no wps (direct download it works)

web/client/epics/layerdownload.js

web/client/selectors/featuregrid.js

MV88 · 2022-04-06T14:14:49Z

web/client/utils/FeatureGridUtils.js

+ * @param {object} customAttributesSettings object containing information about deactivated attributes, see getCustomAttributesSettings
+ * @returns undefined|array
+ */
+export const getAttributesList = (attributes, customAttributesSettings) => {


can this be unit tested?

Added test with different scenarios.

web/client/components/data/download/FeatureEditorButton.jsx

Added unit test for "getAttributesList" function

alexander-fedorenko · 2022-04-06T22:03:59Z

Please skip this problem with readthedocs, I just don't want to merge master back to this branch cause I'm not sure it won't mess up with the changelog of PR. This issue was fixed on master afaik by you @MV88
Please re-review and let me know if the problem with locale availability is still occurs for that tooltip.

MV88 · 2022-04-07T15:35:19Z

@ElenaGallo please test DEV

ElenaGallo · 2022-04-19T14:26:34Z

@alexander-fedorenko sometime it is not possible to download data, in any format and it always happens if I select the attributes I want to export. See the gif attached:

download.mov

download2.mov

tdipisa · 2022-04-20T08:17:42Z

@alexander-fedorenko @ElenaGallo we are too close to the release, therefore I'm going move to the next one (2022.01.02).

alexander-fedorenko · 2022-05-04T19:54:35Z

@ElenaGallo I tried to reproduce problems you had, unfortunately I was not able to do so. As per the screencasts you've shared it really looks like there was some problem with GeoServer running process on any step during file generation. It didn't return meaningful error message in your case, so the only way to trace the error is to reproduce error one more time and check GeoServer logs, or catch a period when problem appears and debug it locally with configured proxy to the dev instance of GeoServer.

ElenaGallo · 2022-05-05T08:09:15Z

Hi @alexander-fedorenko, this issue has been fixed here so I move this pr to test.

tdipisa · 2022-05-25T15:52:25Z

@ElenaGallo let us know if we can backport to 2022.01.xx.

ElenaGallo · 2022-05-26T09:35:29Z

Test passed, @alexander-fedorenko please backport to 2022.01.xx. Thanks

…counted when user exports grid data (geosolutions-it#7845) * geosolutions-it#1945 FeatureGrid layer data export will exclude hidden attributes. * Added tooltips in few places * - Keep attributes list undefined if no attribute is excluded. - Keep geometry in exported file when custom propertyName value is passed to the request. * Update for WPS export: using custom flow if layer is vector and custom attributes list is passed to the action. New observables to use for custom attributes list. * Minor changes required to make geometry properly exported when it's included into attributes list. * Using process reference to get results of gs:Download as an input for vec:Query * Minor corrections: selector name, variable name Added unit test for "getAttributesList" function (cherry picked from commit f62a9f3)

…er exports grid data (#7845) (#8232) * #1945 FeatureGrid layer data export will exclude hidden attributes. * Added tooltips in few places * - Keep attributes list undefined if no attribute is excluded. - Keep geometry in exported file when custom propertyName value is passed to the request. * Update for WPS export: using custom flow if layer is vector and custom attributes list is passed to the action. New observables to use for custom attributes list. * Minor changes required to make geometry properly exported when it's included into attributes list. * Using process reference to get results of gs:Download as an input for vec:Query * Minor corrections: selector name, variable name Added unit test for "getAttributesList" function (cherry picked from commit f62a9f3)

alexander-fedorenko added 2 commits February 7, 2022 23:16

geosolutions-it#1945 FeatureGrid layer data export will exclude hidde…

01d98d0

…n attributes.

Added tooltips in few places

2521f4d

alexander-fedorenko added this to the 2022.01.00 milestone Feb 7, 2022

alexander-fedorenko self-assigned this Feb 7, 2022

alexander-fedorenko added enhancement Good first issue Priority: Low labels Feb 7, 2022

- Keep attributes list undefined if no attribute is excluded.

14e9bd4

- Keep geometry in exported file when custom propertyName value is passed to the request.

alexander-fedorenko force-pushed the 1945-property_selection_featuregrid_export branch from 15d00e5 to 14e9bd4 Compare February 8, 2022 22:05

tdipisa added tmp and removed tmp labels Feb 11, 2022

Update for WPS export: using custom flow if layer is vector and custo…

6d2ae53

…m attributes list is passed to the action. New observables to use for custom attributes list.

tdipisa requested a review from MV88 February 22, 2022 11:49

tdipisa added the Blocked label Feb 25, 2022

tdipisa requested review from offtherailz and removed request for MV88 February 25, 2022 14:24

offtherailz marked this pull request as draft February 25, 2022 15:35

Minor changes required to make geometry properly exported when it's i…

d659d0b

…ncluded into attributes list.

Using process reference to get results of gs:Download as an input for…

5ec5a1c

… vec:Query

alexander-fedorenko marked this pull request as ready for review February 25, 2022 20:15

alexander-fedorenko changed the title ~~#1945 Make FeatureGrid attributes visibility selector counted when user exports grid data [WIP]~~ #1945 Make FeatureGrid attributes visibility selector counted when user exports grid data Feb 25, 2022

tdipisa removed the Blocked label Feb 28, 2022

tdipisa removed this from the 2022.01.00 milestone Mar 4, 2022

tdipisa added BackportNeeded Commits provided for an issue need to be backported to the milestone's stable branch Blocked labels Mar 7, 2022

tdipisa removed the Blocked label Mar 18, 2022

tdipisa requested review from MV88 and removed request for offtherailz March 28, 2022 14:58

MV88 requested changes Apr 6, 2022

View reviewed changes

Minor corrections: selector name, variable name

3610057

Added unit test for "getAttributesList" function

alexander-fedorenko requested a review from MV88 April 6, 2022 21:55

MV88 approved these changes Apr 7, 2022

View reviewed changes

MV88 merged commit f62a9f3 into geosolutions-it:master Apr 7, 2022

alexander-fedorenko mentioned this pull request May 26, 2022

[Backport 2022.01.xx] #1945 Make FeatureGrid attributes visibility selector counted when user exports grid data (#7845) #8232

Merged

alexander-fedorenko removed the BackportNeeded Commits provided for an issue need to be backported to the milestone's stable branch label May 26, 2022

MV88 added this to the v2022.01.02 milestone Jun 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#1945 Make FeatureGrid attributes visibility selector counted when user exports grid data #7845

#1945 Make FeatureGrid attributes visibility selector counted when user exports grid data #7845

alexander-fedorenko commented Feb 7, 2022

alexander-fedorenko commented Feb 13, 2022

tdipisa commented Feb 14, 2022

MV88 commented Feb 24, 2022

tdipisa commented Feb 25, 2022 •

edited

Loading

offtherailz commented Feb 25, 2022 •

edited

Loading

offtherailz commented Feb 25, 2022

offtherailz commented Feb 25, 2022 via email

alexander-fedorenko commented Feb 25, 2022

offtherailz commented Feb 25, 2022 •

edited

Loading

alexander-fedorenko commented Feb 25, 2022 •

edited

Loading

MV88 left a comment

MV88 Apr 6, 2022

alexander-fedorenko Apr 6, 2022

alexander-fedorenko commented Apr 6, 2022 •

edited

Loading

MV88 commented Apr 7, 2022 •

edited

Loading

ElenaGallo commented Apr 19, 2022

tdipisa commented Apr 20, 2022

alexander-fedorenko commented May 4, 2022

ElenaGallo commented May 5, 2022

tdipisa commented May 25, 2022

ElenaGallo commented May 26, 2022

#1945 Make FeatureGrid attributes visibility selector counted when user exports grid data #7845

#1945 Make FeatureGrid attributes visibility selector counted when user exports grid data #7845

Conversation

alexander-fedorenko commented Feb 7, 2022

Description

Issue

Breaking change

Other useful information

alexander-fedorenko commented Feb 13, 2022

tdipisa commented Feb 14, 2022

MV88 commented Feb 24, 2022

tdipisa commented Feb 25, 2022 • edited Loading

offtherailz commented Feb 25, 2022 • edited Loading

offtherailz commented Feb 25, 2022

offtherailz commented Feb 25, 2022 via email

alexander-fedorenko commented Feb 25, 2022

offtherailz commented Feb 25, 2022 • edited Loading

alexander-fedorenko commented Feb 25, 2022 • edited Loading

MV88 left a comment

Choose a reason for hiding this comment

MV88 Apr 6, 2022

Choose a reason for hiding this comment

alexander-fedorenko Apr 6, 2022

Choose a reason for hiding this comment

alexander-fedorenko commented Apr 6, 2022 • edited Loading

MV88 commented Apr 7, 2022 • edited Loading

ElenaGallo commented Apr 19, 2022

tdipisa commented Apr 20, 2022

alexander-fedorenko commented May 4, 2022

ElenaGallo commented May 5, 2022

tdipisa commented May 25, 2022

ElenaGallo commented May 26, 2022

tdipisa commented Feb 25, 2022 •

edited

Loading

offtherailz commented Feb 25, 2022 •

edited

Loading

offtherailz commented Feb 25, 2022 •

edited

Loading

alexander-fedorenko commented Feb 25, 2022 •

edited

Loading

alexander-fedorenko commented Apr 6, 2022 •

edited

Loading

MV88 commented Apr 7, 2022 •

edited

Loading