memorycpu720 highio720 restapi not returning data

[vlimant]

this

https://cms-gwmsmon.cern.ch/prodview/json/historynew/memorycpu720/pdmvserv_task_BPH-RunIIFall18GS-00244__v1_T_200122_123649_5454/BPH-RunIIFall18GS-00244_0/success

is returning "No JSON object could be decoded".
that leaves unified unable to make any tuning of job requirements.

See "No JSON object could be decoded" from
https://cms-unified.web.cern.ch/cms-unified//logs/equalizor/last.log

[vlimant]

@juztas can you please point to the line in the code that forms the query that is broken at the moment?
@vkuznet would there be an equivalent query/api to kibana that would give the same result?

[vkuznet]

Jean-Roch, there is a grafana proxy which allows query data in ES and InfluxDB. But form your post it is unclear what is "the same result" means since ticket discusses "No JSON object could be decoded" issue. In other words, there is programmatic interface to Kibana, if you need concrete query please specify the query, and the data-source, and if possible open up CMSMONIT Jira ticket.

[juztas]

@vlimant The problem is not the code, but configuration file. Due to reinstall (not working for more than 6 months) it overwrote the config file and username/password was lost to access es-cms with password.
Queries are here: https://github.com/dmwm/gwmsmon/blob/master/src/prodview/app.py#L22-L28
CERN Ticket for pass - RQF1440737

[sharad1126]

@juztas @vlimant @vkuznet do we have an update here?
https://github.com/CMSCompOps/WmAgentScripts/blob/master/Unified/equalizor.py#L97
https://github.com/CMSCompOps/WmAgentScripts/blob/master/Unified/equalizor.py#L138
https://github.com/CMSCompOps/WmAgentScripts/blob/master/Unified/equalizor.py#L226

looks like these gwmsmon calls are still returning No JSON object could be decoded.

[vkuznet]

Well, if you take an output of curl call, e.g.

curl -L -k --key ~/.globus/userkey.pem --cert ~/.globus/usercert.pem "https://cms-gwmsmon.cern.ch/prodview/json/historynew/memorycpu720/pdmvserv_task_BPH-RunIIFall18GS-00244__v1_T_200122_123649_5454/BPH-RunIIFall18GS-00244_0/success"

you will see that it simply returns an HTML to the CERN Shibboleth authentication page. Therefore if you want programmatic access to this site your client should properly handle the Shibboleth authentication handshake. Since I have no idea how it was working before I can't say for sure what has been changed. But your python code which in fact relies on curl will not work due to lack of authentication. May be before this service accepted X509 certs and it is not the case anymore.

As I wrote already we have programmatic access to MONIT/ES/InfluxDB, the question is do we have similar data over there. This is a separate issue and you should provide concrete details of what you need and we can compare if this information is present or can be extracted from MONIT.

[vkuznet]

Probably you should start from Job Monitoring CMS dashboard, if you need to look-up concrete document from specific ES collection we already provide tools for that, e.g. here is a recipe to query MONIT/ES for WMArchive document of specific task name

# login to lxplus and use monit tool
/cvmfs/cms.cern.ch/cmsmon/monit -h

# prepare your query
cat > q.json << EOF
{"query":{"match":{"data.task": "/pdmvserv_task_TOP-RunIIAutumn18NanoAODv7-00113__v1_T_200419_152318_6256/TOP-RunIIAutumn18NanoAODv7-00113_0"}}, "from": 0, "size": 10}
EOF

# run your query with monit tool, for that you need to have valid CMS monitoring token
/cvmfs/cms.cern.ch/cmsmon/monit -token token -query=q.json -dbname=WMArchive

# and it will produce results in JSON format (similar as we get them in MONIT)

The new monit tool (we're planning to advertise it widely on up-coming C&O meeting) is available through cvmfs and works without any setup. If you need a valid token we generate them per user request, therefore please open the appropriate CMSMONIT Jira ticket and explain your use case.

[sharad1126]

https://cms-gwmsmon.cern.ch/prodview/json/historynew/memorycpu720/pdmvserv_task_BPH-RunIIFall18GS-00244__v1_T_200122_123649_5454/BPH-RunIIFall18GS-00244_0

returns no json could be decoded and we get the same through our CURL command as well. I don't think this is an authentication issue but the issue of json file not being created properly which is needed.

[sharad1126]

@vkuznet we won't like to complicate this and just prefer continue using gwmsmon. we want someone to check this and confirm why the json files aren't being available. personally, I don't want to add another dependency on monit in unified. @drkovalskyi @amaltaro @juztas do we know how we can get someone to fix this issue for us? @jenimal @z4027163 @dpiparo fyi

[drkovalskyi]

We need to think it through and see what we need and for what. I'm not familiar with such usages of gwmsmon and I don't know what it's trying to extract. We need to assess how critical this information is for Unified and Ops in general and then look for a solution. Let's talk about it offline.