Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GitHub actions fail #183

Closed
ClaraBuettner opened this issue Mar 25, 2021 · 5 comments · May be fixed by #187
Closed

GitHub actions fail #183

ClaraBuettner opened this issue Mar 25, 2021 · 5 comments · May be fixed by #187
Assignees
@gnn
Labels
🧪 testing Tests and CI

Comments

@ClaraBuettner
Copy link
Contributor

ClaraBuettner commented Mar 25, 2021 •
edited
Loading

All GitHub actions of PR that include the current dev fail with the following error:

> 
>   ==================================== ERRORS ====================================
>   _ ERROR collecting .tox/py37-nocov-linux/lib/python3.7/site-packages/egon/data/airflow/dags/pipeline.py _
>   .tox/py37-nocov-linux/lib/python3.7/site-packages/egon/data/airflow/dags/pipeline.py:30: in <module>
>       airflow_db_connection()
>   .tox/py37-nocov-linux/lib/python3.7/site-packages/egon/data/db.py:108: in airflow_db_connection
>       cred = credentials()
>   .tox/py37-nocov-linux/lib/python3.7/site-packages/egon/data/db.py:28: in credentials
>       custom = config.paths(pid="*")[0]
>   E   IndexError: list index out of range
@ClaraBuettner ClaraBuettner added the 🧪 testing Tests and CI label Mar 25, 2021
@gnn
Copy link
Collaborator

gnn commented Mar 25, 2021

Ah, OK. I know what's happening here. The function egon.data.db.airflow_db_connection gets executed when egon.data.airflow.dags.pipeline gets imported, which is pretty much before anything else happens. This wasn't a problem when the connection parameters where read from packaged files. Now that they're read from the configuration file though, the problem is that this configuration file isn't yet generated at the point where airflow_db_connection tries to read the connection parameters, which triggers the error. I'll fix that.

@EvaWie
Copy link
Contributor

EvaWie commented Mar 26, 2021

There is another issue regarding
custom = config.paths(pid="*")[0]
IndexError: list index out of range:
On my system the egon-data.pid-6751.configuration.yaml file disappears, when I stop airflow in terminal, but on @ClaraBuettner 's system it does not. That means for me, I cannot test my code in spyder, when airflow is not running.
Can we do something about it? Thanks for your help!

@gnn gnn mentioned this issue Mar 29, 2021
Open
gnn added a commit that referenced this issue Mar 30, 2021
@gnn
Adjust tests to standalone egon_data function
68c3a0c 
I didn't adjust the tests when separating the `egon-data` logic from
`main` which obviously made the tests fail. Now the tests no longer
invoke `main`, which no longer is a `command`, but they invoke
`egon_data`, which is the new name of the main command containing the
`egon-data` logic, so now the tests should pass again.
This actually fixes #183.
@gnn
Copy link
Collaborator

gnn commented Mar 30, 2021

On my system the egon-data.pid-6751.configuration.yaml file disappears, when I stop airflow in terminal,

That is intentional. The "egon-data.pid-N.configuration.yaml" file is specific to the process with process id N, i.e. a specifig egon-data invocation, and is intended to live only as long as this process.

but on @ClaraBuettner 's system it does not.

Then this is a bug. Either there is still a running egon-data process to whom this file belongs, or a previous process failed to clean up and delete its file. Do you receive the message:

Ignoring supplied options. Found a configuration file belonging to a
different `egon-data` process. Using that one.

when running egon-data, @ClaraBuettner? Also, I tried to fix the issue at hand in PR #187 so could you please check if the branch fixes/#183-make-build-pass-again solves the issue you filed here?

That means for me, I cannot test my code in spyder, when airflow is not running.
Can we do something about it? Thanks for your help!

Sure. First, check whether your problem is still present on the fixes/#183-make-build-pass-again branch. If that's the case, post a detailed stack trace of the error, so that I can figure out where the line triggering the error is still used so that I can change it.

@gplssm
Copy link
Contributor

gplssm commented Mar 30, 2021

I ran into E IndexError: list index out of range as well, even when I executed a script directly from command-line (during development). Using fixes/#183-make-build-pass-again this is gone. Thanks 👏

@gnn gnn closed this as completed in 472ac8c Mar 31, 2021
@ClaraBuettner
Copy link
Contributor Author

Then this is a bug. Either there is still a running egon-data process to whom this file belongs, or a previous process failed to clean up and delete its file. Do you receive the message:

Ignoring supplied options. Found a configuration file belonging to a
different `egon-data` process. Using that one.

when running egon-data, @ClaraBuettner? Also, I tried to fix the issue at hand in PR #187 so could you please check if the branch fixes/#183-make-build-pass-again solves the issue you filed here?

Yes, I received this message before. But when I check out to the branch of PR #187, it is gone and the egon-data.pid-....configuration.yaml file is deleted when I stop airflow and recreated when I start it again. So this is fixed.

khelfen pushed a commit that referenced this issue Nov 13, 2024
@nailend
Fix changelog entry #183
3dd8c79
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gnn gnn

Labels
🧪 testing Tests and CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix automatic checks openego/eGon-data
4 participants
@gnn @gplssm @EvaWie @ClaraBuettner