Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tweak/enhance auto data install #50

Open
wants to merge 3 commits into
base: develop
Choose a base branch
from
Open

tweak/enhance auto data install #50

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0a1d8f4
adjust/enhance auto data install: if data was already installed, no n…
mperrin Dec 18, 2024
eff743f
update install docs for data auto install
mperrin Dec 18, 2024
df4a7c9
remove get_webbpsf_data_path compatibility layer
mperrin Dec 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 8 additions & 15 deletions docs/installation.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,14 +40,6 @@ STPSF and its underlying optical library POPPY may be installed from the `Python

Successfully installed stpsf

Note that ``pip install stpsf`` only installs the program code. **If you install via pip, you must manually download and install the data files, as** :ref:`described <data_install>` **below.**
To obtain source spectra for calculations, you should also follow :ref:`installation instructions for synphot <synphot_install>`.

.. note::
Installation through conda is not available as of STPSF version 1.1.0. Conda
users should instead follow the insructions in the preceding section to
install via pip.


.. _synphot_install:

Expand All @@ -61,21 +53,22 @@ Stsynphot is an optional dependency, but is highly recommended. Its installation
Installing the Required Data Files
----------------------------------

*If you install via pip or manually*, you must install the data files yourself.
.. note::
The necessary data files will be installed automatically in a default path, ``$HOME/data/stpsf-data``, and there is no user action required unless you want to install them somewhere else.

.. (If you install via Conda, the data files are automatically installed, in
which case you can skip this section.) [uncomment once conda installation is
available again]
Files containing such information as the JWST pupil shape, instrument throughputs, and aperture positions are distributed separately from STPSF. These sum to about 100 MB in size.

Files containing such information as the JWST pupil shape, instrument throughputs, and aperture positions are distributed separately from STPSF. To run STPSF, you must download these files and tell STPSF where to find them using the ``STPSF_PATH`` environment variable.
STPSF will now **automatically download and install these files** the first time it is run. By default these will be downloaded and installed into a folder ``$HOME/data/stpsf-data`` within the user's home directory. This directory will be created automatically if it doesn't exist already. If this path is acceptable to you, then no further action is needed on your part, and it'll all work automatically.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good addition! Thanks for cleaning up the docs!


*If you would like to place the data files at some other location*, you can manually install the data files yourself.

1. Download the following file: `stpsf-data-LATEST.tar.gz <https://stsci.box.com/shared/static/kqfolg2bfzqc4mjkgmujo06d3iaymahv.gz>`_ [approx. 70 MB]
2. Untar ``stpsf-data-LATEST.tar.gz`` into a directory of your choosing.
3. Set the environment variable ``STPSF_PATH`` to point to that directory. e.g. ::

export STPSF_PATH=$HOME/data/stpsf-data

for bash. (You will probably want to add this to your ``.bashrc``.)
for bash. (You will probably want to add this to your ``.bashrc``, or equivalent for the shell of your choice.)

You should now be able to successfully ``import stpsf`` in a Python session.

Expand All @@ -87,13 +80,13 @@ You should now be able to successfully ``import stpsf`` in a Python session.
.. Note::

**For STScI Users Only:** Users at STScI may access the required data files from the Central Storage network. Set the following environment variables in your ``bash`` shell. (You will probably want to add this to your ``.bashrc``.) ::

export STPSF_PATH="/grp/stpsf/stpsf-data"
export PYSYN_CDBS="/grp/hst/cdbs"

Software Requirements
---------------------


See `the requirements.txt specification file <https://github.com/spacetelescope/stpsf/blob/develop/requirements.txt>`_ for the required package dependencies.

**Required Python version**: STPSF 1.1 and above require Python 3.10 or higher.
Expand Down
38 changes: 22 additions & 16 deletions stpsf/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import sys
import warnings
from collections import OrderedDict
from pathlib import Path

import astropy.io.fits as fits
import astropy.units as u
Expand Down Expand Up @@ -179,16 +180,18 @@ def setup_logging(level='INFO', filename=None):
****************************************************************************
"""

def get_default_data_path():
return Path.home() / "data" / "stpsf-data"


def auto_download_stpsf_data():
import os
import tarfile
from pathlib import Path
from tempfile import TemporaryDirectory
from urllib.request import urlretrieve

# Create a default directory for the data files
default_path = Path.home() / "data" / "stpsf-data"
default_path = get_default_data_path()
default_path.mkdir(parents=True, exist_ok=True)

os.environ["STPSF_PATH"] = str(default_path)
Expand All @@ -206,6 +209,7 @@ def auto_download_stpsf_data():
# Extract the tarball
with tarfile.open(filename, "r:gz") as tar:
tar.extractall(default_path.parent, filter="fully_trusted")
warnings.warn(f"STPSF data files downloaded and installed to {default_path}.")

if not any(default_path.iterdir()):
raise IOError(f"Failed to get and extract STPSF data files to {default_path}")
Expand All @@ -216,30 +220,32 @@ def auto_download_stpsf_data():
def get_stpsf_data_path(data_version_min=None, return_version=False):
"""Get the STPSF data path

Simply checking an environment variable is not always enough, since
for packaging this code as a Mac .app bundle, environment variables are
not available since .apps run outside the Terminal or X11 environments.

Therefore, check first the environment variable STPSF_PATH, and secondly
Check first the environment variable STPSF_PATH, and secondly
check the configuration file in the user's home directory.

If data_version_min is supplied (as a 3-tuple of integers), it will be
compared with the version number from version.txt in the STPSF data
package.
"""
import os
from pathlib import Path

path_from_config = conf.STPSF_PATH # read from astropy configuration
if path_from_config == 'from_environment_variable':
path = os.getenv('STPSF_PATH')
if path is None:
message = (
'Environment variable $STPSF_PATH is not set!\n'
f'{MISSING_STPSF_DATA_MESSAGE} searching default location..s'
)
warnings.warn(message)
path = auto_download_stpsf_data()

# Check if the data were already downloaded to the default path
default_path = get_default_data_path()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change isn't necessary as this logic already exists within auto_download_stpsf_data (which also sets the STPSF_PATH which is necessary)

auto_download_stpsf_data() will make a new default directory (only if one doesn't exist), and download the data (only if the directory is empty).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My point here was intentionally refactoring to allow checking for a prior successful call to auto_download_stpsf_data, prior to displaying the big giant warning message about trying to download the data. If the data has already been downloaded successfully, it’s incorrect to display a warning that says “stpsf will try to download the data”.

The current code will display the warning text every time the package is imported in a new Python session, even if the data is already indeed present and fine in the default directory. That’s excess warnings for no benefit to the user.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see your concern (deleted my previous reply because I misunderstood what download message you were referring to)

I think its important that the Environment variable $STPSF_PATH is not set! warning is still displayed.

May just make more sense to move the MISSING_STPSF_DATA_MESSAGE inside auto_download_stpsf_data inside the if not any(default_path.iterdir()): codeblock.

if os.path.exists(default_path):
# If so, let's look there
path = default_path
else:
# If not, let's try to download the data
message = (
'Environment variable $STPSF_PATH is not set!\n'
f'{MISSING_STPSF_DATA_MESSAGE} searching default location.'
)
warnings.warn(message)
path = auto_download_stpsf_data()
else:
path = path_from_config

Expand Down Expand Up @@ -1069,4 +1075,4 @@ def get_target_phase_map_filename(apername):
raise ValueError("File wss_target_phase_{}.fits, \
not found under {}.".format(apername.split('_')[1].lower(),path))

return was_targ_file
return was_targ_file
Loading