Better understanding the behavior of follow_symlinks

[forsyth2]

I created a script to test different cases involving links.

Expand for test script

# Test symlinks
# follow_symlinks introduced: https://github.com/E3SM-Project/zstash/pull/261
# follow_symlinks modifying-source-directory error fixed: https://github.com/E3SM-Project/zstash/pull/343

script_path="/global/homes/f/forsyth/ez/zstash_runs/symlink_scripts"

# 1) Test this using a zstash dev environment
# git fetch upstream main
# Choose one of the next 3:
# git checkout -b test-pre-261 98cd756e9fba2b657c5b1bfa727c91d4471eea7e
# git checkout -b test-post-261 a0947968ea9686d08fd8af6b64c5c96e78a76bc7
# git checkout -b test-post-343 6894798a9c4b762758444a8d1b8860ef13c7fb56
# Then:
# mamba clean --all
# mamba env create -f conda/dev.yml -n zstash_dev_{branch_name}
# conda activate zstash_dev_{branch_name}
# pip install .

# 2) Test with this set to true and false
follow_symlinks=true

# Run this test with every combination of 1 & 2 (3*2-1=5 runs because follow_symlinks won't work pre-261)
# Each time we try archiving and extracting for 6 cases:
# A symlink/hard link to a file in the directory being archived
# A symlink/hard link to a file in a different directory
# A symlink/hard link to a file that doesn't exist.

setup_dirs()
{
    # Set up the workspace
    rm -rf workdir1 workdir2 workdir3
    mkdir workdir1 workdir2 workdir3
    cd workdir1
    mkdir -p zstash_src/d1 zstash_src/d2 non-src
}

produce_output()
{
    
    # Begin viewing output
    echo ""
    echo "Output: zstash_src before create"
    echo "> ls -l  zstash_src/d2"
    ls -l  zstash_src/d2

    echo ""
    if [[ "${follow_symlinks,,}" == "true" ]]; then
        zstash create --hpss=none --follow-symlinks --cache ${script_path}/workdir2 zstash_src/d2 > /dev/null 2>&1
    else
        zstash create --hpss=none --cache ${script_path}/workdir2 zstash_src/d2 > /dev/null 2>&1
    fi

    echo ""
    echo "Output: zstash_src after create"
    echo "> ls -l  zstash_src/d2"
    ls -l  zstash_src/d2

    cd ../workdir3
    zstash extract --hpss=none --cache /global/homes/f/forsyth/ez/zstash_runs/symlink_scripts/workdir2 > /dev/null 2>&1

    cd ..
    echo ""
    echo "Output: zstash_src after extract"
    echo "> ls -l workdir3"
    ls -l workdir3

    echo ""
    echo "Output: make sure the contents are copied"
    echo "> cat workdir3/*"
    cat workdir3/*
}

case_symlink_internal()
{
    echo ""
    echo "CASE: internal symlink ##################################################"
    echo "text" > zstash_src/d1/symlink_src_in_zstash_src.txt
    ln -s ${script_path}/workdir1/zstash_src/d1/symlink_src_in_zstash_src.txt zstash_src/d2/symlink_src_in_zstash_src.txt
}

case_symlink_external()
{
    echo ""
    echo "CASE: external symlink ##################################################"
    echo "text" > non-src/symlink_src_not_in_zstash_src.txt
    ln -s ${script_path}/workdir1/non-src/symlink_src_not_in_zstash_src.txt zstash_src/d2/symlink_src_not_in_zstash_src.txt
}

case_symlink_broken()
{
    echo ""
    echo "CASE: broken symlink ##################################################"
    echo "text" > non-src/symlink_src_to_be_deleted.txt
    ln -s ${script_path}/workdir1/non-src/symlink_src_to_be_deleted.txt zstash_src/d2/symlink_src_to_be_deleted.txt
    rm -rf ${script_path}/workdir1/non-src/symlink_src_to_be_deleted.txt
}

case_hard_link_internal()
{
    echo ""
    echo "CASE: internal hard link ##################################################"
    echo "text" >zstash_src/d1/hard_link_src_in_zstash_src.txt
    ln ${script_path}/workdir1/zstash_src/d1/hard_link_src_in_zstash_src.txt zstash_src/d2/hard_link_src_in_zstash_src.txt
}

case_hard_link_external()
{
    echo ""
    echo "CASE: external hard link ##################################################"
    echo "text" > non-src/hard_link_src_not_in_zstash_src.txt
    ln ${script_path}/workdir1/non-src/hard_link_src_not_in_zstash_src.txt zstash_src/d2/hard_link_src_not_in_zstash_src.txt
}

case_hard_link_broken()
{
    echo ""
    echo "CASE: broken hard link ##################################################"
    echo "text" > non-src/hard_link_src_to_be_deleted.txt
    ln ${script_path}/workdir1/non-src/hard_link_src_to_be_deleted.txt zstash_src/d2/hard_link_src_to_be_deleted.txt
    rm -rf ${script_path}/workdir1/non-src/hard_link_src_to_be_deleted.txt
}

# Print what we're testing
cd  /global/homes/f/forsyth/ez/zstash
echo ""
echo "1) The current branch:"
git branch | grep "*"
cd - > /dev/null 2>&1
echo "2) follow_symlinks=${follow_symlinks}"
echo "We will archive workdir1/src"
echo "We will use workdir2 as our cache"
echo "We will extract into workdir3"

setup_dirs
case_symlink_internal
produce_output

setup_dirs
case_symlink_external
produce_output

setup_dirs
case_symlink_broken
produce_output

setup_dirs
case_hard_link_internal
produce_output

setup_dirs
case_hard_link_external
produce_output

setup_dirs
case_hard_link_broken
produce_output

Two pull requests are relevant:

• follow_symlinks introduced: Include symlinks #261
• follow_symlinks modifying-source-directory error fixed: Fix symlink behavior #343

Legend for the tables below:

First 3 output columns:

• 🔗 -> is present in the ls -l output, indicating a link
• ❌ -> is not present in the ls -l output, indicating not a link

Last output column:

• ✅ the word "text" is printed from running cat on the extracted output, indicating the contents of the file is accessible
• ❌ the word "text" is not present.

Using follow_symlinks, the ideal output row looks like: 🔗 🔗 ❌ ✅ (src unaffected, src unaffected, extraction is a copy, data is accessible).

Post-343

Post-343, using follow_symlinks

Case	zstash_src before create	zstash_src after create	extracted from zstash archive	make sure the file contents are copied
internal symlink	🔗	🔗	❌	✅
external symlink	🔗	🔗	❌	✅
broken symlink	🔗	🔗	N/A	No such file or directory
internal hard link	❌	❌	❌	✅
external hard link	❌	❌	❌	✅
broken hard link	❌	❌	❌	✅

Post-343, not using follow_symlinks

Case	zstash_src before create	zstash_src after create	extracted from zstash archive	make sure the file contents are copied
internal symlink	🔗	🔗	🔗	✅
external symlink	🔗	🔗	🔗	✅
broken symlink	🔗	🔗	🔗	No such file or directory
internal hard link	❌	❌	❌	✅
external hard link	❌	❌	❌	✅
broken hard link	❌	❌	❌	✅

Notice that the symlink is still present in the extracted output (second-to-last column). The only reason the file contents are accessible (last column) is because I archived on the same machine (i.e., the file path is still accessible).

Post-261

Post-261, using follow_symlinks

Case	zstash_src before create	zstash_src after create	extracted from zstash archive	make sure the file contents are copied
internal symlink	🔗	❌	❌	✅
external symlink	🔗	❌	❌	✅
broken symlink	🔗	Nothing in zstash_src/d2	Nothing in workdir3	No such file or directory
internal hard link	❌	❌	❌	✅
external hard link	❌	❌	❌	✅
broken hard link	❌	❌	❌	✅

In the first two rows, we see that zstash_src was altered by zstash create: the file is no longer a symlink! This was fixed in the post-343 output.

In the third row, we see that before #343, if we deleted a symlink's source before running zstash create, then zstash create would actually also end up deleting the symlink in the directory being archived as well (rather than keeping the broken link).

The hard links were unaffected by #343.

Post-261, not using follow_symlinks

Case	zstash_src before create	zstash_src after create	extracted from zstash archive	make sure the file contents are copied
internal symlink	🔗	🔗	🔗	✅
external symlink	🔗	🔗	🔗	✅
broken symlink	🔗	🔗	🔗	No such file or directory
internal hard link	❌	❌	❌	✅
external hard link	❌	❌	❌	✅
broken hard link	❌	❌	❌	✅

There is no difference at all between the post-261 and post-343 output when we don't use follow_symlinks.

Pre-261

Pre-261, not using follow_symlinks (because that option doesn't exist yet)

Case	zstash_src before create	zstash_src after create	extracted from zstash archive	make sure the file contents are copied
internal symlink	🔗	🔗	🔗	✅
external symlink	🔗	🔗	🔗	✅
broken symlink	🔗	🔗	🔗	No such file or directory
internal hard link	❌	❌	❌	✅
external hard link	❌	❌	❌	✅
broken hard link	❌	❌	❌	✅

There is no difference at all between the pre-261 and post-261 (and post-343) output when we don't use follow_symlinks.

Summarize the current changes

So, from pre-261 to post-343 what changed?

When not using follow_symlinks, nothing changed.

When using follow_symlinks:

Case	zstash_src before create	zstash_src after create	extracted from zstash archive	make sure the file contents are copied
internal symlink	🔗	🔗	🔗->❌	✅
external symlink	🔗	🔗	🔗->❌	✅
broken symlink	🔗	🔗	🔗->N/A	No such file or directory
internal hard link	❌	❌	❌	✅
external hard link	❌	❌	❌	✅
broken hard link	❌	❌	❌	✅

In summary, hard links aren't affected by follow_symlinks, and symlinks are left unaltered in the directory being archived but end up being actual copies in the extracted output (rather than links to some file that might not even be accessible). If the symlink's source is deleted before zstash create, the file doesn't get archived at all (not even a broken symlink).

[forsyth2]

@TonyB9000 @golaz I have summarized the behavior of follow_symlinks over time above.

[TonyB9000]

@forsyth2 That is an outstanding array of tests!

My only concern is the definition of "external" symlink (in the case of NO "--follow-symlinks"). The quintessential example being

/local_path_to/project/dir_1/fileX
/local_path_to/project/dir_2/symlink-for-fileX

The question is whether the symlink was defined by

symlink-for-fileX  -->   /local_path_to/project/dir_1/fileX    (absolute link)
or
symlink-for-fileX  -->   ../dir_1/fileX    (relative link)

When archiving WITHOUT --follow-symlinks, but archiving the whole of "project", the actual content of "fileX" will be captured in the archive under both absolute and relative links, But upon extraction, even though the fileX will be extracted, the link in directory "dir_2" will not work if the absolute link was used, because "local_path-to/" will not (necessarily) exist at the destination.

My question is whether your definition of "external link" applies to this absolute link. The relative link is certainly "internal".

[forsyth2]

The external symlink in the script is defined as:

ln -s ${script_path}/workdir1/non-src/symlink_src_not_in_zstash_src.txt zstash_src/d2/symlink_src_not_in_zstash_src.txt

That's using an absolute link. No relative links are used in the test script at all.

[forsyth2]

Similarly, the internal symlink is defined with an absolute path:

ln -s ${script_path}/workdir1/zstash_src/d1/symlink_src_in_zstash_src.txt zstash_src/d2/symlink_src_in_zstash_src.txt

[forsyth2]

Other notes

• I think we can assume the checksums are working fine, as the zstash extract command is functional.
• Should I add the tables above for using HPSS (instead of --hpss=none)?
• Can we do any sort of assertion in the zstash code to make sure it hasn't modified the directory being archived? git diff would work nicely if we were archiving git directories, but we're usually not.

[TonyB9000]

A directory can change for a variety of reasons (file was modified, added, removed, file permissions altered, directory permissions altered, etc.) I think perhaps pre- and post-examination of directory "mtime" might work. Simply accessing (reading) a directory will change the "atime" but not the "mtime". These values are accessible via the "stat" module.

[forsyth2]

I'm not familiar with many of these commands, but I think we definitely need to look into this.

[TonyB9000]

Briefly:

import os, stat
mtime = os.stat(source_dir)[stat.ST_MTIME]

If you check mtime before and after processing, and the values are equal, then no content changes to the directory have occurred,

[forsyth2]

@TonyB9000 I automated the testing script described above -- see #348.

@golaz Regarding behavior of tar dereferencing:

https://docs.python.org/3/library/tarfile.html states:

If dereference is False, add symbolic and hard links to the archive. If it is True, add the content of the target files to the archive. This has no effect on systems that do not support symbolic links.

This is the mechanism zstash is now using after the bug fix.

So, with that usage description + the added tests, I think we can say this is performing correctly now.

[TonyB9000]

Agreed! Thanks Ryan!