Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: provide log retrieval commands #5

Merged
merged 4 commits into from
Aug 29, 2023

Conversation

jacderida
Copy link
Contributor

@jacderida jacderida commented Aug 26, 2023

  • d526a54 feat: provide log retrieval commands

    Log retrieval has two components: obtaining the files from S3 and reassembling them. There is a
    separate subcommand for each.

    Obtaining the files is a recursive process that involves visiting the hierarchy of folders in the
    bucket. Since there are lots of folders, this can be time consuming for a large testnet.

    Logstash is forwarding the files to the bucket in parts, every 30 minutes. For this reason, they
    need to be reassembled back into a single file. This is based on the part numbers in the file names.
    The use of different GUIDs on each file part is done intentionally by Logstash.

    The reason these have been implemented as separate subcommands is so that we can continually sync
    the logs from S3 if that is necessary, without re-downloading the parts we've already retrieved. The
    reassembly process removes the original part files.

    There has also been some simple logging enabled in this commit, just using the env_logger crate,
    since we don't really need anything sophisticated for this program.

  • b15451a chore: remove angus from key list

  • dedd402 feat: provide log copy command

    Use an Ansible playbook to copy logs from all the remote machines to the logs directory on the
    local machine. There are a few redundant directories that get copied as part of the directory
    structure, so there's a Python script that clears these up.

    This also switches on trace logging for the services by using SN_LOG=all.

  • 7a9f824 feat: provide logs rm command

    Remove the logs from a previous testnet run by deleting the entire folder from S3.

    When a new testnet is created we check to see if logs from a previous run with the same name already
    exist. We won't proceed unless the user deletes those logs. We offer the choice to delete or
    retrieve them before deletion.

    If we did proceed, the logs from the two testnets would be intermingled with each other and this
    would lead to a confusing situation.

Log retrieval has two components: obtaining the files from S3 and reassembling them. There is a
separate subcommand for each.

Obtaining the files is a recursive process that involves visiting the hierarchy of folders in the
bucket. Since there are lots of folders, this can be time consuming for a large testnet.

Logstash is forwarding the files to the bucket in parts, every 30 minutes. For this reason, they
need to be reassembled back into a single file. This is based on the part numbers in the file names.
The use of different GUIDs on each file part is done intentionally by Logstash.

The reason these have been implemented as separate subcommands is so that we can continually sync
the logs from S3 if that is necessary, without re-downloading the parts we've already retrieved. The
reassembly process removes the original part files.

There has also been some simple logging enabled in this commit, just using the `env_logger` crate,
since we don't really need anything sophisticated for this program.
Use an Ansible playbook to copy logs from all the remote machines to the `logs` directory on the
local machine. There are a few redundant directories that get copied as part of the directory
structure, so there's a Python script that clears these up.

This also switches on trace logging for the services by using `SN_LOG=all`.
Remove the logs from a previous testnet run by deleting the entire folder from S3.

When a new testnet is created we check to see if logs from a previous run with the same name already
exist. We won't proceed unless the user deletes those logs. We offer the choice to delete or
retrieve them before deletion.

If we did proceed, the logs from the two testnets would be intermingled with each other and this
would lead to a confusing situation.
@jacderida jacderida merged commit 49f2e11 into maidsafe:main Aug 29, 2023
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants