Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: provide log retrieval commands #5

Merged
merged 4 commits into from
Aug 29, 2023

Commits on Aug 25, 2023

  1. feat: provide log retrieval commands

    Log retrieval has two components: obtaining the files from S3 and reassembling them. There is a
    separate subcommand for each.
    
    Obtaining the files is a recursive process that involves visiting the hierarchy of folders in the
    bucket. Since there are lots of folders, this can be time consuming for a large testnet.
    
    Logstash is forwarding the files to the bucket in parts, every 30 minutes. For this reason, they
    need to be reassembled back into a single file. This is based on the part numbers in the file names.
    The use of different GUIDs on each file part is done intentionally by Logstash.
    
    The reason these have been implemented as separate subcommands is so that we can continually sync
    the logs from S3 if that is necessary, without re-downloading the parts we've already retrieved. The
    reassembly process removes the original part files.
    
    There has also been some simple logging enabled in this commit, just using the `env_logger` crate,
    since we don't really need anything sophisticated for this program.
    jacderida committed Aug 25, 2023
    Configuration menu
    Copy the full SHA
    d526a54 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b15451a View commit details
    Browse the repository at this point in the history
  3. feat: provide log copy command

    Use an Ansible playbook to copy logs from all the remote machines to the `logs` directory on the
    local machine. There are a few redundant directories that get copied as part of the directory
    structure, so there's a Python script that clears these up.
    
    This also switches on trace logging for the services by using `SN_LOG=all`.
    jacderida committed Aug 25, 2023
    Configuration menu
    Copy the full SHA
    dedd402 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2023

  1. feat: provide logs rm command

    Remove the logs from a previous testnet run by deleting the entire folder from S3.
    
    When a new testnet is created we check to see if logs from a previous run with the same name already
    exist. We won't proceed unless the user deletes those logs. We offer the choice to delete or
    retrieve them before deletion.
    
    If we did proceed, the logs from the two testnets would be intermingled with each other and this
    would lead to a confusing situation.
    jacderida committed Aug 29, 2023
    Configuration menu
    Copy the full SHA
    bfe8a93 View commit details
    Browse the repository at this point in the history