From 179bfd82e0f818f96a0887871e33078407a32fb0 Mon Sep 17 00:00:00 2001 From: Meliox Date: Sun, 18 Feb 2024 01:06:50 +0100 Subject: [PATCH] Update (#34) Add sftp support Heavy code cleanup and commenting Fixes multiple broken functions Update download urls for dependencies --- LICENCE | 2 +- README.md | 43 +- dependencies/ftp_list.sh | 96 --- dependencies/ftp_login.sh | 46 - dependencies/ftp_main.sh | 812 ----------------- dependencies/ftp_online_test.sh | 99 --- dependencies/help.sh | 253 +++--- dependencies/server_alive_test.sh | 106 +++ dependencies/server_list.sh | 100 +++ dependencies/server_login.sh | 104 +++ ...anagement.sh => server_size_management.sh} | 65 +- dependencies/setup.sh | 181 ++-- dependencies/sorting.sh | 87 +- dependencies/transfer_main.sh | 815 ++++++++++++++++++ ftpauto.sh | 770 +++++++++-------- install.sh | 427 +++++---- plugins/largefile.sh | 43 +- plugins/pushover.sh | 57 +- plugins/videofile.sh | 216 ++--- 19 files changed, 2296 insertions(+), 2026 deletions(-) delete mode 100644 dependencies/ftp_list.sh delete mode 100644 dependencies/ftp_login.sh delete mode 100644 dependencies/ftp_main.sh delete mode 100644 dependencies/ftp_online_test.sh create mode 100644 dependencies/server_alive_test.sh create mode 100644 dependencies/server_list.sh create mode 100644 dependencies/server_login.sh rename dependencies/{ftp_size_management.sh => server_size_management.sh} (53%) create mode 100644 dependencies/transfer_main.sh diff --git a/LICENCE b/LICENCE index b2ab207..88e7952 100644 --- a/LICENCE +++ b/LICENCE @@ -1,4 +1,4 @@ -Copyright (C) 2013-2018, Meliox +Copyright (C) 2013-2024, Meliox Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal diff --git a/README.md b/README.md index 026cea5..e092a90 100644 --- a/README.md +++ b/README.md @@ -1,21 +1,21 @@ # FTPauto -FTPauto is a simple, but highly advanced and configurable FTP-client wrap-around written in #Bash for Unix. It is based on [lftp](http://lftp.yar.ru/) and helps to automate simple transfers - yet FTPauto allows much more. +FTPauto is a simple, but highly advanced and configurable FTP-client wrap-around written in #Bash for Unix. It is based on [lftp](https://github.com/lavv17/lftp) and helps to manage transfers. # Features -* Send files easily with lftp (To and from FTP, and between serveres - FXP) -* Highly customizable command line -* Monitor free space or server status on FTP-server +* Send files easily with lftp (To and from FTP(s), and between serveres - FXP, and SFTP) +* Highly customizable command line optopns +* Monitor free space or server status on the remote server (pre/post check) * Progressbar with estimated time of transferes -* History (Transfer logs) -* Queue system of transfers (failed transferes, queue) +* History (Transfer logs, total transfered etc.) +* Queue manager for failed, queued transfere * Packing/splitting directory/files into rar-files. Including sfv to verify files at end-server. -* Multiple users (different configuration) +* Multiple user support * Delay transfer to start at a specific time * Support pre/post transfer using external scripts * Sorting (regex-based or manually) -* Exclution of files (regex-based) -* Seamless unpack of rar-files in stored in store-mode using rarmount +* Regex-based exclution of files or folders +* Seamless read-content from rar files using rar2fs fuse * Send push notification to phones etc. with [Pushover](https://pushover.net/) * Automatic transfers with the use of [FlexGet](http://flexget.com/) @@ -53,7 +53,7 @@ You should have User and sudo or root access to use and install, respectively. Script is mainly written to Debian/Ubuntu and is guaranteed to work under these! ## Installation -There are several way to install FTPauto. Briefly summarized: The installer does just about everything for you. This is also the most stable version. You can also get the most recent version from git by pulling it. +There are several way to install FTPauto. Sudo is necesary to install some dependent tools ### The installer (recommended) To get FTPauto, download and execute the installer: [download](https://raw.github.com/Meliox/FTPauto/master/install.sh) @@ -64,6 +64,11 @@ mkdir FTPauto && cd FTPauto wget https://raw.github.com/Meliox/FTPauto/master/install.sh bash install.sh install ``` +and update +```bash +bash install.sh update +``` + Follow the instructions to set up a user and then you're ready to use FTPauto! If you skipped user setup or need help go to [configuration](https://github.com/Meliox/FTPauto#configuration) to set up a user. @@ -77,21 +82,11 @@ Run bash install.sh install ``` -### Manual install -If you prefer to do everything manually, read the install.sh script. - -### Upgrading -If you want to upgrade to newest version, simply run -```bash -bash install.sh update -``` - -The same almost goes for git +and update ```bash git pull bash install.sh update ``` -NOTE: Running lastest version, doesn't mean it's 100% stable. # Configuration First thing that need to be done is to create a user and edit the users settings. The setting that is to be edited is shown in [settings](https://github.com/Meliox/FTPauto#settings). @@ -119,8 +114,8 @@ The most important setting is "transferetype". Depending on the solution you can FTPauto can do ``` FTP - SERVER --> client : upftp - server <-- CLIENT : downftp + SERVER --> client : upftp(s) or upsftp + server <-- CLIENT : downftp(s) FXP SERVER <-> server : fxp ``` @@ -198,7 +193,7 @@ Here's an overview as well --verbose | Debugs to console ``` ## Debugging -If the script for some reason should fail, it is easy to debug. Debugging can either be permanently set if the error comes and goes. This setting can be in ftpauto.sh by altering the line 3: +If the script for some reason should fail, it is easy to debug. Debugging can either be permanently set if the error comes and goes. This setting can be in ftpauto.sh by altering the line 3: ```bash verbose="0" #0 Normal info | 1 debug console | 2 debug into logfile ``` diff --git a/dependencies/ftp_list.sh b/dependencies/ftp_list.sh deleted file mode 100644 index 2d7ac43..0000000 --- a/dependencies/ftp_list.sh +++ /dev/null @@ -1,96 +0,0 @@ -#!/bin/bash -function ftp_list { - echo "INFO: Looking up content on ftp..." - if [[ -z $dir ]]; then - dir="/" - fi - while true; do - get_content "$dir" - read -p "Enter number to expand directory or download by also adding a \"d\", fx. 1d ? (x to exit) " - # make sure number is inserted - while ! [[ "$REPLY" =~ [0-9] ]] && [[ $REPLY != "x" ]]; do - read -p " Enter number to expand directory or download by also adding a \"d\", fx. 1d ? (x to exit) " - done - number=$REPLY - if [[ "$number" == "x" ]]; then - break - elif [[ "$number" == "0" ]]; then - # go to top path - dir="/" - continue - elif [[ "$number" == "1" ]]; then - # remove last path component - dir=$(dirname "$dir") - continue - elif [[ "$number" =~ ^[0-9]+d ]]; then - # extract number - number=${number%d} - # make download. - path="$dir$(echo ${array_list[$number]} | awk '{print $9}')" - download_argument+=("--user=$username") - download_argument+=("--path=/$path") - download_argument+=("--source=Manually") - # change option - option[0]="download" - option[1]="queue" - echo " Adding $path to queue" - main - # unset the added path and stay in current directory. - unset path download_argument option - else - # do normal listing - # make sure its a directory. Their size is generally 4096 bytes - if [[ "$(echo ${array_list[$number]} | awk '{print $5}')" -gt 4096 ]]; then - # not a directory - echo -e "\e[00;31m Cannot expand file: $(echo ${array_list[$number]} | awk '{print $9}')\e[00m" - continue - fi - dir="$dir$(echo ${array_list[$number]} | awk '{print $9}')/" - continue - fi - done - # Ask to start transfer if something is in queue - if [[ -e "$queue_file" ]]; then - read -p " Do you want to start the download (y/n)? " - if [[ $REPLY == "y" ]]; then - read -p " Do you wish to execute it as a background thread (y/n)? " - if [[ $REPLY == "y" ]]; then - background=true - fi - start_ftpmain - fi - fi -} -function get_content { - # remove old file first - rm -f "$ftplist_file" - loadDependency DFtpLogin && ftp_login 1 # we need to generate a new each time as download removes it - cat "$ftplogin_file1" >> "$ftplist_file" - echo "ls -aFl \"${dir}\" > ~/../..$ftp_content" >> "$ftplist_file" - echo "quit" >> "$ftplist_file" - $lftp -f "$ftplist_file" &> /dev/null - if [[ $? -eq 0 ]]; then - echo -e "\n\e[00;32mINFO: Listing content(s):\e[00m" - echo "Current path: $dir" - old_dir="$dir" - array_list=( ) - readarray array_list < "$ftp_content" - array_list=( "X X X X 4096 Feb XX XX ." "X X X X 4096 Feb XX XX .." "${array_list[@]}" ) - i=0 - # find length of array and add zero - for value in "${array_list[@]}"; do - if [[ $(echo $value | awk '{print $5}') -eq 4096 ]]; then - # directory - printf "%-8s\n" "$i : $(($(echo $value | awk '{print $5}')/(1024*1024)))MB $(echo -e "\e[00;34m$(echo $value | awk '{print $9}')\e[00m")" - else - # file - printf "%-8s\n" "$i : $(($(echo $value | awk '{print $5}')/(1024*1024)))MB $(echo $value | awk '{print $9}')" - fi - let i++ - done | column -c 1 -t - fi - # cleanup - rm -f "$ftplist_file" - rm -f "$ftp_content" - rm -f "$ftplogin_file1" -} diff --git a/dependencies/ftp_login.sh b/dependencies/ftp_login.sh deleted file mode 100644 index 5c4cb0d..0000000 --- a/dependencies/ftp_login.sh +++ /dev/null @@ -1,46 +0,0 @@ -#!/bin/bash -function ftp_login { - # generate ftp login details based in input. The argument passed is (1 or 2). - local number OIFS IFS ftpcustom ftpssl ftpuser ftppass ftphost ftppass ftploginfile - number="$1" - # correct variables so that they can be loaded properly later - ftpcustom="ftpcustom${number}" - ftpssl="ftpssl${number}" - ftpuser="ftpuser${number}" - ftppass="ftppass${number}" - ftphost="ftphost${number}" - ftpport="ftpport${number}" - ftploginfile="ftplogin_file${number}" - # set timeout settings - echo "set net:timeout 10" >> "${!ftploginfile}" - echo "set net:max-retries 3" >> "${!ftploginfile}" - echo "set net:reconnect-interval-base 10" >> "${!ftploginfile}" - echo "set net:reconnect-interval-multiplier 1" >> "${!ftploginfile}" - echo "set net:reconnect-interval-max 60" >> "${!ftploginfile}" - # write custom configurations to file, will overwrite any of above settings - if ((verbose)); then - echo "debug 8 -t -o $lftpdebug" >> "${!ftploginfile}" - fi - # write ssl setting to file - if [[ "${!ftpssl}" == true ]]; then - echo "set ftp:ssl-force true" >> "${!ftploginfile}" - echo "set ssl:verify-certificate false" >> "${!ftploginfile}" - fi - if [[ -n "${!ftpcustom}" ]]; then - OIFS="$IFS" - IFS=';' - for i in "${!ftpcustom[@]}"; do - echo "$i" >> "${!ftploginfile}" - done - IFS="$OIFS" - fi - # only allow the normal transfere types - if [[ "$transferetype" =~ "upftp" ]] || [[ "$transferetype" =~ "downftp" ]] || [[ "$transferetype" =~ "fxp" ]]; then - echo "open -u ${!ftpuser},${!ftppass} ${!ftphost} -p ${!ftpport}" >> "${!ftploginfile}" - else - echo -e "\e[00;31mERROR: Transfer-option \"$transferetype\" not recognized. Have a look on your config (--user=$user --edit)!\e[00m\n" - cleanup sesssion - cleanup end - exit 1 - fi -} diff --git a/dependencies/ftp_main.sh b/dependencies/ftp_main.sh deleted file mode 100644 index ed699a1..0000000 --- a/dependencies/ftp_main.sh +++ /dev/null @@ -1,812 +0,0 @@ -#!/bin/bash -function delay { - # if --delay is set, wait until it ends. If start/end time is set in config use them. Delay overrules everything - local current_epoch target_epoch sleep_seconds timediff - if [[ -n $delay ]]; then - current_epoch=$(date +%s) - target_epoch=$(date -d "$delay" +%s) - if [[ $target_epoch -gt $current_epoch ]]; then - sleep_seconds=$(( target_epoch - current_epoch )) - if [[ $test_mode != "true" ]]; then - echo "INFO: Transfere has been postponed until $delay" - timediff=$(printf '%2d:%2d:%2d' "$((sleep_seconds/(60*60)))" "$(((sleep_seconds/60)%60))" "$((sleep_seconds%60))") - countdown "$timediff" - else - echo -e "\e[00;31mTESTMODE: Would delay until $delay\e[00m" - fi - else - echo -e "\e[00;31mERROR: Time is older than current time, now $(date '+%m/%d/%y %H:%M') vs. delay $delay . Format should be mm/dd/yy hh:mm.\e[00m\n" - cleanup session - cleanup end - exit 1 - fi - elif [[ -n $transfer_start ]] && [[ -n $transfer_end ]] && [[ $force == "false" ]]; then - tranfere_timeframe - fi -} - -function countdown { - # calculate and show countdown in display - local OLD_IFS arr seconds start end cur left IFS - OLD_IFS="${IFS}" - IFS=":" - arr=( $1 ) - seconds=$(( (arr[0] * 60 * 60) + (arr[1] * 60) + arr[2] )) - start=$(date +%s) - end=$((start + seconds)) - cur=$start - while [[ $cur -lt $end ]]; do - cur=$(date +%s) - left=$((end-cur)) - printf "\r%02d:%02d:%02d" \ - $((left/3600)) $(((left/60)%60)) $((left%60)) - sleep 1 - done - IFS="${OLD_IFS}" - echo " " -} - -function queue { - # queuesystem. If something already is running for the user, add it to queue. - local option i old_id - option=$2 - case "$1" in - "add" ) - if [[ $queue_running == true ]]; then - # task has been started from queue, no need to add it - true - else - # figure out ID - id_old=$id - if [[ -e "$queue_file" ]]; then - #get last id - id=$(( $(tail -1 "$queue_file" | cut -d'|' -f1) + 1 )) - else - #assume this is the first one - id="1" - fi - get_size "$filepath" &> /dev/null - if [[ -e "$queue_file" ]] && [[ -n $(cat "$queue_file" | grep "$filepath") ]] && [[ -z $option ]]; then - # passing an item which is already in queue, do nothing - echo -e "INFO: Item already in queue. Doing nothing...\n" - exit 0 - elif [[ "$option" == failed ]]; then - # passing a failed item, remove it, and add it with the status failed - failed="true" - # remove ID from queue - sed "/^"$id_old"/d" -i "$queue_file" - echo "$id|$source|$filepath|$sortto|${size}MB|true|$(date '+%d/%m/%y-%a-%H:%M:%S')" >> "$queue_file" - echo -e "\e[00;33mINFO: Failing item: $(basename "$filepath")\e[00m" - elif [[ "$option" == end ]]; then - # passed item should only be queued, then exit - source="${source}Q" - echo "$id|$source|$filepath|$sortto|${size}MB|false|$(date '+%d/%m/%y-%a-%H:%M:%S')" >> "$queue_file" - echo -e "INFO: Queueing: $(basename "$filepath"), id=$id\n" - exit 0 - else - # passed item should be queued, e.g. when something already is being transferred - echo "INFO: Queueid: $id" - echo "$id|$source|$filepath|$sortto|${size}MB|false|$(date '+%d/%m/%y-%a-%H:%M:%S')" >> "$queue_file" - fi - fi - ;; - "remove" ) - #remove item according to id - sed "/^"$id"/d" -i "$queue_file" - # if queue is true then continue to run else stop - if [[ $continue_queue == true ]]; then - queue next - else - cleanup end - fi - ;; - "next" ) - # Process next item in queue from top - if [[ -f "$queue_file" ]] && [[ -n $(cat "$queue_file") ]]; then - i="1" - failed="true" - # look for non failed items - while [[ $failed == true ]]; do - # load next item from top - id=$(awk 'BEGIN{FS="|";OFS=" "}NR=='$i'{print $1}' "$queue_file") - # check if ID has failed - failed=$(awk 'BEGIN{FS="|";OFS=" "}NR=='$i'{print $6}' "$queue_file") - let i++ - done - if [[ $failed == false ]]; then - # found a non failed item, which will be downloaded - i=$(grep -n "^${id}|" "$queue_file" | grep -Eo '^[^:]+') - source=$(awk 'BEGIN{FS="|";OFS=" "}NR=='$i'{print $2}' "$queue_file") - filepath=$(awk 'BEGIN{FS="|";OFS=" "}NR=='$i'{print $3}' "$queue_file") - sort=$(awk 'BEGIN{FS="|";OFS=" "}NR=='$i'{print $4}' "$queue_file") - # execute main script again - queue_running="true" - if [[ -f "$lockfile" ]]; then - # ensure that lockfile isn't created running queue - lockfileRunning="true" - fi - echo "---------------------- Running queue ----------------------" - echo "Transfering id=$id, $(basename "$filepath")" - start_main --path="$filepath" --user="$username" --sortto="$sort" - else - # all items in the queue are marked as failed, e.g. nothing to transfer - echo "---------------------- Failed queue -----------------------" - while read line; do - id=$(echo $line | cut -d'|' -f1) - source=$(echo $line | cut -d'|' -f2) - path=$(echo $line | cut -d'|' -f3) - sort=$(echo $line | cut -d'|' -f4) - size=$(echo $line | cut -d'|' -f5) - time=$(echo $line | cut -d'|' -f7) - echo "ID|PATH|SORT TO|SIZE(MB)|TIME" - echo "$id|$source|$path|$sort|$size|$time" - done < "$queue_file" - echo -e "\nINFO: Queue does not contain non failed items. Program will end\n" - cleanup end - exit 1 - fi - else - # no queuefile found, e.g. nothing to transfer - echo "----------------------- Empty queue -----------------------" - if [[ -f "$queue_file" ]]; then rm "$queue_file"; fi - echo -e "INFO: Queue is empty. Program will end\n" - cleanup end - exit 0 - fi - ;; - "fail" ) - # failed item, remove it, and add it with the status failed - # remove ID from queue - sed "/^"$id"/d" -i "$queue_file" - echo "$id|$source|$filepath|$sortto|${size}MB|true|$(date '+%d/%m/%y-%a-%H:%M:%S')" >> "$queue_file" - echo -e "\e[00;33mINFO: Failing item: $(basename "$filepath")\e[00m" - ;; - esac -} - -function ftp_transfer_process { - # used to start and stop the lftp transfer and progressbar - local pid_f_process - case "$1" in - "start" ) #start progressbar and transfer - TransferStartTime=$(date +%s) - ftp_processbar & - pid_f_process=$! - sed "3c $pid_f_process" -i "$lockfile" - echo -e "\e[00;37mINFO: \e[00;32mTransfer started: $(date --date=@$TransferStartTime '+%d/%m/%y-%a-%H:%M:%S')\n\e[00m" - $lftp -f "$ftptransfere_file" &> /dev/null & - pid_transfer=$! - sed "2c $pid_transfer" -i "$lockfile" - wait $pid_transfer 2>/dev/null - pid_transfer_status=$? - TransferEndTime=$(date +%s) - ;; - "stop-process-bar" ) - kill $(sed -n '3p' $lockfile) &> /dev/null - wait $(sed -n '3p' "$lockfile") 2>/dev/null - kill $(sed -n '4p' $lockfile) &> /dev/null - wait $(sed -n '4p' "$lockfile") 2>/dev/null - - ;; - esac -} - -function ftp_transfere { - local lftp_exclude quittime waittime - #prepare new transfer - { - # Write regexp to config for directory transferes - if [[ "${#exclude_array[@]}" -gt 0 ]] && [[ -n "${exclude_array[@]}" ]] && ( [[ $transfer_type = directory ]] || [[ -d "$transfer_path" ]] ); then - for ((i=0;i<${#exclude_array[@]};i++)); do - if [[ $i -gt 0 ]]; then - lftp_exclude="$lftp_exclude|" - fi - lftp_exclude="$lftp_exclude^.*${exclude_array[i]}*" - done - lftp_exclude="$lftp_exclude$" - echo "set mirror:exclude-regex \"$lftp_exclude\"" >> "$ftptransfere_file" - echo "set mirror:no-empty-dirs true" >> "$ftptransfere_file" - fi - if [[ $transferetype == "downftp" ]]; then - # handle lftp transfere for downftp - { - cat "$ftplogin_file1" >> "$ftptransfere_file" - # create final directories if they don't exists - echo "!mkdir -p \"${ftpcomplete}\"" >> "$ftptransfere_file" - if [[ -n $ftpincomplete ]]; then - echo "!mkdir -p \"${ftpincomplete}\"" >> "$ftptransfere_file" - fi - # fail if transfers fails - echo "set cmd:fail-exit true" >> "$ftptransfere_file" - # from get_size we know if its a file or a path! - if [[ $transfer_type = file ]]; then - if [[ -n $ftpincomplete ]]; then - echo "queue get -c -O \"${ftpincomplete}\" \"${transfer_path}\"" >> "$ftptransfere_file" - elif [[ -z $ftpincomplete ]]; then - echo "queue get -c -O \"${ftpcomplete}\" \"${transfer_path}\"" >> "$ftptransfere_file" - fi - elif [[ $transfer_type = directory ]]; then - if [[ -n $ftpincomplete ]]; then - echo "queue mirror --no-umask -p --parallel=$parallel -c \"${transfer_path}\" \"${ftpincomplete}\"" >> "$ftptransfere_file" - elif [[ -z $ftpincomplete ]]; then - echo "queue mirror --no-umask -p --parallel=$parallel -c \"${transfer_path}\" \"${ftpcomplete}\"" >> "$ftptransfere_file" - fi - fi - # wait for transferes to finish - echo "wait" >> "$ftptransfere_file" - # moving part, locally - if [[ -n $ftpincomplete ]]; then - if [[ $transfer_type = file ]]; then - echo "queue !mv \"${ftpincomplete}${orig_name}\" \"${ftpcomplete}\"" >> "$ftptransfere_file" - elif [[ $transfer_type = directory ]]; then - echo "queue !mv \"${ftpincomplete}${orig_name}/\" \"${ftpcomplete}/\"" >> "$ftptransfere_file" - fi - echo "wait" >> "$ftptransfere_file" - fi - echo "wait" >> "$ftptransfere_file" - } - elif [[ $transferetype == "upftp" ]]; then - # handle lftp transfere for upftp - { - cat "$ftplogin_file1" >> "$ftptransfere_file" - echo "mkdir -p \"${ftpcomplete}\"" >> "$ftptransfere_file" - # handle files for transfer - if [[ -n "${ftpincomplete}" ]]; then - echo "mkdir -p \"${ftpincomplete}\"" >> "$ftptransfere_file" - fi - # fail if transfers fails - echo "set cmd:fail-exit true" >> "$ftptransfere_file" - if [[ -f "$transfer_path" ]]; then - # single file - if [[ -n "$ftpincomplete" ]]; then - echo "queue put -c -O \"$ftpincomplete\" \"${transfer_path}\" " >> "$ftptransfere_file" - elif [[ -z "$ftpincomplete" ]]; then - echo "queue put -c -O \"$ftpcomplete\" \"${transfer_path}\" " >> "$ftptransfere_file" - fi - elif [[ -d "$transfer_path" ]]; then - # directory - if [[ -n "$ftpincomplete" ]]; then - echo "queue mirror --no-umask -p --parallel=$parallel -c -RL \"${transfer_path}\" \"${ftpincomplete}\"" >> "$ftptransfere_file" #needs fixing - elif [[ -z "$ftpincomplete" ]]; then - echo "queue mirror --no-umask -p --parallel=$parallel -c -RL \"${transfer_path}\" \"${ftpcomplete}\"" >> "$ftptransfere_file" #needs fixing - fi - fi - # wait for transferes to finish - echo "wait" >> "$ftptransfere_file" - # moving part, remotely, if ftpincomplete directory is used - if [[ -n "$ftpincomplete" ]]; then - # correction for file and path - if [[ -f "$filepath" ]]; then - echo "queue mv \"${ftpincomplete}${orig_name}\" \"${ftpcomplete}\"" >> "$ftptransfere_file" - elif [[ -d "$filepath" ]]; then - echo "queue mv \"${ftpincomplete}${orig_name}/\" \"${ftpcomplete}\"" >> "$ftptransfere_file" - fi - echo "wait" >> "$ftptransfere_file" - fi - } - elif [[ $transferetype == "fxp" ]]; then - # handle lftp transfere for fxp - ftp_login 2 - cat "$ftplogin_file2" >> "$ftptransfere_file" - # first login and create final directories if they don't exists on ftphost2 - echo "mkdir -p \"$ftpcomplete\"" >> "$ftptransfere_file" - if [[ -n $ftpincomplete ]]; then - echo "mkdir -p \"$ftpincomplete\"" >> "$ftptransfere_file" - fi - # fail if transfers fails - echo "set cmd:fail-exit true" >> "$ftptransfere_file" - # from get_size we know if its a file or a path! - if [[ $transfer_type = file ]]; then - # single file - if [[ -n "$ftpincomplete" ]]; then - echo "queue get -c ftp://$ftpuser1:$ftppass1@$ftphost1:$ftpport1:\"$transfer_path\" -o ftp://$ftpuser2:$ftppass2@$ftphost2:$ftpport2:\"$ftpincomplete\"" >> "$ftptransfere_file" - elif [[ -z "$ftpincomplete" ]]; then - echo "queue get -c ftp://$ftpuser1:$ftppass1@$ftphost1:$ftpport1:\"$transfer_path\" -o ftp://$ftpuser2:$ftppass2@$ftphost2:$ftpport2:\"$ftpcomplete\"" >> "$ftptransfere_file" - fi - elif [[ $transfer_type = directory ]]; then - # directory - if [[ -n "$ftpincomplete" ]]; then - echo "queue mirror --no-umask -p --parallel=$parallel -c -RL ftp://$ftpuser1:$ftppass1@$ftphost1:$ftpport1:\"${transfer_path}\" ftp://$ftpuser2:$ftppass2@$ftphost2:$ftpport2:\"${ftpincomplete}\"" >> "$ftptransfere_file" #needs fixing - elif [[ -z "$ftpincomplete" ]]; then - echo "queue mirror --no-umask -p --parallel=$parallel -c -RL ftp://$ftpuser1:$ftppass1@$ftphost1:$ftpport1:\"${transfer_path}\" ftp://$ftpuser2:$ftppass2@$ftphost2:$ftpport2:\"${ftpincomplete}\"" >> "$ftptransfere_file" #needs fixing - fi - fi - # wait for transferes to finish - echo "wait" >> "$ftptransfere_file" - # moving part, remotely, if ftpincomplete directory is used - if [[ -n "$ftpincomplete" ]]; then - # correction for file and path - if [[ $transfer_type = file ]]; then - echo "queue mv \"${ftpincomplete}${orig_name}\" \"${ftpcomplete}\"" >> "$ftptransfere_file" - elif [[ $transfer_type = directory ]]; then - echo "queue mv \"${ftpincomplete}${orig_name}/\" \"${ftpcomplete}\"" >> "$ftptransfere_file" - fi - echo "wait" >> "$ftptransfere_file" - fi - else - echo -e "\e[00;31mERROR: FTP setting not recognized\e[00m\n" - cleanup die - fi - echo "quit" >> "$ftptransfere_file" - } - #start transferring - { - if [[ $test_mode != "true" ]]; then - ftp_transfer_process start - #did lftp end properly - while [[ $pid_transfer_status -ne 0 ]]; do - quittime=$(( ScriptStartTime + retry_download_max*60 )) #minutes - if [[ $(date +%s) -gt $quittime ]]; then - echo -e "\e[00;31mERROR: FTP transfer failed after max ($retry_download_max minutes)!\e[00m" - # remove processbar processes - ftp_transfer_process "stop-process-bar" - # mark transfer as failed - queue add failed - break - else - echo -e "\e[00;31mERROR: FTP transfer failed for some reason!\e[00m" - echo "INFO: Retrying until $(date --date=@$quittime '+%d/%m/%y-%a-%H:%M:%S')" - # Kill processbar - ftp_transfer_process "stop-process-bar" - echo "INFO: Pausing session and trying again in 60s" - sed "3s#.*#*************************** FTP INFO: DOWNLOAD POSTPONED! Trying again in ${retry_download}mins#" -i "$logfile" - sleep 60 - # restart transfer - ftp_transfer_process start - fi - done - #remove processbar processes - ftp_transfer_process "stop-process-bar" - echo -e "\n\e[00;37mINFO: \e[00;32mTransfer ended: $(date --date=@$TransferEndTime '+%d/%m/%y-%a-%H:%M:%S')\e[00m" - else - echo -e "\e[00;31mTESTMODE: LFTP-transfer NOT STARTED\e[00m" - echo "Would execute the following in lftp:" - cat "$ftptransfere_file" | (while read; do echo " $REPLY"; done) - fi - } -} - -function ftp_processbar { #Showing how download is proceeding - local transfered_size ProgressTimeNew TransferredNew TransferredNewMB TotalTimeDiff TimeDiff percentage speed eta etatime SpeedOld sum SpeedAverage cols percentagebarlength string string2 TransferredOld ProgressTimeOld - if [[ $test_mode != "true" ]]; then - sleep 5 #wait for transfer to start - if [[ $transferetype == "downftp" ]]; then - transfered_size="du -s \"$ftpincomplete$changed_name\" > \"$proccess_bar_file\"" - if [[ $transfer_type = file ]]; then - echo "du -s \"$ftpincomplete${orig_name}\" > ~/../..$proccess_bar_file" >> "$ftptransfere_processbar" - elif [[ $transfer_type = directory ]]; then - echo "du -s \"$ftpincomplete$orig_name\" > ~/../..$proccess_bar_file" >> "$ftptransfere_processbar" - fi - elif [[ $transferetype == "upftp" ]]; then - #Create configfile for lftp processbar - cat "$ftplogin_file1" >> "$ftptransfere_processbar" - # ~ is /home/USER/ - if [[ -f "$filepath" ]]; then - echo "du -s \"$ftpincomplete${orig_name}\" > ~/../..$proccess_bar_file" >> "$ftptransfere_processbar" - elif [[ -d "$filepath" ]]; then - echo "du -s \"$ftpincomplete$orig_name\" > ~/../..$proccess_bar_file" >> "$ftptransfere_processbar" - fi - echo "quit" >> "$ftptransfere_processbar" - elif [[ $transferetype == "fxp" ]]; then - #Create configfile for lftp processbar - cat "$ftplogin_file2" >> "$ftptransfere_processbar" - # ~ is /home/USER/ - if [[ $transfer_type = file ]]; then - echo "du -s \"$ftpincomplete${orig_name}\" > ~/../..$proccess_bar_file" >> "$ftptransfere_processbar" - elif [[ $transfer_type = directory ]]; then - echo "du -s \"$ftpincomplete$orig_name\" > ~/../..$proccess_bar_file" >> "$ftptransfere_processbar" - fi - echo "quit" >> "$ftptransfere_processbar" - fi - { #run processbar loop - while :; do - if [[ $transferetype == "downftp" ]]; then - eval $transfered_size - elif [[ $transferetype == "upftp" ]] || [[ $transferetype == "fxp" ]]; then - $lftp -f "$ftptransfere_processbar" &> /dev/null & - pid_process=$! - sed "4c $pid_process" -i "$lockfile" - wait $pid_process - fi - # get first time and size. First time, set time, restart loop - if [[ ! -a "$proccess_bar_file" ]]; then - # no transferred information - continue - elif [[ -z "$TransferredOld" ]] && [[ -a "$proccess_bar_file" ]]; then - # transferred information available - TransferredOld=$(cat $proccess_bar_file | awk '{print $1}') - ProgressTimeOld=$(date +%s) - rm "$proccess_bar_file" - continue - fi - # Feedback received - if [[ -a "$proccess_bar_file" ]]; then - # set current time - ProgressTimeNew=$(date +%s) - # Get new transferred information - TransferredNew=$(cat "$proccess_bar_file" | awk '{print $1}') - TransferredNewMB=$(echo $TransferredNew / 1024 | bc) - TotalTimeDiff=$(( ProgressTimeNew - TransferStartTime )) - # calculate data - TimeDiff=$(printf '%02dh:%02dm:%02ds' "$((TotalTimeDiff/(60*60)))" "$(((TotalTimeDiff/60)%60))" "$((TotalTimeDiff%60))") - # Ensure value are valid - if [[ "$(( $TransferredNew - $TransferredOld ))" -ge "1" ]] && [[ "$(( $TransferredNew - $TransferredOld ))" =~ ^[0-9]+$ ]]; then - percentage=$(echo "scale=4; ( $TransferredNew / ( $directorysize / ( 1024 ) ) ) * 100" | bc) - percentage=$(echo $percentage | sed 's/\(.*\)../\1/') - speed=$(echo "scale=2; ( ($TransferredNew - $TransferredOld) / 1024 ) / ( $ProgressTimeNew - $ProgressTimeOld )" | bc) # MB/s - eta=$(echo "( ($directorysize / 1024 ) - $TransferredNew ) / ($speed * 1024 )" | bc) - etatime=$(printf '%02dh:%02dm:%02ds' "$(($eta/(60*60)))" "$((($eta/60)%60))" "$(($eta%60))") - # Calculate average speed. Needs to be calculated each time as transfer stops ftp_processbar - SpeedOld+=( "$speed" ) - if [[ -n "${#SpeedOld[@]}" ]]; then - sum="0" - for i in "${SpeedOld[@]}"; do - sum=$(echo "( $sum + $i )" | bc) - done - SpeedAverage=$(echo "scale=2; $sum / ${#SpeedOld[@]}" | bc) - sed "5c $SpeedAverage" -i "$lockfile" - # we can start overwriting progresline - tput cuu 1; tput el1 - fi - else - speed="?" - SpeedAverage="?" - percentage="0" - etatime="?" - fi - #update file and output the current line - sed "5s#.*#*************************** Transferring: ${orig_name}, $percentage\%, in $TimeDiff, $speed MB/s(current), ETA: $etatime, ${SpeedAverage} MB/s (avg) #" -i "$logfile" - cols=$(($(tput cols) - 2)) - percentagebarlength=$(echo "scale=0; $percentage * $cols / 100" | bc) - string="$(eval printf "=%.0s" '{1..'"$percentagebarlength"\})" - string2="$(eval printf "\ %.0s" '{1..'"$(($cols - $percentagebarlength - 1))"\})" - if [[ $percentagebarlength -eq 0 ]]; then - printf "\r[$string2] (no transfere information yet) ($(date '+%H:%M:%S'))" - elif [[ $(echo "scale=0; $cols - $percentagebarlength - 1" | bc) -eq 0 ]]; then - printf "\r[$string>] $percentage%% ETA ${etatime}@${speed}MB/s. ${TransferredNewMB}MB@${SpeedAverage}MB/s(avg). ($(date '+%H:%M:%S'))" - else - printf "\r[$string>$string2] $percentage%% ETA ${etatime}@${speed}MB/s. ${TransferredNewMB}MB@${SpeedAverage}MB/s(avg). ($(date '+%H:%M:%S'))" - fi - fi - # update variables and wait - TransferredOld="$TransferredNew" - ProgressTimeOld="$ProgressTimeNew" - rm "$proccess_bar_file" - sleep $sleeptime - done - } - #new line - echo -ne '\n' - else - echo -e "\e[00;31mTESTMODE: LFTP-processbar NOT STARTED\e[00m" - fi -} - -function logrotate { - if [[ $test_mode != "true" ]]; then - transferTime=$(( $TransferEndTime - $TransferStartTime )) - transferTime2=$(printf '%02dh:%02dm:%02ds' "$(($transferTime/(60*60)))" "$((($transferTime/60)%60))" "$(($transferTime%60))") - SpeedAverage=$(sed -n 5p "$lockfile") - #Adds new info to 7th line - sed "7i $(date --date=@$ScriptStartTime '+%d/%m/%y-%a-%H:%M:%S')|${source}|${orig_name}|${size}MB|${transferTime2}|${SpeedAverage}MB/s" -i "$logfile" - lognumber=$((7 + $lognumber )) - #Add text to old file - if [[ $logrotate == "true" ]]; then - if [[ -n $(sed -n $lognumber,'$p' "$logfile") ]]; then - sed -n $lognumber,'$p' "$logfile" >> "$oldlogfile" - fi - fi - #Remove text from old file - if [ "$lognumber" -ne 0 ]; then - sed $lognumber,'$d' -i "$logfile" - fi - totaldl=$(awk 'BEGIN{FS="|";OFS=" "}NR==2{print $1}' "$logfile" | cut -d' ' -f4) - totaldl=${totaldl%MB} - if [[ -z "$totaldl" ]]; then - totaldl="0" - fi - totaldl=$(echo "$totaldl + $size" | bc) - totalrls=$(awk 'BEGIN{FS="|";OFS=" "}NR==2{print $1}' "$logfile" | cut -d' ' -f6) - totalrls=$(echo "$totalrls + 1" | bc) - totaldltime=$(awk 'BEGIN{FS="|";OFS=" "}NR==2{print $1}' "$logfile" | cut -d' ' -f10) - totaldltime_seconds=$(awk 'BEGIN{split("'$totaldltime'",a,":"); print a[1]*(60*60*24)+a[2]*(60*60)+a[3]*60+a[4];}') - totaldltime=$(echo "$totaldltime_seconds + $transferTime" | bc) - totaldltime=$(printf '%02dd:%02dh:%02dm:%02ds' "$(($totaldltime/(60*60*24)))" "$(($totaldltime/(60*60)%24))" "$((($totaldltime/60)%60))" "$(($totaldltime%60))") - - sed "1s#.*#***** FTPauto ${s_version}#" -i "$logfile" - sed "2s#.*#***** STATS: ${totaldl}MB in ${totalrls} transfers in ${totaldltime}#" -i "$logfile" - sed "3s#.*#***** FTP INFO: N/A#" -i "$logfile" - sed "4s#.*#***** LASTDL: $(date)|${orig_name}|${SpeedAverage}MB/s#" -i "$logfile" - sed "5s#.*#***** #" -i "$logfile" - else - echo -e "\e[00;31mTESTMODE: LOGGING NOT STARTED\e[00m" - fi -} - -function loadConfig { - # reload config - loadDependency DConfig - - #load paths to everything - setup -} - -function lockfile { - # upon start (from queue or --path) no option is available, hence create lockfile - if [[ -f "$lockfile" ]] && [[ $force != "true" ]]; then - # The file exists, find PID, transfere, confirm it still is running - mypid_script=$(sed -n 1p "$lockfile") - mypid=$(sed -n 2p "$lockfile") - alreadyinprogres=$(sed -n 3p "$lockfile") - kill -0 $mypid_script &> /dev/null - if [[ $? -eq 1 ]]; then - #Process is not running, continue - echo "INFO: Old lockfile found, but process is not running" - rm -f "$lockfile" - else - echo "INFO: Already running" - echo " The script pid: $mypid_script" - echo " The transfere pid: $alreadyinprogres" - echo " Transfer: $(sed -n 5p < "$logfile" | cut -d',' -f1 | cut -d' ' -f2)" - echo " Lockfile: $lockfile" - echo " See --help on how to stop it" - queue add end - fi - fi - # allocate pids - echo >> "$lockfile" # bash pid - echo >> "$lockfile" # lftp transfer pid - echo >> "$lockfile" # bash progress pid - echo >> "$lockfile" # lftp process pid - echo >> "$lockfile" # speedaverage - sed "1c $BASHPID" -i "$lockfile" - echo "INFO: Process id: $BASHPID" -} - -function main { - #setting paths - filepath="$1" - transfer_path="$filepath" - orig_name=$(basename "$filepath") - # if filepath is a file, correct temppath - if [[ -f "$filepath" ]]; then - tempdir="$scriptdir/run/$username-temp/${orig_name%.*}-temp/" - elif [[ -d "$filepath" ]]; then - tempdir="$scriptdir/run/$username-temp/${orig_name}-temp/" - fi - ScriptStartTime=$(date +%s) - echo "INFO: Process start-time: $(date --date=@$ScriptStartTime '+%d/%m/%y-%a-%H:%M:%S')" - echo "INFO: Preparing transfer: $filepath" - echo "INFO: Lunched from: $source" - - #add to queue file, to get ID initialized - queue add - echo "INFO: Simultaneous transfers: $parallel" - #Checking transferesize - get_size "$filepath" - #Execute preexternal command - if [[ -n "$exec_pre" ]]; then - if [[ $test_mode != "true" ]]; then - echo "INFO: Executing external command - START" - echo " $exec_pre" - eval "$exec_pre" | (while read; do echo " $REPLY"; done) - else - echo -e "\e[00;31mTESTMODE: Would execute external command: \"$exec_pre\"\e[00m" - fi - echo "INFO: Executing external command - ENDED" - fi - - #Prepare login - loadDependency DFtpLogin && ftp_login 1 - - #confirm server is online - if [[ $confirm_online == "true" ]]; then - loadDependency DFtpOnlineTest && online_test - fi - - #Check if enough free space on ftp - if [[ "$ftpsizemanagement" == "true" ]]; then - if [[ $transferetype == "upftp" ]]; then - loadDependency DFtpSizeManagement && ftp_sizemanagement check - elif [[ $transferetype == "downftp" ]]; then - freesize=$(( $(df -P "$ftpincomplete" | tail -1 | awk '{ print $3}') / (1024*1024) )) - freespaceneeded="$size" - while [[ "$freesize" -lt "$freespaceneeded" ]]; do - echo "INFO: Not enough free space" - echo "INFO: Trying again in 1 min" - sleep 60 - # recalculate free space - freesize=$(( $(df -P "$ftpincomplete" | tail -1 | awk '{ print $3}') / (1024*1024) )) - done - fi - fi - - ## Sendoption - echo "INFO: Sendoption: $send_option" - #Is largest file too large - if [[ "$send_option" == "split" ]]; then - if [[ -n $(builtin type -p rar) ]] || [[ -n $(builtin type -p cksfv) ]]; then - if [[ $transferetype == "upftp" ]]; then - loadDependency DLargeFile && largefile "$filepath" "exclude_array[@]" - elif [[ $transferetype == "downftp" ]]; then - echo -e "\e[00;33mERROR: send_option=split is not supported in mode=$transferetype. Exiting ...\e[00m" - cleanup session; cleanup end - echo -e "INFO: Program has ended\n" - exit 1 - fi - else - echo -e "\e[00;33mERROR: send_option=split is not supported as rar or cksfv is missing. Exiting ...\e[00m" - cleanup session; cleanup end - echo -e "INFO: Program has ended\n" - exit 1 - fi - # Try to only send videofile - elif [[ "$send_option" == "video" ]]; then - if [[ -n $(builtin type -p rar2fs) ]]; then - if [[ $transferetype == "upftp" ]]; then - loadDependency DVideoFile && videoFile - elif [[ $transferetype == "downftp" ]]; then - echo -e "\e[00;33mERROR: send_option=video is not supported in mode=$transferetype. Exiting ...\e[00m" - cleanup session; cleanup end - echo -e "INFO: Program has ended\n" - exit 1 - fi - else - echo -e "\e[00;33mERROR: send_option=video is not supported as rarfs is missing. Exiting ...\e[00m" - cleanup session; cleanup end - echo -e "INFO: Program has ended\n" - exit 1 - fi - fi - - # Try to sort files - if [[ "$sort" == "true" ]] || [[ -n "$sortto" ]]; then - loadDependency DSort && sortFiles "$sortto" - fi - - # Delay transfer if needed - delay - - # Transfer files - ftp_transfere - - # Checking for remaining space - if [[ "$ftpsizemanagement" == "true" ]] && [[ $failed != true ]]; then - ftp_sizemanagement info # already loaded previously - fi - - # Update logfile - if [[ $failed != true ]]; then - logrotate - fi - - # Clean up current session - cleanup session - - #send push notification - if [[ -n $push_user ]]; then - if [[ $test_mode ]]; then - echo -e "\e[00;31mTESTMODE: Would send notification \""$orig_name" "Sendoption=$send_option Size=$size MB Time=$transferTime2 Average speed=$SpeedAverage MB/s Path=$ftpcomplete"\" to token=$push_token and user=$push_user \e[00m" - elif [[ $failed == true ]]; then - loadDependency DPushOver && Pushover "Failed: $orig_name" "Sendoption: $send_option -Size: $size MB -Path: $ftpcomplete" - else - loadDependency DPushOver && Pushover "$orig_name" "Sendoption: $send_option -Size: $size MB -Time: $transferTime2 -Average speed: $SpeedAverage MB/s -Path: $ftpcomplete" - fi - fi - - #Execute external command - if [[ -n $exec_post ]] && [[ $failed != true ]]; then - if [[ $test_mode != "true" ]]; then - if [[ $allow_background == "true" ]]; then - echo "INFO: Executing external command(In background) - START" - echo " $exec_post" - eval $exec_post & - else - echo "INFO: Executing external command - START:" - echo " $exec_post" - eval $exec_post | (while read; do echo " $REPLY"; done) - echo "INFO: Executing external command - ENDED" - fi - else - echo -e "\e[00;31mTESTMODE: Would execute external command: \"$exec_post\"\e[00m" - fi - fi - - # final - ScriptEndTime=$(date +%s) - TotalTransferTime=$(( $ScriptEndTime - $ScriptStartTime )) - if [[ $failed == true ]]; then - echo -e "\e[00;37mINFO: \e[00;31mTransfere failed\e[00m" - else - echo -e "\e[00;37mINFO: \e[00;32mTransfere finished\e[00m" - fi - echo " Name: $orig_name" - echo " Size: $size MB" - echo " Speed: $SpeedAverage MB/s" - echo " Transfer time: $transferTime2" - echo " Start time: $(date --date=@$ScriptStartTime '+%d/%m/%y-%a-%H:%M:%S')" - echo " End time: $(date --date=@$ScriptEndTime '+%d/%m/%y-%a-%H:%M:%S')" - echo " Total time: $(printf '%02dh:%02dm:%02ds' "$(($TotalTransferTime/(60*60)))" "$((($TotalTransferTime/60)%60))" "$(($TotalTransferTime%60))")" - - # Remove finished one - if [[ $failed != true ]]; then - queue remove - fi - # Run queue - queue next -} - -function start_main { - #Look for which options has been used - while :; do - case "$1" in - --path=* ) filepath="${1#--path=}"; shift;; - --user=* ) user="${1#--user=}"; shift;; - --exec_post=* ) exec_post="${1#--exec_post=}"; shift;; - --exec_pre=* ) exec_pre="${1#--exec_pre=}"; shift;; - --delay=* ) delay="${1#--delay=}"; shift;; - --force | -f ) force=true; shift;; - --queue ) queue=true; shift;; - --source=* ) source="${1#--source=}"; shift;; - --sortto=* ) sortto="${1#--sortto=}"; shift;; - --test ) test_mode="true"; echo "INFO: Running in TESTMODE, no changes are made!"; shift;; - * ) break ;; - --) shift; break;; - esac - done - - # main program starts here - - # confirm filepath - if [[ -z "$filepath" ]]; then - # if --path is not used, try and run queue - queue next - elif [[ -z "$(find "$filepath" -type d 2>/dev/null)" ]] && [[ -z "$(find "$filepath" -type f -print | head -n 1 2>/dev/null)" ]] || [[ -z "$(find "$filepath" -type f -print | head -n 1 2>/dev/null)" ]]; then - # path with files or file not found - if [[ "$transferetype" == "downftp" ]] || [[ "$transferetype" == fxp ]]; then - # server <-- client, assume path is OK - we will know for sure when size is found - true - elif [[ "$transferetype" == "upftp" ]]; then - # server --> client - echo -e "\e[00;31mERROR: Option --path is required with existing path (with file(s)), or file does not exists:\n $filepath\n This cannot be transfered!\e[00m\n" - queue fail - queue next - else - echo -e "\e[00;31mERROR: Transfer-option \"$transferetype\" not recognized. Have a look on your config (--user=$user --edit)!\e[00m\n" - exit 1 - fi - fi - # Save transfer to queue and exit - if [[ $queue == true ]]; then - queue add end - fi - - # Create lockfile - if [[ "$lockfileRunning" == "true" ]]; then - echo "INFO: Updating lockfile" - else - lockfile - fi - - echo "INFO: Transfer-option: $transferetype" - - #Load dependencies - loadDependency DSetup - - # Load user config - loadConfig - - # OK nothing running and --path is real, lets continue - # fix spaces: "/This\ is\ a\ path" - # Note: The use of normal backslashes is NOT supported - filepath="$(echo "$filepath" | sed 's/\\./ /g')" - - #start program - main "$filepath" -} diff --git a/dependencies/ftp_online_test.sh b/dependencies/ftp_online_test.sh deleted file mode 100644 index 5b82689..0000000 --- a/dependencies/ftp_online_test.sh +++ /dev/null @@ -1,99 +0,0 @@ -#!/bin/bash -function online_test { #confirm that server is alive and writable - online - if [[ $? -eq 0 ]]; then - writeable # Check if server is writable - else # not online - if [[ -f "$ftpalive_file" ]]; then rm "$ftpalive_file"; fi; - echo -e "\e[00;31m [RETRYING]\e[00m" - retry_count="0" - # before download - if [[ $confirm_online == "true" ]]; then - # continue trying acording to setttings - quittime=$(( $scriptstart + $retry_download_max*60*60 )) #hours - echo "INFO: Keep trying until $(date --date=@$quittime)" - if [[ $(date +%s) -lt $quittime ]]; then - echo -e "INFO: Stopping session and trying again $retry_download"mins" later\n" - cleanup session - waittime=$(($retry_download*60)) - sed "3s#.*#*************************** FTP INFO: SERVER OFFLINE: DOWNLOAD POSTPONED! Trying again in "$waittime"mins#" -i "$logfile" - sleep 10 - queue run #running new session - fi - fi - # online test - while [[ $retry_count -lt $retries ]]; do - echo -e "\e[00;31m [RETRYING]\e[00m" - let retry_count++ - online - if [[ $? -eq 0 ]]; then - writeable - is_online="0" - break - fi - done - # ok we failed - if [[ $is_online -ne 0 ]]; then - sed "3s#.*#*************************** FTP INFO: SERVER OFFLINE#" -i $logfile - is_online="1" - fi - fi -} - -function writeable_test { - if [[ -f "$ftpalive_file" ]]; then rm "$ftpalive_file"; fi; - echo -e "\e[00;32m [OK]\e[00m" - # Confirm that path is writeable etc. - echo -n "INFO: Checking if set path is writeable..." - echo "Testing ftp settings for ftpautodownload" > "$ftpcheck_testfile" - cat "$ftplogin_file1" >> "$ftpcheck_file" - echo "put -O \"$ftpincomplete\" \"$ftpcheck_file\"" >> "$ftpcheck_file" - echo "rm \"$ftpincomplete$(basename $ftpcheck_file)\"" >> "$ftpcheck_file" - echo "quit" >> "$ftpcheck_file" - $lftp -f "$ftpcheck_file" &> /dev/null -} - -function online { - echo -n "INFO: Checking if server is alive..." - #returns 1 if ftp server is offline, takes up to 1 min! - cat "$ftplogin_file1" >> $ftpalive_file - echo "ls" >> $ftpalive_file - echo "quit" >> $ftpalive_file - $lftp -f "$ftpalive_file" &> /dev/null -} - -function writeable { - writeable_test - if [[ -f "$ftpcheck_file" ]]; then rm "$ftpcheck_file"; fi; - if [[ $? -eq 0 ]]; then - echo -e "\e[00;32m [OK]\e[00m" - is_online="0" - else - echo -e "\e[00;32m [RETRYING]\e[00m" - retry_count="0" - # before download - if [[ $confirm_online == "true" ]]; then - # continue trying acording to settings - quittime=$(( $scriptstart + $retry_download_max*60*60 )) #hours - echo "INFO: Keep trying until $(date --date=@$quittime)" - if [[ $(date +%s) -lt $quittime ]]; then - echo -e "INFO: Stopping session and trying again $retry_download"mins" later\n" - cleanup session - waittime=$(($retry_download*60)) - sed "3s#.*#*************************** FTP INFO: SERVER OFFLINE: DOWNLOAD POSTPONED! Trying again in "$waittime"mins#" -i "$logfile" - sleep 10 - let retry_count++ - queue run #running new session - fi - fi - # online test - while [[ $retry_count -lt $retries ]]; do - echo -e "\e[00;31m [RETRYING]\e[00m" - let retry_count++ - writeable_test - done - # ok we failed - sed "3s#.*#*************************** FTP INFO: SERVER NOT WRITEABLE#" -i $logfile - is_online="1" - fi -} \ No newline at end of file diff --git a/dependencies/help.sh b/dependencies/help.sh index 3b065d4..30753a8 100644 --- a/dependencies/help.sh +++ b/dependencies/help.sh @@ -1,20 +1,33 @@ #!/bin/bash +# Function to write configuration file based on the show_example function function write_config { local config - # write show_example to config - echo -n "Preparing config ..." + + # Prepare config file + echo "Preparing config ..." + + # Set default username if not provided if [[ -z "$username" ]]; then - user=default + username=default fi + + # Define config file path config="$scriptdir/users/$username/config" + + # Create directory if not exists mkdir -p "$scriptdir/users/$username" + + # Check if config file exists if [[ -f "$config" ]]; then - read -p " Config already exits($config), do you want to overwrite(y/n)? " + # Prompt to overwrite if config file already exists + read -p " Config already exists ($config), do you want to overwrite (y/n)? " if [[ "$REPLY" == "y" ]]; then + # Write config using show_example function show_example > "$config" echo "Config written to \"$config\"" fi else + # Write config using show_example function show_example > "$config" echo "Config written to \"$config\"" fi @@ -23,10 +36,10 @@ function write_config { function show_help { echo -e " \033[1mHomepage\033[0m -\e[04;34mhttps://bitbucket.org/teamsilent/ftpautodownload/\e[00m +\e[04;34mhttps://github.com/Meliox/FTPauto/\e[00m \033[1mDescription\033[0m - FTPauto is a bash command line tool to send files from a local server to\n a remote easily. Numerous options may be specified such a checking free space for \n transferring, multiple user support, delay of transfer and category support. It is\n especially powerful in combination with Flexget + FTPauto is a bash command line tool to send files from a local server to\n a remote easily (ftp, ftps, fxp or sftp). Numerous options may be specified such a checking free space for \n transferring, multiple user support, delay of transfer and category support. It is\n especially powerful in combination with Flexget For more information read the README or see homepage. @@ -50,15 +63,9 @@ IMPORTANT: Default is always used if --user isn't used! --retry | Resets all failed downloads. Retry with --start. --forget | Remove from queue --path= | used to transfer now! - --queue | Sends to queue WITHOUT starting script if autostart=false in config. + --queue | Sends to queue NOTE that --path is required for this to work. - Might also be used to start transfer in background if autostart=true - --source= | Source is used to show how the download has been started. The - following is possible: - MANDL=manual download(if nothing is used) - WEBDL=download from webpage - FLXDL=autodownload from flexget - other can be used as well... + --source= | Source is used to show how the download has been started, e.g. from YOURPROGRAM --up | Move Up --sort | Sorts transfer into passed directory. Usage --sort=somedir/somedir2/ This will overwrite automatic sorting. @@ -90,108 +97,122 @@ IMPORTANT: Default is always used if --user isn't used! } function show_example { -#shows an example for configuration - echo - echo "#This the the configuration file for FTPauto" - echo "config_version=\"6\"" - echo "#Place the file in $scriptdir/run/'$username'/config and load with --user='$username'" - echo - echo "#HOWTO: Edit the info between the qoutes \"TEST\", here the word TEST" - echo - echo "#### FTP server Setup ####" - echo " # If you just want the server to send to your ftp, edit the options below" - echo "transferetype=\"(upftp|downftp|fxp)\" # Determine how to transfer file: Either send or receive from ftp or fxp them to another server" - echo "" - echo " # These directories are where you want to download/send the item. REMEMBER TRAILING SLASH" - echo "ftpincomplete=\"~/somedirectory/incomplete/\" # incomplete directory. Leave empty if no incomplete directory should be used" - echo "ftpcomplete=\"~/somedirectory/complete/\" # complete directory." - echo "" - echo "#### DOWN/UP MODE ####" - echo " # If you just want to send/receive items, change these" - echo "ftpuser1=\"user\" # username" - echo "ftppass1=\"pass\" # password" - echo "ftphost1=\"ip\" # ip address for ftp server" - echo "ftpport1=\"port\" # ftp port" - echo "ftpssl1=\"false\" # Set to true to use ftps else set it to false" - echo "ftpcustom1=\"\" # Enter settings separated by ';', eg. set cache:expire 1;set cache:size 2" - echo "" - echo "#### FXP MODE ####" - echo " # If you just want to send/receive items from one server to another, change these also! In FXP mode, ftphost1 is source and ftphost2 receiver" - echo "ftpuser2=\"user\" # username" - echo "ftppass2=\"pass\" # password" - echo "ftphost2=\"ip\" # ip address for ftp server" - echo "ftpport2=\"port\" # ftp port" - echo "ftpssl2=\"false\" # use ftps or not" - echo "ftpcustom2=\"\" # Enter settings separated by ';', eg. set cache:expire 1; set cache:size 2" - echo - echo "#### Log settings ###" - echo "logrotate=\"false\" #enabled logrotating to move old to log.old" - echo "lognumber=\"50\" #how many transfers to save in log before moving to log.old. 0 for disabled" - echo - echo "#### Transfer settings ####" - echo - echo "### Filehandling" - echo " # Splitting files if filesize exceed MB. Some FTP servers disconnect after a certain amount of time is there is no" - echo " # activity. These settings only work if the server handling the script also sends the files, i.e. in upftp and upfxp mode!" - echo "send_option=\"(video|split|default)\" # Can be configured to send only videofile, split files according to settings or simply transfer the , default. If videofile or sizelimit are not met, then the files will be transfered as default - without any modifications." - echo "splitsize=\"100\" # How large the rarparts should be in MB" - echo "create_sfv=\"false\" # Create sfv for rarfiles" - echo "" - echo "### Transfer settings" - echo - echo "## General settings" - echo "parallel=\"3\" # how many simultaneous transfers to download with" - echo "continue_queue=\"true\" # Script will continue downloading if something is queued" - echo "retries=\"3\" # How many times should the transfer be tried, before giving up" - echo "retry_download=\"10\" # retry again in minutes after minimum space is reached OR server is offline." - echo "retry_download_max=\"30\" # retry for how many minites, before quitting. Recommended 30 mins. For each try 3 tries to establish connection with furthermore be tried" - echo - echo "## Extra settings" - echo "force=\"false\" # Transfer regardless of lockfiles/other transfers" - echo "confirm_transfer=\"false\" # Try to confirm transfer" - echo "confirm_online=\"false\" # Try to confirm that server is online/writeable before doing anything" - echo "exclude_array=( ) # Ignore certain files with name matching, format is ( \"word1\" \"word2\" )" - echo - echo "## Extra settings" - echo " # To enable FTP space info, ftpsizemanagement has to be set to true" - echo "ftpsizemanagement=\"false\" # will confirm enough free space in dir according to settings" - echo "totalmb=\"14950\" # total ftp space in mb" - echo "critical=\"100\" # minimum space before aborting transfer in mb" - echo - echo "## Processbar settings" - echo " # Processbar shows how the transfer is proceeding, gives eta. etc." - echo "sleeptime=\"60\" # how often to provide progress information. Time in seconds" - echo - echo "## Miscellaneous settings" - echo "exec_post=\"\" #Execute external command upon finish. See --help exec_pre for more info" - echo "allow_background=\"false\" # don't wait for exec to finish. ONLY for exec_post" - echo "exec_pre=\"\" #Execute external command before starting. See --help for more info" - echo "sort=\"false\" # Sort files into DVD/TV/etc/ or like defined in --sort=DIRECTORY. Changes can be made in the file /dependencies/sorting.sh" - echo - echo "#### Push notifications ####" - echo " # Create a user at https://pushover.net/ and enter details below" - echo " # Leave push_user empty if you don't use it" - echo "push_token=\"\"" - echo "push_user=\"\"" + # Shows an example for configuration + echo + echo "# This the the configuration file for FTPauto" + echo "config_version=\"7\"" + echo "# Place the file in $scriptdir/run/'$username'/config and load with --user='$username'" + echo + echo "# HOWTO: Edit the info between the quotes \"TEST\", here the word TEST" + echo + echo "#### FTP server Setup ####" + echo "# If you just want the server to send to your ftp, edit the options below" + echo "transferetype=\"(upftp|downftp|fxp|upsftp)\" # Determine how to transfer file: Either send or receive from ftp or fxp them to another server" + echo + echo "# These directories are where you want to download/send the item. REMEMBER TRAILING SLASH" + echo "incomplete=\"~/somedirectory/incomplete/\" # incomplete directory. Leave empty if no incomplete directory should be used" + echo "complete=\"~/somedirectory/complete/\" # complete directory." + echo + echo "#### DOWN/UP MODE ####" + echo "# If you just want to send/receive items, change these" + echo "ftpuser1=\"user\" # username" + echo "ftppass1=\"pass\" # password" + echo "ftphost1=\"ip\" # ip address for ftp server" + echo "ftpport1=\"port\" # ftp port" + echo "ftpssl1=\"false\" # Set to true to use ftps else set it to false" + echo "ftpcustom1=\"\" # Enter settings separated by ';', eg. set cache:expire 1;set cache:size 2" + echo + echo "#### FXP MODE ####" + echo "# If you just want to send/receive items from one server to another, change these also! In FXP mode, ftphost1 is source and ftphost2 receiver" + echo "ftpuser2=\"user\" # username" + echo "ftppass2=\"pass\" # password" + echo "ftphost2=\"ip\" # ip address for ftp server" + echo "ftpport2=\"port\" # ftp port" + echo "ftpssl2=\"false\" # use ftps or not" + echo "ftpcustom2=\"\" # Enter settings separated by ';', eg. set cache:expire 1; set cache:size 2" + echo + echo "#### SFTP server Setup ####" + echo "# For SFTP server, you need to specify the following parameters" + echo "sftpuser1=\"user\" # SFTP username" + echo "sftppass1=\"pass\" # SFTP password" + echo "sftphost1=\"ip\" # SFTP host address" + echo "sftpport1=\"port\" # SFTP port" + echo "sftpcustom1=\"\" # Enter additional settings separated by ';'" + echo + echo "#### Log settings ###" + echo + echo "logrotate=\"false\" # enabled logrotating to move old to log.old" + echo "lognumber=\"50\" # how many transfers to save in log before moving to log.old. 0 for disabled" + echo + echo "#### Transfer settings ####" + echo + echo "### Filehandling" + echo "# Splitting files if filesize exceed MB. Some FTP servers disconnect after a certain amount of time is there is no" + echo "# activity. These settings only work if the server handling the script also sends the files, i.e. in upftp and upfxp mode!" + echo "send_option=\"(video|split|default)\" # Can be configured to send only videofile, split files according to settings or simply transfer the , default. If videofile or sizelimit are not met, then the files will be transfered as default - without any modifications." + echo "splitsize=\"100\" # How large the rarparts should be in MB" + echo "create_sfv=\"false\" # Create sfv for rarfiles" + echo + echo "### Transfer settings" + echo + echo "## General settings" + echo "parallel=\"3\" # how many simultaneous transfers to download with" + echo "continue_queue=\"true\" # Script will continue downloading if something is queued" + echo "retries=\"3\" # How many times should the transfer be tried, before giving up" + echo "retry_download=\"10\" # retry again in minutes after minimum space is reached OR server is offline." + echo "retry_download_max=\"30\" # retry for how many minites, before quitting. Recommended 30 mins. For each try 3 tries to establish connection with furthermore be tried" + echo + echo "## Extra settings" + echo "force=\"false\" # Transfer regardless of lockfiles/other transfers" + echo "confirm_transfer=\"false\" # Try to confirm transfer" + echo "confirm_online=\"false\" # Try to confirm that server is online/writeable before doing anything" + echo "exclude_array=( ) # Ignore certain files with name matching, format is ( \"word1\" \"word2\" )" + echo + echo "## Extra settings" + echo "# To enable Server space info, serversizemanagement has to be set to true" + echo "serversizemanagement=\"false\" # will confirm enough free space in dir according to settings" + echo "totalmb=\"14950\" # total server space in mb" + echo "critical=\"100\" # minimum space before aborting transfer in mb" + echo + echo "## Processbar settings" + echo "# Processbar shows how the transfer is proceeding, gives eta. etc." + echo "sleeptime=\"60\" # how often to provide progress information. Time in seconds" + echo + echo "## Miscellaneous settings" + echo "# Execute external command upon finish. See --help exec_pre for more info" + echo "exec_post=\"\"" + echo "# Don't wait for exec to finish. ONLY for exec_post" + echo "allow_background=\"false\"" + echo "# Execute external command before starting. See --help for more info" + echo "exec_pre=\"\"" + echo "sort=\"false\" # Sort files into DVD/TV/etc/ or like defined in --sort=DIRECTORY. Changes can be made in the file /dependencies/sorting.sh" + echo + echo "#### Push notifications ####" + echo "# Create a user at https://pushover.net/ and enter details below" + echo "# Leave push_user empty if you don't use it" + echo "push_token=\"\"" + echo "push_user=\"\"" } + function create_log_file { - if [[ ! -e "$logfile" ]]; then - echo "INFO: First time usage. Logfile is created" - echo "*************************** FTPauto - $s_version" >> "$logfile" - echo "*************************** STATS: 0MB in 0 transfers in 00d:00h:00m:00s" >> "$logfile" - if [[ $ftpsizemanagement == true ]]; then - echo "*************************** FTP INFO: 0/${totalmb}MB (Free ${freemb}MB)" >> "$logfile" - else - echo "*************************** FTP INFO: (not used yet)" >> "$logfile" - fi - echo "*************************** LASTDL: nothing" >> "$logfile" - echo "*************************** " >> "$logfile" - echo "**********************************************************************************************************************************" >> "$logfile" - echo "" >> "$logfile" - else - echo "INFO: Logfile: "$logfile"" - # clean log file - echo "*************************** FTP INFO:" >> "$logfile" - fi -} \ No newline at end of file + # Check if the logfile exists, if not, create it + if [[ ! -e "$logfile" ]]; then + serversizemanagement="false" + # First time usage, create the logfile + echo "INFO: First time usage. Logfile is created" + echo "*************************** FTPauto - $s_version" >> "$logfile" + echo "*************************** STATS: 0MB in 0 transfers in 00d:00h:00m:00s" >> "$logfile" + # Display server info based on configuration + echo "*************************** SERVER INFO: (not used yet)" >> "$logfile" + echo "*************************** LAST TRANSFER: nothing" >> "$logfile" + echo "***************************" >> "$logfile" + echo "**********************************************************************************************************************************" >> "$logfile" + echo "" >> "$logfile" + else + # Logfile exists, indicate its location + echo "INFO: Logfile: $logfile" + # Clean the existing logfile + echo "*************************** SERVER INFO:" >> "$logfile" + fi +} diff --git a/dependencies/server_alive_test.sh b/dependencies/server_alive_test.sh new file mode 100644 index 0000000..18fbce1 --- /dev/null +++ b/dependencies/server_alive_test.sh @@ -0,0 +1,106 @@ +#!/bin/bash + +# Function to confirm that the server is alive and writable +function online_test { + loadDependency DServerLogin && load_login 1 + online + if [[ $? -eq 0 ]]; then + writeable # Check if server is writable + else + if [[ -f "$server_alive_file" ]]; then rm "$server_alive_file"; fi; + echo -e "\e[00;31m [RETRYING]\e[00m" + retry_count="0" + # Before download + if [[ $confirm_online == "true" ]]; then + # Continue trying according to settings + quittime=$(( $scriptstart + $retry_download_max*60*60 )) # Hours + echo "INFO: Keep trying until $(date --date=@$quittime)" + if [[ $(date +%s) -lt $quittime ]]; then + echo -e "INFO: Stopping session and trying again $retry_download"mins" later\n" + cleanup session + waittime=$(($retry_download*60)) + sed "3s#.*#*************************** SERVER INFO: SERVER OFFLINE: DOWNLOAD POSTPONED! Trying again in "$waittime"mins#" -i "$logfile" + sleep 10 + queue run # Running new session + fi + fi + # Online test + while [[ $retry_count -lt $retries ]]; do + echo -e "\e[00;31m [RETRYING]\e[00m" + let retry_count++ + online + if [[ $? -eq 0 ]]; then + writeable + is_online="0" + break + fi + done + # If still offline + if [[ $is_online -ne 0 ]]; then + sed "3s#.*#*************************** SERVER INFO: SERVER OFFLINE#" -i $logfile + is_online="1" + fi + fi + cleanup session +} + +# Function to check if the server is writable +function writeable_test { + if [[ -f "$server_alive_file" ]]; then rm "$server_alive_file"; fi; + echo -e "\e[00;32m [OK]\e[00m" + # Confirm that path is writeable etc. + echo -n "INFO: Checking if set path is writeable..." + echo "Testing server settings for download" > "$server_check_testfile" + cat "$login_file1" >> "$server_check_file" + echo "put -O \"$incomplete\" \"$server_check_file\"" >> "$server_check_file" + echo "rm \"$incomplete$(basename $server_check_file)\"" >> "$server_check_file" + echo "quit" >> "$server_check_file" + $lftp -f "$server_check_file" &> /dev/null +} + +# Function to check if the server is online +function online { + echo -n "INFO: Checking if server is alive..." + # Returns 1 if server is offline, takes up to 1 min! + cat "$login_file1" >> $server_alive_file + echo "ls" >> $server_alive_file + echo "quit" >> $server_alive_file + $lftp -f "$server_alive_file" &> /dev/null +} + +# Function to check if the server is writable +function writeable { + writeable_test + if [[ -f "$server_check_file" ]]; then rm "$server_check_file"; fi; + if [[ $? -eq 0 ]]; then + echo -e "\e[00;32m [OK]\e[00m" + is_online="0" + else + echo -e "\e[00;32m [RETRYING]\e[00m" + retry_count="0" + # Before download + if [[ $confirm_online == "true" ]]; then + # Continue trying according to settings + quittime=$(( $scriptstart + $retry_download_max*60*60 )) # Hours + echo "INFO: Keep trying until $(date --date=@$quittime)" + if [[ $(date +%s) -lt $quittime ]]; then + echo -e "INFO: Stopping session and trying again $retry_download"mins" later\n" + cleanup session + waittime=$(($retry_download*60)) + sed "3s#.*#*************************** SERVER INFO: SERVER OFFLINE: DOWNLOAD POSTPONED! Trying again in "$waittime"mins#" -i "$logfile" + sleep 10 + let retry_count++ + queue run # Running new session + fi + fi + # Online test + while [[ $retry_count -lt $retries ]]; do + echo -e "\e[00;31m [RETRYING]\e[00m" + let retry_count++ + writeable_test + done + # If still offline + sed "3s#.*#*************************** SERVER INFO: SERVER NOT WRITEABLE#" -i $logfile + is_online="1" + fi +} \ No newline at end of file diff --git a/dependencies/server_list.sh b/dependencies/server_list.sh new file mode 100644 index 0000000..1c144ba --- /dev/null +++ b/dependencies/server_list.sh @@ -0,0 +1,100 @@ +#!/bin/bash +# Function to list content on a remote server and interactively navigate it + +function remote_server_list { + echo "INFO: Looking up content on remote server..." + # Set default directory to root if not specified + [[ -z $dir ]] && dir="/" + + while true; do + # Get content of current directory + get_content "$dir" + + # Prompt user for input + read -p "Enter number to expand directory or download by also adding a \"d\", e.g., 1d ? (x to exit) " + + # Ensure valid input + while ! [[ "$REPLY" =~ [0-9] ]] && [[ $REPLY != "x" ]]; do + read -p "Enter number to expand directory or download by also adding a \"d\", e.g., 1d ? (x to exit) " + done + + number=$REPLY + + # Exit if user inputs 'x' + if [[ "$number" == "x" ]]; then + break + elif [[ "$number" == "0" ]]; then + # Go to top path + dir="/" + continue + elif [[ "$number" == "1" ]]; then + # Move up one directory + dir=$(dirname "$dir") + continue + elif [[ "$number" =~ ^[0-9]+d ]]; then + # Extract number and prepare for download + number=${number%d} + path="$dir$(echo ${array_list[$number]} | awk '{print $9}')" + download_argument+=("--user=$username" "--path=/$path" "--source=Manually") + option=("download" "queue") + echo " Adding $path to queue" + main + unset path download_argument option # Unset variables + else + # Check if selected item is a directory + if [[ "$(echo ${array_list[$number]} | awk '{print $5}')" -gt 4096 ]]; then + echo -e "\e[00;31m Cannot expand file: $(echo ${array_list[$number]} | awk '{print $9}')\e[00m" + continue + fi + # Expand selected directory + dir="$dir$(echo ${array_list[$number]} | awk '{print $9}')/" + continue + fi + done + + # Prompt to start download if queue file exists + if [[ -e "$queue_file" ]]; then + read -p "Do you want to start the download (y/n)? " + if [[ $REPLY == "y" ]]; then + read -p "Do you wish to execute it as a background thread (y/n)? " + [[ $REPLY == "y" ]] && background=true + start_transfermain + fi + fi +} + +# Function to retrieve content from a remote server and display it +function get_content { + # Remove old files first + rm -f "$server_list_file" + loadDependency DServerLogin && load_login 1 # Generate a new login file as download removes it + cat "$login_file1" >> "$server_list_file" + echo "ls -aFl \"${dir}\" > ~/../..$server_content" >> "$server_list_file" + echo "quit" >> "$server_list_file" + $lftp -f "$server_list_file" &> /dev/null + + # Check if listing was successful + if [[ $? -eq 0 ]]; then + echo -e "\n\e[00;32mINFO: Listing content(s):\e[00m" + echo "Current path: $dir" + old_dir="$dir" + array_list=( ) + readarray array_list < "$server_content" + array_list=( "X X X X 4096 Feb XX XX ." "X X X X 4096 Feb XX XX .." "${array_list[@]}" ) + i=0 + # Iterate over array and display content + for value in "${array_list[@]}"; do + if [[ $(echo $value | awk '{print $5}') -eq 4096 ]]; then + # Directory + printf "%-8s\n" "$i : $(($(echo $value | awk '{print $5}')/(1024*1024)))MB $(echo -e "\e[00;34m$(echo $value | awk '{print $9}')\e[00m")" + else + # File + printf "%-8s\n" "$i : $(($(echo $value | awk '{print $5}')/(1024*1024)))MB $(echo $value | awk '{print $9}')" + fi + let i++ + done | column -c 1 -t + fi + + # Cleanup + cleanup session +} diff --git a/dependencies/server_login.sh b/dependencies/server_login.sh new file mode 100644 index 0000000..a7fee2f --- /dev/null +++ b/dependencies/server_login.sh @@ -0,0 +1,104 @@ +#!/bin/bash + +# Prepare login based on transfer type +function load_login { + if [[ "$transferetype" == "upftp" || "$transferetype" == "downftp" || "$transferetype" == "fxp" ]]; then + ftp_login $1 + elif [[ "$transferetype" == "upsftp" ]]; then + sftp_login $1 + fi +} + +# Function to generate FTP login details based on input (1 or 2) +function ftp_login { + local number OIFS IFS custom ssl user pass host pass loginfile + number="$1" + + # Set variables based on input + custom="ftpcustom${number}" + ssl="ftpssl${number}" + user="ftpuser${number}" + pass="ftppass${number}" + host="ftphost${number}" + port="ftpport${number}" + login_file="login_file${number}" + + # Set timeout settings + echo "set net:timeout 10" >> "${!login_file}" + echo "set net:max-retries 3" >> "${!login_file}" + echo "set net:reconnect-interval-base 10" >> "${!login_file}" + echo "set net:reconnect-interval-multiplier 1" >> "${!login_file}" + echo "set net:reconnect-interval-max 60" >> "${!login_file}" + + # Write debug settings to file if verbose mode is enabled + if ((verbose)); then + echo "debug 8 -t -o $lftpdebug" >> "${!login_file}" + fi + + # Write SSL settings to file + if [[ "${!ssl}" == true ]]; then + echo "set ftp:ssl-force true" >> "${!login_file}" + echo "set ssl:verify-certificate false" >> "${!login_file}" + fi + + # Write custom configurations to file + if [[ -n "${!custom}" ]]; then + OIFS="$IFS" + IFS=';' + for i in "${!custom[@]}"; do + echo "$i" >> "${!login_file}" + done + IFS="$OIFS" + fi + + # Allow only normal transfer types + if [[ "$transferetype" =~ "upftp" ]] || [[ "$transferetype" =~ "downftp" ]] || [[ "$transferetype" =~ "fxp" ]]; then + echo "open -u ${!user},${!pass} ${!host} -p ${!port}" >> "${!login_file}" + else + echo -e "\e[00;31mERROR: Transfer-option \"$transferetype\" not recognized. Check your config (--user=$user --edit)!\e[00m\n" + cleanup session + cleanup end + exit 1 + fi +} + +# Function to generate SFTP login details +function sftp_login { + local number OIFS IFS custom ssl user pass host pass loginfile + number="$1" + + # Set variables based on input + custom="sftpcustom${number}" + ssl="sftpssl${number}" + user="sftpuser${number}" + pass="sftppass${number}" + host="sftphost${number}" + port="sftpport${number}" + login_file="login_file${number}" + + # Set timeout settings + echo "set net:timeout 10" >> "${!login_file}" + echo "set net:max-retries 3" >> "${!login_file}" + echo "set net:reconnect-interval-base 10" >> "${!login_file}" + echo "set net:reconnect-interval-multiplier 1" >> "${!login_file}" + echo "set net:reconnect-interval-max 60" >> "${!login_file}" + + echo "set sftp:auto-confirm true" >> "${!login_file}" + + # Write custom configurations to file + if [[ -n "${!custom}" ]]; then + for option in "${!custom[@]}"; do + echo "$option" >> "${!login_file}" + done + fi + + # Check if username and password are provided + if [[ "$transferetype" =~ "upsftp" ]]; then + echo "open -u ${!user},${!pass} sftp://${!host} -p ${!port}" >> "${!login_file}" + else + echo -e "\e[00;31mERROR: Transfer-option \"$transferetype\" not recognized. Check your config (--user=$user --edit)!\e[00m\n" + cleanup session + cleanup end + exit 1 + fi +} \ No newline at end of file diff --git a/dependencies/ftp_size_management.sh b/dependencies/server_size_management.sh similarity index 53% rename from dependencies/ftp_size_management.sh rename to dependencies/server_size_management.sh index 00e3e56..a3ca52c 100644 --- a/dependencies/ftp_size_management.sh +++ b/dependencies/server_size_management.sh @@ -1,17 +1,19 @@ #!/bin/bash -function ftp_sizemanagement { #checking if freespace is sufficient before filetransfer, show ftpspace used +#checking if freespace is sufficient before filetransfer, show space used +function server_sizemanagement { + loadDependency DServerLogin && load_login 1 if [[ $test_mode != "true" ]]; then local mode=$1 - cat "$ftplogin_file1" >> "$ftpfreespace_file" - echo "du -s $ftpincomplete > ~/../../$ftp_size_file" >> "$ftpfreespace_file" - echo "wait" >> "$ftpfreespace_file" - echo "du -s $ftpcomplete > ~/../../$ftp_size_file" >> "$ftpfreespace_file" - echo "quit" >> "$ftpfreespace_file" + cat "$login_file1" >> "$server_freespace_file" + echo "du -s $incomplete >> ~/../..$server_size_file" >> "$server_freespace_file" + echo "wait" >> "$server_freespace_file" + echo "du -s $complete >> ~/../..$server_size_file" >> "$server_freespace_file" + echo "quit" >> "$server_freespace_file" echo "INFO: Looking up used space, this may take a while......" - $lftp -f "$ftpfreespace_file" &> /dev/null + $lftp -f "$server_freespace_file" &> /dev/null if [[ $? -eq 0 ]]; then # server online, continue to find size - ftp_getsize $mode + server_getsize $mode else retry_count="0" if [[ $mode == "info" ]]; then @@ -20,52 +22,53 @@ function ftp_sizemanagement { #checking if freespace is sufficient before filetr echo -e "\e[00;31mTransfer terminated: $(date '+%d/%m/%y-%a-%H:%M:%S')\e[00m" waittime=$(($retry_download*60)) echo "INFO: Pausing session and trying again $retry_download"mins" later" - sed "3s#.*#*************************** FTP INFO: SERVER OFFLINE: DOWNLOAD POSTPONED! Trying again in "$waittime"mins#" -i $logfile + sed "3s#.*#*************************** SERVER INFO: SERVER OFFLINE: DOWNLOAD POSTPONED! Trying again in "$waittime"mins#" -i $logfile sleep $waittime let retry_count++ - $lftp -f "$ftpfreespace_file" &> /dev/null + $lftp -f "$server_freespace_file" &> /dev/null if [[ $? -eq 0 ]]; then # server online, continue to find size - ftp_getsize $mode + server_getsize $mode is_online="0" - break + break fi done elif [[ $mode == "check" ]]; then quittime=$(( $scriptstart + $retry_download_max*60*60 )) #hours - echo "INFO: Keep trying until $(date --date=@$quittime)" + echo "INFO: Keep trying until $(date --date=@$quittime)" while [[ $(date +%s) -lt $quittime ]]; do echo -e "\e[00;31mERROR: Looking up free space failed for some reason!\e[00m" echo -e "\e[00;31mTransfer terminated: $(date '+%d/%m/%y-%a-%H:%M:%S')\e[00m" waittime=$(($retry_download*60)) echo "INFO: Pausing session and trying again $retry_download"mins" later" - sed "3s#.*#*************************** FTP INFO: SERVER OFFLINE: DOWNLOAD POSTPONED! Trying again in "$waittime"mins#" -i $logfile + sed "3s#.*#*************************** SERVER INFO: SERVER OFFLINE: DOWNLOAD POSTPONED! Trying again in "$waittime"mins#" -i $logfile sleep $waittime let retry_count++ - $lftp -f "$ftpfreespace_file" &> /dev/null + $lftp -f "$server_freespace_file" &> /dev/null if [[ $? -eq 0 ]]; then # server online, continue to find size - ftp_getsize $mode + server_getsize $mode is_online="0" - break + break fi - done + done fi fi if [[ $is_online -ne 0 ]]; then # ok we failed - sed "3s#.*#*************************** FTP INFO: SERVER OFFLINE#" -i $logfile + sed "3s#.*#*************************** SERVER INFO: SERVER OFFLINE#" -i $logfile is_online="1" fi else echo -e "\e[00;31mTESTMODE: LFTP-sizemanagement NOT STARTED\e[00m" - echo "Would look up free space at FTP host rootdir" + echo "Would look up free space at server host rootdir" fi + cleanup session } -function ftp_getsize { - usedkb=$(cat "$ftp_size_file" | awk -F ":" '{sum+=$NF} END { printf ("%0.0f\n", sum)}') - rm $ftpfreespace_file +function server_getsize { + usedkb=$(cat "$server_size_file" | awk -F ":" '{sum+=$NF} END { printf ("%0.0f\n", sum)}') + rm $server_freespace_file if [[ $usedkb -ge "1" ]]; then usedmb=$(( $usedkb / 1024 )) else @@ -74,29 +77,27 @@ function ftp_getsize { freemb=$(( $totalmb - $usedmb )) case "$1" in "info" ) - sed "3s#.*#*************************** FTP INFO: "$usedmb"\/"$totalmb"MB (Free "$freemb"MB)#" -i $logfile + sed "3s#.*#*************************** SERVER INFO: "$usedmb"\/"$totalmb"MB (Free "$freemb"MB)#" -i $logfile echo "INFO: Free space: "$freemb"MB ("$usedmb"/"$totalmb"MB used)" - cleanup session; ;; "check" ) - sed "3s#.*#*************************** FTP INFO: "$usedmb"\/"$totalmb"MB (Free "$freemb"MB)#" -i $logfile + sed "3s#.*#*************************** SERVER INFO: "$usedmb"\/"$totalmb"MB (Free "$freemb"MB)#" -i $logfile if [[ "$( echo "$freemb < $critical" | bc)" -eq "1" ]] || [[ "$( echo "$size > $freemb" | bc)" -eq "1" ]]; then - sed "3s#.*#*************************** FTP INFO: "$usedmb"\/"$totalmb"MB - FREE SPACE IS CRITICAL! DOWNLOAD POSTPONED!#" -i $logfile + sed "3s#.*#*************************** SERVER INFO: "$usedmb"\/"$totalmb"MB - FREE SPACE IS CRITICAL! DOWNLOAD POSTPONED!#" -i $logfile freespaceneeded=$(echo $size - $freemb | bc) sed "5s#.*#*************************** PENDING: "$orig_name" needs "$freespaceneeded"MB additional free space#" -i $logfile - echo "INFO: FTPSERVER: "$usedmb"/"$totalmb"MB Used" - echo -e "\e[00;31mERROR: FTPSERVER: Free space: "$freemb"MB\e[00m" + echo "INFO: SERVER: "$usedmb"/"$totalmb"MB Used" + echo -e "\e[00;31mERROR: SERVER: Free space: "$freemb"MB\e[00m" echo -e "\e[00;31mERROR: "$orig_name" needs "$freespaceneeded"MB additional free space\e[00m" echo "INFO: Stopping session and trying again $retry_download"mins" later" - cleanup session echo -e "INFO: Exiting current session\n" waittime=$(($retry_download*60)) - sed "3s#.*#*************************** FTP INFO: "$usedmb"\/"$totalmb"MB - FREE SPACE IS CRITICAL! DOWNLOAD POSTPONED! Trying again in "$waittime"mins#" -i $logfile + sed "3s#.*#*************************** SERVER INFO: "$usedmb"\/"$totalmb"MB - FREE SPACE IS CRITICAL! DOWNLOAD POSTPONED! Trying again in "$waittime"mins#" -i $logfile sleep $waittime queue run #running new session fi usedcrit=$(( $totalmb - $critical )) - echo "INFO: FTP OK - free space: "$freemb"MB ("$usedmb"/"$totalmb"MB used)" + echo "INFO: SERVER OK - free space: "$freemb"MB ("$usedmb"/"$totalmb"MB used)" ;; esac } \ No newline at end of file diff --git a/dependencies/setup.sh b/dependencies/setup.sh index 3e7f8e4..0a01f4e 100644 --- a/dependencies/setup.sh +++ b/dependencies/setup.sh @@ -7,155 +7,161 @@ function setup { lftp=$(which lftp) rar2fs=$(which rar2fs) # paths for internal files - queue_file="$scriptdir/run/$username.queue" + login_file1="$scriptdir/run/$username.login1" + login_file2="$scriptdir/run/$username.login2" + lftpdebug="$scriptdir/run/$username.lftpdebug" + lftptransfersize="$scriptdir/run/$username.transfersize" + lftptransfersize2="$scriptdir/run/$username.lftptransfersize2" lockfile="$scriptdir/run/$username.lck" + log_control="$scriptdir/run/$username.controllog" logfile="$scriptdir/users/$username/log" + maindebugfile="$scriptdir/run/$username.main.debug" oldlogfile="$scriptdir/users/$username/log.old" - ftptransfere_file="$scriptdir/run/$username.ftptransfere" - ftptransfere_processbar="$scriptdir/run/$username.ftpprocessbar" - ftplogin_file1="$scriptdir/run/$username.ftplogin1" - ftplogin_file2="$scriptdir/run/$username.ftplogin2" - ftpfreespace_file="$scriptdir/run/$username.ftpfreespace" - ftpalive_file="$scriptdir/run/$username.ftpalive" - ftpcheck_testfile="$scriptdir/run/$username.testfile" - ftpcheck_file="$scriptdir/run/$username.ftpcheck" proccess_bar_file="$scriptdir/run/$username.transfered.info" - ftp_size_file="$scriptdir/run/$username.ftpsize.info" - log_control="$scriptdir/run/$username.controllog" - ftpmaindebugfile="$scriptdir/run/$username.ftpmain.debug" - lftpdebug="$scriptdir/run/$username.lftpdebug" - lftptransfersize="$scriptdir/run/$username.ftptransfersize" + queue_file="$scriptdir/run/$username.queue" + server_alive_file="$scriptdir/run/$username.serveralive" + server_check_file="$scriptdir/run/$username.servercheck" + server_check_testfile="$scriptdir/run/$username.servertestfile" + server_content="$scriptdir/run/$username.servercontent" + server_freespace_file="$scriptdir/run/$username.serverfreespace" + server_list_file="$scriptdir/run/$username.lftplist" + server_size_file="$scriptdir/run/$username.serversize.info" + transfere_file="$scriptdir/run/$username.transfere" + transfere_processbar="$scriptdir/run/$username.processbar" transfersize="$scriptdir/run/$username.transfersize" transfersize2="$scriptdir/run/$username.transfersize2" - lftptransfersize2="$scriptdir/run/$username.lftptransfersize2" - ftplist_file="$scriptdir/run/$username.lftplist" - ftp_content="$scriptdir/run/$username.ftpcontent" } +# Function to get the size of a file or directory function get_size { - # called with $filepath + # Called with $filepath local dir n count i exp path t_size + dir="$1" - if [[ "$transferetype" == downftp ]] || [[ "$transferetype" == fxp ]]; then - #client + + if [[ "$transferetype" == downftp || "$transferetype" == fxp ]]; then + # Client-side transfer loadDependency DFtpLogin && ftp_login 1 - cat "$ftplogin_file1" > "$lftptransfersize" >> "$lftptransfersize" + cat "$login_file1" > "$lftptransfersize" >> "$lftptransfersize" echo "du -bs \"$dir\" > ~/../..$transfersize" >> "$lftptransfersize" echo "ls -lR \"$dir\" > ~/../..$transfersize2" >> "$lftptransfersize" echo "exit" >> "$lftptransfersize" "$lftp" -f "$lftptransfersize" &> /dev/null - # figure out if it is a file or directory + + # Determine if it's a file or directory local count=0 - while read line; do + while read -r line; do let ++count done <"$transfersize2" if [[ $count -gt 0 ]]; then - echo "INFO: Transfering a directory" + echo "INFO: Transferring a directory" transfer_type="directory" else - echo "INFO: Transfering a file" + echo "INFO: Transferring a file" transfer_type="file" fi - if [[ "${#exclude_array[@]}" -gt 0 ]] && [[-n "${exclude_array[@]}" ]]; then - # size lookup if expression is used - # prepare regex - local exp=() n="0" + + if [[ "${#exclude_array[@]}" -gt 0 && -n "${exclude_array[@]}" ]]; then + # Size lookup if expression is used + # Prepare regex + local exp=() n=0 for i in "${exclude_array[@]}"; do - if [[ $n -lt "${#exclude_array[@]}" ]] && [[ $n -ge 1 ]]; then - exp+="|" - fi + (( n++ )) + [[ $n -lt ${#exclude_array[@]} && $n -ge 1 ]] && exp+="|" exp+="$i" - let n++ done - # loop through result from lftp + + # Loop through result from lftp size=0 - while read line; do + while read -r line; do if [[ -n $(echo "$line" | egrep '('$exp')') ]]; then - # ignore files in regex + # Ignore files in regex continue fi if [[ -n $(echo "$line" | egrep '\/(.*):') ]]; then - # catch subdirectories path to add to the files + # Catch subdirectories path to add to the files path="$line" path=${path%:} path="$path/" continue fi - if [[ -n "$line" ]]; then - # make sure line contains something - if [[ "$(echo "$line" | awk {'print $5'})" -gt "0" ]]; then - # remove entries without size - if [[ "$(echo "$line" | awk {'print $2'})" == "2" ]]; then - # catch directories - t_size=$(echo "$line" | awk {'print $5'}) - full_path="$path$(echo "$line" | awk {'print $9'})/" - fi + if [[ -n "$line" && $(echo "$line" | awk '{print $5}') -gt 0 ]]; then + # Make sure line contains something + # Remove entries without size + if [[ $(echo "$line" | awk '{print $2}') == "2" ]]; then + # Catch directories + t_size=$(echo "$line" | awk '{print $5}') + full_path="$path$(echo "$line" | awk '{print $9}')/" fi - t_size=$(echo "$line" | awk {'print $5'}) - #full_path="$path$(echo "$line" | awk {'print $9'})" - size=$(echo $size + $t_size | bc) + t_size=$(echo "$line" | awk '{print $5}') + size=$(echo "$size + $t_size" | bc) fi done < "$transfersize2" else - size=$(cat "$transfersize" | awk '{print $1}') + size=$(awk '{print $1}' < "$transfersize") fi + directorysize="$size" - size=$(echo "scale=2; "$size" / (1024*1024)" | bc) + size=$(echo "scale=2; $size / (1024 * 1024)" | bc) if [[ ${#exclude_array[@]} -eq 0 ]]; then - echo "INFO: Size to transfere: ${size}MB" + echo "INFO: Size to transfer: ${size}MB" else - echo "INFO: Size to transfere(regex used): ${size}MB" + echo "INFO: Size to transfer (regex used): ${size}MB" fi + cleanup session - elif [[ "$transferetype" == upftp ]]; then - # look up directory or file size locally + elif [[ "$transferetype" == upftp || "$transferetype" == upsftp ]]; then + # Look up directory or file size locally directorysize=$(du -bsL "$dir" | awk '{print $1}') - size=$(echo "scale=2; "$directorysize" / (1024*1024)" | bc) - echo "INFO: Size to transfere: ${size}MB" - # exclude files matching passed regex - if [[ "${#exclude_array[@]}" -gt 0 ]] && [[ -n "${exclude_array[@]}" ]]; then + size=$(echo "scale=2; $directorysize / (1024 * 1024)" | bc) + echo "INFO: Size to transfer: ${size}MB" + + # Exclude files matching passed regex + if [[ "${#exclude_array[@]}" -gt 0 && -n "${exclude_array[@]}" ]]; then exclude_expression=() - n="1" + n=1 for i in "${exclude_array[@]}"; do exclude_expression+=("-iname *$i*") - #add -or if not finished - if [[ "$n" -lt "${#exclude_array[@]}" ]]; then + # Add -or if not finished + if [[ "$n" -lt ${#exclude_array[@]} ]]; then exclude_expression+=("-or") fi - let n++ + (( n++ )) done exclude_expression="${exclude_expression[@]}" directorysize=$(find -L "$dir" \! \( $exclude_expression \) -type f -printf '%s\n' | awk -F ":" '{sum+=$NF} END { printf ("%0.0f\n", sum)}') # in bytes - size=$(echo "scale=2; $directorysize / (1024*1024)" | bc) - echo "INFO: Updated size to transfere(regex used): ${size}MB" + size=$(echo "scale=2; $directorysize / (1024 * 1024)" | bc) + echo "INFO: Updated size to transfer (regex used): ${size}MB" fi fi } +# Function to remove files from an array function removeClean { local array array=("$@") - # removes passed files + # Iterate through array and remove files for i in "${array[@]}"; do rm -f "$i" done } +# Function to perform cleanup operations function cleanup { local array case "$1" in - "die" ) #used when script stops on user input + "die" ) # Used when script stops on user input echo -e "\n*** Ouch! Exiting ***\n" stty sane if [[ "$safelock" != "true" ]]; then - # 1) stop processes, 2) cleanup session, 3) remove lockfile/end session + # Stop processes, cleanup session, remove lockfile/end session cleanup stop-safe cleanup session cleanup end - sed "5s#.*#*************************** Transfer: Aborted #" -i $logfile + sed "5s#.*#*************************** Transfer: Aborted #" -i "$logfile" fi exit 1;; - "session" ) #use to end session of transfer + "session" ) # Use to end session of transfer if [[ $test_mode == "true" ]]; then read -sn 1 -p "Press ANY button to continue cleanup..." fi @@ -163,27 +169,36 @@ function cleanup { mountsystem umount unset mount_in_use fi - # removal of all files creates - array=( "$ftplogin_file1" "$ftplogin_file2" "$ftptransfere_file" "$ftp_size_file" "$ftpfreespace_file" "$lftptransfersize" "$lftptransfersize2" "$transfersize" "$transfersize2" "$proccess_bar_file" "$ftpalive_file" "$ftpcheck_file" "$ftpcheck_testfile" "$ftptransfere_processbar" ) + # Define array of files to remove + array=( "$login_file1" "$login_file2" "$lftpdebug" "$lftptransfersize" "$lftptransfersize2" "$lockfile" "$log_control" "$maindebugfile" "$oldlogfile" "$proccess_bar_file" "$server_alive_file" "$server_check_file" "$server_check_testfile" "$server_content" "$server_freespace_file" "$server_list_file" "$server_size_file" "$transfere_file" "$transfere_processbar" "$transfersize" "$transfersize2" ) + # Call removeClean function to remove files removeClean "${array[@]}" - # removal of tempdirs - if [[ -d "$scriptdir/run/$username-temp" ]]; then rm -r "$scriptdir/run/$username-temp"; fi; - echo -e "INFO: Cleanup done" + # Remove tempdirs + if [[ -d "$scriptdir/run/$username-temp" ]]; then + rm -r "$scriptdir/run/$username-temp" + fi + echo -e "INFO: Session cleanup done" ;; - "end" ) #use to end script + "end" ) # Use to end script removeClean "$lockfile" - sed "5s#.*#*************************** #" -i $logfile + sed "5s#.*#*************************** #" -i "$logfile" echo -e "\e[00;32mExiting successfully...\e[00m\n" ;; - "stop-safe" ) #use to terminate all pids except main process + "stop-safe" ) # Use to terminate all pids except main process for i in {2..4}; do - if [[ -n "$(sed -n "${i}p" $lockfile)" ]]; then kill $(sed -n "${i}p" $lockfile) &> /dev/null; wait $(sed -n "$i p" $lockfile) 2>/dev/null; fi + if [[ -n "$(sed -n "${i}p" "$lockfile")" ]]; then + kill "$(sed -n "${i}p" "$lockfile")" &> /dev/null + wait "$(sed -n "$i p" "$lockfile")" 2>/dev/null + fi done ;; - "stop" ) #use to terminate all pids found in the lockfile - for i in {1..4}; do - if [[ -n "$(sed -n "${i}p" $lockfile)" ]]; then kill $(sed -n "${i}p" $lockfile) &> /dev/null; wait $(sed -n "$i p" $lockfile) 2>/dev/null; fi - done + "stop" ) # Use to terminate all pids found in the lockfile + for i in {1..4}; do + if [[ -n "$(sed -n "${i}p" "$lockfile")" ]]; then + kill "$(sed -n "${i}p" "$lockfile")" &> /dev/null + wait "$(sed -n "$i p" "$lockfile")" 2>/dev/null + fi + done ;; esac } diff --git a/dependencies/sorting.sh b/dependencies/sorting.sh index a909021..1859f29 100644 --- a/dependencies/sorting.sh +++ b/dependencies/sorting.sh @@ -1,44 +1,47 @@ #!/bin/bash -#version 0.3 +# Version 0.4 + +# Function to sort files based on their names function sortFiles { - # $1 is the argument passed using --sort=. No trailing slash is needed - if [[ -n "$1" ]]; then - if [[ "$1" == "nosort" ]]; then - echo "INFO: No sorting. Transferring to default complete directory: \"$ftpcomplete\"" - else - ftpcomplete="${ftpcomplete}${1}/" - echo "INFO: Sorted to ${ftpcomplete}" - fi - elif [[ $orig_name =~ (\.(x86|x64|mac|android|iphone)) ]]; then - ftpcomplete=$ftpcomplete"Appz/" - echo "INFO: Sorted to ${ftpcomplete}" - elif [[ $orig_name =~ (PAL|NTSC).+DVDR. ]]; then - ftpcomplete=$ftpcomplete"DVD/" - echo "INFO: Sorted to ${ftpcomplete}" - elif [[ $orig_name =~ (BDRip|DVDRip).+(XviD|x264) ]] && [[ ! $orig_name =~ (S[0-9][0-9]) ]] && [[ ! $orig_name =~ (S[0-9][0-9]E[0-9][0-9]) ]] && [[ ! $orig_name =~ (E[0-9][0-9]) ]] && [[ ! $orig_name =~ (S[0-9][0-9]) ]] && [[ ! $orig_name =~ (\.(episode|ep|e|Part|pt)\.(([0-9][0-9]?)|(I|II|III|IV|V|VI|VII|VIII|IIX|IX|X|XI|XII))\.) ]] && [[ ! $orig_name =~ ([[:digit:]](x|of)[[:digit:]]) ]]; then - ftpcomplete=$ftpcomplete"XViD/" - echo "INFO: Sorted to ${ftpcomplete}" - elif [[ $orig_name =~ (HDDVD|BluRay|WEB-DL|WEB).+(x264|x265) ]] && [[ ! $orig_name =~ (S[0-9][0-9]) ]]; then - ftpcomplete=$ftpcomplete"HD/" - echo "INFO: Sorted to ${ftpcomplete}" - elif [[ $orig_name =~ \.XXX\. ]]; then - ftpcomplete=$ftpcomplete"XXX/" - echo "INFO: Sorted to ${ftpcomplete}" - elif [[ $orig_name =~ (S[0-9][0-9]E[0-9][0-9]) ]] || [[ $orig_name =~ (\.E[0-9][0-9]?[0-9]?\.) ]] || [[ $orig_name =~ (S[0-9][0-9]) ]] || [[ $orig_name =~ (\.(episode|ep|e|Part|pt)\.(([0-9][0-9]?)|(I|II|III|IV|V|VI|VII|VIII|IIX|IX|X|XI|XII))\.) ]] || [[ $orig_name =~ ([[:digit:]](x|of)[[:digit:]]) ]] || [[ $orig_name =~ ([[:digit:]]{4}.[[:digit:]]{2}.[[:digit:]]{2})|([[:digit:]]{2}.[[:digit:]]{2}.[[:digit:]]{4}) ]]; then - # Sort to series folders if format matches - local series_name="${orig_name%%.S[[:digit:]]*}" - if [[ $orig_name == $series_name ]]; then - ftpcomplete=$ftpcomplete"TV/" - echo "INFO: Series name could not be found" - echo "INFO: Sorted to: ${ftpcomplete}" - else - series_name="${series_name//./ }" # replace dots with spaces - echo -e "\e[00;37mINFO: Series name: \e[00;32m$series_name\e[00m" - ftpcomplete=$ftpcomplete"TV/$series_name/" - echo "INFO: Sorted to: ${ftpcomplete}" - fi - else - echo -e "\e[00;33mINFO: Category could not be parsed\e[00m" - echo "INFO: Moving files upon finish to default directory: \"$ftpcomplete\"" - fi -} + # $1 is the argument passed using --sort=. No trailing slash is needed + if [[ -n "$1" ]]; then + if [[ "$1" == "nosort" ]]; then + echo "INFO: No sorting. Transferring to default complete directory: \"$ftpcomplete\"" + else + ftpcomplete="${ftpcomplete}${1}/" + echo "INFO: Sorted to ${ftpcomplete}" + fi + # Check file name patterns and sort accordingly + elif [[ $orig_name =~ (\.(x86|x64|mac|android|iphone)) ]]; then + ftpcomplete=$ftpcomplete"Appz/" + echo "INFO: Sorted to ${ftpcomplete}" + elif [[ $orig_name =~ (PAL|NTSC).+DVDR. ]]; then + ftpcomplete=$ftpcomplete"DVD/" + echo "INFO: Sorted to ${ftpcomplete}" + elif [[ $orig_name =~ (BDRip|DVDRip).+(XviD|x264) ]] && [[ ! $orig_name =~ (S[0-9][0-9]) ]] && [[ ! $orig_name =~ (S[0-9][0-9]E[0-9][0-9]) ]] && [[ ! $orig_name =~ (E[0-9][0-9]) ]] && [[ ! $orig_name =~ (S[0-9][0-9]) ]] && [[ ! $orig_name =~ (\.(episode|ep|e|Part|pt)\.(([0-9][0-9]?)|(I|II|III|IV|V|VI|VII|VIII|IIX|IX|X|XI|XII))\.) ]] && [[ ! $orig_name =~ ([[:digit:]](x|of)[[:digit:]]) ]]; then + ftpcomplete=$ftpcomplete"XViD/" + echo "INFO: Sorted to ${ftpcomplete}" + elif [[ $orig_name =~ (HDDVD|BluRay|WEB-DL|WEB).+(x264|x265|H264|H265) ]] && [[ ! $orig_name =~ (S[0-9][0-9]) ]]; then + ftpcomplete=$ftpcomplete"HD/" + echo "INFO: Sorted to ${ftpcomplete}" + elif [[ $orig_name =~ \.XXX\. ]]; then + ftpcomplete=$ftpcomplete"XXX/" + echo "INFO: Sorted to ${ftpcomplete}" + elif [[ $orig_name =~ (S[0-9][0-9]E[0-9][0-9]) ]] || [[ $orig_name =~ (\.E[0-9][0-9]?[0-9]?\.) ]] || [[ $orig_name =~ (S[0-9][0-9]) ]] || [[ $orig_name =~ (\.(episode|ep|e|Part|pt)\.(([0-9][0-9]?)|(I|II|III|IV|V|VI|VII|VIII|IIX|IX|X|XI|XII))\.) ]] || [[ $orig_name =~ ([[:digit:]](x|of)[[:digit:]]) ]] || [[ $orig_name =~ ([[:digit:]]{4}.[[:digit:]]{2}.[[:digit:]]{2})|([[:digit:]]{2}.[[:digit:]]{2}.[[:digit:]]{4}) ]]; then + # Sort to series folders if format matches + local series_name="${orig_name%%.S[[:digit:]]*}" + if [[ $orig_name == $series_name ]]; then + ftpcomplete=$ftpcomplete"TV/" + echo "INFO: Series name could not be found" + echo "INFO: Sorted to: ${ftpcomplete}" + else + series_name="${series_name//./ }" # replace dots with spaces + echo -e "\e[00;37mINFO: Series name: \e[00;32m$series_name\e[00m" + ftpcomplete=$ftpcomplete"TV/$series_name/" + echo "INFO: Sorted to: ${ftpcomplete}" + fi + else + echo -e "\e[00;33mINFO: Category could not be parsed\e[00m" + echo "INFO: Moving files upon finish to default directory: \"$ftpcomplete\"" + fi +} \ No newline at end of file diff --git a/dependencies/transfer_main.sh b/dependencies/transfer_main.sh new file mode 100644 index 0000000..d6e5170 --- /dev/null +++ b/dependencies/transfer_main.sh @@ -0,0 +1,815 @@ +#!/bin/bash +function delay { + # if --delay is set, wait until it ends. If start/end time is set in config use them. Delay overrules everything + local current_epoch target_epoch sleep_seconds timediff + if [[ -n $delay ]]; then + current_epoch=$(date +%s) + target_epoch=$(date -d "$delay" +%s) + if [[ $target_epoch -gt $current_epoch ]]; then + sleep_seconds=$(( target_epoch - current_epoch )) + if [[ $test_mode != "true" ]]; then + echo "INFO: Transfere has been postponed until $delay" + timediff=$(printf '%2d:%2d:%2d' "$((sleep_seconds/(60*60)))" "$(((sleep_seconds/60)%60))" "$((sleep_seconds%60))") + countdown "$timediff" + else + echo -e "\e[00;31mTESTMODE: Would delay until $delay\e[00m" + fi + else + echo -e "\e[00;31mERROR: Time is older than current time, now $(date '+%m/%d/%y %H:%M') vs. delay $delay . Format should be mm/dd/yy hh:mm.\e[00m\n" + cleanup session + cleanup end + exit 1 + fi + elif [[ -n $transfer_start ]] && [[ -n $transfer_end ]] && [[ $force == "false" ]]; then + tranfere_timeframe + fi +} + +function countdown { + # calculate and show countdown in display + local OLD_IFS arr seconds start end cur left IFS + OLD_IFS="${IFS}" + IFS=":" + arr=( $1 ) + seconds=$(( (arr[0] * 60 * 60) + (arr[1] * 60) + arr[2] )) + start=$(date +%s) + end=$((start + seconds)) + cur=$start + while [[ $cur -lt $end ]]; do + cur=$(date +%s) + left=$((end-cur)) + printf "\r%02d:%02d:%02d" \ + $((left/3600)) $(((left/60)%60)) $((left%60)) + sleep 1 + done + IFS="${OLD_IFS}" + echo " " +} + +function queue { + # Queue system: If something is already running for the user, add it to the queue. + local option i old_id + option=$2 + case "$1" in + "add" ) + if [[ $queue_running == true ]]; then + # Task has been started from queue, no need to add it. + : + else + # Figure out ID. + id_old=$id + if [[ -e "$queue_file" ]]; then + # Get last ID. + id=$(( $(tail -1 "$queue_file" | cut -d'|' -f1) + 1 )) + else + # Assume this is the first one. + id="1" + fi + get_size "$filepath" &> /dev/null + if [[ -e "$queue_file" && -n $(cat "$queue_file" | grep "$filepath") ]]; then + # Passing an item which is already in the queue, do nothing. + echo -e "INFO: Item already in queue. Doing nothing...\n" + exit 0 + elif [[ "$option" == failed ]]; then + # Passing a failed item, remove it, and add it with the status failed. + failed="true" + # Remove ID from queue. + sed "/^"$id_old"/d" -i "$queue_file" + echo "$id|$source|$filepath|$sortto|${size}MB|true|$(date '+%d/%m/%y-%a-%H:%M:%S')" >> "$queue_file" + echo -e "\e[00;33mINFO: Failing item: $(basename "$filepath")\e[00m" + elif [[ "$option" == end ]]; then + # Passed item should only be queued, then exit. + source="${source}Q" + echo "$id|$source|$filepath|$sortto|${size}MB|false|$(date '+%d/%m/%y-%a-%H:%M:%S')" >> "$queue_file" + echo -e "INFO: Queueing: $(basename "$filepath"), id=$id\n" + exit 0 + else + # Passed item should be queued, e.g. when something already is being transferred. + echo "INFO: Queueid: $id" + echo "$id|$source|$filepath|$sortto|${size}MB|false|$(date '+%d/%m/%y-%a-%H:%M:%S')" >> "$queue_file" + fi + fi + ;; + "remove" ) + # Remove item according to ID. + sed "/^"$id"/d" -i "$queue_file" + # If queue is true then continue to run else stop. + if [[ $continue_queue == true ]]; then + queue next + else + cleanup end + fi + ;; + "next" ) + # Process next item in queue from top. + if [[ -f "$queue_file" ]] && [[ -n $(cat "$queue_file") ]]; then + i="1" + failed="true" + # Look for non-failed items. + while [[ $failed == true ]]; do + # Load next item from top. + id=$(awk 'BEGIN{FS="|";OFS=" "}NR=='$i'{print $1}' "$queue_file") + # Check if ID has failed. + failed=$(awk 'BEGIN{FS="|";OFS=" "}NR=='$i'{print $6}' "$queue_file") + let i++ + done + if [[ $failed == false ]]; then + # Found a non-failed item, which will be downloaded. + i=$(grep -n "^${id}|" "$queue_file" | grep -Eo '^[^:]+') + source=$(awk 'BEGIN{FS="|";OFS=" "}NR=='$i'{print $2}' "$queue_file") + filepath=$(awk 'BEGIN{FS="|";OFS=" "}NR=='$i'{print $3}' "$queue_file") + sort=$(awk 'BEGIN{FS="|";OFS=" "}NR=='$i'{print $4}' "$queue_file") + # Execute main script again. + queue_running="true" + if [[ -f "$lockfile" ]]; then + # Ensure that lockfile isn't created running queue. + lockfileRunning="true" + fi + echo "---------------------- Running queue ----------------------" + echo "Transfering id=$id, $(basename "$filepath")" + start_main --path="$filepath" --user="$username" --sortto="$sort" + else + # All items in the queue are marked as failed, e.g. nothing to transfer. + echo "---------------------- Failed queue -----------------------" + while read line; do + id=$(echo $line | cut -d'|' -f1) + source=$(echo $line | cut -d'|' -f2) + path=$(echo $line | cut -d'|' -f3) + sort=$(echo $line | cut -d'|' -f4) + size=$(echo $line | cut -d'|' -f5) + time=$(echo $line | cut -d'|' -f7) + echo "ID|PATH|SORT TO|SIZE(MB)|TIME" + echo "$id|$source|$path|$sort|$size|$time" + done < "$queue_file" + echo -e "\nINFO: Queue does not contain non-failed items. Program will end\n" + cleanup end + exit 1 + fi + else + # No queuefile found, e.g. nothing to transfer. + echo "----------------------- Empty queue -----------------------" + if [[ -f "$queue_file" ]]; then rm "$queue_file"; fi + echo -e "INFO: Queue is empty. Program will end\n" + cleanup end + exit 0 + fi + ;; + "fail" ) + # Failed item, remove it, and add it with the status failed. + # Remove ID from queue. + sed "/^"$id"/d" -i "$queue_file" + echo "$id|$source|$filepath|$sortto|${size}MB|true|$(date '+%d/%m/%y-%a-%H:%M:%S')" >> "$queue_file" + echo -e "\e[00;33mINFO: Failing item: $(basename "$filepath")\e[00m" + ;; + esac +} + +function transfer_process { + # used to start and stop the lftp transfer and progressbar + local pid_f_process + case "$1" in + "start" ) #start progressbar and transfer + TransferStartTime=$(date +%s) + transfer_process_bar & + pid_f_process=$! + sed "3c $pid_f_process" -i "$lockfile" + echo -e "\e[00;37mINFO: \e[00;32mTransfer started: $(date --date=@$TransferStartTime '+%d/%m/%y-%a-%H:%M:%S')\n\e[00m" + $lftp -f "$transfere_file" &> /dev/null & + pid_transfer=$! + sed "2c $pid_transfer" -i "$lockfile" + wait $pid_transfer 2>/dev/null + pid_transfer_status=$? + TransferEndTime=$(date +%s) + ;; + "stop-process-bar" ) + kill $(sed -n '3p' $lockfile) &> /dev/null + wait $(sed -n '3p' "$lockfile") 2>/dev/null + kill $(sed -n '4p' $lockfile) &> /dev/null + wait $(sed -n '4p' "$lockfile") 2>/dev/null + + ;; + esac +} + +function transfer { + local lftp_exclude quittime waittime + + # Prepare new transfer + { + loadDependency DServerLogin && load_login 1 + + # Write regexp to config for directory transfers + if [[ "${#exclude_array[@]}" -gt 0 && -n "${exclude_array[@]}" && ($transfer_type = "directory" || -d "$transfer_path") ]]; then + for ((i=0; i<${#exclude_array[@]}; i++)); do + [[ $i -gt 0 ]] && lftp_exclude+="|" + lftp_exclude+="^.*${exclude_array[i]}*" + done + lftp_exclude+="\$" + echo "set mirror:exclude-regex \"$lftp_exclude\"" >> "$transfere_file" + echo "set mirror:no-empty-dirs true" >> "$transfere_file" + fi + + if [[ $transferetype == "downftp" ]]; then + # Handle lftp transfer for downftp + cat "$login_file1" >> "$transfere_file" + # Create final directories if they don't exist + echo "!mkdir -p \"${complete}\"" >> "$transfere_file" + [[ -n $incomplete ]] && echo "!mkdir -p \"${incomplete}\"" >> "$transfere_file" + echo "set cmd:fail-exit true" >> "$transfere_file" + + # Determine transfer type (file or directory) + if [[ $transfer_type = file ]]; then + [[ -n $incomplete ]] && echo "queue get -c -O \"${incomplete}\" \"${transfer_path}\"" >> "$transfere_file" + [[ -z $incomplete ]] && echo "queue get -c -O \"${complete}\" \"${transfer_path}\"" >> "$transfere_file" + elif [[ $transfer_type = directory ]]; then + [[ -n $incomplete ]] && echo "queue mirror --no-umask -p --parallel=$parallel -c \"${transfer_path}\" \"${incomplete}\"" >> "$transfere_file" + [[ -z $incomplete ]] && echo "queue mirror --no-umask -p --parallel=$parallel -c \"${transfer_path}\" \"${complete}\"" >> "$transfere_file" + fi + + echo "wait" >> "$transfere_file" + + # Move files locally if incomplete directory is used + [[ -n $incomplete ]] && echo "queue !mv \"${incomplete}${orig_name}\" \"${complete}\"" >> "$transfere_file" + echo "wait" >> "$transfere_file" + elif [[ $transferetype == "upftp" || $transferetype == "upsftp" ]]; then + # Handle lftp transfer for upftp + cat "$login_file1" >> "$transfere_file" + echo "mkdir -p \"${complete}\"" >> "$transfere_file" + # handle files for transfer + if [[ -n "${incomplete}" ]]; then + echo "mkdir -p \"${incomplete}\"" >> "$transfere_file" + fi + # fail if transfers fails + echo "set cmd:fail-exit true" >> "$transfere_file" + + # Determine transfer type (file or directory) + if [[ -f "$transfer_path" ]]; then + [[ -n "$incomplete" ]] && echo "queue put -c -O \"$incomplete\" \"${transfer_path}\"" >> "$transfere_file" + [[ -z "$incomplete" ]] && echo "queue put -c -O \"$complete\" \"${transfer_path}\"" >> "$transfere_file" + elif [[ -d "$transfer_path" ]]; then + [[ -n "$incomplete" ]] && echo "queue mirror --no-umask -p --parallel=$parallel -c -RL \"${transfer_path}\" \"${incomplete}\"" >> "$transfere_file" + [[ -z "$incomplete" ]] && echo "queue mirror --no-umask -p --parallel=$parallel -c -RL \"${transfer_path}\" \"${complete}\"" >> "$transfere_file" + fi + + echo "wait" >> "$transfere_file" + + # Move files remotely if incomplete directory is used + [[ -n "$incomplete" ]] && echo "queue mv \"${incomplete}${orig_name}\" \"${complete}\"" >> "$transfere_file" + echo "wait" >> "$transfere_file" + elif [[ $transferetype == "fxp" ]]; then + # Handle lftp transfer for fxp + server_login 2 + cat "$login_file2" >> "$transfere_file" + echo "mkdir -p \"$complete\"" >> "$transfere_file" + [[ -n $incomplete ]] && echo "mkdir -p \"$incomplete\"" >> "$transfere_file" + echo "set cmd:fail-exit true" >> "$transfere_file" + + # Determine transfer type (file or directory) + if [[ $transfer_type = file ]]; then + [[ -n "$incomplete" ]] && echo "queue get -c ftp://$ftpuser1:$ftppass1@$ftphost1:$ftpport1:\"$transfer_path\" -o ftp://$ftpuser2:$ftppass2@$ftphost2:$ftpport2:\"$incomplete\"" >> "$transfere_file" + [[ -z "$incomplete" ]] && echo "queue get -c ftp://$ftpuser1:$ftppass1@$ftphost1:$ftpport1:\"$transfer_path\" -o ftp://$ftpuser2:$ftppass2@$ftphost2:$ftpport2:\"$complete\"" >> "$transfere_file" + elif [[ $transfer_type = directory ]]; then + [[ -n "$incomplete" ]] && echo "queue mirror --no-umask -p --parallel=$parallel -c -RL ftp://$ftpuser1:$ftppass1@$ftphost1:$ftpport1:\"${transfer_path}\" ftp://$ftpuser2:$ftppass2@$ftphost2:$ftpport2:\"${incomplete}\"" >> "$transfere_file" + [[ -z "$incomplete" ]] && echo "queue mirror --no-umask -p --parallel=$parallel -c -RL ftp://$ftpuser1:$ftppass1@$ftphost1:$ftpport1:\"${transfer_path}\" ftp://$ftpuser2:$ftppass2@$ftphost2:$ftpport2:\"${complete}\"" >> "$transfere_file" + fi + + echo "wait" >> "$transfere_file" + + # Move files remotely if incomplete directory is used + [[ -n "$incomplete" ]] && echo "queue mv \"${incomplete}${orig_name}\" \"${complete}\"" >> "$transfere_file" + echo "wait" >> "$transfere_file" + else + echo -e "\e[00;31mERROR: Transfer setting not recognized\e[00m\n" + cleanup die + fi + + echo "quit" >> "$transfere_file" + } + + # Start transferring + { + if [[ $test_mode != "true" ]]; then + # Start the transfer process + transfer_process start + + # Loop until transfer completes or timeout is reached + while [[ $pid_transfer_status -ne 0 ]]; do + quittime=$(( ScriptStartTime + retry_download_max*60 )) + + # Check if it's time to quit + if [[ $(date +%s) -gt $quittime ]]; then + echo -e "\e[00;31mERROR: Transfer failed after reaching the maximum retry time ($retry_download_max minutes)!\e[00m" + + # Stop the transfer process and add to queue + transfer_process "stop-process-bar" + queue add failed + break + else + echo -e "\e[00;31mERROR: Transfer failed for some reason!\e[00m" + echo "INFO: Retrying until $(date --date=@$quittime '+%d/%m/%y-%a-%H:%M:%S')" + + # Stop the transfer process, pause session, and retry after a delay + transfer_process "stop-process-bar" + echo "INFO: Pausing session and trying again in 60 seconds" + sed "3s#.*#*************************** SERVER INFO: DOWNLOAD POSTPONED! Trying again in ${retry_download} minutes#" -i "$logfile" + sleep 60 + transfer_process start + fi + done + + # Stop the transfer process and display end message + transfer_process "stop-process-bar" + echo -e "\n\e[00;37mINFO: \e[00;32mTransfer ended: $(date --date=@$TransferEndTime '+%d/%m/%y-%a-%H:%M:%S')\e[00m" + else + # Test mode: Display the lftp commands that would be executed + echo -e "\e[00;31mTESTMODE: LFTP-transfer NOT STARTED\e[00m" + echo "Would execute the following in lftp:" + cat "$transfere_file" | (while read; do echo " $REPLY"; done) + fi + } + cleanup session +} + +# Function to show the progress of the transfer +function transfer_process_bar { + if [[ $test_mode != "true" ]]; then + sleep 5 # Wait for transfer to start + + if [[ $transferetype == "downftp" ]]; then + # Get the transferred size for download + transfered_size="du -s \"$incomplete$changed_name\" > \"$proccess_bar_file\"" + if [[ $transfer_type = file ]]; then + echo "du -s \"$incomplete${orig_name}\" > ~/../..$proccess_bar_file" >> "$transfere_processbar" + elif [[ $transfer_type = directory ]]; then + echo "du -s \"$incomplete$orig_name\" > ~/../..$proccess_bar_file" >> "$transfere_processbar" + fi + elif [[ $transferetype == "upftp" || $transferetype == "fxp" || $transferetype == "upsftp" ]]; then + # Create a config file for lftp process bar + cat "$login_file1" >> "$transfere_processbar" + # Determine whether it's a file or directory + if [[ -f "$filepath" ]]; then + echo "du -s \"$incomplete${orig_name}\" > ~/../..$proccess_bar_file" >> "$transfere_processbar" + elif [[ -d "$filepath" ]]; then + echo "du -s \"$incomplete$orig_name\" > ~/../..$proccess_bar_file" >> "$transfere_processbar" + fi + echo "quit" >> "$transfere_processbar" + else + echo "Progressbar not supported" + return + fi + + # Run the process bar loop + while :; do + if [[ $transferetype == "downftp" ]]; then + eval $transfered_size + elif [[ $transferetype == "upftp" || $transferetype == "fxp" || $transferetype == "upsftp" ]]; then + $lftp -f "$transfere_processbar" &> /dev/null & + pid_process=$! + sed "4c $pid_process" -i "$lockfile" + wait $pid_process + fi + + # Get current time and transferred information + if [[ ! -a "$proccess_bar_file" ]]; then + continue + elif [[ -z "$TransferredOld" ]] && [[ -a "$proccess_bar_file" ]]; then + TransferredOld=$(cat $proccess_bar_file | awk '{print $1}') + ProgressTimeOld=$(date +%s) + rm "$proccess_bar_file" + continue + fi + + # Feedback received + if [[ -a "$proccess_bar_file" ]]; then + ProgressTimeNew=$(date +%s) + TransferredNew=$(cat "$proccess_bar_file" | awk '{print $1}') + TransferredNewMB=$(echo $TransferredNew / 1024 | bc) + TotalTimeDiff=$(( ProgressTimeNew - TransferStartTime )) + TimeDiff=$(printf '%02dh:%02dm:%02ds' "$((TotalTimeDiff/(60*60)))" "$(((TotalTimeDiff/60)%60))" "$((TotalTimeDiff%60))") + + if [[ "$(( $TransferredNew - $TransferredOld ))" -ge "1" ]] && [[ "$(( $TransferredNew - $TransferredOld ))" =~ ^[0-9]+$ ]]; then + percentage=$(echo "scale=4; ( $TransferredNew / ( $directorysize / ( 1024 ) ) ) * 100" | bc) + percentage=$(echo $percentage | sed 's/\(.*\)../\1/') + speed=$(echo "scale=2; ( ($TransferredNew - $TransferredOld) / 1024 ) / ( $ProgressTimeNew - $ProgressTimeOld )" | bc) + eta=$(echo "( ($directorysize / 1024 ) - $TransferredNew ) / ($speed * 1024 )" | bc) + etatime=$(printf '%02dh:%02dm:%02ds' "$(($eta/(60*60)))" "$((($eta/60)%60))" "$(($eta%60))") + + SpeedOld+=( "$speed" ) + if [[ -n "${#SpeedOld[@]}" ]]; then + sum="0" + for i in "${SpeedOld[@]}"; do + sum=$(echo "( $sum + $i )" | bc) + done + SpeedAverage=$(echo "scale=2; $sum / ${#SpeedOld[@]}" | bc) + sed "5c $SpeedAverage" -i "$lockfile" + tput cuu 1 + tput el1 + fi + else + speed="?" + SpeedAverage="?" + percentage="0" + etatime="?" + fi + + # Update file and output the current line + sed "5s#.*#*************************** Transferring: ${orig_name}, $percentage\%, in $TimeDiff, $speed MB/s(current), ETA: $etatime, ${SpeedAverage} MB/s (avg) #" -i "$logfile" + cols=$(($(tput cols) - 2)) + percentagebarlength=$(echo "scale=0; $percentage * $cols / 100" | bc) + string="$(eval printf "=%.0s" '{1..'"$percentagebarlength"\})" + string2="$(eval printf "\ %.0s" '{1..'"$(($cols - $percentagebarlength - 1))"\})" + + if [[ $percentagebarlength -eq 0 ]]; then + printf "\r[$string2] (no transferred information yet) ($(date '+%H:%M:%S'))" + elif [[ $(echo "scale=0; $cols - $percentagebarlength - 1" | bc) -eq 0 ]]; then + printf "\r[$string>] $percentage%% ETA ${etatime}@${speed}MB/s. ${TransferredNewMB}MB@${SpeedAverage}MB/s(avg). ($(date '+%H:%M:%S'))" + else + printf "\r[$string>$string2] $percentage%% ETA ${etatime}@${speed}MB/s. ${TransferredNewMB}MB@${SpeedAverage}MB/s(avg). ($(date '+%H:%M:%S'))" + fi + fi + + # Update variables and wait + TransferredOld="$TransferredNew" + ProgressTimeOld="$ProgressTimeNew" + rm "$proccess_bar_file" + sleep $sleeptime + done + else + echo -e "\e[00;31mTESTMODE: LFTP-processbar NOT STARTED\e[00m" + fi +} + +# Function to rotate and manage logs +function logrotate { + # Check if test mode is not enabled + if [[ $test_mode != "true" ]]; then + # Calculate transfer time + transferTime=$(( $TransferEndTime - $TransferStartTime )) + transferTime2=$(printf '%02dh:%02dm:%02ds' "$(($transferTime/(60*60)))" "$((($transferTime/60)%60))" "$(($transferTime%60))") + + # Get average speed from lockfile + SpeedAverage=$(sed -n 5p "$lockfile") + + # Add new info to 7th line of logfile + sed "7i $(date --date=@$ScriptStartTime '+%d/%m/%y-%a-%H:%M:%S')|${source}|${orig_name}|${size}MB|${transferTime2}|${SpeedAverage}MB/s" -i "$logfile" + lognumber=$((7 + $lognumber )) + + # Add text to oldlogfile if log rotation is enabled + if [[ $logrotate == "true" ]]; then + if [[ -n $(sed -n $lognumber,'$p' "$logfile") ]]; then + sed -n $lognumber,'$p' "$logfile" >> "$oldlogfile" + fi + fi + + # Remove text from old file + if [ "$lognumber" -ne 0 ]; then + sed $lognumber,'$d' -i "$logfile" + fi + + # Calculate total downloaded size + totaldl=$(awk 'BEGIN{FS="|";OFS=" "}NR==2{print $1}' "$logfile" | cut -d' ' -f4) + totaldl=${totaldl%MB} + if [[ -z "$totaldl" ]]; then + totaldl="0" + fi + totaldl=$(echo "$totaldl + $size" | bc) + + # Increment total number of transfers + totalrls=$(awk 'BEGIN{FS="|";OFS=" "}NR==2{print $1}' "$logfile" | cut -d' ' -f6) + totalrls=$(echo "$totalrls + 1" | bc) + + # Calculate total download time + totaldltime=$(awk 'BEGIN{FS="|";OFS=" "}NR==2{print $1}' "$logfile" | cut -d' ' -f10) + totaldltime_seconds=$(awk 'BEGIN{split("'$totaldltime'",a,":"); print a[1]*(60*60*24)+a[2]*(60*60)+a[3]*60+a[4];}') + totaldltime=$(echo "$totaldltime_seconds + $transferTime" | bc) + totaldltime=$(printf '%02dd:%02dh:%02dm:%02ds' "$(($totaldltime/(60*60*24)))" "$(($totaldltime/(60*60)%24))" "$((($totaldltime/60)%60))" "$(($totaldltime%60))") + + # Update logfile header and stats + sed "1s#.*#***** FTPauto ${s_version}#" -i "$logfile" + sed "2s#.*#***** STATS: ${totaldl}MB in ${totalrls} transfers in ${totaldltime}#" -i "$logfile" + sed "3s#.*#***** FTP INFO: N/A#" -i "$logfile" + sed "4s#.*#***** LAST TRANSFER: $(date)|${orig_name}|${SpeedAverage}MB/s#" -i "$logfile" + sed "5s#.*#***** #" -i "$logfile" + else + # If test mode, just print message + echo -e "\e[00;31mTESTMODE: LOGGING NOT STARTED\e[00m" + fi +} + +function loadConfig { + # reload config + loadDependency DConfig + + #load paths to everything + setup +} + +function lockfile { + # Check if lockfile exists and process is still running + if [[ -f "$lockfile" ]] && [[ $force != "true" ]]; then + # Lockfile exists, check if process is still running + mypid_script=$(sed -n 1p "$lockfile") + mypid=$(sed -n 2p "$lockfile") + alreadyinprogress=$(sed -n 3p "$lockfile") + + # Check if the script process is still running + kill -0 $mypid_script &> /dev/null + if [[ $? -eq 1 ]]; then + # Process is not running, continue + echo "INFO: Old lockfile found, but process is not running" + rm -f "$lockfile" + else + # Process is still running, display information and exit + echo "INFO: Another instance of the script is already running:" + echo " Script PID: $mypid_script" + echo " Transfer PID: $alreadyinprogress" + echo " Transfer: $(sed -n 5p < "$logfile" | cut -d',' -f1 | cut -d' ' -f2)" + echo " Lockfile: $lockfile" + echo " Use --help to learn how to stop it" + queue add end + return + fi + fi + + # Allocate PIDs and update lockfile + echo "$BASHPID" >> "$lockfile" # Bash PID + echo >> "$lockfile" # LFTP transfer PID (to be allocated later) + echo >> "$lockfile" # Bash progress PID (to be allocated later) + echo >> "$lockfile" # LFTP process PID (to be allocated later) + echo >> "$lockfile" # Speed average (to be calculated later) + + echo "INFO: Process ID: $BASHPID" +} + +function main { + # Set paths + filepath="$1" + transfer_path="$filepath" + orig_name=$(basename "$filepath") + + # Define temp directory based on file type + if [[ -f "$filepath" ]]; then + tempdir="$scriptdir/run/$username-temp/${orig_name%.*}-temp/" + elif [[ -d "$filepath" ]]; then + tempdir="$scriptdir/run/$username-temp/${orig_name}-temp/" + fi + + # Record script start time and display relevant information + ScriptStartTime=$(date +%s) + echo "INFO: Process start-time: $(date --date=@$ScriptStartTime '+%d/%m/%y-%a-%H:%M:%S')" + echo "INFO: Preparing transfer: $filepath" + echo "INFO: Launched from: $source" + + # Add to queue file to initialize ID + queue add + echo "INFO: Simultaneous transfers: $parallel" + + # Check transfer size + get_size "$filepath" + + # Execute pre-external command if specified + if [[ -n "$exec_pre" ]]; then + if [[ $test_mode != "true" ]]; then + echo "INFO: Executing external command - START" + echo " $exec_pre" + eval "$exec_pre" | (while read; do echo " $REPLY"; done) + else + echo -e "\e[00;31mTESTMODE: Would execute external command: \"$exec_pre\"\e[00m" + fi + echo "INFO: Executing external command - ENDED" + fi + + #confirm server is online + if [[ $confirm_online == "true" ]]; then + loadDependency DServerAliveTest && online_test + fi + + #Check if enough free space on remote server + if [[ "$serversizemanagement" == "true" ]]; then + if [[ $transferetype == "upftp" ]]; then + loadDependency DServerSizeManagement && server_sizemanagement check + elif [[ $transferetype == "downftp" ]]; then + freesize=$(( $(df -P "$incomplete" | tail -1 | awk '{ print $3}') / (1024*1024) )) + freespaceneeded="$size" + while [[ "$freesize" -lt "$freespaceneeded" ]]; do + echo "INFO: Not enough free space" + echo "INFO: Trying again in 1 min" + sleep 60 + # recalculate free space + freesize=$(( $(df -P "$incomplete" | tail -1 | awk '{ print $3}') / (1024*1024) )) + done + fi + fi + + # Check send option + echo "INFO: Send option: $send_option" + if [[ "$send_option" == "split" ]]; then + # Check if rar or cksfv is available + if [[ -n $(command -v rar) || -n $(command -v cksfv) ]]; then + # Check transfer type + if [[ "$transferetype" == "upftp" || "$transferetype" == "upsftp" ]]; then + loadDependency DLargeFile && largefile "$filepath" "exclude_array[@]" + elif [[ "$transferetype" == "downftp" ]]; then + echo -e "\e[00;33mERROR: send_option=split is not supported in mode=$transferetype. Exiting ...\e[00m" + cleanup session + cleanup end + echo -e "INFO: Program has ended\n" + exit 1 + fi + else + echo -e "\e[00;33mERROR: send_option=split is not supported as rar or cksfv is missing. Exiting ...\e[00m" + cleanup session + cleanup end + echo -e "INFO: Program has ended\n" + exit 1 + fi + elif [[ "$send_option" == "video" ]]; then + # Check if rar2fs is available + if [[ -n $(command -v rar2fs) ]]; then + # Check transfer type + if [[ "$transferetype" == "upftp" || "$transferetype" == "upsftp" ]]; then + loadDependency DVideoFile && videoFile + elif [[ "$transferetype" == "downftp" ]]; then + echo -e "\e[00;33mERROR: send_option=video is not supported in mode=$transferetype. Exiting ...\e[00m" + cleanup session + cleanup end + echo -e "INFO: Program has ended\n" + exit 1 + fi + else + echo -e "\e[00;33mERROR: send_option=video is not supported as rarfs is missing. Exiting ...\e[00m" + cleanup session + cleanup end + echo -e "INFO: Program has ended\n" + exit 1 + fi + elif [[ "$send_option" == "default" ]]; then + : + else + echo -e "\e[00;33mERROR: send_option=$send_option is not supported Exiting ...\e[00m" + cleanup session + cleanup end + echo -e "INFO: Program has ended\n" + exit 1 + fi + + # Try to sort files + if [[ "$sort" == "true" ]] || [[ -n "$sortto" ]]; then + loadDependency DSort && sortFiles "$sortto" + fi + + # Delay transfer if needed + delay + + # Transfer files + transfer + + # Checking for remaining space + if [[ "$serversizemanagement" == "true" && $failed != true ]]; then + if [[ $transferetype == "upftp" ]]; then + loadDependency DServerSizeManagement && server_sizemanagement info + fi + fi + + # Update logfile + if [[ $failed != true ]]; then + logrotate + fi + + # Clean up current session + cleanup session + + # Send push notification if push user is specified + if [[ -n $push_user ]]; then + if [[ $test_mode ]]; then + # Display notification details in test mode + echo -e "\e[00;31mTESTMODE: Would send notification \""$orig_name" "Sendoption=$send_option Size=$size MB Time=$transferTime2 Average speed=$SpeedAverage MB/s Path=$complete"\" to token=$push_token and user=$push_user \e[00m" + elif [[ $failed == true ]]; then + # Send notification if transfer failed + loadDependency DPushOver && Pushover "Failed: $orig_name" "Sendoption: $send_option + Size: $size MB + Path: $complete" + else + # Send notification if transfer succeeded + loadDependency DPushOver && Pushover "$orig_name" "Sendoption: $send_option + Size: $size MB + Time: $transferTime2 + Average speed: $SpeedAverage MB/s + Path: $complete" + fi + fi + + # Execute external command if specified and transfer did not fail + if [[ -n $exec_post ]] && [[ $failed != true ]]; then + if [[ $test_mode != "true" ]]; then + if [[ $allow_background == "true" ]]; then + # Execute command in background if allowed + echo "INFO: Executing external command (in background) - START" + echo " $exec_post" + eval $exec_post & + else + # Execute command and display output + echo "INFO: Executing external command - START:" + echo " $exec_post" + eval $exec_post | (while read; do echo " $REPLY"; done) + echo "INFO: Executing external command - ENDED" + fi + else + # Display the command that would be executed in test mode + echo -e "\e[00;31mTESTMODE: Would execute external command: \"$exec_post\"\e[00m" + fi + fi + + # Finalize transfer and display summary + ScriptEndTime=$(date +%s) + TotalTransferTime=$(( $ScriptEndTime - $ScriptStartTime )) + if [[ $failed == true ]]; then + # Display transfer failure message + echo -e "\e[00;37mINFO: \e[00;31mTransfer failed\e[00m" + else + # Display transfer success message + echo -e "\e[00;37mINFO: \e[00;32mTransfer finished\e[00m" + fi + echo " Name: $orig_name" + echo " Size: $size MB" + echo " Speed: $SpeedAverage MB/s" + echo " Transfer time: $transferTime2" + echo " Start time: $(date --date=@$ScriptStartTime '+%d/%m/%y-%a-%H:%M:%S')" + echo " End time: $(date --date=@$ScriptEndTime '+%d/%m/%y-%a-%H:%M:%S')" + echo " Total time: $(printf '%02dh:%02dm:%02ds' "$(($TotalTransferTime/(60*60)))" "$((($TotalTransferTime/60)%60))" "$(($TotalTransferTime%60))")" + + # Remove completed item from the queue + if [[ $failed != true ]]; then + queue remove + fi + + # Move to the next item in the queue + queue next +} + +function start_main { + # Parse command-line options + while [[ "$1" ]]; do + case "$1" in + --path=* ) filepath="${1#--path=}"; shift;; + --user=* ) user="${1#--user=}"; shift;; + --exec_post=* ) exec_post="${1#--exec_post=}"; shift;; + --exec_pre=* ) exec_pre="${1#--exec_pre=}"; shift;; + --delay=* ) delay="${1#--delay=}"; shift;; + --force | -f ) force=true; shift;; + --queue ) queue=true; shift;; + --source=* ) source="${1#--source=}"; shift;; + --sortto=* ) sortto="${1#--sortto=}"; shift;; + --test ) test_mode="true"; echo "INFO: Running in TESTMODE, no changes are made!"; shift;; + * ) break ;; + esac + done + + # Main program starts here + + # Confirm filepath + if [[ -z "$filepath" ]]; then + # If --path is not used, try to run from the queue + queue next + elif [[ -z "$(find "$filepath" -type d 2>/dev/null)" ]] && [[ -z "$(find "$filepath" -type f -print | head -n 1 2>/dev/null)" ]] || [[ -z "$(find "$filepath" -type f -print | head -n 1 2>/dev/null)" ]]; then + # Path with files or file not found + if [[ "$transferetype" == "downftp" || "$transferetype" == "fxp" ]]; then + # Server <-- Client: Assume path is OK, we will know for sure when size is found + true + elif [[ "$transferetype" == "upftp" ]]; then + # Server --> Client: Display error if path not found + echo -e "\e[00;31mERROR: Option --path is required with an existing path (with file(s)), or file does not exist:\n $filepath\n This cannot be transferred!\e[00m\n" + queue fail + queue next + else + echo -e "\e[00;31mERROR: Transfer-option \"$transferetype\" not recognized. Check your configuration (--user=$user --edit)!\e[00m\n" + exit 1 + fi + fi + + # Save transfer to queue and exit + if [[ $queue == true ]]; then + queue add end + fi + + # Create lockfile if not already running + if [[ "$lockfileRunning" != "true" ]]; then + lockfile + else + echo "INFO: Updating lockfile" + fi + + echo "INFO: Transfer-option: $transferetype" + + # Load dependencies + loadDependency DSetup + + # Load user config + loadConfig + + # Fix spaces in filepath + filepath="$(echo "$filepath" | sed 's/\\./ /g')" + + # Start main program + main "$filepath" +} \ No newline at end of file diff --git a/ftpauto.sh b/ftpauto.sh index c871c18..1781054 100644 --- a/ftpauto.sh +++ b/ftpauto.sh @@ -1,5 +1,5 @@ #!/bin/bash -s_version="0.5.8" +s_version="0.6.0" verbose="0" #0 Normal info | 1 debug console | 2 debug into logfile script="$(readlink -f "$0")" scriptdir=$(dirname $script) @@ -14,407 +14,461 @@ control_c() { trap control_c SIGINT function verbose { - #todo fix verbose in external scripts + # Function to control verbosity of script output + + # Check if 'quiet' flag is set if [[ $quiet ]]; then - #silent + # If 'quiet' flag is set, redirect all output to /dev/null exec > /dev/null 2>&1 + # Check if 'verbose' flag is set to 1 elif [[ ! $quiet ]] && [[ $verbose -eq 1 ]]; then + # If 'verbose' flag is set to 1, enable shell tracing and display starting message echo "STARTING PID=$BASHPID" set -x + # Check if 'verbose' flag is set to 2 elif [[ ! $quiet ]] && [[ $verbose -eq 2 ]]; then - #verbose + # If 'verbose' flag is set to 2, redirect stderr to debug logfile, enable shell tracing, and display starting message echo "INFO: Debugging. All input is redirected to logfile. Script is finished when console is idle again. Please wait!" exec 2>> "$scriptdir/run/$username.ftpauto.debug" echo "STARTING PID=$BASHPID" set -x + # Check if 'quiet' flag is set and 'verbose' flag is not set to 0 elif [[ $quiet ]] && [[ $verbose -ne 0 ]]; then + # If both 'quiet' and 'verbose' flags are set, display an error message and exit echo -e "\e[00;31mERROR: Verbose and silent can't be used at the same time\e[00m\n" exit 0 fi } + # load verbose verbose function updateChecker { -if [[ $(date +'%s') -gt $(echo $lastUpdate + 14*24*60*60 | bc) ]]; then - echo -n "INFO: Checking for update ..." - local release=$(curl --silent https://github.com/Meliox/FTPauto/releases | egrep -o 'FTPauto-v(.*)+tar.gz' | sort -n | tail -1) - release_version=${release#$"FTPauto-v"} - release_version=${release_version%.tar.gz} - if [[ "$release_version" == "$s_version" ]]; then - echo -e " \e[00;32m [Latest]\e[00m" - sed "7c message=\"\"" -i "$script" + # Check if it's time to perform an update check + if [[ $(date +'%s') -gt $((lastUpdate + 14*24*60*60)) ]]; then + echo -n "INFO: Checking for update ..." + # Retrieve the latest release version from GitHub + local release=$(curl --silent https://github.com/Meliox/FTPauto/releases | grep -o 'FTPauto-v[0-9.]*tar.gz' | sort -V | tail -1) + release_version=${release#"FTPauto-v"} + release_version=${release_version%.tar.gz} + # Compare the latest release version with the installed version + if [[ "$release_version" == "$s_version" ]]; then + # If the versions match, notify that it's the latest version + echo -e " \e[00;32m [Latest]\e[00m" + # Clear any previous update messages + sed "7c message=\"\"" -i "$script" + else + # Split version numbers into arrays for comparison + local IFS=. + local n1=($release_version) n2=($s_version) + for i in "${!n1[@]}"; do + if [[ "${n1[i]}" -gt "${n2[i]}" ]]; then + # If a newer version is found, notify and update the message + echo -e "\e[00;33m [$release_version available. Consider updating]\e[00m" + sed "7c message=\"INFO: New version ($release_version) available. Consider updating (execute bash install.sh update)\"" -i "$script" + break + elif [[ "${n1[i]}" -lt "${n2[i]}" ]]; then + # If the installed version is newer, notify it's the latest version and clear the message + echo -e " \e[00;32m [Latest]\e[00m" + sed "7c message=\"\"" -i "$script" + break + fi + done + fi + # Update the last update timestamp + sed "6c lastUpdate=$(date +'%s')" -i "$script" else - local IFS=. - local n1=($release_version) n2=($s_version) - for i in "${!n1[@]}"; do - if [[ "${n1[i]}" -gt "${n2[i]}" ]]; then - echo -e "\e[00;33m [$release_version available. Consider updating]\e[00m" - sed "7c message=\"INFO: New version ($release_version) available. Consider updating (execute bash install.sh update)\"" -i "$script" - break - elif [[ "${n1[i]}" -lt "${n2[i]}" ]]; then - echo -e " \e[00;32m [Latest]\e[00m" - sed "7c message=\"\"" -i "$script" - fi - done - fi - sed "6c lastUpdate=$(date +'%s')" -i "$script" -else - if [[ -n $message ]]; then - echo -e "\e[00;33m$message\e[00m" + # If it's not time for an update check, display any existing message + if [[ -n $message ]]; then + echo -e "\e[00;33m$message\e[00m" + fi fi -fi } +# Function to load dependencies based on input function loadDependency { - # common function to load all dependencies when needed - local LoadPath="" - case "$1" in - DConfig) LoadPath="$scriptdir/users/$username/config";; - DFtpList) LoadPath="$scriptdir/dependencies/ftp_list.sh";; - DFtpLogin) LoadPath="$scriptdir/dependencies/ftp_login.sh";; - DFtpMain) LoadPath="$scriptdir/dependencies/ftp_main.sh";; - DFtpOnlineTest) LoadPath="$scriptdir/dependencies/ftp_online_test.sh";; - DLargeFile) LoadPath="$scriptdir/plugins/largefile.sh";; - DFtpSizeManagement) LoadPath="$scriptdir/dependencies/ftp_size_management.sh";; - DHelp) LoadPath="$scriptdir/dependencies/help.sh";; - DPushOver) LoadPath="$scriptdir/plugins/pushover.sh";; - DSetup) LoadPath="$scriptdir/dependencies/setup.sh";; - DSort) LoadPath="$scriptdir/dependencies/sorting.sh";; - DVideoFile) LoadPath="$scriptdir/plugins/videofile.sh";; - esac - source "$LoadPath" + # Set the path for the dependency based on the input + local LoadPath="" + case "$1" in + DConfig) LoadPath="$scriptdir/users/$username/config";; + DServerList) LoadPath="$scriptdir/dependencies/server_list.sh";; + DServerLogin) LoadPath="$scriptdir/dependencies/server_login.sh";; + DServerMain) LoadPath="$scriptdir/dependencies/transfer_main.sh";; + DServerAliveTest) LoadPath="$scriptdir/dependencies/server_alive_test.sh";; + DLargeFile) LoadPath="$scriptdir/plugins/largefile.sh";; + DServerSizeManagement) LoadPath="$scriptdir/dependencies/server_size_management.sh";; + DHelp) LoadPath="$scriptdir/dependencies/help.sh";; + DPushOver) LoadPath="$scriptdir/plugins/pushover.sh";; + DSetup) LoadPath="$scriptdir/dependencies/setup.sh";; + DSort) LoadPath="$scriptdir/dependencies/sorting.sh";; + DVideoFile) LoadPath="$scriptdir/plugins/videofile.sh";; + esac + # Source the dependency + source "$LoadPath" } -function start_ftpmain { - # used to start ftpmain script and set proper debug level - loadDependency DFtpMain - if [[ $verbose -eq 1 ]]; then - start_main "${download_argument[@]}" - elif [[ $verbose -eq 2 ]]; then - start_main "${download_argument[@]}" >> "$ftpmaindebugfile" - else - # run in background or run normal - if [[ $background == "true" ]]; then - start_main "${download_argument[@]}" &> /dev/null & - else - start_main "${download_argument[@]}" - fi - fi +# Function to start the transfer main script with appropriate debug level +function start_transfermain { + # Load the transfer main dependency + loadDependency DServerMain + + # Check the verbose level and start the main script accordingly + if [[ $verbose -eq 1 ]]; then + start_main "${download_argument[@]}" + elif [[ $verbose -eq 2 ]]; then + start_main "${download_argument[@]}" >> "$maindebugfile" + else + # Run in background or run normally + if [[ $background == "true" ]]; then + start_main "${download_argument[@]}" &> /dev/null & + else + start_main "${download_argument[@]}" + fi + fi } +# Function to confirm the existence of files or locks function confirm { - case "$1" in - queue_file ) - if [[ ! -f "$queue_file" ]]; then - message "$2" "$3" - fi - ;; - lock_file ) - if [[ ! -f "$lockfile" ]]; then - message "$2" "3" - fi - ;; - esac + case "$1" in + queue_file ) + # Check if the queue file exists and display message if it doesn't + if [[ ! -f "$queue_file" ]]; then + message "$2" "$3" + fi + ;; + lock_file ) + # Check if the lock file exists and display message if it doesn't + if [[ ! -f "$lockfile" ]]; then + message "$2" "$3" + fi + ;; + esac } +# Function to display messages with different colors based on the severity level function message { - if [[ "$2" -eq "1" ]]; then - echo -e "\e[00;31m$(date '+%d/%m/%y-%a-%H:%M:%S'): $1\e[00m\n" - else - echo -e "\e[00;32m$(date '+%d/%m/%y-%a-%H:%M:%S'): $1\e[00m\n" - fi - exit "$2" + if [[ "$2" -eq "1" ]]; then + # Display message in red for error + echo -e "\e[00;31m$(date '+%d/%m/%y-%a-%H:%M:%S'): $1\e[00m\n" + else + # Display message in green for success + echo -e "\e[00;32m$(date '+%d/%m/%y-%a-%H:%M:%S'): $1\e[00m\n" + fi + # Exit with the provided exit code + exit "$2" } +# Function to load user configuration function load_user { - if [[ -z "$username" ]] && [[ -f "$scriptdir/users/default/config" ]]; then - username="default" - config_name="$scriptdir/users/default/config" - loadDependency DConfig - echo "INFO: User: $username" - elif [[ -n "$username" ]] && [[ -f "$scriptdir/users/$username/config" ]]; then - username="$username" - config_name="$scriptdir/users/$username/default" - loadDependency DConfig - echo "INFO: User: $username" - elif [[ $option == "add" ]]; then - # manually add user - if [[ -z "$username" ]]; then - username="default" - else - username="$username" - fi - else - # user used not found, want to create them - echo -e "\e[00;31mERROR: No user found. See --help for more info.\e[00m\n" - exit 1 - fi - # confirm that config is most recent version - if [[ $config_version -ne "6" ]] && [[ $option != "add" ]] && [[ $option != "edit" ]]; then - echo -e "\e[00;31mERROR: Please update it to version 6. See --help for more info!\e[00m\n" - exit 1 - fi + if [[ -z "$username" && -f "$scriptdir/users/default/config" ]]; then + # If username is not provided and default configuration exists, use default user + username="default" + config_name="$scriptdir/users/default/config" + loadDependency DConfig + echo "INFO: User: $username" + elif [[ -n "$username" && -f "$scriptdir/users/$username/config" ]]; then + # If username is provided and corresponding configuration exists, use provided user + username="$username" + config_name="$scriptdir/users/$username/default" + loadDependency DConfig + echo "INFO: User: $username" + elif [[ $option == "add" ]]; then + # Manually add user + if [[ -z "$username" ]]; then + username="default" + else + username="$username" + fi + else + # User used not found, prompt to create them + echo -e "\e[00;31mERROR: No user found. See --help for more info.\e[00m\n" + exit 1 + fi + # Confirm that config is most recent version + if [[ $config_version -ne "7" && $option != "add" && $option != "edit" ]]; then + echo -e "\e[00;31mERROR: Please update it to version 6. See --help for more info!\e[00m\n" + exit 1 + fi } +# Function to handle invalid arguments function invalid_arg { -echo -e "\e[00;31mERROR: Invalid input for argument '$@'. See --help for more info.\e[00m\n" -exit 1 + echo -e "\e[00;31mERROR: Invalid input for argument '$@'. See --help for more info.\e[00m\n" + exit 1 } +# Function to manage options function option_manage { -if [[ -z ${option[0]} ]]; then - option="$1" -else - echo -e "\e[00;31mERROR: An option, --${option[0]} is already used. One can only be used.\e[00m\n" - exit 1 -fi + if [[ -z ${option[0]} ]]; then + option="$1" + else + echo -e "\e[00;31mERROR: An option, --${option[0]} is already used. Only one can be used.\e[00m\n" + exit 1 + fi } +# Function to handle main operations function main { -safelock="true" -case "${option[0]}" in - "add" ) # add user - loadDependency DHelp; write_config - read -p " Do you want to configure that user now(y/n)? " - if [[ "$REPLY" == "y" ]]; then - nano "$scriptdir/users/$username/config" - else - echo "You can edit the user later, by editing \"$scriptdir/users/$username/config\"" - fi - # create the user's logfile - create_log_file - message "User=$username added." "0" - ;; - "edit" ) # edit user config - nano "$scriptdir/users/$username/config" - message "User=$username edited." "0" - ;; - "remove" ) # remove all userfiles generated files. Does not remove config - rm -rf "$scriptdir/run/$username" - rm "$scriptdir/users/$username/log" - message "Userfiles removed for $username." "0" - ;; - "purge" ) # remove all userfiles log files and config from /run and /user/$username/ - confirm lock_file "Can't remove $username while session is running." "1" - rm -rf "$scriptdir/users/$username" - rm -rf "$scriptdir/run/$username" - message "User=$username removed." "0" - ;; - "pause" ) # Stop transfer - safelock="false" - confirm lock_file "Error, lockfile couldn't be found. Nothing could be done!" "1" - cleanup stop - cleanup session - cleanup end - message "Session has been terminated." "0" - ;; - "stop" ) # Stop transfer and remove queue - safelock="false" - confirm lock_file "Error, lockfile couldn't be found. Nothing could be done!" "1" - cleanup stop - cleanup session - cleanup end - rm "$queue_file" - message "Session has been terminated." "0" - ;; - "start" ) # start session from queue file - safelock="false" - if [[ ! -e "$queue_file" ]]; then - message "Nothing in queue." "1" - fi - start_ftpmain - if [[ $background == "true" ]]; then - message "Session has started." "0" - elif [[ $? -eq 1 ]]; then - message "Succeeded." "0" - else - message "Failed." "1" - fi - ;; - "retry" ) # resets all failed downloads so that they can be retried - confirm queue_file "Empty queue!" "0" - mv "$queue_file" "${queue_file}.old" - while read line; do - id=$(echo $line | cut -d'|' -f1) - source=$(echo $line | cut -d'|' -f2) - path=$(echo $line | cut -d'|' -f3) - sort=$(echo $line | cut -d'|' -f4) - size=$(echo $line | cut -d'|' -f5) - failed=$(echo $line | cut -d'|' -f6) - time=$(echo $line | cut -d'|' -f7) - echo "$id|$source|$path|$sort|$size|false|$time" >> "$queue_file" - done < "${queue_file}.old" - rm "${queue_file}.old" - message "OK." "0" - ;; - "download" ) - safelock="false" - # set source - if [[ -z $source ]]; then - source="CONSOLE" - else - source="$source" - fi - # start download right away - if [[ ${option[1]} == "start" ]]; then - start_ftpmain - if [[ $background == "true" ]]; then - message "Session in background has started." "0" - elif [[ $? -eq 0 ]]; then - message "Succeeded." "0" - else - message "Failed." "1" - fi - # queue download - # TODO: Add options to queuefile as well - elif [[ ${option[1]} == "queue" ]]; then - start_ftpmain "${download_argument[@]}" --queue - fi - ;; - "list" ) # list content of queue file - confirm queue_file "Empty queue!" "0" - echo "ID|PATH|SORT TO|SIZE(MB)|FAILED STATUS|TIME" - while read line; do - id=$(echo $line | cut -d'|' -f1) - source=$(echo $line | cut -d'|' -f2) - path=$(echo $line | cut -d'|' -f3) - sort=$(echo $line | cut -d'|' -f4) - size=$(echo $line | cut -d'|' -f5) - failed=$(echo $line | cut -d'|' -f6) - time=$(echo $line | cut -d'|' -f7) - echo "$id|$source|$path|$sort|$size|$failed|$time" - done < "$queue_file" - message "List has been shown." "0" - ;; - "clear" ) # clear content of queue file - confirm queue_file "Error, queue could not be found." "1" - rm "$queue_file" - message "Queue removed." "0" - ;; - "forget" ) # remove item with from queue - confirm queue_file "Error, queuefile couldn't be found. Nothing could be removed!" "1" - if [[ -n "$id" ]] && [[ -n $(cat "$queue_file" | grep "^$id") ]]; then - #make sure id exists and is present in queue - echo "Removing id=$id" - sed "/^"$id"/d" -i "$queue_file" #ex -s -c '%s/^[0-9]*//|wq' file.txt if your ex is actually symlinked to the installed vim, then you can use \d and \+ - message "Id=$id removed from queue." "0" - else - message "No Id=$id selected/in queue." "1" - fi - ;; - "up" ) # Move item with 1 up in queue - confirm queue_file "Error, queuefile couldn't be found. Nothing could be moved!" "1" - if [[ -n "$id" ]] && [[ -n $(cat "$queue_file" | grep "^$id") ]]; then - line_info=$(cat "$queue_file" | grep "^$id") - line_number=$(cat "$queue_file" | grep -ne "^$id" | cut -d':' -f1) - previous_line_number=$(($line_number -1)) - if [[ "$line_number" -lt "2" ]]; then - #if id is the first, keep it there - message "Id, $id, is at top." "0" - else - sed "/^"$id"/d" -i "$queue_file" - sed "$previous_line_number i $line_info" -i "$queue_file" - message "Moved Id=$id, up." "0" - fi - else - message "No Id=$id, selected/in queue." "1" - fi - ;; - "down" ) # Move item with 1 down in queue - confirm queue_file "Error, queuefile couldn't be found. Nothing could be moved!" "1" - if [[ -n "$id" ]] && [[ -n $(cat "$queue_file" | grep "^$id") ]]; then - line_info=$(cat "$queue_file" | grep "^$id") - line_number=$(cat "$queue_file" | grep -ne "^$id" | cut -d':' -f1) - next_line_number=$(($line_number +1)) - last_line=$(cat "$queue_file" | grep -ne '' | cut -d':' -f1 | tail -n1 ) - if [[ $next_line_number -eq $last_line ]]; then - #add id to the end of file - sed "/^"$id"/d" -i "$queue_file" - echo $line_info >> "$queue_file" - message "Id=$id, is at the buttom." "0" - elif [[ $next_line_number -gt $last_line ]]; then - #if id is the last, do nothing - message ": Id=$id, is at the buttom." "1" - else - #any other cases - sed "/^"$id"/d" -i "$queue_file" - sed "$next_line_number i $line_info" -i "$queue_file" - message "$option: Moved Id=$id, down." "0" - fi - else - message "$option: No Id, $id, selected/in queue." "1" - fi - ;; - "online" ) # Perform server test - loadDependency DFtpLogin && ftp_login 1 - loadDependency DFtpOnlineTest && online_test - cleanup session - if [[ $is_online -eq 0 ]]; then - message "Server is OK" "0" - else - message "Server is NOT OK" "1" - fi - ;; - "freespace" ) # check free space - loadDependency DFtpLogin && ftp_login 1 - loadDependency DFtpSizeManagement && ftp_sizemanagement info - cleanup session - if [[ $is_online -eq 1 ]]; then - message "$option: Could " "1" - else - message "$option: Server is responding" "0" - fi - ;; - "progress" ) # write out download progress - confirm lock_file "Error, lockfile couldn't be found. Nothing is being transferred!" "1" - echo "INFO: Keeps updating every $sleeptime second. (Exit with \"x\")" - if [ -t 0 ]; then stty -echo -icanon time 0 min 0; fi - keypress="" - local count=0 - while [[ "x$keypress" == "x" ]]; do - local title="$(sed -n 5p < "$logfile" | cut -d',' -f1 | cut -d' ' -f2)" - local percentage="$(sed -n 5p < "$logfile" | cut -d',' -f2 | cut -d' ' -f2)" - percentage="${percentage%\%}" - local TimeDiff="$(sed -n 5p < "$logfile" | cut -d',' -f3 | cut -d' ' -f3)" - local Speed="$(sed -n 5p < "$logfile" | cut -d',' -f4 | cut -d' ' -f2)" - local SpeedAverage="$(sed -n 5p < "$logfile" | cut -d',' -f6 | cut -d' ' -f2)" - local etatime="$(sed -n 5p < "$logfile" | cut -d',' -f5 | cut -d' ' -f3)" - if [[ -z "$(sed -n 5p < "$logfile" | grep Transferring)" ]]; then - echo "INFO: Nothing is being transferred!" - break - fi - if [[ $count -gt 0 ]]; then - # we can start overwriting progresline - tput cuu 1; tput el1 + safelock="true" + case "${option[0]}" in + "add" ) # Add user + loadDependency DHelp + write_config + read -p " Do you want to configure that user now(y/n)? " + if [[ "$REPLY" == "y" ]]; then + nano "$scriptdir/users/$username/config" + else + echo "You can edit the user later, by editing \"$scriptdir/users/$username/config\"" + fi + create_log_file + message "User=$username added." "0" + ;; + "edit" ) # Edit user config + nano "$scriptdir/users/$username/config" + message "User=$username edited." "0" + ;; + "remove" ) # Remove all userfiles generated files. Does not remove config + rm -rf "$scriptdir/run/$username*" + rm "$scriptdir/users/$username/log" + message "Userfiles removed for $username." "0" + ;; + "purge" ) # Remove all userfiles log files and config from /run and /user/$username/ + rm -rf "$scriptdir/users/$username" + rm -rf "$scriptdir/run/$username*" + message "User=$username removed." "0" + ;; + "pause" ) # Stop transfer + safelock="false" + confirm lock_file "Error, lockfile couldn't be found. Nothing could be done!" "1" + cleanup stop + cleanup session + cleanup end + message "Session has been terminated." "0" + ;; + "stop" ) # Stop transfer and remove queue + safelock="false" + confirm lock_file "Error, lockfile couldn't be found. Nothing could be done!" "1" + cleanup stop + cleanup session + cleanup end + rm "$queue_file" + message "Session has been terminated." "0" + ;; + "start" ) # Start session from queue file + safelock="false" + if [[ ! -e "$queue_file" ]]; then + message "Nothing in queue." "1" + fi + start_transfermain + if [[ $background == "true" ]]; then + message "Session has started." "0" + elif [[ $? -eq 1 ]]; then + message "Succeeded." "0" + else + message "Failed." "1" + fi + ;; + "retry" ) # Resets all failed downloads so that they can be retried + confirm queue_file "Empty queue!" "0" + mv "$queue_file" "${queue_file}.old" + while read line; do + id=$(echo $line | cut -d'|' -f1) + source=$(echo $line | cut -d'|' -f2) + path=$(echo $line | cut -d'|' -f3) + sort=$(echo $line | cut -d'|' -f4) + size=$(echo $line | cut -d'|' -f5) + failed=$(echo $line | cut -d'|' -f6) + time=$(echo $line | cut -d'|' -f7) + echo "$id|$source|$path|$sort|$size|false|$time" >> "$queue_file" + done < "${queue_file}.old" + rm "${queue_file}.old" + message "OK." "0" + ;; + "download" ) # Download or queue download + safelock="false" + if [[ -z $source ]]; then + source="CONSOLE" + else + source="$source" + fi + if [[ ${option[1]} == "start" ]]; then + start_transfermain + if [[ $background == "true" ]]; then + message "Session in background has started." "0" + elif [[ $? -eq 0 ]]; then + message "Succeeded." "0" + else + message "Failed." "1" + fi + elif [[ ${option[1]} == "queue" ]]; then + start_transfermain "${download_argument[@]}" --queue + fi + ;; + "list" ) # List content of queue file + confirm queue_file "Empty queue!" "0" + echo "ID|PATH|SORT TO|SIZE(MB)|FAILED STATUS|TIME" + while read line; do + id=$(echo $line | cut -d'|' -f1) + source=$(echo $line | cut -d'|' -f2) + path=$(echo $line | cut -d'|' -f3) + sort=$(echo $line | cut -d'|' -f4) + size=$(echo $line | cut -d'|' -f5) + failed=$(echo $line | cut -d'|' -f6) + time=$(echo $line | cut -d'|' -f7) + echo "$id|$source|$path|$sort|$size|$failed|$time" + done < "$queue_file" + message "List has been shown." "0" + ;; + "clear" ) # Clear content of queue file + confirm queue_file "Error, queue could not be found." "1" + rm "$queue_file" + message "Queue removed." "0" + ;; + "forget" ) # Remove item with from queue + confirm queue_file "Error, queuefile couldn't be found. Nothing could be removed!" "1" + if [[ -n "$id" ]] && [[ -n $(cat "$queue_file" | grep "^$id") ]]; then + echo "Removing id=$id" + sed "/^"$id"/d" -i "$queue_file" + message "Id=$id removed from queue." "0" + else + message "No Id=$id selected/in queue." "1" + fi + ;; + "up" ) # Move item with 1 up in queue + confirm queue_file "Error, queuefile couldn't be found. Nothing could be moved!" "1" + if [[ -n "$id" ]] && [[ -n $(cat "$queue_file" | grep "^$id") ]]; then + line_info=$(cat "$queue_file" | grep "^$id") + line_number=$(cat "$queue_file" | grep -ne "^$id" | cut -d':' -f1) + previous_line_number=$(($line_number -1)) + if [[ "$line_number" -lt "2" ]]; then + message "Id, $id, is at top." "0" + else + sed "/^"$id"/d" -i "$queue_file" + sed "$previous_line_number i $line_info" -i "$queue_file" + message "Moved Id=$id, up." "0" + fi + else + message "No Id=$id, selected/in queue." "1" + fi + ;; + "down" ) # Move item with 1 down in queue + confirm queue_file "Error, queuefile couldn't be found. Nothing could be moved!" "1" + if [[ -n "$id" ]] && [[ -n $(cat "$queue_file" | grep "^$id") ]]; then + line_info=$(cat "$queue_file" | grep "^$id") + line_number=$(cat "$queue_file" | grep -ne "^$id" | cut -d':' -f1) + next_line_number=$(($line_number +1)) + last_line=$(cat "$queue_file" | grep -ne '' | cut -d':' -f1 | tail -n1 ) + if [[ $next_line_number -eq $last_line ]]; then + sed "/^"$id"/d" -i "$queue_file" + echo $line_info >> "$queue_file" + message "Id=$id, is at the buttom." "0" + elif [[ $next_line_number -gt $last_line ]]; then + message ": Id=$id, is at the buttom." "1" + else + sed "/^"$id"/d" -i "$queue_file" + sed "$next_line_number i $line_info" -i "$queue_file" + message "$option: Moved Id=$id, down." "0" + fi + else + message "$option: No Id, $id, selected/in queue." "1" + fi + ;; + "online" ) # Perform server test + loadDependency DServerAliveTest && online_test + if [[ $is_online -eq 0 ]]; then + message "Server is OK" "0" + else + message "Server is NOT OK" "1" + fi + ;; + "freespace" ) # Check free space + loadDependency DServerSizeManagement && server_sizemanagement info + if [[ $is_online -eq 1 ]]; then + message "$option: Could " "1" + else + message "$option: Server is responding" "0" + fi + ;; + "progress" ) # Write out download progress + # Confirm the existence of the lock file; if not found, display an error message + confirm lock_file "Error: Lockfile not found. Nothing is being transferred!" "1" + + echo "INFO: Transferring: $(sed -n 5p < "$logfile" | cut -d',' -f1 | cut -d' ' -f3)" + + # Display information about the update frequency + echo "INFO: Updates every $sleeptime second. (Exit with 'x')" + + # Disable echo and canonical mode for user input + if [ -t 0 ]; then + stty -echo -icanon time 0 min 0 fi - local cols=$(($(tput cols) - 2)) - local percentagebarlength=$(echo "scale=0; $percentage * $cols / 100" | bc) - local string="$(eval printf "=%.0s" '{1..'"$percentagebarlength"\})" - local string2="$(eval printf "\ %.0s" '{1..'"$(($cols - $percentagebarlength - 1))"\})" - if [[ $percentagebarlength -eq 0 ]]; then - printf "\r[$string2] (no transfere information yet) ($(date '+%H:%M:%S'))" - else - printf "\r[$string>$string2] $percentage%% ETA ${etatime}@${speed}MB/s. ${TransferredNewMB}MB@${SpeedAverage}MB/s(avg). ($(date '+%H:%M:%S'))" + + # Initialize keypress variable and count + keypress="" + local count=0 + + # Loop until 'x' key is pressed + while [[ "x$keypress" == "x" ]]; do + # Extract transfer progress information from the logfile + local title="$(sed -n 5p < "$logfile" | cut -d',' -f1 | cut -d' ' -f2)" + local percentage="$(sed -n 5p < "$logfile" | cut -d',' -f2 | cut -d' ' -f2)" + percentage="${percentage%\%}" + local TimeDiff="$(sed -n 5p < "$logfile" | cut -d',' -f3 | cut -d' ' -f3)" + local Speed="$(sed -n 5p < "$logfile" | cut -d',' -f4 | cut -d' ' -f2)" + local SpeedAverage="$(sed -n 5p < "$logfile" | cut -d',' -f6 | cut -d' ' -f2)" + local etatime="$(sed -n 5p < "$logfile" | cut -d',' -f5 | cut -d' ' -f3)" + + # Check if any transfer is in progress + if [[ -z "$(sed -n 5p < "$logfile" | grep Transferring)" ]]; then + echo "INFO: No ongoing transfer." + break + fi + + # Update progress bar + if [[ $count -gt 0 ]]; then + tput cuu 1 + tput el1 + fi + local cols=$(($(tput cols) - 2)) + local percentagebarlength=$(echo "scale=0; $percentage * $cols / 100" | bc) + local string="$(eval printf "=%.0s" '{1..'"$percentagebarlength"'}')" + local string2="$(eval printf "\ %.0s" '{1..'"$(($cols - $percentagebarlength - 1))"'}')" + if [[ $percentagebarlength -eq 0 ]]; then + printf "\r[$string2] (no transfer information yet) ($(date '+%H:%M:%S'))" + else + printf "\r[$string>$string2] $percentage%% ETA ${etatime}@${speed}MB/s. ${TransferredNewMB}MB@${SpeedAverage}MB/s(avg). ($(date '+%H:%M:%S'))" + fi + + # Increment count and wait for 1 second + let count++ + sleep 1 + + # Read user input (keypress) + read keypress + done + + # Restore terminal settings if applicable + if [ -t 0 ]; then + stty sane fi - let count++ - sleep 1 - read keypress - done - if [ -t 0 ]; then stty sane; fi - echo -e '\n' - message "Progress finished" "0" - ;; - "dir" ) # list content of ftpserver and download it - loadDependency DFtpList && ftp_list - message "Closing FTP filebrowser" "0" - ;; - * ) - message "No options selected." "1" - ;; -esac + + # Print newline character for clarity + echo -e '\n' + + # Display completion message + message "Progress finished" "0" + ;; + "dir" ) # List content of server and download it + loadDependency DServerList && remote_server_list + message "Closing filebrowser" "0" + ;; + * ) + message "No options selected." "1" + ;; + esac } -################################################### CODE BELOW ####################################################### + +################################################### CODE BELOW ######################################################## echo -e "\n\e[00;34mFTPauto script - $s_version\e[00m\n" @@ -442,7 +496,7 @@ while :; do --id=* ) id=${1#--id=}; shift;; --clear ) option_manage clear; shift;; # Options - --queue ) option[1]=queue; shift;; + --queue ) download_argument+=("--queue"); shift;; --delay ) if (($# > 1 )); then delay="$2"; download_argument+=("--delay=$delay"); else invalid_arg "$@"; fi; shift 2;; --delay=* ) delay=${1#--delay=}; download_argument+=("--delay=$delay"); shift;; --sort ) if (($# > 1 )); then sortto="$2"; download_argument+=("--sortto=$sortto"); else invalid_arg "$@"; fi; shift 2;; @@ -490,10 +544,10 @@ loadDependency DSetup setup # make sure user has a log file -if [ ! -e "$logfile" ] && [[ "${option[0]}" != add ]]; then +if [[ ! -e "$logfile" && "${option[0]}" != add ]]; then loadDependency DHelp; create_log_file fi # Execute the given option echo "INFO: Option(s): ${option[@]}" -main +main \ No newline at end of file diff --git a/install.sh b/install.sh index 3498d84..15c9963 100644 --- a/install.sh +++ b/install.sh @@ -1,21 +1,22 @@ #!/bin/bash -### + +# Get the directory of the script scriptdir=$(dirname "$(readlink -f "$0")") -# -### code below -# + +# Function to get the current version of FTPauto function getCurrentVersion { -if [[ -f "$scriptdir/ftpauto.sh" ]]; then - # local version - i_version=$(sed -n '2p' "$scriptdir/ftpauto.sh") - i_version=${i_version#$"s_version=\""} - i_version=${i_version%\"} -else - i_version=0 -fi + if [[ -f "$scriptdir/ftpauto.sh" ]]; then + # Get local version + i_version=$(sed -n '2p' "$scriptdir/ftpauto.sh") + i_version=${i_version#$"s_version=\""} + i_version=${i_version%\"} + else + i_version=0 + fi } -control_c() { - # run if user hits control-c + +# Function to handle Ctrl+C interruption +function control_c() { echo -ne '\n' echo -e " \e[00;31mUser hit CTRL+C, exiting...\e[00m" echo -e " Run install.sh uninstall to remove created files before trying again!\n" @@ -23,120 +24,151 @@ control_c() { } trap control_c SIGINT -#programs +# Function to compare versions function version_compare { - if [[ "$1" == "$2" ]]; then - # Uptodate - new_version="false" - else - local IFS=. - local n1=($1) n2=($2) - for i in "${!n1[@]}"; do - if [[ ${n1[i]} -gt ${n2[i]} ]]; then - new_version="true" - break - elif [[ ${n1[i]} -lt ${n2[i]} ]]; then - new_version="false" - fi - done - fi + if [[ "$1" == "$2" ]]; then + new_version="false" + else + local IFS=. + local n1=($1) n2=($2) + local len=${#n1[@]} + for ((i=0; i<$len; i++)); do + if [[ ${n1[i]:-0} -gt ${n2[i]:-0} ]]; then + new_version="true" + break + elif [[ ${n1[i]:-0} -lt ${n2[i]:-0} ]]; then + new_version="false" + break + fi + done + fi } +# Function to install/update lftp function lftp_update { - argument="$1" - if [[ -n $(builtin type -p lftp) ]]; then - # lftp is installed, compare installed version to newest version - local lftpversion=$(curl --silent http://lftp.tech/ftp/ | grep -Eo '>lftp.(.*).+tar.gz<' | tail -1) - lftpversion=${lftpversion#\>lftp\-} - lftpversion=${lftpversion%.tar.gz<} - # get current lftp version - local c_lftpversion=$(lftp --version | grep -Eo 'Version\ [0-9].[0-9].[0-9]' | cut -d' ' -f2) - version_compare "$lftpversion" "$c_lftpversion" - if [[ "$new_version" == "true" ]]; then - echo -e " [\e[00;33m$lftpversion available, current version $c_lftpversion\e[00m]" - read -p " Do you wish to update(y/n)? " - if [[ "$REPLY" == "y" ]]; then - echo -n "Removing old version ..." - sudo apt-get -y remove lftp &> /dev/null - # remove compiled version - sudo rm -rf "$scriptdir/dependencies/lftp*" - argument="install" - else - echo -e " lftp update ... [\e[00;33mSKIPPED\e[00m]" - fi - else - c_lftpversion=$(lftp --version | grep -Eo 'Version\ [0-9].[0-9].[0-9]' | cut -d' ' -f2) - echo -e " \e[00;32m [Latest - v$c_lftpversion]\e[00m" - fi - fi + argument="$1" - if [[ "$argument" == "install" ]]; then - sudo apt-get -y build-dep lftp &> /dev/null - sudo apt-get -y install gcc openssl build-essential automake readline-common libreadline-dev pkg-config ncurses-dev libssl-dev libncurses5-dev libreadline-dev zlib1g-dev &> /dev/null - # find latest lftp - cd "$scriptdir/dependencies" || exit - local lftpversion=$(curl --silent http://lftp.tech/ftp/ | grep -Eo '>lftp.(.*).+tar.gz' | tail -1) - lftpversion=${lftpversion#\>lftp\-} - lftpversion=${lftpversion%.tar.gz} - wget "http://lftp.tech/ftp/lftp-$lftpversion.tar.gz" &> /dev/null - tar -xzvf "lftp-$lftpversion.tar.gz" &> /dev/null - rm "$scriptdir/dependencies/lftp-$lftpversion.tar.gz" - cd "lftp-$lftpversion" && ./configure --with-openssl --silent &> /dev/null && make --silent &> /dev/null && sudo checkinstall -y &> /dev/null - echo -n " Checking for lftp ..." - if [[ -n $(builtin type -p lftp) ]]; then - echo -e " \e[00;32m [Latest - v$lftpversion]\e[00m" - else - echo -e "INFO: Could not install program using sudo.\nYou have to install \"lftp\" manually... Exiting\n"; exit 0 - fi - fi + # Check if lftp is installed + if [[ -n $(command -v lftp) ]]; then + # Get the latest version of lftp from GitHub + local lftpversion=$(curl -s https://github.com/lavv17/lftp/tags | grep -oP '(?<=tags/v)\d+\.\d+\.\d+' | sort -V | tail -n 1) + # Get the current version of lftp + local c_lftpversion=$(lftp --version | grep -Eo 'Version\ [0-9].[0-9].[0-9]' | cut -d' ' -f2) + # Compare versions + version_compare "$lftpversion" "$c_lftpversion" + + if [[ "$new_version" == "true" ]]; then + # Prompt user to update lftp + echo -e " [\e[00;33m$lftpversion available, current version $c_lftpversion\e[00m]" + read -p " Do you wish to update (y/n)? " + + if [[ "$REPLY" == "y" ]]; then + # Remove old version of lftp + echo -n "Removing old version ..." + sudo apt-get -y remove lftp &> /dev/null + sudo rm -rf "$scriptdir/dependencies/lftp*" + argument="install" + else + echo -e " lftp update ... [\e[00;33mSKIPPED\e[00m]" + fi + else + # Display that lftp is up to date + echo -e " \e[00;32m [Latest - v$c_lftpversion]\e[00m" + fi + fi + + # Install lftp if specified or if it's not installed + if [[ "$argument" == "install" ]]; then + # Install required dependencies + sudo apt-get -y build-dep lftp &> /dev/null + sudo apt-get -y install gcc openssl build-essential automake readline-common libreadline-dev pkg-config ncurses-dev libssl-dev libncurses5-dev libreadline-dev zlib1g-dev &> /dev/null + + # Download and install lftp + cd "$scriptdir/dependencies" || exit + local lftpversion=$(curl -s https://github.com/lavv17/lftp/tags | grep -oP '(?<=tags/v)\d+\.\d+\.\d+' | sort -V | tail -n 1) + wget "https://github.com/lavv17/lftp/archive/refs/tags/v$lftpversion.tar.gz" &> /dev/null + mv "$scriptdir/dependencies/v$lftpversion.tar.gz" "$scriptdir/dependencies/lftp-$lftpversion.tar.gz" + tar -xzvf "lftp-$lftpversion.tar.gz" &> /dev/null + rm "$scriptdir/dependencies/lftp-$lftpversion.tar.gz" + cd "lftp-$lftpversion" && ./configure --with-openssl --silent &> /dev/null && make --silent &> /dev/null && sudo checkinstall -y &> /dev/null + + # Check if lftp is installed + echo -n " Checking for lftp ..." + if [[ -n $(command -v lftp) ]]; then + echo -e " \e[00;32m [Latest - v$lftpversion]\e[00m" + else + echo -e "INFO: Could not install program using sudo.\nYou have to install \"lftp\" manually... Exiting\n" + exit 0 + fi + fi } +# Function to install/update lftp function install_lftp { - echo -n " Checking/updating lftp ..." - if [[ -z $(builtin type -p lftp) ]]; then - echo -e "lftp is not installed! lftp is required for FTPauto to work!" - read -p " Do you want to install it(y/n)? " - if [[ "$REPLY" == "y" ]]; then - read -p " Do you want latest version(needs to be compiled - SLOW - Any installed version will be removed) or the package from repo(y/n)? " - if [[ "$REPLY" == "y" ]]; then - lftp_update install - else - sudo apt-get -y install lftp &> /dev/null - if [[ $? -eq 1 ]]; then - echo "ERROR: Installation of lftp failed. Please install manually before continuing." - echo -e "... Exiting\n"; exit 1 - fi - fi - echo -n "Checking lftp .." - if [[ -z $(builtin type -p lftp) ]]; then - echo -e " \e[00;32m [OK]\e[00m" - fi - else - echo "FTPauto will not work without lftp. Exiting..." - echo -e " Run install.sh uninstall to remove created files\n" - exit 0 - fi - else - # lftp is installed, compare installed version to newest version - lftp_update - fi + echo -n " Checking/updating lftp ..." + + # Check if lftp is installed + if [[ -z $(command -v lftp) ]]; then + # Prompt user to install lftp if not installed + echo -e "lftp is not installed! It is required for FTPauto to work!" + read -p " Do you want to install it (y/n)? " + + if [[ "$REPLY" == "y" ]]; then + # Ask user for installation method + read -p " Do you want latest version (needs to be compiled - SLOW - Any installed version will be removed) or the package from the repo (y/n)? " + + if [[ "$REPLY" == "y" ]]; then + # Install lftp using latest version + lftp_update install + else + # Install lftp from repository + sudo apt-get -y install lftp &> /dev/null + + # Check if installation was successful + if [[ $? -eq 1 ]]; then + echo "ERROR: Installation of lftp failed. Please install manually before continuing." + echo -e "... Exiting\n" + exit 1 + fi + fi + + # Check if lftp is installed after installation + echo -n "Checking lftp .." + if [[ -z $(command -v lftp) ]]; then + echo -e " \e[00;32m [OK]\e[00m" + fi + else + # Inform user that FTPauto will not work without lftp and exit + echo "FTPauto will not work without lftp. Exiting..." + echo -e " Run install.sh uninstall to remove created files\n" + exit 0 + fi + else + # Update lftp if it is already installed + lftp_update + fi } +# Function to install/update rar2fs function rar2fs_update { argument="$1" + # Check if rar2fs is installed if [[ -n $(builtin type -p rar2fs) ]]; then - # rar2fs is installed, compare installed version to newest version - c_rar2fsversion=$(rar2fs --version 2> /dev/null | grep -Eo 'rar2fs\sv[0-9][0-9]?\.[0-9][0-9]?\.[0-9][0-9]?' | cut -d' ' -f2 | cut -d'v' -f2) - rar2fsversion=$(curl -s https://api.github.com/repos/hasse69/rar2fs/releases | grep browser_download_url | head -n 1 | cut -d '"' -f 4 | cut -d '/' -f8 | cut -d'v' -f2) + # Get the current version of rar2fs + local c_rar2fsversion=$(rar2fs --version 2> /dev/null | grep -Eo 'rar2fs\sv[0-9][0-9]?\.[0-9][0-9]?\.[0-9][0-9]?' | cut -d' ' -f2 | cut -d'v' -f2) + # Get the latest release version of rar2fs from GitHub + local rar2fsversion=$(curl -s https://api.github.com/repos/hasse69/rar2fs/releases | grep browser_download_url | head -n 1 | cut -d '"' -f 4 | cut -d '/' -f8 | cut -d'v' -f2) + # Compare the versions version_compare "$rar2fsversion" "$c_rar2fsversion" if [[ "$new_version" == "true" ]]; then + # Prompt user to update if a new version is available echo -e " [\e[00;33m$rar2fsversion available, current version $c_rar2fsversion\e[00m]" read -p " Do you wish to update(y/n)? " if [[ "$REPLY" == "y" ]]; then if [[ -n $(builtin type -p rar2fs) ]]; then + # Remove old version and proceed with installation echo -n "Removing old version ..." sudo apt-get -y remove rar2fs &> /dev/null - # remove compiled version sudo rm -rf "$scriptdir/dependencies/rar2fs*" argument="install" fi @@ -147,20 +179,18 @@ function rar2fs_update { echo -e " \e[00;32m [Latest -v$rar2fsversion]\e[00m" fi fi + # Install rar2fs if specified or if it's not installed if [[ "$argument" == "install" ]]; then - # install cd "$scriptdir/dependencies/" - # get latest stable release of rar2fs - var=$(curl -s https://api.github.com/repos/hasse69/rar2fs/releases | grep browser_download_url | head -n 1 | cut -d '"' -f 4) + # Download and install rar2fs + local var=$(curl -s https://api.github.com/repos/hasse69/rar2fs/releases | grep browser_download_url | head -n 1 | cut -d '"' -f 4) wget -q "$var" - name=$(basename "$var") + local name=$(basename "$var") tar zxf "$name" && rm "$name" name=${name::-7} cd "$name" - #get unrar dependency for rar2fs - wget -q http://www.rarlab.com/rar/unrarsrc-5.6.3.tar.gz && tar -zxf unrarsrc-5.6.3.tar.gz && rm unrarsrc-5.6.3.tar.gz + wget -q https://www.rarlab.com/rar/unrarsrc-6.2.12.tar.gz && tar -zxf unrarsrc-6.2.12.tar.gz && rm unrarsrc-6.2.12.tar.gz cd unrar && make lib &> /dev/null && sudo make install-lib &> /dev/null && cd .. - #compile rar2fs autoreconf -f -i &> /dev/null && ./configure --silent && make --silent && sudo checkinstall -y &> /dev/null echo -n " Checking for rar2fs ..." if [[ -n $(builtin type -p rar2fs) ]]; then @@ -172,49 +202,57 @@ function rar2fs_update { fi } +# Function to install/update rar2fs function install_rar2fs { echo -n " Checking/updating rar2fs ..." + # Check if rar2fs is installed if [[ -z $(builtin type -p rar2fs) ]]; then - echo -e "\n \"rar2fs\" is not installed! It is needed to mount rarfiles in order to send videofile only. The file(s) will be transferred in original format otherwise" + echo -e "\n \"rar2fs\" is not installed! It is needed to mount rarfiles in order to send video files only. The file(s) will be transferred in original format otherwise" read -p " Do you want to install it(needs to be compiled - SLOW)(y/n)? " if [[ "$REPLY" == "y" ]]; then + # Install required dependencies sudo apt-get -y install libfuse-dev autoconf &> /dev/null if [[ $? -eq 1 ]]; then - echo -e "INFO: Could not install program using sudo.\nYou have to install \"$i\" manually using root, typing \"su root\"; \"apt-get install $i\"\n... Exiting\n"; exit 0 + echo -e "INFO: Could not install program using sudo.\nYou have to install \"$i\" manually using root, typing \"su root\"; \"apt-get install $i\"\n... Exiting\n" + exit 0 else + # Proceed with rar2fs installation rar2fs_update install fi else echo -e "Checking rar2fs ... [\e[00;33mSKIPPED\e[00m] NOTE: \"video\" will not work as send option" fi + # If rar2fs is installed, check for updates elif [[ -n $(builtin type -p rar2fs) ]]; then - # rar2fs is installed, compare installed version to newest version rar2fs_update fi } +# Function to install dependencies function installDependencies { + # Create necessary directories echo "Installing dependency tools ..." - # create directories - echo -n " Creating directories ..." mkdir -p "$scriptdir/run" "$scriptdir/users" + echo -n " Creating directories ..." echo -e "\e[00;32m [OK]\e[00m" - echo -n " Checking dependencies files ..." - local main_files=("ftpauto.sh" "dependencies/ftp_online_test.sh" "dependencies/ftp_list.sh" "dependencies/ftp_size_management.sh" "dependencies/help.sh" "plugins/largefile.sh" "dependencies/setup.sh" "dependencies/sorting.sh") - # Confirm all files are present + + # Check for existence of main files + local main_files=("ftpauto.sh" "dependencies/transfer_main.sh" "dependencies/setup.sh" "dependencies/server_alive_test.sh" "dependencies/server_login.sh" "dependencies/server_list.sh" "dependencies/server_size_management.sh" "dependencies/help.sh" "plugins/largefile.sh" "dependencies/setup.sh" "dependencies/sorting.sh") for i in "${main_files[@]}"; do if [[ ! -f "$scriptdir/$i" ]]; then - echo -e "\n\e[00;31m$i not found.\nScript will not work without... Try reinstalling it. Exiting...\e[00m\n"; exit 1; + echo -e "\n\e[00;31m$i not found.\nScript will not work without... Try reinstalling it. Exiting...\e[00m\n" + exit 1 fi done + echo -n " Checking files ..." echo -e "\e[00;32m [OK]\e[00m" - # lftp + # Install lftp install_lftp + # Install optional tools echo "Installing optional tools ..." - # split files - programs=( "rar" "cksfv" ) + local programs=("rar" "cksfv") for i in "${programs[@]}"; do echo -n " Checking $i ..." if [[ -z $(builtin type -p "$i") ]]; then @@ -237,14 +275,15 @@ function installDependencies { echo -e "\e[00;32m [OK]\e[00m" fi done - # for mounting to work + + # Install rar2fs install_rar2fs + # Prompt for user installation echo "Finalizing ..." - # Install default user read -p " Do you want to install a user? (You can add user later on)(y/n)? " if [[ "$REPLY" == "y" ]]; then - read -p " What username do you want to use (leave empty for "default")? " + read -p " What username do you want to use (leave empty for 'default')? " if [[ -n "$REPLY" ]]; then username="$REPLY" else @@ -271,79 +310,102 @@ function installDependencies { exit 0 } +# Function to handle installation function install { - read -p " Do you wish to install(y/n)? " + # Prompt the user to confirm installation + read -p "Do you wish to install? (y/n): " if [[ "$REPLY" == "n" ]]; then - echo -e "... Exiting\n"; exit 0 + echo -e "... Exiting\n" + exit 0 fi - # Install mandatory things that is needed for rest to work + # Install minimum required tools echo -e "\nInstalling minimum required tools ..." - programs=( "bc" "curl" "openssl" "date" "tail" "awk" "mkdir" "sed" "tput" "nano" "cut" "checkinstall") + local programs=( "bc" "curl" "openssl" "date" "tail" "awk" "mkdir" "sed" "tput" "nano" "cut" "checkinstall") for i in "${programs[@]}"; do - echo -n " Checking for $i ..." + echo -n "Checking for $i ..." if [[ -z $(builtin type -p "$i") ]]; then - echo -e "\e[00;31m[Not found]\e[00m" + echo -e "[Not found]" sudo apt-get -y install "$i" &> /dev/null if [[ $? -eq 1 ]]; then - echo -e "INFO: Could not install program using sudo.\nYou have to install \"$i\"... Exiting\n"; exit 0 + echo -e "INFO: Could not install program using sudo.\nYou have to install \"$i\"... Exiting\n" + exit 0 fi echo -n " Checking $i ..." if [[ -n $(builtin type -p "$i") ]]; then - echo -e "\e[00;32m [OK]\e[00m" + echo -e "[OK]" else - echo -e "\e[00;31mScript will not work without... exiting\e[00m\n" + echo -e "Script will not work without... exiting\n" exit 0 fi else - echo -e "\e[00;32m [OK]\e[00m" + echo -e "[OK]" fi done - # ok we know have the required tools to update script + + # Update the script to the latest version updateScript installNew - # continue part2 of the installation + + # Install dependencies installDependencies } +# Function to uninstall FTPauto and its dependencies function uninstall { + # List of programs to be removed local programs=("lftp" "rar" "cksfv" "rar2fs" "pkg-config" "automake" "libfuse-dev" "checkinstall" "libssl-dev" "libncurses5-dev" "libreadline-dev" "zlib1g-dev" "autoconf") - echo "The following will be removed: ${programs[@]}" - read -p " Do you want to remove all or one by one(all=y/one-by-one(safe)=n)? " + + # Prompt the user to confirm uninstallation method + echo "The following programs will be removed: ${programs[@]}" + read -p "Do you want to remove all at once (y) or one by one (n)? (all=y/one-by-one(safe)=n): " + + # If user chooses to remove all at once if [[ "$REPLY" == "y" ]]; then - echo -e "\n\e[00;31mWARNING: THIS WILL REMOVE ALL PROGRAMS, SOME MIGHT ALSO BE NEEDED BY OTHER PROGRAMS ON YOUR SYSTEM :WARNING.\e[00m\n" - read -p " ARE YOU SURE?(y/n)? " + # Confirm the user's intention to remove all programs + read -p "WARNING: THIS WILL REMOVE ALL PROGRAMS. ARE YOU SURE? (y/n): " if [[ "$REPLY" == "y" ]]; then + # Remove all programs for i in "${programs[@]}"; do sudo apt-get -y remove "$i" &> /dev/null - echo -e "$i \e[00;32m[REMOVED]\e[00m" + echo -e "$i [REMOVED]" done fi else + # If user chooses to remove one by one for i in "${programs[@]}"; do + # Check if program exists before attempting to remove it if builtin type -p "$i" &>/dev/null; then echo -n "Removing $i ..." - read -p " Do you want to remove it(y/n)? " + # Prompt the user to confirm removal of each program + read -p "Do you want to remove it? (y/n): " if [[ "$REPLY" == "y" ]]; then sudo apt-get -y remove "$i" &> /dev/null - echo -e "\e[00;32m [REMOVED]\e[00m" + echo -e "[REMOVED]" else - echo -e "\e[00;33m [KEPT]\e[00m" + echo -e "[KEPT]" fi fi done fi - echo -n "Removing sourcefiles ..." + + # Remove FTPauto and its dependencies + echo -n "Removing source files ..." sudo rm -rf "$scriptdir/dependencies/" "$scriptdir/plugins/" rm -f "$scriptdir/ftpauto.sh" "$scriptdir/LICENCE" "$scriptdir/README.md" "$scriptdir/ftpauto.sh" - echo -e "\e[00;32m [OK]\e[00m" - echo -n "Removing userfiles ..." + echo -e "[OK]" + + echo -n "Removing user files ..." rm -rf "$scriptdir/run" "$scriptdir/users" + echo -n "Removing remainder of FTPauto ..." rm -rf "$scriptdir" - echo -e "\e[00;32m [OK]\e[00m\nComplete removal of FTPauto and dependencies complete!\n" + + # Inform user of successful removal + echo -e "[OK]\nComplete removal of FTPauto and dependencies!\n" exit 0 } +# Function to handle downloading the script function downloadScript { echo -n " Downloading FTPauto..." wget -q "https://github.com/Meliox/FTPauto/archive/FTPauto-v$release_version.tar.gz" @@ -358,56 +420,53 @@ function downloadScript { exit 0; } +# Function to handle updating the script function updateScript { argument="$1" getCurrentVersion - # get most recent stable version from git echo -n " Checking/updating FTPauto ..." - local release=$(curl --silent https://github.com/Meliox/FTPauto/releases | grep -Eo 'FTPauto-v(.*)+tar.gz' | sort -n | tail -1) - release_version=${release#$"FTPauto-v"} - release_version=${release_version%.tar.gz} + local release_version=$(curl -s https://api.github.com/repos/Meliox/ftpauto/tags | grep -oP '"name": "\KFTPauto-v\d+\.\d+\.\d+' | sort -V | tail -n 1 | cut -d'v' -f2) version_compare "$release_version" "$i_version" - # compare to local version if [[ "$i_version" == "0" ]] && [[ $argument != installNew ]]; then echo -e "\e[00;31m [ERROR]\e[00m\nNo installation found. Execute script with install as argument instead. Exiting.\n" exit 0 elif [[ "$new_version" == "true" ]] && [[ $argument == installNew ]]; then echo -e "\e[00;32m [Found - v$release_version]\e[00m" - downloadScript - elif [[ "$new_version" == "true" ]]; then - echo -e "\e[00;33m [$release_version available]\e[00m" - read -p " Do you want to update your version(y/n)? " + read -p " Do you wish to update(y/n)? " + if [[ "$REPLY" == "y" ]]; then + downloadScript + else + echo -e " FTPauto update ... [\e[00;33mSKIPPED\e[00m]" + fi + elif [[ "$new_version" == "true" ]] && [[ $argument != installNew ]]; then + echo -e "\e[00;33m [Found - v$release_version]\e[00m" + read -p " Do you wish to update(y/n)? " if [[ "$REPLY" == "y" ]]; then downloadScript else - echo -e "\e[00;33m [Local version kept]\e[00m" + echo -e " FTPauto update ... [\e[00;33mSKIPPED\e[00m]" fi else - echo -e "\e[00;32m [Latest - v$i_version]\e[00m" - sed "6c lastUpdate=$(date +'%s')" -i "$scriptdir/ftpauto.sh" # set update time - sed "7c message=\"\"" -i "$scriptdir/ftpauto.sh" # reset update message + if [[ "$argument" == "installNew" ]]; then + echo -e " \e[00;32m[Local - v$i_version, Latest - v$release_version]\e[00m" + else + echo -e " \e[00;32m [Local - v$i_version, Latest - v$i_version]\e[00m" + fi fi } -function update { - updateScript - install_rar2fs - install_lftp - echo -e "\n\e[00;32m[Update done]\e[00m\nEnjoy! Start using FTPauto by using ftpauto.sh --help\n" - exit 0 -} - -function startupmessage { - echo -e "\nAutoinstaller for FTPauto\n" -} - -getCurrentVersion case "$1" in - uninstall) startupmessage; uninstall;; - install) startupmessage; install;; - update) startupmessage; update;; + installNew) + updateScript installNew + ;; + install) + install + ;; + uninstall) + uninstall + ;; *) - startupmessage; echo -e "\nUsage: $0 (install | uninstall | update)\nExecute uninstall first to clean up everything if you encounter any problems.\n"; + echo "Usage: $0 {install|uninstall}" >&2 exit 1 ;; -esac +esac \ No newline at end of file diff --git a/plugins/largefile.sh b/plugins/largefile.sh index 1f7a2b3..289010e 100644 --- a/plugins/largefile.sh +++ b/plugins/largefile.sh @@ -1,24 +1,39 @@ #!/bin/bash function largefile { + # Function to split large files and create SFV checkfiles + # Splitting process - echo "INFO: Splitting files into volumes $tempdir" - sed "5s#.*#*************************** Transferring: ${orig_name} - Rar splitting, ${splitsize} MB pieces, in process#" -i $logfile - if [[ -f "$filepath" ]]; then - tempdir="$scriptdir/run/$username-temp/${orig_name%.*}/" - mkdir -p "$tempdir" - rar a -r -"v${splitsize}M" -vn -m0 "${tempdir}${orig_name%.*}.rar" "$filepath" &> /dev/null - elif [[ -d "$filepath" ]]; then - tempdir="$scriptdir/run/$username-temp/${orig_name}" - mkdir -p "$tempdir" - rar a -r -v"${splitsize}M" -vn -m0 "${tempdir}/${orig_name}.rar" "$filepath" &> /dev/null - fi - transfer_path="$tempdir" # update transfer path - echo "INFO: Splitting into volumes done" - # sfv process + echo "INFO: Splitting files into volumes $tempdir" + + # Update log file with information about the current process + sed "5s#.*#*************************** Transferring: ${orig_name} - Rar splitting, ${splitsize} MB pieces, in process#" -i $logfile + + # Check if the file path exists + if [[ -f "$filepath" ]]; then + # If it's a file, create a temporary directory and split the file into volumes + tempdir="$scriptdir/run/$username-temp/${orig_name%.*}/" + mkdir -p "$tempdir" + rar a -r -"v${splitsize}M" -vn -m0 "${tempdir}${orig_name%.*}.rar" "$filepath" &> /dev/null + elif [[ -d "$filepath" ]]; then + # If it's a directory, create a temporary directory and split the directory into volumes + tempdir="$scriptdir/run/$username-temp/${orig_name}" + mkdir -p "$tempdir" + rar a -r -v"${splitsize}M" -vn -m0 "${tempdir}/${orig_name}.rar" "$filepath" &> /dev/null + fi + + # Update transfer path + transfer_path="$tempdir" + + echo "INFO: Splitting into volumes done" + + # SFV process if [[ "$create_sfv" == "true" ]]; then echo "INFO: Creating sfv checkfile" + # Update log file with information about the current process sed "5s#.*#*************************** Transferring: ${orig_name} - Creating sfv #" -i $logfile + + # Create SFV checkfile if [[ -f "$filepath" ]]; then cksfv -b "$tempdir"* > "${tempdir}/${orig_name%.*}.sfv" echo "INFO: ${orig_name%.*}.sfv created" diff --git a/plugins/pushover.sh b/plugins/pushover.sh index 94c23e2..92ff94b 100644 --- a/plugins/pushover.sh +++ b/plugins/pushover.sh @@ -1,24 +1,39 @@ #!/bin/bash -#Shell-script wrapper around curl for sending messages through PushOver -#For more info, see https://pushover.net/ + +# Shell-script wrapper around curl for sending messages through PushOver +# For more info, see https://pushover.net/ function Pushover { -version="1.2" -push_title="$1" -push_message="$2" -#send message through curl -CURL="$(which curl)" -PUSHOVER_URL="https://api.pushover.net/1/messages" -if [[ -n "$push_token" ]] && [[ -n "$push_user" ]] && [[ -n "$push_message" ]]; then - echo "INFO: Sending push-notification" - curl_cmd="\"${CURL}\" -s \ - -F \"token=${push_token}\" \ - -F \"user=${push_user}\" \ - -F \"title=${push_title}\" \ - -F \"message=${push_message}\" \ - ${PUSHOVER_URL} 2>&1 >/dev/null || echo \"$0: Failed to send message\" >&2" - eval "${curl_cmd}" - echo "INFO: Push-notification sent" -else - echo -e "\e[00;31mERROR: All settings are not set.\e[00m" -fi + # Function to send a message through PushOver + + # Version of the script + version="1.2" + + # Extract parameters + push_title="$1" + push_message="$2" + + # Find the location of curl + CURL="$(which curl)" + + # PushOver API URL + PUSHOVER_URL="https://api.pushover.net/1/messages" + + # Check if all necessary parameters are provided + if [[ -n "$push_token" ]] && [[ -n "$push_user" ]] && [[ -n "$push_message" ]]; then + # If all parameters are provided, send the push notification + echo "INFO: Sending push-notification" + # Construct the curl command + curl_cmd="\"${CURL}\" -s \ + -F \"token=${push_token}\" \ + -F \"user=${push_user}\" \ + -F \"title=${push_title}\" \ + -F \"message=${push_message}\" \ + ${PUSHOVER_URL} 2>&1 >/dev/null || echo \"$0: Failed to send message\" >&2" + # Execute the curl command + eval "${curl_cmd}" + echo "INFO: Push-notification sent" + else + # If any of the parameters are missing, display an error message + echo -e "\e[00;31mERROR: All settings are not set.\e[00m" + fi } diff --git a/plugins/videofile.sh b/plugins/videofile.sh index bd17f11..b439dd9 100644 --- a/plugins/videofile.sh +++ b/plugins/videofile.sh @@ -1,27 +1,35 @@ #!/bin/bash + function videoFile { + # Function to handle video files + local found_files found_file_size_total found_file_size found_file_percentage + + # Check if the file is a single file and not a RAR archive if [[ -f "$filepath" ]] && ! [[ "$filepath" =~ (\.(rar)$) ]]; then - # a single file is passed, and not rar, so no need to evaluate that + # Single file passed, not compressed, continue echo "INFO: Single file passed (not compressed). Continuing..." elif [[ -f "$filepath" ]] && [[ "$filepath" =~ (\.(rar)$) ]]; then - # a single rar file is passed investigate using rar mount - echo "INFO: A rar file is passed. Looking inside rarfile for videofile(s)..." + # A single RAR file is passed, investigate using rar mount + echo "INFO: A RAR file is passed. Looking inside RAR file for video file(s)..." echo -e "\e[00;33mINFO: Size calculations may be incorrect.\e[00m" mountsystem mount elif [[ -d "$filepath" ]]; then - # search and calculate total size of all video files found in found_file to get what percentage of transfer consists of videofiles + # Search and calculate the total size of all video files found to get the percentage of transfer that consists of video files found_files=() + + # Check for exclusions if [[ "${#exclude_array[@]}" -gt 0 ]] && [[ -n "${exclude_array[@]}" ]]; then while IFS= read -r -d $'\0'; do found_files+=("$REPLY") done < <(find "$filepath" \( -type f \) -and \( -name "*.avi" -or -name "*.mkv" -or -name "*.img" -or -name "*.iso" -or -name "*.mp4" \) -and \! \( $exclude_expression \) -print0) else - while IFS= read -r -d $'\0'; do - found_files+=("$REPLY") - done < <(find "$filepath" \( -type f \) -and \( -name "*.avi" -or -name "*.mkv" -or -name "*.img" -or -name "*.iso" -or -name "*.mp4" \) -print0) + while IFS= read -r -d $'\0'; do + found_files+=("$REPLY") + done < <(find "$filepath" \( -type f \) -and \( -name "*.avi" -or -name "*.mkv" -or -name "*.img" -or -name "*.iso" -or -name "*.mp4" \) -print0) fi - # for found video files, evaluate their total size and if nothing is found try rar mount + + # For found video files, evaluate their total size and if nothing is found try rar mount if [[ "${#found_files[@]}" -gt 0 ]]; then echo "INFO: ${#found_files[@]} video file(s) found:" found_file_size_total="0" @@ -30,136 +38,148 @@ function videoFile { found_file_size_total=$(echo "$found_file_size_total + $found_file_size" | bc) echo " $(basename "$n") $(echo "scale=2; $found_file_size / (1024*1024)" | bc)MB" done - echo "INFO: Total size found: $(echo "scale=2; $found_file_size_total / (1024*1024)" | bc)MB. " + echo "INFO: Total size found: $(echo "scale=2; $found_file_size_total / (1024*1024)" | bc)MB." found_file_percentage=$(echo "scale=3; $found_file_size_total / $directorysize * 100" | bc | cut -d'.' -f1) if [[ $found_file_percentage -gt 80 ]]; then echo "INFO: File(s) found is ${found_file_percentage}% > 80%. Everything OK" - # update path and correct filename(s) + + # Update path and correct filename(s) if [[ "${#found_files[@]}" -eq 1 ]] && [[ -d "$filepath" ]]; then - # one file is found in a directory - # rename file to top directory + # One file is found in a directory + # Rename file to top directory mkdir -p "$tempdir$orig_name" ln -s "${found_files[0]}" "$tempdir$orig_name/$(basename "$(dirname "${found_files[0]}")").${found_files[0]##*.}" elif [[ "${#found_files[@]}" -gt 1 ]]; then - # multiple files found, rename those to top directory + # Multiple files found, rename those to top directory for n in "${found_files[@]}"; do mkdir -p "$tempdir$orig_name" ln -s "$n" "$tempdir$orig_name/$(basename "$n")" done fi - transfer_path="$tempdir$orig_name" # update transfer path - #Update filesize to be transferred + + transfer_path="$tempdir$orig_name" # Update transfer path + + # Update filesize to be transferred get_size "$transfer_path" else - echo "INFO: No large videofile(s) found (> 80% of total size). Looking for rarfiles containing videofiles..." + echo "INFO: No large video file(s) found (> 80% of total size). Looking for RAR files containing video files..." mountsystem mount fi else - echo "INFO: No videofile(s) found. Looking for rarfiles containing videofiles..." + echo "INFO: No video file(s) found. Looking for RAR files containing video files..." mountsystem mount fi fi } function mountsystem { + # Function to handle mounting and unmounting RAR files + if [[ -n "$rar2fs" ]]; then case "$1" in - "mount" ) - # search filepath for rarfile(s) - local rarset found_files found_file_size_total found_file_size found_file_percentage - rarset=() - if [[ "${#exclude_array[@]}" -gt 0 ]] && [[ -n "${exclude_array[@]}" ]]; then - while IFS= read -r -d $'\0'; do - rarset+=("$REPLY") - done < <(find "$filepath" \! \( $exclude_expression \) -and \( -name '*.rar' \) -print0 | sort -z) - else - while IFS= read -r -d $'\0'; do - rarset+=("$REPLY") - done < <(find "$filepath" \( -name '*.rar' \) -print0 | sort -z) - fi - if [[ "${#rarset[@]}" -gt 0 ]]; then - echo "INFO: ${#rarset[@]} rarfile(s) found:" - for i in "${rarset[@]}"; do - echo " $(basename $i)" - done - # mount filepath with rar2fs in tempdir - mkdir -p "$tempdir$orig_name-rarmount" - "$rar2fs" "$filepath" "$tempdir$orig_name-rarmount" --seek-length=2 &> /dev/null - # used to unmount - mount_in_use="true" - # search tempdir for videofiles - found_files=() + "mount" ) + # Search filepath for RAR file(s) + local rarset found_files found_file_size_total found_file_size found_file_percentage + rarset=() if [[ "${#exclude_array[@]}" -gt 0 ]] && [[ -n "${exclude_array[@]}" ]]; then while IFS= read -r -d $'\0'; do - found_files+=("$REPLY") - done < <(find "$tempdir$orig_name-rarmount" \( -type f \) -and \( -name "*.avi" -or -name "*.mkv" -or -name "*.img" -or -name "*.iso" -or -name "*.mp4" \) -and \! \( $exclude_expression \) -print0) + rarset+=("$REPLY") + done < <(find "$filepath" \! \( $exclude_expression \) -and \( -name '*.rar' \) -print0 | sort -z) else while IFS= read -r -d $'\0'; do - found_files+=("$REPLY") - done < <(find "$tempdir$orig_name-rarmount" \( -type f \) -and \( -name "*.avi" -or -name "*.mkv" -or -name "*.img" -or -name "*.iso" -or -name "*.mp4" \) -print0) + rarset+=("$REPLY") + done < <(find "$filepath" \( -name '*.rar' \) -print0 | sort -z) fi - # verify that they make up 80% of everything - if [[ "${#found_files[@]}" -gt 0 ]]; then - echo "INFO: ${#found_files[@]} video file(s) found:" - found_file_size_total="0" - for n in "${found_files[@]}"; do - found_file_size=$(du -bs "$n" | awk '{print $1}') - found_file_size_total=$(echo "$found_file_size_total + $found_file_size" | bc) - echo " $(basename "$n") $(echo "scale=2; $found_file_size / (1024*1024)" | bc)MB" + if [[ "${#rarset[@]}" -gt 0 ]]; then + echo "INFO: ${#rarset[@]} RAR file(s) found:" + for i in "${rarset[@]}"; do + echo " $(basename $i)" done - found_file_percentage=$(echo "scale=3; $found_file_size_total / $directorysize * 100" | bc | cut -d'.' -f1) - if [[ $found_file_percentage -gt 80 ]]; then - # videofiles were succesfully found - echo "INFO: File(s) found is ${found_file_percentage}% > 80%. Everything OK" - # All videos files are not correctly named, so that needs to be fixed. - # A new directory is created and all files are symlinked with correct name + + # Mount filepath with rar2fs in tempdir + mkdir -p "$tempdir$orig_name-rarmount" + "$rar2fs" "$filepath" "$tempdir$orig_name-rarmount" --seek-length=2 &> /dev/null + + # Used to unmount + mount_in_use="true" + + # Search tempdir for video files + found_files=() + if [[ "${#exclude_array[@]}" -gt 0 ]] && [[ -n "${exclude_array[@]}" ]]; then + while IFS= read -r -d $'\0'; do + found_files+=("$REPLY") + done < <(find "$tempdir$orig_name-rarmount" \( -type f \) -and \( -name "*.avi" -or -name "*.mkv" -or -name "*.img" -or -name "*.iso" -or -name "*.mp4" \) -and \! \( $exclude_expression \) -print0) + else + while IFS= read -r -d $'\0'; do + found_files+=("$REPLY") + done < <(find "$tempdir$orig_name-rarmount" \( -type f \) -and \( -name "*.avi" -or -name "*.mkv" -or -name "*.img" -or -name "*.iso" -or -name "*.mp4" \) -print0) + fi + + # Verify that they make up 80% of everything + if [[ "${#found_files[@]}" -gt 0 ]]; then + echo "INFO: ${#found_files[@]} video file(s) found:" + found_file_size_total="0" for n in "${found_files[@]}"; do - mkdir -p "$tempdir$orig_name/" - directoryname="$(basename "$(dirname "$n")")" - directoryname="${directoryname%-rarmount}" - ln -s "$n" "$tempdir$orig_name/$directoryname.${n##*.}" + found_file_size=$(du -bs "$n" | awk '{print $1}') + found_file_size_total=$(echo "$found_file_size_total + $found_file_size" | bc) + echo " $(basename "$n") $(echo "scale=2; $found_file_size / (1024*1024)" | bc)MB" done - transfer_path="$tempdir$orig_name" # update transfer path - get_size "$transfer_path" + found_file_percentage=$(echo "scale=3; $found_file_size_total / $directorysize * 100" | bc | cut -d'.' -f1) + if [[ $found_file_percentage -gt 80 ]]; then + # Video files were successfully found + echo "INFO: File(s) found is ${found_file_percentage}% > 80%. Everything OK" + + # All video files are not correctly named, so that needs to be fixed. + # A new directory is created and all files are symlinked with correct name + for n in "${found_files[@]}"; do + mkdir -p "$tempdir$orig_name/" + directoryname="$(basename "$(dirname "$n")")" + directoryname="${directoryname%-rarmount}" + ln -s "$n" "$tempdir$orig_name/$directoryname.${n##*.}" + done + + transfer_path="$tempdir$orig_name" # Update transfer path + + get_size "$transfer_path" + else + echo -e "\e[00;33mINFO: No large video file(s) found (> 80% of total size).\e[00m" + fi else - echo -e "\e[00;33mINFO: No large videofile(s) found (> 80% of total size).\e[00m" + echo -e "\e[00;33mINFO: No video file(s) found inside RAR file(s). File(s) will be transferred instead.\e[00m" fi else - echo -e "\e[00;33mINFO: No videofile(s) found inside rarfile(s). File(s) will be transferred instead.\e[00m" + echo -e "\e[00;33mINFO: No RAR file(s) are found in path.\e[00m" fi - else - echo -e "\e[00;33mINFO: No rarfile(s) are found in path.\e[00m" - fi - if [[ $mount_in_use != "true" ]]; then - # system was unsuccessful in finding video files using mount, so it will revert to default transfer instead - mountsystem umount - echo "INFO: Ignoring send_option=video. Path will be transferred normally." - send_option="default" - echo "INFO: Sendoption: $send_option" - fi - ;; - "umount" ) - # attempt and unmount directory - local i - i=1 - while :; do - fusermount -u "$tempdir$orig_name-rarmount" - local status=$? - if [[ $status -eq 0 ]]; then - echo "INFO: Mount directory has been unmounted" - break - elif [[ $status -ne 0 ]] && [[ $i -eq 3 ]]; then - echo -e "\e[00;33m\nINFO: Umounting failed. Could not umount files (try manually fusermount -u): \"$tempdir$orig_name-rarmount\" \e[00m" - break + if [[ $mount_in_use != "true" ]]; then + # System was unsuccessful in finding video files using mount, so it will revert to default transfer instead + mountsystem umount + echo "INFO: Ignoring send_option=video. Path will be transferred normally." + send_option="default" + echo "INFO: Sendoption: $send_option" fi - echo -e "\e[00;33m\nINFO: Umounting failed - $i attemp(s). Retrying in 10 seconds... \e[00m" - sleep 10 - let i++ - done - ;; + ;; + "umount" ) + # Attempt to unmount directory + local i + i=1 + while :; do + fusermount -u "$tempdir$orig_name-rarmount" + local status=$? + if [[ $status -eq 0 ]]; then + echo "INFO: Mount directory has been unmounted" + break + elif [[ $status -ne 0 ]] && [[ $i -eq 3 ]]; then + echo -e "\e[00;33m\nINFO: Unmounting failed. Could not unmount files (try manually fusermount -u): \"$tempdir$orig_name-rarmount\" \e[00m" + break + fi + echo -e "\e[00;33m\nINFO: Unmounting failed - $i attempt(s). Retrying in 10 seconds... \e[00m" + sleep 10 + let i++ + done + ;; esac else echo -e "\e[00;31mERROR: rar2fs not found. Ignoring mount and transferring everything as normal\e[00m" echo -e "\e[00;36mINFO: Rerun installer to install rar2fs\e[00m" fi -} +} \ No newline at end of file