No one now, must sit by the computer, and insert credentials manualy.
The demand was to log on automatically to the SFTP servers, and download the files by its name generated by the yesterday date. If the current day is Monday, simply download the files from Last Friday.
The data were supposed to be downloaded from several servers. The servers were not configured in the same way. One of them supported authentication using Public key and another not. Therefore the script had to take into account this fact. The environment on the client server, to whom the data should be downloaded to was quite tightened. No option to install any 3rd party application, that would allow to connect and authenticate to the SFTP server was allowed. Therefore, no Expect, no LFTP, or any other clients.
That was quite easy and quick. I simply generated the pair of Private / Public key. As the job is running during the night, there is no password used in the keys. The Public key was sent to the administrator of the SFTP server, and I kept mine on the server. Several times, I just finetuned some loops and everything started to work swiftly without any issues.
Another server unfortunately did not supported this method of authentication. because of that, there was required to perform another solution. Because of tightened security, no mean to install anything else than core of the Operating System was possible. I googled a lot, and I finally found a workaround.
Indeed, it is not as secure as the first option, but it works. The download of the data by own job is possible if you use e.g. CURL instead of unsupported keys.
#!/bin/bash
#purpose: script for downloading of files from SFTP using SSH public keys or using CURL if source server does not support SSH public/private keys combination
YdayYMD
=
$(date --date="yesterday" +"%Y%m%d") #date format like 20221115
YdayDM
=
$(date --date="yesterday" +"%d%m") #date format like 1511
Now
=
$(date --date "today" +"%d. %m. %Y %H:%M:%S") #date time format like 15. 11. 2022 19:09:13
Day
=
$(date --date="yesterday" +%u) #1 - for Monday (depends on operation system configuration)
LFridayYMD
=
$(date -d 'last Friday' +'%Y%m%d') #20221028 - for last friday in format YYYYMMDD
LFridayDM
=
$(date -d 'last Friday' +'%d%m') #2810 - for last friday in format DDMM
LFolder
=
"/johndoe/admin621/data/indices/" #local folder for download of the files from SFTP servers
LogFile
=
"/johndoe/admin621/data/indices/log.txt" #text log files for info about downloaded files
MailRecipient
=
"distributionlist@domain.com" #distribution group - recipients of the report and attached files
#MailRecipient="firstname.surname@domain.com" #e-mail address for development/debugging of the script
MailRecipientC
=
"firstname.surname@domain.com" #recipient to CC of the email message
FTP1Login
=
"JohnDoe987" #Username of the source SFTP server (CURL usage)
FTP1Pass
=
"789EodNhoj" #Password of the source SFTP server (CURL usage)
FTP1Port
=22 #TCP port - specified usage only, if the SFTP listens on non-standard port
#FTP1="${FTP1Login}@data.server1.com" #for use once the SFTP public key will work as expected
FTP1
="data.server1.com"
#FQDN of the SFTP server
FTP2Login
=
"MelodicJerry//23" #Username of the source SFTP server
FTP2Port
=22 #TCP port - specified usage only, if the SFTP listens on non-standard port
FTP2
=
"${FTP2Login}@subdomain.domain.tld" #filling the server login for further use
FTPs
=
([0]="${FTP1}" [1]="${FTP2}") #filling the array with the list of servers, the script will connect to
FTPsLen
=
${#FTPs[@]} #SFTPs array length
FTP1Files
=
(/data/eu/100a${YdayDM}.csv
/data/eu/vall${YdayDM}.csv
/data/eu/trad${YdayDM}.csv)
#Filling the array with filenames to download in common days - yesterday timestamp in filename
FTP2Files=
(./Products/SP_US_EOD_NCS/${YdayYMD}_SP500_NCS_CLS.SDC
./Inbox/${YdayYMD}_SPASX200_NCS_CLS.SDC
./Inbox/${YdayYMD}_SPTSX.SDE
./Inbox/${YdayYMD}_SPTSX_CLS.SDC
./Products/SP_US_EOD_NCS/${YdayYMD}_SP500_NCS.SDE
./Inbox/${YdayYMD}_SPASX200_NCS.SDE)
#Filling the array for common weekdays (yesterday's data)
FTP1FilesLen=
${#FTP1Files[@]} #Length of the array with files for download for further iteration
FTP2FilesLen=
${#FTP2Files[@]} #Length of the array with files for download for further iteration
StatusTxt=
"" #Report intiialization for sending the mail message
Yesterday=
$(date --date="yesterday" +%u) #Day of week generation - number format
Day=
$(date +%u) #Day of week generation - number format (Monday = 1)
if [ ${Yesterday} -eq 5 ] || [ ${Yesterday} -eq 6 ]; then #If yesterday was weekend
exit 0 #Stop script run
fi
if [ ${Day} -eq 1 ]; then #If today is Monday (1), use filenames containing dates from last Friday
FTP2Files=
(./Products/SP_US_EOD_NCS/${LFridayYMD}_SP500_NCS_CLS.SDC
./Inbox/${LFridayYMD}_SPASX200_NCS_CLS.SDC
./Inbox/${LFridayYMD}_SPTSX.SDE
./Inbox/${LFridayYMD}_SPTSX_CLS.SDC
./Products/SP_US_EOD_NCS/${LFridayYMD}_SP500_NCS.SDE
./Inbox/${LFridayYMD}_SPASX200_NCS.SDE)
FTP1Files=
(/data/uk_indices/ftse100/history/100a${LFridayDM}.csv
/data/valuation/uk_all_share/history/vall${LFridayDM}.csv
/data/tracker/uk_trader/history/trad${LFridayDM}.csv)
fi
rm ${LogFile} 2> /dev/null #Remove log file
echo "Following files were downloaded: " | tee -a ${LogFile} > /dev/null
#Writing to the log file - header with downloaded files
echo " " | tee -a ${LogFile} > /dev/null
#New line in log
State=0 #Error checking initialization
for ((h=0; h<=${FTPsLen}; h++)); #For each configured SFTP server
do
if [[ ${FTPs[$h]} == *"${FTP2}"* ]]; then
#Public SSH keys supported - SFTP used, next block follows
for ((j=0; j<=${FTP2FilesLen}; j++)); #For each file to download from the array
do
FileNameLength=${#FTP2Files[$j]} #Filename length checking
if [ "${FileNameLength}" -gt 0 ]; then #If the filename is longer than 0
echo "@get ${FTP2Files[$j]} ${LFolder}" | sftp -i "/isfo/.ssh/id_rsa2" -b - ${FTPs[$h]} 2> /dev/null
#Connect to SFTP server using ID and check if file exists
if [ $? -eq 0 ]; then #If there is no error during check of the filesize
FileToCopy=$(basename "${FTP2Files[$j]}") #Cut the path and keep filename only
echo ${FileToCopy} | tee -a ${LogFile} > /dev/null #Write the filename to the log
else #If there was an error
FileToCopy=$(basename "${FTP2Files[$j]}")
echo "${FileToCopy} - File not found." | tee -a ${LogFile} > /dev/null
#File not found - write to the log
State=$((State+1)) #Increase the Error level value
fi
echo "bye" > /dev/null
#The end of the communication with SFTP server
echo "EOF" > /dev/null #Logout from the SFTP server
fi
done
fi
if [[ ${FTPs[$h]} == *"${FTP1}"*
]]; then #No support of the SSH public keys - CURL used instead, next block follows
for ((j=0; j<=${FTP1FilesLen}; j++)); #For each file to download loop
do
FileNameLength=${#FTP1Files[$j]} #Filename including path length
if [ "${FileNameLength}" -gt 0 ]; then #If the filename length is longer than 0
FileToCopy=$(basename "${FTP1Files[$j]}") #Cut the path and keep filename only
curl -k "sftp://${FTP1}:${FTP1Port}${FTP1Files[$j]}" --user "${FTP1Login}:${FTP1Pass}" -o "${LFolder}${FileToCopy}" 2> /dev/null #Connect to SFTP server using CURL and get the file if exists
if [ $? -eq 0 ]; then #If there was no error
echo ${FileToCopy} | tee -a ${LogFile} > /dev/null #Write to the log
else
echo "${FileToCopy} - File not found." | tee -a ${LogFile} > /dev/null #Else write the error to the log
State=$((State+1)) #Increase the error level value
fi
fi
done
fi
done
cd ${LFolder} #Open local folder with downloaded files
chmod -R 777 "${LFolder}" 2> /dev/null #Change privileges to each downloaded file to 777, so everyone can access them
echo " " | tee -a ${LogFile} > /dev/null #New line written to the log
if [ ${Stav} -gt 0 ]; then #If the error value is greater than 0 => error
StatusTxt="The resulting state: NOK" #Some errors appeared - Final value equals to NOK
else
StatusTxt="The resulting state: OK" #Everything is OK - Final value equals to OK
fi
echo ${StavTxt} | tee -a ${LogFile} > /dev/null #Append Final value to the log
echo "Date of report ${Now}." | tee -a ${LogFile} > /dev/null
#Append also current datetime, when files were downloaded
if [ ${Day} -eq 1 ]; then #If week-day is Monday, attach files from last Friday
filestoattach=(/${LFolder}${LFridayYMD}_SP500_NCS.SDE
/${LFolder}${LFridayYMD}_SPTSX.SDE)
for f in "${filestoattach[@]}"; do
[[ -f "$f" ]] && attachments+=(-a "$f")
done
else #For other week-day, use yesterday's files
filestoattach=(/${LFolder}${YdayYMD}_SP500_NCS.SDE
/${LFolder}${YdayYMD}_SPTSX.SDE)
for f in "${filestoattach[@]}"; do
[[ -f "$f" ]] && attachments+=(-a "$f")
done
fi
#Send an e-mail message with the Log content and provide info about errors detected (to configured recipients). Also attach selected files to the mail message
mail -s "Downloaded files report - state as per $(date --date="yesterday" +"%d. %m. %Y")"
"${attachments[@]}" -c "${MailRecipientC}" "${MailRecipient}" < /${LogFile}
exit 0
#End of the script run
