One of the more advanced topics when it comes to MOVEit Automation and in use of FTP and SFTP is that of directory parsing.
Tools such as FileZilla, WinSCP and MOVEit Automation go a long way to making file transfer protocol as accessible as possible, especially when it comes to translating the output of a server directory listing into a format suitable for a Graphical User Interface (GUI). But what happens on those few occasions when they need a helping hand?
When MOVEit Automation connects to an FTP or SFTP server and successfully authenticates, the server will send an output of what is contained in the connected directory. It looks something like this:
drwxr-x--- $superuser $superuser 0 Wed, 02 Mar 2022 06:26:12 GMT inbound
drwxr-x--- $superuser $superuser 0 Wed, 02 Mar 2022 06:26:20 GMT outbound
It is then the job of MOVEit Automation to determine which of the columns – based on position, spacing between and format – relates to parameters such as whether it is a directory or a file; when it was created; and what the directory or filename is.
So that it can be presented like this:
In particular, the designation of folder or file; and what the filename are critical, as if these items are incorrectly presented, it will impact the ability to navigate the directory structure and download files.
For example, look at the output below, when directory parsing has not successfully determined the correct filename:
The date/time and time zone have incorrectly been assumed as part of the filename, which means that should MOVEit Automation attempt to download the first file in the list, it will issue a command to download “/outbound/08:47:56 GMT 57001_SomeFileName.tb2”.
This will then fail as the server will report that there are no files which exist with this filename. Therefore, in this scenario we would need to enable some form of custom directory parsing.
If you need to enable custom directory parsing in MOVEit Automation, you can do this for both FTP and SFTP host connectors.
For FTP hosts, you simply need to open the options of the FTP host connector and select Directory List Parsing close to the bottom of the list.
For SFTP hosts, it is a little more complicated. Versions of MOVEit Automation prior to 2020.0.x (12.0.x) had the Directory List Parsing option much like FTP. From this version and onward, MOVEit Automation switched their SFTP library and so hid the Directory List Parsing option.
This can be revealed again by exporting the MOVEit Automaton configuration into a text file; adding the line SFTPLib=“WeOnlyDo” to the host; and then uploading the modified configuration back into MOVEit Automation.
Note, a mistake in this configuration file can halt all MOVEit Automation task executions and should only be attempted with great caution.
It is possible that by revering to the “WeOnlyDo” SFTP library that your directory parsing issues may have been resolved, so make sure to try this before proceeding with scripting.
Once you have the Directory List Parsing option enabled on your FTP or SFTP host, you will have two options for custom directory parsing.
Parsing parameters allow you to specify which lines to skip – an example of this would be if there is a header or footer line in the output – and which columns specify the filename and date.
Parsing scripts are scripts which have been uploaded into the scripting interface under resources and have a greater degree of flexibility and manipulation.
These scripts are written in VBScript, with a sample script provided by Progress in their administration guide - https://docs.ipswitch.com/MOVEit/Automation2020/Help/Admin/en/index.htm#30463.htm
Directory parsing can be a difficult and intimidating topic even for those who have worked with Managed File Transfer (MFT) solutions for many years.
As experts in this space, we are here to help. We have assisted with over 800 MFT projects over 18 years; many of which have required directory parsing customisations. So, should you need us to assist you with custom directory parsing, feel free to reach out to the Sales team for more information.