Direct retrieval solves the bottlenecks that can occur when processing a large number of files in between retrieval cycles. It also helps in managing the data stream’s execution log, which can be a challenge when the number of files that accumulate between processing cycles is in the thousands.
Direct retrieval takes all the files that accumulate between retrievals and merges them into one large file. This process reduces the job queue into a single processing job, which makes the processing action faster and easier to manage and maintain.
If you use this option, make sure that your files are of an “incremental nature.” Data from one file doesn’t replace data in other files, but rather adds to the data in other files. For example, if you have the same key in two separate files, both of the files’ rows appear in the merged file. When the file is uploaded to the database, the measurements are summed together. If the accumulated files are supposed to replace each other’s data, direct retrieval doesn’t work.
