Remove Redundant Projections

To improve memory usage and performance costs, remove unnecessary projections from your queries and load only the data required. If you have to perform an operation, include pre-projection statements as needed.

Here's an example of a query with an unnecessary projection.

q = load "Superstore";
q = foreach q generate 'Category';
q = group q by 'Category';
q = foreach q generate 'Category', count() as 'count';

The first foreach statement projects the Category field, which is already included in the dataset. Since we're not performing any operation on the field, we can remove it.

q = load "Superstore";
q = group q by 'Category';
q = foreach q generate 'Category', count() as 'count';

Here's an example with an implicit cogroup.

a = load "Customer_Data";
a = foreach a generate 'Customer_Name';
b = load "Superstore";
b = foreach b generate 'Customer_Name';
a = group a by 'Customer_Name' full, b by 'Customer_Name';
a = foreach a generate coalesce(a.'Customer_Name', b.'Customer_Name') as 'Customer_Name', count('a') as 'Superstore', count('b') as 'Customer_data';

In this example, the foreach statements that follow loading the “Customer_Data” and the “Superstore” datasets are unnecessary, since they’re projecting the Customer_Name fields without any additional action. You can group the fields pre-projection.

a = load “Customer_Data”;
b = load “Superstore”;
a = group a by ‘Customer_Name’ full, b by 'Customer_Name';
a = foreach a generate coalesce(a.'Customer_Name', b.'Customer_Name') as 'Customer_Name', count('a') as 'Superstore', count('b') as 'Customer_data';