Apache Pig : UDF

-

Apache Pig : Writing a User Defined Function (UDF)





Apache Pig : Writing a User Defined Function (UDF)



Preface:

In this post we will write a basic/demo custom function for Apache Pig, called as UDF (User Defined Function).
Pig’s Java UDF extends functionalities of EvalFunc. This abstract class have an abstract method “exec” which user needs to implement in concrete class with appropriate functionality.

Problem Statement:

Lets write a simple Java UDF which takes input as Tuple of two DataBag and check whether second databag(set) is subset of first databag(set).
For example, Assume you have been given tuple of two databags. Each DataBag contains elements(tuples) as number.
Input:
Databag1 : {(10),(4),(21),(9),(50)}
Databag2 : {(9),(4),(50)}
Output:
True
Then function should return true as Databag2 is subset of Databag1.

From implemetation point of view

As we are extending abstract class EvalFucn, we will be implementing exec function. In this function we’ll write logic to find is given set is subset of other or not. We will also override function outputSchema to specify output schema ( boolean : true or false ).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
import java.io.IOException;
import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Set;
import org.apache.pig.EvalFunc;
import org.apache.pig.data.DataBag;
import org.apache.pig.data.DataType;
import org.apache.pig.data.Tuple;
import org.apache.pig.impl.logicalLayer.schema.Schema;
import org.apache.pig.impl.logicalLayer.schema.Schema.FieldSchema;
/**
 * Find the whether given SetB is subset of SetA.
 *
 *  input:
 *
setA : {(10),(4),(21),(9),(50)}
 *
setB : {(9),(4),(50)}
 *
 *  output:
 *
true
 *
 *
 *
 */
public class IsSubSet extends EvalFunc {
    @Override
    public Schema outputSchema(Schema input) {
        if(input.size()!=2){
            throw new IllegalArgumentException("input should contains two elements!");
        }
        List fields = input.getFields();
        for(FieldSchema f : fields){
            if(f.type != DataType.BAG){
                throw new IllegalArgumentException("input fields should be bag!"); 
            }
        }
        return new Schema(new FieldSchema("isSubset",DataType.BOOLEAN));
    }
    private Set populateSet(DataBag dataBag){
        HashSet set = new HashSet();
        Iterator iter = dataBag.iterator();
        while(iter.hasNext()){
            set.add(iter.next());
        }
        return set;
    }
    @Override
    public Boolean exec(Tuple input) throws IOException {
        Set setA = populateSet((DataBag) input.get(0));
        Set setB = populateSet((DataBag) input.get(1));
        return setA.containsAll(setB) ? Boolean.TRUE : Boolean.FALSE;
    }
}

A Quick Test

Lets test our UDF to find whether given set is subset of other set or not.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
-- Register jar which contains UDF.
register '/home/hadoop/udf.jar';
-- Define function for use.
define isSubset IsSubSet();
-- lets assume we have dataset as following :
 dump datset;
--({(10),(4),(21),(9),(50)},{(9),(4),(50)})
--({(50),(78),(45),(7),(4)},{(7),(45),(50)})
--({(1),(2),(3),(4),(5)},{(4),(3),(50)})
-- lets check subset function
result = foreach dataset generate $0,$1,isSubset($0,$1);
dump result;
--({(10),(4),(21),(9),(50)},{(9),(4),(50)},true)
--({(50),(78),(45),(7),(4)},{(7),(45),(50)},false)
--({(1),(2),(3),(4),(5)},{(4),(3),(50)},false)
About







Posted on 

About these ads

Comments

Post a Comment

Popular posts from this blog

SharePoint 2007 - Simple Task Dashboard

-
Image
example of creating a simple Task Dashboard. This is a pretty basic example of using all OOTB (out of the box) features and combining them to quickly build a solution. Scenario: We want to provide a way for users to easily see a summary of the current project tasks. We want them to be able to see the following information: Open Tasks Assigned to Them Overdue Tasks Tasks Due This Week Task Calendar Tasks Assigned to Others Solution: To build this solution we will be using several different tools available in SharePoint, all included in the list below: Task List Task List Views We will create views for each of the different ways we want to see the data displayed on our dashboard. Web Part Page List View Web Parts Custom Navigation The first thing we will do is create our task lists and then configure all the views that we need. To create the list, go to View All Site Content, Create and then select the Task list. Next, we will need to configure our custom views. To do this, w
Read more

MERGE transformation in SSIS

-
Image
In this post we are gonna discuss about MERGE transformation. MERGE in SSIS is equal to UNION ALL in SQL Server. This transformation unions two datasets/tables.The merge Transformation combines two  sorted  dataset into single dataset. Highlighted the text SORTED in last statement because “It is not possible to use MERGE when the inputs are NOT SORTED”. There is one more transformation which is very similar to this i.e UNION ALL. The Merge Transformation is similar to the union all transformations. Use the union all transformation instead of the merge transformation in the following situations. The transformation inputs are not sorted The combined output does not need to be sorted. The transformation has more than two inputs. MERGE takes ONLY TWO inputs where as UNION ALL can take more than two. Now lets see how to configure MERGE transformation with an example. I created TWO tables with names MergeA and MergeB and inserted few records into each table as shown below.
Read more