command line arguments to Pig scripts

Parameter Placeholder

First, we need to create a place holder for the parameter that needs to be replaced inside the Pig script. Let’s say you have the following line in your Pig script where you are loading an input file.

INPUT = LOAD '/data/input/20130326'

In the above statement, if you want to replace date part dynamically, then have to create a placeholder for it.

INPUT = LOAD '/data/input/$date'

Individual Parameters

To pass individual parameters to the Pig script we can use the -param option while invoking the Pig script. So the syntax would be

pig -param date=20130326 -f myfile.pig

If you want to pass two parameters then you can add one more -param option.

pig -param date=20130326 -param date2=20130426 -f myfile.pig

Param File

If there are lot of parameters that needs to be passed, or if we needed a more flexible way to do it, then we can place all of them in a single file and pass the file name using the -param_file option.

The param file uses the simple ini file format where every line contains the param name and the value. We can specify comments using the # character.

date=20130326
date2=20130426

We can pass the param file using the following syntax

pig -param_file=myfile.ini -f myfile.pig

Default Statement

We can also assign a default value to a parameter inside the Pig script using the default statement like below

%default date '20130326'

Processing Order

One good thing about parameter substitution in Pig is that you can pass in value for the same parameter using multiple options simultaneously. Pig will pick them up in the following order.

• The default statement takes the lowest precedence.
• The values passed using -param_file takes the next precedence.
  • If there are multiple entries for the same param is present in a file, then the one which comes later takes more precedence.
  • If there are multiple param files, then the files that are specified later will take more precedence.
• The values that are passed using the -param option takes the next precedence.
  • If multiple values are specified for the same param, then the ones which are specified later takes more precedence.

Debugging

Sometimes, the precedence might be little confusing, especially if you have multiple files and multiple params. Pig also provides a -debug option to debug this kind of scenario’s. If you invoke Pig with this option, then it will generate a file with extension .substitued in the current directory with the place holders replaced with the correct values.

I specify a default value using the default statement and then pass actual values using the -param_fileoption. If I am in a hurry and just want to test something locally, then I use -param option, but generally I try to put them in a separate ini file so that I can check-in the options as well.

Pig UDF : Eval

Apache Pig : UDF

Apache Pig : Writing a User Defined Function (UDF)

Apache Pig : Writing a User Defined Function (UDF)

Preface:

In this post we will write a basic/demo custom function for Apache Pig, called as UDF (User Defined Function).
Pig’s Java UDF extends functionalities of EvalFunc. This abstract class have an abstract method “exec” which user needs to implement in concrete class with appropriate functionality.

Problem Statement:

Lets write a simple Java UDF which takes input as Tuple of two DataBag and check whether second databag(set) is subset of first databag(set).
For example, Assume you have been given tuple of two databags. Each DataBag contains elements(tuples) as number.

Input:
Databag1 : {(10),(4),(21),(9),(50)}
Databag2 : {(9),(4),(50)}
Output:
True

Then function should return true as Databag2 is subset of Databag1.

From implemetation point of view

As we are extending abstract class EvalFucn, we will be implementing exec function. In this function we’ll write logic to find is given set is subset of other or not. We will also override function outputSchema to specify output schema ( boolean : true or false ).

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63

import java.io.IOException; import java.util.HashSet; import java.util.Iterator; import java.util.List; import java.util.Set; import org.apache.pig.EvalFunc; import org.apache.pig.data.DataBag; import org.apache.pig.data.DataType; import org.apache.pig.data.Tuple; import org.apache.pig.impl.logicalLayer.schema.Schema; import org.apache.pig.impl.logicalLayer.schema.Schema.FieldSchema; /**  * Find the whether given SetB is subset of SetA.  *  *  input:  * setA : {(10),(4),(21),(9),(50)}  * setB : {(9),(4),(50)}  *  *  output:  * true  *

 *

 *

 */

public class IsSubSet extends EvalFunc {

    @Override

    public Schema outputSchema(Schema input) {

        if(input.size()!=2){

            throw new IllegalArgumentException("input should contains two elements!");

        }

        List fields = input.getFields();

        for(FieldSchema f : fields){

            if(f.type != DataType.BAG){

                throw new IllegalArgumentException("input fields should be bag!"); 

            }

        }

        return new Schema(new FieldSchema("isSubset",DataType.BOOLEAN));

    }

    private Set populateSet(DataBag dataBag){

        HashSet set = new HashSet();

        Iterator iter = dataBag.iterator();

        while(iter.hasNext()){

            set.add(iter.next());

        }

        return set;

    }

    @Override

    public Boolean exec(Tuple input) throws IOException {

        Set setA = populateSet((DataBag) input.get(0));

        Set setB = populateSet((DataBag) input.get(1));

        return setA.containsAll(setB) ? Boolean.TRUE : Boolean.FALSE;

    }

}

A Quick Test

Lets test our UDF to find whether given set is subset of other set or not.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

-- Register jar which contains UDF. register '/home/hadoop/udf.jar'; -- Define function for use. define isSubset IsSubSet(); -- lets assume we have dataset as following :  dump datset; --({(10),(4),(21),(9),(50)},{(9),(4),(50)}) --({(50),(78),(45),(7),(4)},{(7),(45),(50)}) --({(1),(2),(3),(4),(5)},{(4),(3),(50)}) -- lets check subset function result = foreach dataset generate $0,$1,isSubset($0,$1); dump result; --({(10),(4),(21),(9),(50)},{(9),(4),(50)},true) --({(50),(78),(45),(7),(4)},{(7),(45),(50)},false) --({(1),(2),(3),(4),(5)},{(4),(3),(50)},false)

---

About

Posted on 

About these ads