problems_pig
1 error1
Error log:
Failed to parse: Pig script failed to parse:
<file /x/home/pp_dt_cmpln_batch/test/CRR_rollupreg_extract.pig, line 20, column 46> Failed to generate logical plan. Nested exception: org.apache.pig.backend.executionengine.ExecException: ERROR 1070: Could not resolve concat using imports: [, java.lang., org.apache.pig.builtin., org.apache.pig.impl.builtin.]
Reason: please refer to the following pig statement. pig's concat function should be uppercase, should be CONTACT().
data = FOREACH data GENERATE RollupregExtract(concat(payload, ${logical_date}));
2 error2
Error log:
Caused by: org.apache.pig.impl.logicalLayer.validators.TypeCheckerException: ERROR 1045:
<file /x/home/pp_dt_cmpln_batch/test/CRR_rollupreg_extract.pig, line 20, column 46> Could not infer the matching function for org.apache.pig.builtin.CONCAT as multiple or none of them fit. Please use an explicit cast.
Reason: pls refer to the incorrect grammer below:
data = FOREACH data GENERATE RollupregExtract(CONCAT(field1, ${logical_date}));
the ${logical_date} should be cast in to chararray implicitly.
Correct grammar:
data = FOREACH data GENERATE RollupregExtract(CONCAT(payload, (chararray)${logical_date}));
data = FOREACH data GENERATE RollupregExtract(CONCAT(payload, 'logical_date:${logical_date}'));
3 error3
ERROR LOG3 :
java.lang.Exception: org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing [POUserFunc (Name: POUserFunc(com.paypal.rollupreg.RollupregExtract)[chararray] - scope-8 Operator Key: scope-8) children: null at []]: java.time.format.DateTimeParseException: Text '' could not be parsed at index 0
Reason:
The grammer(data = FOREACH data GENERATE RollupregExtract(CONCAT(payload, 'logical_date:${logical_date}'))😉 is correct, we can transfer the logical_date to Pig UDF function. It is because Pig got some empty lines. For empty lines, Pig read the whole directory. There are many files in the directory. Pig will handle all the files in the directory, only one file is the correct source file. Other files have the illegal data, pig will ignore it, and output an empty line.
Solution:
put the source data into an specific directory, and set the directory as the Pig src directory.
