UDF函数 解码url

背景

URL 的编码 是ASCII十六进制格式。数仓接受到前端上报的URL，要对URL字段解码。

如要将

https%3A%2F%2Fmywebsite%2Fdocs%2Fenglish%2Fsite%2"Fmybook.do%3Frequest_type%3D%26type%3Dprivate

解码为:

https://mywebsite/docs/english/site/mybook.do?request_type=&type=private

 

方式

hive sql中，通过反射调用java.net.URLDecoder方法

select reflect('java.net.URLDecoder', 'decode',url, 'UTF-8') from table_name;

问题

实践中发现，部分https开头的url解析不完整。例如解码下面的URL，

https%253A%252F%252Fwww.cnblogs.com/drjava

通过上面的方法，结果是：

https%3A%2F%2Fwww.cnblogs.com/drjava

 

解决方案

通过google找到了可行的方法，解码两次，由于sql写起来比较难以阅读，所以封装了udf。代码如下：

import org.apache.hadoop.hive.ql.exec.UDF;
import java.io.UnsupportedEncodingException;
import java.net.URLDecoder;

public class UrlDecode extends UDF {
    /**
     * 
     *
     * @param component 编码的url
     * @return 解码url
     */
    public  String evaluate(String component) {
        if(component == null || component.length() <= 0){
            return "";
        }

        String result = "";
        component = component.replaceAll("%(?![0-9a-fA-F]{2})", "%25");
        try {
            result = URLDecoder.decode(component, "UTF-8");
            result = URLDecoder.decode(result, "UTF-8");
        } catch (UnsupportedEncodingException e) {
            result = component;
        }
        return result;
    }
}

做一下单元测试

import org.junit.Assert;
import org.junit.Test;
import udf.UrlDecode;

public class UrlDecodeTest {
    @Test
    public void UrlDecodeMethod(){
        UrlDecode ud = new UrlDecode();
        Assert.assertEquals(ud.evaluate("https%3A%2F%2Fmywebsite%2Fdocs%2Fenglish%2Fsite%2" +
                "Fmybook.do%3Frequest_type%3D%26type%3Dprivate"),
                "https://mywebsite/docs/english/site/mybook.do?request_type=&type=private");

        Assert.assertEquals(ud.evaluate("你是谁，为了谁"),"你是谁，为了谁");

        Assert.assertEquals(ud.evaluate(null),"");
    }
}