Springboot 系列 (26) - Springboot+HBase 大数据存储(四)| Springboot 项目通过 HBase API 方式访问 HBase

 

Apache HBase 是 Java 语言编写的一款 Apache 开源的 NoSQL 型数据库,不支持 SQL,不支持事务,不支持 Join 操作,没有表关系。Apache HBase 构建在 Apache Hadoop 和 Apache Zookeeper 之上。

Apache HBase: https://hbase.apache.org/

HBase 的安装配置,请参考 “Springboot 系列 (24) - Springboot+HBase 大数据存储(二)| 安装配置 Apache HBase 和 Apache Zookeeper”。

HBase API 操作表的相关命令,请参考 “Springboot 系列 (25) - Springboot+HBase 大数据存储(三)| HBase Shell ”。

本文将创建一个 SpringBoot 项目,演示通过 HBase API 方式访问 HBase。

 

1. HBase 运行环境

    操作系统:Ubuntu 20.04
    Hadoop 版本:3.2.2
    Zookeeper 版本:3.6.3
    HBase 版本:2.4.4
    HBase 所在路径:~/apps/hbase-2.4.4/

    本文使用的 HBase 部署在伪分布式 Hadoop 架构上(主机名:hadoop-master-vm),在 HBase + Zookeeper (独立的) 模式下运行,Zookeeper 使用端口 2182。

 

2. 创建 Springboot 项目

    Windows版本:Windows 10 Home (20H2)   
    IntelliJ IDEA:Community Edition for Windows 2020.1.4
    Apache Maven:3.8.1

    注:Spring 开发环境的搭建,可以参考 “ Spring基础知识(1)- Spring简介、Spring体系结构和开发环境配置 ”。

    1) 运行 IDEA 创建项目
   
        点击菜单 New 创建 Project:
        
        New Project -> Project Type: Maven -> Project SDK: 1.8 -> Check "Create from archtype" -> select "org.apache.maven.archtypes:maven-archtype-quickstart" -> Next

            Name: SpringbootExample21
            GroupId: com.example
            ArtifactId: SpringbootExample21

        -> Finish

    2) 修改 pom.xml

        <?xml version="1.0" encoding="UTF-8"?>

        <project xmlns="http://maven.apache.org/POM/4.0.0"
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                                    http://maven.apache.org/xsd/maven-4.0.0.xsd">
            <modelVersion>4.0.0</modelVersion>

            <groupId>com.example</groupId>
            <artifactId>SpringbootExample21</artifactId>
            <version>1.0-SNAPSHOT</version>

            <name>SpringbootExample21</name>
            <!-- FIXME change it to the project's website -->
            <url>http://www.example.com</url>

            <properties>
                <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
                <maven.compiler.source>1.8</maven.compiler.source>
                <maven.compiler.target>1.8</maven.compiler.target>
            </properties>

            <parent>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-starter-parent</artifactId>
                <version>2.6.6</version>
                <relativePath/> <!-- lookup parent from repository -->
            </parent>

            <dependencies>
                <dependency>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-starter</artifactId>
                </dependency>
                <dependency>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-starter-test</artifactId>
                    <scope>test</scope>
                </dependency>
                <dependency>
                    <groupId>junit</groupId>
                    <artifactId>junit</artifactId>
                    <version>4.11</version>
                    <scope>test</scope>
                </dependency>
            </dependencies>
            
            <build>
                <plugins>
                    <plugin>
                        <groupId>org.springframework.boot</groupId>
                        <artifactId>spring-boot-maven-plugin</artifactId>
                        <configuration>
                            <mainClass>com.example.App</mainClass>
                            <layout>JAR</layout>
                        </configuration>
                        <executions>
                            <execution>
                                <goals>
                                    <goal>repackage</goal>
                                </goals>
                            </execution>
                        </executions>
                    </plugin>
                </plugins>

                <pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
                    <plugins>
                        ...
                    </plugins>
                </pluginManagement>
            </build>
        </project>


        在IDE中项目列表 -> SpringbootExample21 -> 点击鼠标右键 -> Maven -> Reload Project

        本文选择了 spring-boot-starter-parent 2.6.6 相关依赖包,spring-boot-starter 和 spring-boot-starter-test 的版本由 spring-boot-starter-parent 控制。

    3) 修改 src/main/java/com/example/App.java 文件

        package com.example;

        import org.springframework.boot.SpringApplication;
        import org.springframework.boot.autoconfigure.SpringBootApplication;

        @SpringBootApplication
        public class App {
            public static void main(String[] args) {
                SpringApplication.run(App.class, args);
                System.out.println("Spring boot jar empty project");
            }
        }


    4) 添加 src/main/resources/application.properties 文件

 

         spring.main.banner-mode=off


    5) 运行

        Run -> Edit configurations -> Click "+" -> Select "Maven"

            Command line: clean spring-boot:run
            Name: SpringbootExample21 [clean,spring-boot:run]

        -> OK

        Run -> Run "SpringbootExample21 [clean,spring-boot:run]"
           
            Spring boot jar empty project

 

3. 添加 hbase-client 依赖

    修改 pom.xml

        <project ... >

            ...

            <dependencies>

                ...

                <dependency>
                    <groupId>org.apache.hbase</groupId>
                    <artifactId>hbase-client</artifactId>
                    <version>2.4.4</version>
                    <exclusions>
                        <exclusion>
                            <groupId>org.slf4j</groupId>
                            <artifactId>slf4j-log4j12</artifactId>
                        </exclusion>
                     </exclusions>
                </dependency>
            </dependencies>

        </project>


    在IDE中项目列表 -> SpringbootExample21 -> 点击鼠标右键 -> Maven -> Reload Project

4. HBase API 示例

    HBase API: https://hbase.apache.org/book.html#hbase_apis
    API Reference : https://hbase.apache.org/apidocs/index.html

    1) 修改 src/main/java/com/example/App.java 文件

        package com.example;

        import java.io.IOException;
        import java.util.ArrayList;
        import java.util.List;

        import org.apache.commons.lang.StringUtils;
        import org.apache.hadoop.hbase.TableName;
        import org.apache.hadoop.hbase.Cell;
        import org.apache.hadoop.hbase.CompareOperator;
        import org.apache.hadoop.hbase.client.*;
        import org.apache.hadoop.hbase.filter.RowFilter;
        import org.apache.hadoop.hbase.filter.SubstringComparator;
        import org.apache.hadoop.hbase.util.Bytes;

        import org.springframework.boot.SpringApplication;
        import org.springframework.boot.autoconfigure.SpringBootApplication;

        @SpringBootApplication
        public class App {

            private static Connection hbaseConnection;
            private static Admin hbaseAdmin;

            public static void main(String[] args) {
                SpringApplication.run(App.class, args);

                //
                try {

                    org.apache.hadoop.conf.Configuration hbaseConf = new org.apache.hadoop.conf.Configuration();
                    hbaseConf.set("hbase.zookeeper.quorum", "hadoop-master-vm");
                    hbaseConf.set("hbase.zookeeper.property.clientPort", "2182");
                    //hbaseConf.setInt("hbase.hconnection.threads.max", 100);
                    //hbaseConf.setInt("hbase.hconnection.threads.core", 50);
                    //hbaseConf.setLong("hbase.hconnection.threads.keepalivetime", 1000);
                    hbaseConf.set("zookeeper.znode.parent", "/hbase");

                    hbaseConnection = ConnectionFactory.createConnection(hbaseConf);
                    hbaseAdmin = hbaseConnection.getAdmin();
                    System.out.println("App -> getAdmin(): hbaseAdmin = " + hbaseAdmin);

                    if (App.tableExists("user")) {
                        System.out.println("App -> tableExists('user') = true");
                    }

                    // Delete table if exists
                    System.out.println("App -> tableExists('user')");
                    if (App.tableExists("user")) {
                        System.out.println("App -> deleteTable('user')");
                        App.deleteTable("user");
                    }

                    // Create table
                    System.out.println("App -> createTable('user', ...)");
                    App.createTable("user", new String[]{"cf1", "cf2", "cf3"});

                    // Insert data
                    System.out.println("App -> insertRecords('user', ... )");
                    App.insertRecords("user", "row1", "cf1", new String[]{"name", "age"}, new String[] {"Tom", "12"});
                    App.insertRecords("user", "row2", "cf1", new String[]{"name", "age"}, new String[] {"Jerry", "9"});
                    App.insertRecord("user", "row1", "cf2", "job", "Student");
                    App.insertRecord("user", "row1", "cf3", "gender", "Male");
                    App.insertRecord("user", "row2", "cf2", "job", "Student2");
                    App.insertRecord("user", "row2", "cf3", "gender", "Female");

                    // Scan table
                    System.out.println("App -> scanTable('user')");
                    System.out.println(App.scanTable("user"));

                    // Get value
                    String value = App.getValue("user", "row1", "cf3", "gender");
                    System.out.println("App -> getValue('user', 'row1', 'cf3', 'gender'): value = " + value);

                    // Delete column
                    System.out.println("App -> deleteColumn('user', 'row1', 'cf3', 'gender')");
                    App.deleteColumn("user", "row1", "cf3", "gender");

                    // Get row
                    System.out.println("App -> getRow('user', 'row1')");
                    System.out.println(App.getRow("user", "row1"));

                    // Delete column family
                    System.out.println("App -> deleteColumnFamily('user', 'row1', 'cf2')");
                    App.deleteColumnFamily("user", "row1", "cf2");

                    // Scan table row
                    System.out.println("App -> scanRow('user', 'row1')");
                    System.out.println(App.scanRow("user", "row1"));

                    // Delete row
                    System.out.println("App -> deleteRow('user', 'row1')");
                    App.deleteRow("user", "row1");

                    // Scan table (2)
                    System.out.println("App -> scanTable('user')");
                    System.out.println(App.scanTable("user"));

                    hbaseAdmin.close();
                    hbaseConnection.close();

                } catch (IOException e) {

                }

            }

            public static boolean tableExists(String tableName) throws IOException {
                TableName[] tableNames = hbaseAdmin.listTableNames();
                if (tableNames != null && tableNames.length > 0) {
                    for (int i = 0; i < tableNames.length; i++) {
                        if (tableName.equals(tableNames[i].getNameAsString())) {
                            return true;
                        }
                    }
                }
                return false;
            }

            public static void createTable(String tableName, String[] columnFamilies) throws IOException {
                TableName name = TableName.valueOf(tableName);

                TableDescriptorBuilder descriptorBuilder = TableDescriptorBuilder.newBuilder(name);
                List<ColumnFamilyDescriptor> columnFamilyList = new ArrayList<>();
                for (String columnFamily : columnFamilies) {
                    ColumnFamilyDescriptor columnFamilyDescriptor = ColumnFamilyDescriptorBuilder.newBuilder(columnFamily.getBytes()).build();
                    columnFamilyList.add(columnFamilyDescriptor);
                }
                descriptorBuilder.setColumnFamilies(columnFamilyList);
                TableDescriptor tableDescriptor = descriptorBuilder.build();
                hbaseAdmin.createTable(tableDescriptor);
            }

            public static void deleteTable(String tableName) throws IOException {
                boolean isExists = App.tableExists(tableName);
                if (!isExists) {
                    return;
                }

                TableName name = TableName.valueOf(tableName);
                hbaseAdmin.disableTable(name);
                hbaseAdmin.deleteTable(name);
            }

            public static void insertRecord(String tableName, String row, String columnFamily, String column, String value) throws IOException {
                TableName name = TableName.valueOf(tableName);
                Table table = hbaseConnection.getTable(name);
                Put put = new Put(Bytes.toBytes(row));
                put.addColumn(Bytes.toBytes(columnFamily), Bytes.toBytes(column), Bytes.toBytes(value));
                table.put(put);
                table.close();
            }

            public static void insertRecords(String tableName, String row, String columnFamily, String[] columns, String[] values) throws IOException {
                TableName name = TableName.valueOf(tableName);
                Table table = hbaseConnection.getTable(name);
                Put put = new Put(Bytes.toBytes(row));
                for (int i = 0; i < columns.length; i++) {
                    put.addColumn(Bytes.toBytes(columnFamily), Bytes.toBytes(columns[i]), Bytes.toBytes(values[i]));
                    table.put(put);
                }
                table.close();
            }

            public static String getRow(String tableName, String rowKey) throws IOException {
                Table table = hbaseConnection.getTable(TableName.valueOf(tableName));
                Get g = new Get(rowKey.getBytes());
                Result rs = table.get(g);

                String str = "";
                for (Cell cell : rs.rawCells()) {
                    StringBuffer stringBuffer = new StringBuffer()
                            .append(Bytes.toString(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength())).append("\t")
                            .append(Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(), cell.getFamilyLength())).append("\t")
                            .append(Bytes.toString(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength())).append("\t")
                            .append(Bytes.toString(cell.getValueArray(), cell.getValueOffset(), cell.getValueLength())).append("\n");
                    str += stringBuffer.toString();
                }
                table.close();

                return str;
            }

            public static String getValue(String tableName, String rowKey, String family, String column) throws IOException {
                Table table = hbaseConnection.getTable(TableName.valueOf(tableName));

                String value = "";
                if (StringUtils.isBlank(tableName) || StringUtils.isBlank(family)
                        || StringUtils.isBlank(rowKey) || StringUtils.isBlank(column)) {
                    return null;
                }

                Get g = new Get(rowKey.getBytes());
                g.addColumn(family.getBytes(), column.getBytes());
                Result result = table.get(g);
                List<Cell> ceList = result.listCells();
                if (ceList != null && ceList.size() > 0) {
                    for (Cell cell : ceList) {
                        value = Bytes.toString(cell.getValueArray(), cell.getValueOffset(), cell.getValueLength());
                    }
                }
                table.close();

                return value;
            }

            public static String scanTable(String tableName) throws IOException {
                Table table = hbaseConnection.getTable(TableName.valueOf(tableName));
                Scan scan = new Scan();
                ResultScanner scanner = table.getScanner(scan);

                String str = "";
                for (Result result : scanner){
                    for (Cell cell : result.rawCells()) {
                        StringBuffer stringBuffer = new StringBuffer()
                                .append(Bytes.toString(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength())).append("\t")
                                .append(Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(), cell.getFamilyLength())).append("\t")
                                .append(Bytes.toString(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength())).append("\t")
                                .append(Bytes.toString(cell.getValueArray(), cell.getValueOffset(), cell.getValueLength())).append("\n");

                        str += stringBuffer.toString();
                    }
                }

                scanner.close();
                table.close();

                return str;
            }

            public static String scanRow(String tableName, String rowKeyword) throws IOException {
                ArrayList<Object> list = new ArrayList<Object>();

                Table table = hbaseConnection.getTable(TableName.valueOf(tableName));
                Scan scan = new Scan();

                RowFilter rowFilter = new RowFilter(CompareOperator.EQUAL, new SubstringComparator(rowKeyword));
                scan.setFilter(rowFilter);

                ResultScanner scanner = table.getScanner(scan);

                String str = "";
                for (Result result : scanner) {
                    for (Cell cell : result.rawCells()) {
                        StringBuffer stringBuffer = new StringBuffer()
                                .append(Bytes.toString(cell.getRowArray(), cell.getRowOffset(), cell.getRowLength())).append("\t")
                                .append(Bytes.toString(cell.getFamilyArray(), cell.getFamilyOffset(), cell.getFamilyLength())).append("\t")
                                .append(Bytes.toString(cell.getQualifierArray(), cell.getQualifierOffset(), cell.getQualifierLength())).append("\t")
                                .append(Bytes.toString(cell.getValueArray(), cell.getValueOffset(), cell.getValueLength())).append("\n");

                        str += stringBuffer.toString();
                    }
                }

                scanner.close();
                table.close();

                return str;
            }

            public static void deleteRow(String tableName, String rowKey) throws IOException {
                Table table = hbaseConnection.getTable(TableName.valueOf(tableName));
                Delete delete = new Delete(rowKey.getBytes());
                table.delete(delete);
                table.close();
            }

            public static void deleteColumnFamily(String tableName, String rowKey, String columnFamily) throws IOException {
                Table table = hbaseConnection.getTable(TableName.valueOf(tableName));
                Delete delete = new Delete(rowKey.getBytes());
                delete.addFamily(Bytes.toBytes(columnFamily));
                table.delete(delete);
                table.close();
            }

            public static void deleteColumn(String tableName, String rowKey, String columnFamily, String column) throws IOException {
                Table table = hbaseConnection.getTable(TableName.valueOf(tableName));
                Delete delete = new Delete(rowKey.getBytes());
                delete.addColumn(Bytes.toBytes(columnFamily), Bytes.toBytes(column));
                table.delete(delete);
                table.close();
            }

        }

 

    2) 运行

         Run -> Edit configurations -> Click "+" -> Select "Maven"

            Command line: clean spring-boot:run
            Name: SpringbootExample21 [clean,spring-boot:run]

        -> OK

        Run -> Run "SpringbootExample21 [clean,spring-boot:run]"

           ...

            2023-03-30 12:06:00.798  INFO 7989 --- [2182@0x55cb6996] org.apache.zookeeper.common.X509Util     : Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
            2023-03-30 12:06:00.806  INFO 7989 --- [2182@0x55cb6996] org.apache.zookeeper.ClientCnxnSocket    : jute.maxbuffer value is 4194304 Bytes
            2023-03-30 12:06:00.819  INFO 7989 --- [2182@0x55cb6996] org.apache.zookeeper.ClientCnxn          : zookeeper.request.timeout value is 0. feature enabled=
            2023-03-30 12:06:00.853  INFO 7989 --- [2.168.1.4:2182)] org.apache.zookeeper.ClientCnxn          : Opening socket connection to server 192.168.1.5/192.168.1.5:2182. Will not attempt to authenticate using SASL (unknown error)
            2023-03-30 12:06:00.862  INFO 7989 --- [2.168.1.4:2182)] org.apache.zookeeper.ClientCnxn          : Socket connection established, initiating session, client: /192.168.1.5:43262, server: 192.168.1.5/192.168.1.5:2182
            2023-03-30 12:06:00.873  INFO 7989 --- [2.168.1.4:2182)] org.apache.zookeeper.ClientCnxn          : Session establishment complete on server 192.168.1.5/192.168.1.5:2182, sessionid = 0x10000053fd6000c, negotiated timeout = 40000

            App -> getAdmin(): hbaseAdmin = org.apache.hadoop.hbase.client.HBaseAdmin@3e2055d6
            App -> tableExists('user') = true
            App -> tableExists('user')
            App -> deleteTable('user')

            2023-03-30 12:06:01.343  INFO 7989 --- [           main] o.apache.hadoop.hbase.client.HBaseAdmin  : Started disable of user
            2023-03-30 12:06:01.736  INFO 7989 --- [           main] o.apache.hadoop.hbase.client.HBaseAdmin  : Operation: DISABLE, Table Name: default:user, procId: 1135 completed
            2023-03-30 12:06:02.077  INFO 7989 --- [           main] o.apache.hadoop.hbase.client.HBaseAdmin  : Operation: DELETE, Table Name: default:user, procId: 1138 completed
            App -> createTable('user', ...)
            2023-03-30 12:06:02.812  INFO 7989 --- [           main] o.apache.hadoop.hbase.client.HBaseAdmin  : Operation: CREATE, Table Name: default:user, procId: 1139 completed

            App -> insertRecords('user', ... )
            App -> scanTable('user')
            row1    cf1     age     12
            row1    cf1     name    Tom
            row1    cf2     job     Student
            row1    cf3     gender  Male
            row2    cf1     age     9
            row2    cf1     name    Jerry
            row2    cf2     job     Student2
            row2    cf3     gender  Female

            App -> getValue('user', 'row1', 'cf3', 'gender'): value = Male
            App -> deleteColumn('user', 'row1', 'cf3', 'gender')
            App -> getRow('user', 'row1')
            row1    cf1     age     12
            row1    cf1     name    Tom
            row1    cf2     job     Student

            App -> deleteColumnFamily('user', 'row1', 'cf2')
            App -> scanRow('user', 'row1')
            row1    cf1     age     12
            row1    cf1     name    Tom

            App -> deleteRow('user', 'row1')
            App -> scanTable('user')
            row2    cf1     age     9
            row2    cf1     name    Jerry
            row2    cf2     job     Student2
            row2    cf3     gender  Female

            2023-03-30 12:06:03.063  INFO 7989 --- [           main] o.a.h.h.client.ConnectionImplementation  : Closing master protocol: MasterService
        


        注:程序连接 HBase 是通过从 ZooKeeper 取到配置后再进行的,程序里 Zookeeper 的配置信息如下:
        
                hbaseConf.set("hbase.zookeeper.quorum", "hadoop-master-vm");
                hbaseConf.set("hbase.zookeeper.property.clientPort", "2182");
                hbaseConf.set("zookeeper.znode.parent", "/hbase");

            这里 hbase.zookeeper.quorum 使用了 Zookeeper 所在的主机名 hadoop-master-vm, 可以替换成 Zookeeper 所在主机的 IP 地址。
            
            使用主机名 hadoop-master-vm 时,要注意它不是 Public 域名,如果程序不在 Zookeeper 所在主机上运行,解析域名找不到对应的 IP 地址,需要把 hadoop-master-vm 写入主机映射列表。本文 IDEA 运行在 Win 10 下,所以修改 C:\Windows\System32\drivers\etc\hosts 文件,添加如下内容:

                    ...

                    192.168.1.5      hadoop-master-vm
                       
    3) 打包 jar

        菜单 View -> Tool Windows -> Maven -> SpringbootExample21 -> Lifecycle -> Clean & Package

        jar 包生成在目录 target/ 里

            SpringbootExample21-1.0-SNAPSHOT.jar
            SpringbootExample21-1.0-SNAPSHOT.jar.original  

        SpringbootExample21.jar 包含依赖包,可以直接运行。 SpringbootExample21.jar.original 里不包含依赖的包(要手动配置依赖环境),运行前要把文件名上的 “.original” 去掉。
     
        运行如下命令:

            $ java -jar SpringbootExample21-1.0-SNAPSHOT.jar

                ...


posted @ 2023-04-01 12:13  垄山小站  阅读(1048)  评论(0)    收藏  举报