随笔分类 -  Spark学习

spark小文件合并解决多级分区
摘要：package spark10.access import java.text.SimpleDateFormatimport java.util.Date import org.apache.hadoop.fs.{FileSystem, Path}import org.apache.log4j.Lo 阅读全文
posted @ 2019-04-12 11:09 何国秀_xue 阅读(1209) 评论(0) 推荐(0)
spark解决josn乱码问题
摘要：package test import org.apache.hadoop.io.{LongWritable, Text}import org.apache.hadoop.mapred.TextInputFormatimport org.apache.spark.sql.hive.HiveConte 阅读全文
posted @ 2019-04-12 09:03 何国秀_xue 阅读(586) 评论(0) 推荐(0)
spark解决text乱码问题
摘要：package test import org.apache.hadoop.io.{LongWritable, Text}import org.apache.hadoop.mapred.TextInputFormatimport org.apache.spark.{SparkConf, SparkC 阅读全文
posted @ 2019-04-12 09:03 何国秀_xue 阅读(1793) 评论(0) 推荐(0)
spark保存数据到hdfsJ及hive
摘要：package spark88 import org.apache.spark.sql.{DataFrame, Row, SQLContext, SaveMode}import org.apache.spark.sql.hive.HiveContextimport org.apache.spark. 阅读全文
posted @ 2019-04-11 17:24 何国秀_xue 阅读(1873) 评论(0) 推荐(0)
spark小文件合并
摘要：package spark99 import java.io.IOException import org.apache.hadoop.fs.{FileSystem, Path}import org.apache.log4j.Loggerimport org.apache.spark.sql.hiv 阅读全文
posted @ 2019-04-11 17:23 何国秀_xue 阅读(3381) 评论(0) 推荐(0)
sparkstreaming消费direct
摘要：package two /** * Created by on 上午2:19. */import kafka.serializer.StringDecoder import org.apache.spark.streaming._import org.apache.spark.streaming.k 阅读全文
posted @ 2018-12-20 15:30 何国秀_xue 阅读(213) 评论(0) 推荐(0)
sparkstreaming消费receive
摘要：package two /** * Created by zhoucw on 上午2:11. */import java.util.HashMap import org.apache.kafka.clients.producer.{ProducerConfig, KafkaProducer, Pro 阅读全文
posted @ 2018-12-20 15:29 何国秀_xue 阅读(245) 评论(0) 推荐(0)
数据零丢失kafka + checkpoint
摘要：package kafkautils /** * Created on 上午12:48. * * High level comsumer api * * low level comsumer api(simple comsumer api) * * */import kafka.serializer 阅读全文
posted @ 2018-12-20 15:27 何国秀_xue 阅读(1453) 评论(0) 推荐(0)
数据零丢失kafka + zookeeper
摘要：package kafkautils import kafka.ZookeeperHelper.clientimport kafka.common.TopicAndPartitionimport kafka.message.MessageAndMetadataimport kafka.seriali 阅读全文
posted @ 2018-12-20 15:25 何国秀_xue 阅读(630) 评论(0) 推荐(0)
数据零丢失kafka + hbase
摘要：package kafkautils import kafka.message.MessageAndMetadataimport kafka.serializer.StringDecoderimport kafka.utils.ZkUtilsimport kafkautils.KafkaZKMana 阅读全文
posted @ 2018-12-20 15:24 何国秀_xue 阅读(480) 评论(0) 推荐(0)
sparkstreaming + kafka + zookeeper + hbase整体发布说明
摘要：一、sparkstreaming + kafka + zookeeper + hbase整体发布说明 阅读全文
posted @ 2018-12-18 09:58 何国秀_xue 阅读(141) 评论(0) 推荐(0)
SparkSQL
摘要：package com.bnls.test.parseimport org.apache.spark.{SparkConf, SparkContext}import org.apache.spark.sql.hive.HiveContextimport org.apache.spark.stream 阅读全文
posted @ 2018-12-04 15:20 何国秀_xue 阅读(143) 评论(0) 推荐(0)
sparkstreaming + kafka + zookeeper + hbase
摘要：package com.bnls.test.sparkimport com.bnls.test.common.ZookeeperHelperimport com.bnls.test.hbase.HbaseHelperimport org.apache.spark.streaming.dstream. 阅读全文
posted @ 2018-12-04 15:19 何国秀_xue 阅读(396) 评论(0) 推荐(0)
zookeeper
摘要：package com.bnls.test.commonimport kafka.common.TopicAndPartitionimport org.apache.curator.framework.CuratorFrameworkFactoryimport org.apache.curator. 阅读全文
posted @ 2018-12-04 15:18 何国秀_xue 阅读(246) 评论(0) 推荐(0)
Kafka：无丢失提取kafka的值，详解kafka的消费过程
摘要：目录： 1、需求 2、代码步鄹 3、代码展现 4、pom.xml文件 5、结果展现 ——————————————————————————————————– 1、需求 前提：将org.apache.spark.streaming.kafka.KafkaCluster这个类抽出来变成KafkaClust 阅读全文
posted @ 2018-12-04 14:04 何国秀_xue 阅读(296) 评论(0) 推荐(0)