摘要：
Shuffle简介 Shuffle描述着数据从map task输出到reduce task输入的这段过程。shuffle是连接Map和Reduce之间的桥梁，Map的输出要用到Reduce中必须经过shuffle这个环节，shuffle的性能高低直接影响了整个程序的性能和吞吐量。因为在分布式情况下， 阅读全文
posted @ 2022-09-04 00:24
Avery_rainys
阅读(121)
评论(0)
推荐(0)
摘要：
Spark任务调度机制论述 在生产环境下，Spark集群的部署方式一般为YARN-Cluster模式。 Driver线程主要是初始化SparkContext对象，准备运行所需的上下文，然后一方面保持与ApplicationMaster的RPC连接，通过ApplicationMaster申请资源，另一 阅读全文
posted @ 2022-09-03 16:35
Avery_rainys
阅读(435)
评论(0)
推荐(0)
摘要：
{ "job": { "content": [ { "reader": { "name": "mysqlreader", "parameter": { "column": ["id","name","region_id","area_code","iso_code","iso_3166_2"], " 阅读全文
posted @ 2022-08-26 22:46
Avery_rainys
阅读(243)
评论(0)
推荐(0)
摘要：
# Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this wor 阅读全文
posted @ 2022-08-26 22:38
Avery_rainys
阅读(189)
评论(0)
推荐(0)
摘要：
HDFS Sink # sink1 a3.sinks.k1.type = hdfs a3.sinks.k1.hdfs.path = /origin_data/gmall/db/business_db/inc/%{tableName}_inc/%Y-%m-%d a3.sinks.k1.hdfs.fil 阅读全文
posted @ 2022-08-26 22:33
Avery_rainys
阅读(44)
评论(0)
推荐(0)
摘要：
Kafka Channel 生产者 a1.channels.c1.type = org.apache.flume.channel.kafka.KafkaChannel a1.channels.c1.kafka.bootstrap.servers = hadoop102:9092,hadoop103: 阅读全文
posted @ 2022-08-26 22:31
Avery_rainys
阅读(42)
评论(0)
推荐(0)
摘要：
Kafka Sources a3.sources.r1.type = org.apache.flume.source.kafka.KafkaSource a3.sources.r1.batchSize = 5000 a3.sources.r1.batchDurationMillis = 2000 a 阅读全文
posted @ 2022-08-26 22:27
Avery_rainys
阅读(28)
评论(0)
推荐(0)
摘要：
<?xml version="1.0"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the Lic 阅读全文
posted @ 2022-08-26 22:23
Avery_rainys
阅读(237)
评论(0)
推荐(0)
摘要：
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); 阅读全文
posted @ 2022-08-26 22:23
Avery_rainys
阅读(178)
评论(0)
推荐(0)
摘要：
<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Licensed under the Apache License, Version 2.0 ( 阅读全文
posted @ 2022-08-26 22:22
Avery_rainys
阅读(83)
评论(0)
推荐(0)
浙公网安备 33010602011771号