配电网监测数据微批处理的血统链标记容错法
|
摘要
针对分布式配电自动化系统存在数据量井喷、海量监测数据缺乏高效的分布式故障容错机制的问题,提出一种血统链标记容错新方法。利用弹性分布式数据集、微批计算的记录级容错和血统链标记序列融合处理的设计技巧,实现了分布式数据容错中血统链的追溯和条件标记的自动选择。以铁路配电网监测采集的数据为算例,搭建了4机集群的调度监控平台进行容错测试。以发生频次最高的单数据节点故障为例,测试结果表明:对于包含3×10~6条监测数据记录的弹性分布式数据集,血统链标记容错模型的集群CPU平均占用率波动小于1.5%,磁盘占用率下降4.2%;当迭代次数达到600、800次时,迭代运算耗时分别降低24.3%和42.9%;所提方法实现500 ms流处理延时的同时,对集群资源的使用情况也具有较好的优化效果,验证了该方法对分布式集群容错的有效性。
Aiming at the existing problems of monitoring message blowout and lacking of efficient distributed fault-tolerant mechanism for huge amounts of monitoring data in the distributed distribution automation system,a novel li-neage chain mark fault-tolerant method is proposed. The retrospection of lineage chain and the automatic selection of conditional marks in the distributed data fault-tolerant progress are realized by applying the design technique of fusing the resilient distributed data set,the record-level fault-tolerant of micro-batching computing and the lineage chain mark sequence. The monitoring data of railway distribution network are taken as the simulation example and the dispatch monitoring platform of a 4-generator cluster is built to carry out the fault-tolerant test. A single-node fault with the highest frequency is taken as an example,whose test results show that,for the resilient distributed data set including 3×10~6 monitoring data records,the average CPU utilization rate fluctuation is less than 1.5% and the disk utilization rate decreases by 4.2%;when the number of iteration increases to 600 and 800,the time-consuming decreases by 24.3% and 42.9% respectively;the proposed method not only realizes the stream processing delay of 500 ms,but also has a better optimization effect on the usage of cluster resources,verifying the effectiveness of the method for distributed cluster fault-tolerant.
Aiming at the existing problems of monitoring message blowout and lacking of efficient distributed fault-tolerant mechanism for huge amounts of monitoring data in the distributed distribution automation system,a novel li-neage chain mark fault-tolerant method is proposed. The retrospection of lineage chain and the automatic selection of conditional marks in the distributed data fault-tolerant progress are realized by applying the design technique of fusing the resilient distributed data set,the record-level fault-tolerant of micro-batching computing and the lineage chain mark sequence. The monitoring data of railway distribution network are taken as the simulation example and the dispatch monitoring platform of a 4-generator cluster is built to carry out the fault-tolerant test. A single-node fault with the highest frequency is taken as an example,whose test results show that,for the resilient distributed data set including 3×10~6 monitoring data records,the average CPU utilization rate fluctuation is less than 1.5% and the disk utilization rate decreases by 4.2%;when the number of iteration increases to 600 and 800,the time-consuming decreases by 24.3% and 42.9% respectively;the proposed method not only realizes the stream processing delay of 500 ms,but also has a better optimization effect on the usage of cluster resources,verifying the effectiveness of the method for distributed cluster fault-tolerant.
引文
[1] 邓建玲. 能源互联网的概念及发展模式[J]. 电力自动化设备,2016,36(3):1-5. DENG Jianling. Concept of energy internet and its development modes[J]. Electric Power Automation Equipment,2016,36(3):1-5.
[2] 胡丽娟,刁赢龙,刘科研,等. 基于大数据技术的配电网运行可靠性分析[J]. 电网技术,2017,41(1):265-271.HU Lijuan,DIAO Yinglong,LIU Keyan,et al. Operational reliability analysis of distribution network based on big data technology[J]. Power System Technology,2017,41(1):265-271.
[3] 葛磊蛟,王守相,瞿海妮. 智能配用电大数据存储架构设计[J]. 电力自动化设备,2016,36(6):194-202. GE Leijiao,WANG Shouxiang,QU Haini. Design of storage framework for big data of SPDU[J]. Electric Power Automation Equipment,2016,36(6):194-202.
[4] 曲广龙,杨洪耕,熊晓祎. 可在线编辑电网地理接线信息的电能质量监测系统设计与实现[J]. 电力自动化设备,2015,35(11):165-174.QU Guanglong,YANG Honggeng,XIONG Xiaoyi. Design and realization of power quality monitoring system with online editable geographical wiring information of power grid[J]. Electric Power Automation Equipment,2015,35(11):165-174.
[5] 邵振国,吴瑾樱,苏文博. 面向海量历史监测数据的谐波污染用户统计建模方法[J]. 电力自动化设备,2016,36(8):110-114.SHAO Zhenguo,WU Jinying,SU Wenbo. Statistical modeling based on massive historical monitoring data for harmonic pollution customer[J]. Electric Power Automation Equipment,2016,36(8):110-114.
[6] 李洋,蔡志远,刘海涛,等. 广域分布式电源运营管控系统[J]. 电力自动化设备,2018,38(3):102-108.LI Yang,CAI Zhiyuan,LIU Haitao,et al. Management and control system of wide-area distributed generation[J]. Electric Power Automation Equipment,2018,38(3):102-108.
[7] 屈志坚,徐振清,周锐霖,等. 按列存储的配电网监测数据包区间编码正规化压缩处理[J]. 电力自动化设备,2018,38(3):56-62.QU Zhijian,XU Zhenqing,ZHOU Ruilin,et al. Range encoding nor-malization compression processing for distribtion network monitoring data pack with column storage[J]. Electric Power Automation Equipment,2018,38(3):56-62.
[8] 曲广龙,杨洪耕,李兰芳. 主动配电网电能质量实时监测系统设计与实现[J]. 电力系统自动化,2015,39(10):117-123.QU Guanglong,YANG Honggeng,LI Lanfang. Design and realization of power quality real-time monitoring system for active distribution networks[J]. Automation of Electric Power Systems,2015,39(10):117-123.
[9] 宋墩文,温渤婴,杨学涛,等. 广域量测信息大数据特征分析及应用策略[J]. 电网技术,2017,41(1):157-163.SONG Dunwen,WEN Boying,YANG Xuetao,et al. Big data feature analysis and application strategy of wide area measurement information[J]. Power System Technology,2017,41(1):157-163.
[10] 张宇帆,艾芊,肖斐,等. 数据驱动电能质量分析现状及其支撑技术与展望[J]. 电力自动化设备,2018,38(11):187-196.ZHANG Yufan,AI Qian,XIAO Fei,et al. Present situation,suppor-ting technologies and prospect of data driven power quality analysis[J]. Electric Power Automation Equipment,2018,38(11):187-196.
[11] BOUGUERRA M S,TRYSTRAM D,WAGNER F. Complexity ana-lysis of checkpoint scheduling with variable costs[J]. IEEE Tran-sactions on Computers,2013,62(6):1269-1275.
[12] WANG H,CHEN H P,DU Z W,et al. BeTL:MapReduce check-point tactics beneath the task level[J]. IEEE Transactions on Ser-vices Computing,2017,9(1):84-95.
[13] HE Z Z,CHEN Y J,MEN C G,et al. Schedulability of fault tolerant real time system based on local optimum checkpoint under priority mixed strategy[J]. Chinese Journal of Electronics,2015,24(2):236-244.
[14] 王守相,张卫涛,葛磊蛟. 智能配电网统一模型云与应用服务技术系统[J]. 电力自动化设备,2015,35(2):49-54. WANG Shouxiang,ZHANG Weitao,GE Leijiao. Unified model cloud and application service system for smart distribution network[J]. Electric Power Automation Equipment,2015,35(2):49-54.
[15] 范贵生,虞慧群,陈丽琼,等. 基于效用的云计算容错策略和模型[J]. 中国科学(信息科学),2014,44(1):158-176.FAN Guisheng,YU Huiqun,CHEN Liqiong,et al. Utility-based fault tolerant strategy and model for cloud computing[J]. Scientia Sinica(Informationis),2014,44(1):158-176.
[16] 张治斌,李燕歌. 云计算下MapReduce多组容错机制架构的分析与研究[J]. 微电子学与计算机,2014,31(1):52-55,59.ZHANG Zhibin,LI Yange. Analysis and study on under cloud computing multiple sets of fault tolerance strategy architecture of Map-Reduce[J]. Microelectronics & Computer,2014,31(1):52-55,59.
[17] WEI H W,CHEN T Y,CHEN S H,et al. A QoS-aware data recons-truction strategy for a data fault-tolerant storage system[C]//2015 IEEE 7th International Conference on Cloud Computing Technology and Science(CloudCom). Vancouver,BC,Canada:IEEE,2015:274-281.
[18] SINGH S,LIU Y. A cloud service architecture for analyzing big monitoring data[J]. Tsinghua Science and Technology,2016,21(1):55-70.
[19] 王保义,王冬阳,张少敏. 基于Spark和IPPSO_LSSVM的短期分布式电力负荷预测算法[J]. 电力自动化设备,2016,36(1):117-122. WANG Baoyi,WANG Dongyang,ZHANG Shaomin. Distributed short-term load forecasting algorithm based on Spark and IIPPSO_LSSVM[J]. Electric Power Automation Equipment,2016,36(1):117-122.
[20] ALSHEIKH M A,NIYATO D,LIN S W,et al. Mobile big data analytics using deep learning and apache spark[J]. IEEE Network,2016,30(3):22-29.
[21] 刘健,赵树仁,张小庆. 中国配电自动化的进展及若干建议[J]. 电力系统自动化,2012,36(19):6-10,21.LIU Jian,ZHAO Shuren,ZHANG Xiaoqing. Development of distribution automation in China and some suggestions[J]. Automation of Electric Power Systems,2012,36(19):6-10,21.
[2] 胡丽娟,刁赢龙,刘科研,等. 基于大数据技术的配电网运行可靠性分析[J]. 电网技术,2017,41(1):265-271.HU Lijuan,DIAO Yinglong,LIU Keyan,et al. Operational reliability analysis of distribution network based on big data technology[J]. Power System Technology,2017,41(1):265-271.
[3] 葛磊蛟,王守相,瞿海妮. 智能配用电大数据存储架构设计[J]. 电力自动化设备,2016,36(6):194-202. GE Leijiao,WANG Shouxiang,QU Haini. Design of storage framework for big data of SPDU[J]. Electric Power Automation Equipment,2016,36(6):194-202.
[4] 曲广龙,杨洪耕,熊晓祎. 可在线编辑电网地理接线信息的电能质量监测系统设计与实现[J]. 电力自动化设备,2015,35(11):165-174.QU Guanglong,YANG Honggeng,XIONG Xiaoyi. Design and realization of power quality monitoring system with online editable geographical wiring information of power grid[J]. Electric Power Automation Equipment,2015,35(11):165-174.
[5] 邵振国,吴瑾樱,苏文博. 面向海量历史监测数据的谐波污染用户统计建模方法[J]. 电力自动化设备,2016,36(8):110-114.SHAO Zhenguo,WU Jinying,SU Wenbo. Statistical modeling based on massive historical monitoring data for harmonic pollution customer[J]. Electric Power Automation Equipment,2016,36(8):110-114.
[6] 李洋,蔡志远,刘海涛,等. 广域分布式电源运营管控系统[J]. 电力自动化设备,2018,38(3):102-108.LI Yang,CAI Zhiyuan,LIU Haitao,et al. Management and control system of wide-area distributed generation[J]. Electric Power Automation Equipment,2018,38(3):102-108.
[7] 屈志坚,徐振清,周锐霖,等. 按列存储的配电网监测数据包区间编码正规化压缩处理[J]. 电力自动化设备,2018,38(3):56-62.QU Zhijian,XU Zhenqing,ZHOU Ruilin,et al. Range encoding nor-malization compression processing for distribtion network monitoring data pack with column storage[J]. Electric Power Automation Equipment,2018,38(3):56-62.
[8] 曲广龙,杨洪耕,李兰芳. 主动配电网电能质量实时监测系统设计与实现[J]. 电力系统自动化,2015,39(10):117-123.QU Guanglong,YANG Honggeng,LI Lanfang. Design and realization of power quality real-time monitoring system for active distribution networks[J]. Automation of Electric Power Systems,2015,39(10):117-123.
[9] 宋墩文,温渤婴,杨学涛,等. 广域量测信息大数据特征分析及应用策略[J]. 电网技术,2017,41(1):157-163.SONG Dunwen,WEN Boying,YANG Xuetao,et al. Big data feature analysis and application strategy of wide area measurement information[J]. Power System Technology,2017,41(1):157-163.
[10] 张宇帆,艾芊,肖斐,等. 数据驱动电能质量分析现状及其支撑技术与展望[J]. 电力自动化设备,2018,38(11):187-196.ZHANG Yufan,AI Qian,XIAO Fei,et al. Present situation,suppor-ting technologies and prospect of data driven power quality analysis[J]. Electric Power Automation Equipment,2018,38(11):187-196.
[11] BOUGUERRA M S,TRYSTRAM D,WAGNER F. Complexity ana-lysis of checkpoint scheduling with variable costs[J]. IEEE Tran-sactions on Computers,2013,62(6):1269-1275.
[12] WANG H,CHEN H P,DU Z W,et al. BeTL:MapReduce check-point tactics beneath the task level[J]. IEEE Transactions on Ser-vices Computing,2017,9(1):84-95.
[13] HE Z Z,CHEN Y J,MEN C G,et al. Schedulability of fault tolerant real time system based on local optimum checkpoint under priority mixed strategy[J]. Chinese Journal of Electronics,2015,24(2):236-244.
[14] 王守相,张卫涛,葛磊蛟. 智能配电网统一模型云与应用服务技术系统[J]. 电力自动化设备,2015,35(2):49-54. WANG Shouxiang,ZHANG Weitao,GE Leijiao. Unified model cloud and application service system for smart distribution network[J]. Electric Power Automation Equipment,2015,35(2):49-54.
[15] 范贵生,虞慧群,陈丽琼,等. 基于效用的云计算容错策略和模型[J]. 中国科学(信息科学),2014,44(1):158-176.FAN Guisheng,YU Huiqun,CHEN Liqiong,et al. Utility-based fault tolerant strategy and model for cloud computing[J]. Scientia Sinica(Informationis),2014,44(1):158-176.
[16] 张治斌,李燕歌. 云计算下MapReduce多组容错机制架构的分析与研究[J]. 微电子学与计算机,2014,31(1):52-55,59.ZHANG Zhibin,LI Yange. Analysis and study on under cloud computing multiple sets of fault tolerance strategy architecture of Map-Reduce[J]. Microelectronics & Computer,2014,31(1):52-55,59.
[17] WEI H W,CHEN T Y,CHEN S H,et al. A QoS-aware data recons-truction strategy for a data fault-tolerant storage system[C]//2015 IEEE 7th International Conference on Cloud Computing Technology and Science(CloudCom). Vancouver,BC,Canada:IEEE,2015:274-281.
[18] SINGH S,LIU Y. A cloud service architecture for analyzing big monitoring data[J]. Tsinghua Science and Technology,2016,21(1):55-70.
[19] 王保义,王冬阳,张少敏. 基于Spark和IPPSO_LSSVM的短期分布式电力负荷预测算法[J]. 电力自动化设备,2016,36(1):117-122. WANG Baoyi,WANG Dongyang,ZHANG Shaomin. Distributed short-term load forecasting algorithm based on Spark and IIPPSO_LSSVM[J]. Electric Power Automation Equipment,2016,36(1):117-122.
[20] ALSHEIKH M A,NIYATO D,LIN S W,et al. Mobile big data analytics using deep learning and apache spark[J]. IEEE Network,2016,30(3):22-29.
[21] 刘健,赵树仁,张小庆. 中国配电自动化的进展及若干建议[J]. 电力系统自动化,2012,36(19):6-10,21.LIU Jian,ZHAO Shuren,ZHANG Xiaoqing. Development of distribution automation in China and some suggestions[J]. Automation of Electric Power Systems,2012,36(19):6-10,21.