I just used spark streaming. I have a few questions about checkpoint: There are two types of checkpoint, one for meta for driver and one for data. The manual says that the checkpoint of data will be written only if you use stateful transformation. So...
the following code implements tagging the data. After a carPlate appears (set to 2 for the first time), the mark is invalid within 1 hour (set to 1), the first one for more than 1 hour is set to 2, and so on. the main problem is that I have several ta...
as shown in the figure, the following problems occurred during the installation of es. I set the folders to 777 permissions and gave es user permissions. Why do I still report insufficient permissions during execution? Look up on the Internet, it is all ...
use jdbc:phoenix:testdmp3.fengdai.org,testdmp4.fengdai.org,testdmp5.fengdai.org:2181: hbase-secure:dcp@EXPER.ORG: "+ classPath + " dcp.keytab to create a connection, which will result in a long period of non-execution, followed by the following error...
now I m going to implement a life game in a serial way, which is on a 100000 x 100000 chessboard and spawns for 500 generations. I chose to use list to form a two-dimensional array to hold the data on the chessboard, but in the middle of the program, I...
from pyecharts import Pie attr = [ 2dlots, 3d ] v1 = [12,5] pie = Pie ( "Chinese 2017 movie visual type ratio ", title_pos= left ,title_text_size=15,width=1000,height=350) pie.add ( ", attr,v1,center= [50,60], radius= [0,80], is_label_show=Tr...
< H1 > problem description < H1 > I installed CDH6, on three virtual machines to install agent in the web interface, waiting for the newly installed Agent detection signal. This step waited for about 1 minute, prompting the following error: Agent ...
...
purpose: there are two large pieces of data in spark that require join,. Both input data contain the field userid. Now you need to associate them according to userid. I hope to avoid shuffle. completed: I pre-processed two pieces of data into 1w f...
1. Json data is now available as follows { "id ": 11, "data ": [{ "package ": "com.browser1 ", "activetime ": 60000}, { "package ": "com.browser6 ", "activetime ": 1205000}, { "package ": "com.browser7 ", "activetime ": 1205000}]} { "id ": 12...
< H2 > Business scenario < H2 > A large number of json files need to be read and re-parsed and imported into elasticsearch . Json files are saved in different date folders. The size of a single folder is about 80g. The number of json files under the ...
the interviewer asked you a very large English text to count the word frequency, what algorithm should be used and how to answer it? ...
use a simple java client to send a message after simulating the configuration of a kafka server, but without message storage, you can only see the error log all the time (the message is really not stored) kafka.common.KafkaException: Wrong request type...
there are two files, A.txt and B.txt. there are 3000w rows of data in A.txt. The contents separated by spaces between id and username are as follows: id usernmae 1 zhangsan 2 lisi ...... there are 3000w rows of data in B.txt. The contents sep...