-
The problem of Object inheritance in scala
:
class Test{}
object Driver extends Test{
}
:
abstract Fruit{
val name :String
val color :String
}
object Fruit{
object Apple extends Fruit("apple","red")
val menu=List(Apple)
}
question: (1) Why object in scala can al...
-
Why can't you access the properties of an object in the method of spark: scala?
how to understand the content of the green part? Why does it feel so awkward? the feeling in the book is also very vague.
...
-
Syntax understanding of scala function
override def generateJob(time: Time): Option[Job] = {
parent.getOrCompute(time) match {
case Some(rdd) =>
val jobFunc = () => createRDDWithLocalProperties(time, displayInnerRDDOps) {
foreachFunc(rdd, time)
}
...
-
On the understanding of scala Grammar in Spark
val lines: Dataset[String] = session.read.textFile("")
val words: Dataset[String] = lines.flatMap(_.split(" "))
linesdataSetflatMapdataSetIDEAflatMap:
def flatMap[U : Encoder](func: T => Traversabl...
-
How does Scala get all the child objects that inherit trait
trait Base
object A extends Base
object B extends Base
object C extends Base
class D extents Base
object Base {
TODO trait Base object
TODO akka actor event Base
}
when using akka to throw a Event event, I want to in...
-
Kafka.common.KafkaException: Wrong request type 18
use a simple java client to send a message after simulating the configuration of a kafka server, but without message storage, you can only see the error log all the time (the message is really not stored)
kafka.common.KafkaException: Wrong request type...
-
There is a small problem encountered in data analysis and data extraction. The newcomer is looking for an answer.
Today, I was working on a machine learning project and encountered a small problem. I want to extract a column of data using the regular expression and then run it and find that there is no name1 data. This symbol is very puzzling for the Great God s ...
-
Can UDF in sparkSQL use query statements to pass parameters
used sparkSQL to write a UDF, to calculate commission based on performance, but now the demand has changed. The percentage of commission is determined according to position and department. Here s an example
spark.udf.register("mmyjtc", (yj: ...
-
Wrong footnote about hadoop running the word count written by python
this is the article I refer to. The program that runs according to this article https: blog.csdn.net wangato., as shown in the picture, always reports the error in the first sentence. I Baidu has the reason, but has not been able to solve it. 1. Py...
-
Spark sql parses the json of an array of nested objects
1. Json data is now available as follows
{ "id ": 11, "data ": [{ "package ": "com.browser1 ", "activetime ": 60000}, { "package ": "com.browser6 ", "activetime ": 1205000}, { "package ": "com.browser7 ", "activetime ": 1205000}]}
{ "id ": 12...
-
Why does the first gender print out null?
topic description
Why does the first gender print out null
sources of topics and their own ideas
I ve tried val gender: String = "male " , but the results don t seem to be right.
related codes
constructor
zhu
null
derived constructor
zhu
zz...
-
Usage of configuration files in spark project
problem description
sparksql project, the sql script is placed in the resource sql file below (different businesses, there are a lot of scripts); Local write code to load the sql script using this.getClass.getResource (). GetPath method, get the pa...
-
Is the Scala Seq prompt used in Spark Dataframe join not serialized?
I want to use the multi-field join function of dataframe in java spark-sql. Take a look at this interface. If you want to have multiple fields join, you need to pass in a usingColumns. .
public org.apache.spark.sql.DataFrame join(org.apache.spark.sql.Da...
-
How does intellij modify the source code of maven jar packages, what plug-ins or other ways do you use?
how does intellij modify the source code of maven jar packages, what plug-ins or other ways do you use ...
-
Regular expression memory overflow problem
problem description
regular expression memory overflow, JVM can t check memory, accumulate all the time, and then the program hangs
the environmental background of the problems and what methods you have tried
delete the second set of parentheses ...
-
Spark2.1 does cache to the temporary watch, but it doesn't work.
problem description
because the temporary table created by DF needs to be queried many times, so the temporary table is cached, but it doesn t work? the amount of data is relatively large, about 10 billion, so it is necessary to optimize the efficie...
-
Why does the sortBy function of Spark generate 4 MapPartitionsRDD?
execute two programs in spark-shell: first paragraph sortBy:
val list1: List[(String, Int)] = List(("the", 12), ("they", 2), ("do", 4), ("wild", 1), ("and", 5), ("into", 4))
val listRDD1: RDD...
-
An algorithm problem
question: there are two actions for a user to invest and refund (for example, if An invests 100000 yuan, then he will get a rebate of 10 yuan after maturity). When the user does not make any investment within 6 months, the next investment will be regard...