[2023年06月18日] 手に入れよう！最新Associate-Developer-Apache-Spark認定された有効な試験問題集解答
100%合格率保証付きの素晴らしいAssociate-Developer-Apache-Spark試験問題PDF
Databricks Associate-Developer-Apache-Spark 認定試験は、Apache Spark に熟練した開発者を対象としており、そのスキルと知識を検証したいと考えている人々に適しています。この試験は、データサイエンティスト、エンジニア、およびアナリストのための共同作業スペースを提供する統合データ分析プラットフォームである Databricks によって提供されています。この認定試験は、開発者が Apache Spark の専門知識を検証し、プラットフォームを使用してデータ処理パイプライン、データ分析アプリケーション、および機械学習モデルを構築する能力を示すのに役立ちます。
この試験は、データ処理、データ分析、データ可視化にApache Sparkを使用する候補者の知識とスキルをテストするために設計されています。試験は、Sparkアーキテクチャ、データフレームとデータセット、Spark SQL、Sparkストリーミング、グラフ処理、機械学習、Sparkパフォーマンスチューニングなど、幅広いトピックをカバーしています。試験は60問の複数選択肢問題で構成され、受験者は90分で試験を受けることができます。試験に合格するには、少なくとも70％のスコアを取得する必要があります。成功した受験者は、データ重視のアプリケーションを構築するためにApache Sparkを使用する能力を示す証明書を受け取ります。
質問 # 66
Which of the following code blocks selects all rows from DataFrame transactionsDf in which column productId is zero or smaller or equal to 3?
- A. transactionsDf.filter((col("productId")==3) or (col("productId")<1))
- B. transactionsDf.filter((col("productId")==3) | (col("productId")<1))
- C. transactionsDf.filter(productId==3 or productId<1)
- D. transactionsDf.where("productId"=3).or("productId"<1))
- E. transactionsDf.filter(col("productId")==3 | col("productId")<1)
正解：B
解説：
Explanation
This question targets your knowledge about how to chain filtering conditions. Each filtering condition should be in parentheses. The correct operator for "or" is the pipe character (|) and not the word or. Another operator of concern is the equality operator. For the purpose of comparison, equality is expressed as two equal signs (==).
Static notebook | Dynamic notebook: See test 2
質問 # 67
Which of the following code blocks creates a new 6-column DataFrame by appending the rows of the
6-column DataFrame yesterdayTransactionsDf to the rows of the 6-column DataFrame todayTransactionsDf, ignoring that both DataFrames have different column names?
- A. todayTransactionsDf.unionByName(yesterdayTransactionsDf, allowMissingColumns=True)
- B. todayTransactionsDf.unionByName(yesterdayTransactionsDf)
- C. union(todayTransactionsDf, yesterdayTransactionsDf)
- D. todayTransactionsDf.concat(yesterdayTransactionsDf)
- E. todayTransactionsDf.union(yesterdayTransactionsDf)
正解：E
解説：
Explanation
todayTransactionsDf.union(yesterdayTransactionsDf)
Correct. The union command appends rows of yesterdayTransactionsDf to the rows of todayTransactionsDf, ignoring that both DataFrames have different column names. The resulting DataFrame will have the column names of DataFrame todayTransactionsDf.
todayTransactionsDf.unionByName(yesterdayTransactionsDf)
No. unionByName specifically tries to match columns in the two DataFrames by name and only appends values in columns with identical names across the two DataFrames. In the form presented above, the command is a great fit for joining DataFrames that have exactly the same columns, but in a different order. In this case though, the command will fail because the two DataFrames have different columns.
todayTransactionsDf.unionByName(yesterdayTransactionsDf, allowMissingColumns=True) No. The unionByName command is described in the previous explanation. However, with the allowMissingColumns argument set to True, it is no longer an issue that the two DataFrames have different column names. Any columns that do not have a match in the other DataFrame will be filled with null where there is no value. In the case at hand, the resulting DataFrame will have 7 or more columns though, so it this command is not the right answer.
union(todayTransactionsDf, yesterdayTransactionsDf)
No, there is no union method in pyspark.sql.functions.
todayTransactionsDf.concat(yesterdayTransactionsDf)
Wrong, the DataFrame class does not have a concat method.
More info: pyspark.sql.DataFrame.union - PySpark 3.1.2 documentation,
pyspark.sql.DataFrame.unionByName - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 3
質問 # 68
Which of the following code blocks returns a 2-column DataFrame that shows the distinct values in column productId and the number of rows with that productId in DataFrame transactionsDf?
- A. transactionsDf.groupBy("productId").agg(col("value").count())
- B. transactionsDf.groupBy("productId").count()
- C. transactionsDf.count("productId").distinct()
- D. transactionsDf.groupBy("productId").select(count("value"))
- E. transactionsDf.count("productId")
正解：B
解説：
Explanation
transactionsDf.groupBy("productId").count()
Correct. This code block first groups DataFrame transactionsDf by column productId and then counts the rows in each group.
transactionsDf.groupBy("productId").select(count("value"))
Incorrect. You cannot call select on a GroupedData object (the output of a groupBy) statement.
transactionsDf.count("productId")
No. DataFrame.count() does not take any arguments.
transactionsDf.count("productId").distinct()
Wrong. Since DataFrame.count() does not take any arguments, this option cannot be right.
transactionsDf.groupBy("productId").agg(col("value").count())
False. A Column object, as returned by col("value"), does not have a count() method. You can see all available methods for Column object linked in the Spark documentation below.
More info: pyspark.sql.DataFrame.count - PySpark 3.1.2 documentation, pyspark.sql.Column - PySpark
3.1.2 documentation
Static notebook | Dynamic notebook: See test 3
質問 # 69
Which of the following code blocks reads all CSV files in directory filePath into a single DataFrame, with column names defined in the CSV file headers?
Content of directory filePath:
1._SUCCESS
2._committed_2754546451699747124
3._started_2754546451699747124
4.part-00000-tid-2754546451699747124-10eb85bf-8d91-4dd0-b60b-2f3c02eeecaa-298-1-c000.csv.gz
5.part-00001-tid-2754546451699747124-10eb85bf-8d91-4dd0-b60b-2f3c02eeecaa-299-1-c000.csv.gz
6.part-00002-tid-2754546451699747124-10eb85bf-8d91-4dd0-b60b-2f3c02eeecaa-300-1-c000.csv.gz
7.part-00003-tid-2754546451699747124-10eb85bf-8d91-4dd0-b60b-2f3c02eeecaa-301-1-c000.csv.gz spark.option("header",True).csv(filePath)
- A. spark.read().option("header",True).load(filePath)
- B. spark.read.load(filePath)
- C. spark.read.format("csv").option("header",True).load(filePath)
- D. spark.read.format("csv").option("header",True).option("compression","zip").load(filePath)
正解：C
解説：
Explanation
The files in directory filePath are partitions of a DataFrame that have been exported using gzip compression.
Spark automatically recognizes this situation and imports the CSV files as separate partitions into a single DataFrame. It is, however, necessary to specify that Spark should load the file headers in the CSV with the header option, which is set to False by default.
質問 # 70
Which of the following code blocks returns a DataFrame that has all columns of DataFrame transactionsDf and an additional column predErrorSquared which is the squared value of column predError in DataFrame transactionsDf?
- A. transactionsDf.withColumnRenamed("predErrorSquared", pow(predError, 2))
- B. transactionsDf.withColumn("predError", pow(col("predErrorSquared"), 2))
- C. transactionsDf.withColumn("predErrorSquared", pow(predError, lit(2)))
- D. transactionsDf.withColumn("predErrorSquared", "predError"**2)
- E. transactionsDf.withColumn("predErrorSquared", pow(col("predError"), lit(2)))
正解：E
解説：
Explanation
While only one of these code blocks works, the DataFrame API is pretty flexible when it comes to accepting columns into the pow() method. The following code blocks would also work:
transactionsDf.withColumn("predErrorSquared", pow("predError", 2))
transactionsDf.withColumn("predErrorSquared", pow("predError", lit(2))) Static notebook | Dynamic notebook: See test 1 (https://flrs.github.io/spark_practice_tests_code/#1/26.html ,
https://bit.ly/sparkpracticeexams_import_instructions)
質問 # 71
The code block shown below should show information about the data type that column storeId of DataFrame transactionsDf contains. Choose the answer that correctly fills the blanks in the code block to accomplish this.
Code block:
transactionsDf.__1__(__2__).__3__
- A. 1. select
2. "storeId"
3. printSchema() - B. 1. limit
2. 1
3. columns - C. 1. select
2. "storeId"
3. print_schema() - D. 1. select
2. storeId
3. dtypes - E. 1. limit
2. "storeId"
3. printSchema()
正解：B
解説：
Explanation
Correct code block:
transactionsDf.select("storeId").printSchema()
The difficulty of this question is that it is hard to solve with the stepwise first-to-last-gap approach that has worked well for similar questions, since the answer options are so different from one another. Instead, you might want to eliminate answers by looking for patterns of frequently wrong answers.
A first pattern that you may recognize by now is that column names are not expressed in quotes. For this reason, the answer that includes storeId should be eliminated.
By now, you may have understood that the DataFrame.limit() is useful for returning a specified amount of rows. It has nothing to do with specific columns. For this reason, the answer that resolves to limit("storeId") can be eliminated.
Given that we are interested in information about the data type, you should question whether the answer that resolves to limit(1).columns provides you with this information. While DataFrame.columns is a valid call, it will only report back column names, but not column types. So, you can eliminate this option.
The two remaining options either use the printSchema() or print_schema() command. You may remember that DataFrame.printSchema() is the only valid command of the two. The select("storeId") part just returns the storeId column of transactionsDf - this works here, since we are only interested in that column's type anyways.
More info: pyspark.sql.DataFrame.printSchema - PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 3
質問 # 72
Which of the following statements about the differences between actions and transformations is correct?
- A. Actions do not send results to the driver, while transformations do.
- B. Actions are evaluated lazily, while transformations are not evaluated lazily.
- C. Actions generate RDDs, while transformations do not.
- D. Actions can be queued for delayed execution, while transformations can only be processed immediately.
- E. Actions can trigger Adaptive Query Execution, while transformation cannot.
正解：E
解説：
Explanation
Actions can trigger Adaptive Query Execution, while transformation cannot.
Correct. Adaptive Query Execution optimizes queries at runtime. Since transformations are evaluated lazily, Spark does not have any runtime information to optimize the query until an action is called. If Adaptive Query Execution is enabled, Spark will then try to optimize the query based on the feedback it gathers while it is evaluating the query.
Actions can be queued for delayed execution, while transformations can only be processed immediately.
No, there is no such concept as "delayed execution" in Spark. Actions cannot be evaluated lazily, meaning that they are executed immediately.
Actions are evaluated lazily, while transformations are not evaluated lazily.
Incorrect, it is the other way around: Transformations are evaluated lazily and actions trigger their evaluation.
Actions generate RDDs, while transformations do not.
No. Transformations change the data and, since RDDs are immutable, generate new RDDs along the way.
Actions produce outputs in Python and data types (integers, lists, text files,...) based on the RDDs, but they do not generate them.
Here is a great tip on how to differentiate actions from transformations: If an operation returns a DataFrame, Dataset, or an RDD, it is a transformation. Otherwise, it is an action.
Actions do not send results to the driver, while transformations do.
No. Actions send results to the driver. Think about running DataFrame.count(). The result of this command will return a number to the driver. Transformations, however, do not send results back to the driver. They produce RDDs that remain on the worker nodes.
More info: What is the difference between a transformation and an action in Apache Spark? | Bartosz Mikulski, How to Speed up SQL Queries with Adaptive Query Execution
質問 # 73
Which of the following code blocks returns all unique values across all values in columns value and productId in DataFrame transactionsDf in a one-column DataFrame?
- A. transactionsDf.select(col('value'), col('productId')).agg({'*': 'count'})
- B. tranactionsDf.select('value').join(transactionsDf.select('productId'), col('value')==col('productId'),
'outer') - C. transactionsDf.select('value').union(transactionsDf.select('productId')).distinct()
- D. transactionsDf.select('value', 'productId').distinct()
- E. transactionsDf.agg({'value': 'collect_set', 'productId': 'collect_set'})
正解：C
解説：
Explanation
transactionsDf.select('value').union(transactionsDf.select('productId')).distinct() Correct. This code block uses a common pattern for finding the unique values across multiple columns: union and distinct. In fact, it is so common that it is even mentioned in the Spark documentation for the union command (link below).
transactionsDf.select('value', 'productId').distinct()
Wrong. This code block returns unique rows, but not unique values.
transactionsDf.agg({'value': 'collect_set', 'productId': 'collect_set'}) Incorrect. This code block will output a one-row, two-column DataFrame where each cell has an array of unique values in the respective column (even omitting any nulls).
transactionsDf.select(col('value'), col('productId')).agg({'*': 'count'}) No. This command will count the number of rows, but will not return unique values.
transactionsDf.select('value').join(transactionsDf.select('productId'), col('value')==col('productId'), 'outer') Wrong. This command will perform an outer join of the value and productId columns. As such, it will return a two-column DataFrame. If you picked this answer, it might be a good idea for you to read up on the difference between union and join, a link is posted below.
More info: pyspark.sql.DataFrame.union - PySpark 3.1.2 documentation, sql - What is the difference between JOIN and UNION? - Stack Overflow Static notebook | Dynamic notebook: See test 3
質問 # 74
The code block shown below should return a two-column DataFrame with columns transactionId and supplier, with combined information from DataFrames itemsDf and transactionsDf. The code block should merge rows in which column productId of DataFrame transactionsDf matches the value of column itemId in DataFrame itemsDf, but only where column storeId of DataFrame transactionsDf does not match column itemId of DataFrame itemsDf. Choose the answer that correctly fills the blanks in the code block to accomplish this.
Code block:
transactionsDf.__1__(itemsDf, __2__).__3__(__4__)
- A. 1. join
2. [transactionsDf.productId==itemsDf.itemId, transactionsDf.storeId!=itemsDf.itemId]
3. select
4. "transactionId", "supplier" - B. 1. filter
2. "transactionId", "supplier"
3. join
4. "transactionsDf.storeId!=itemsDf.itemId, transactionsDf.productId==itemsDf.itemId" - C. 1. select
2. "transactionId", "supplier"
3. join
4. [transactionsDf.storeId!=itemsDf.itemId, transactionsDf.productId==itemsDf.itemId] - D. 1. join
2. transactionsDf.productId==itemsDf.itemId, transactionsDf.storeId!=itemsDf.itemId
3. filter
4. "transactionId", "supplier" - E. 1. join
2. transactionsDf.productId==itemsDf.itemId, how="inner"
3. select
4. "transactionId", "supplier"
正解：A
解説：
Explanation
This question is pretty complex and, in its complexity, is probably above what you would encounter in the exam. However, reading the question carefully, you can use your logic skills to weed out the wrong answers here.
First, you should examine the join statement which is common to all answers. The first argument of the join() operator (documentation linked below) is the DataFrame to be joined with. Where join is in gap 3, the first argument of gap 4 should therefore be another DataFrame. For none of the questions where join is in the third gap, this is the case. So you can immediately discard two answers.
For all other answers, join is in gap 1, followed by .(itemsDf, according to the code block. Given how the join() operator is called, there are now three remaining candidates.
Looking further at the join() statement, the second argument (on=) expects "a string for the join column name, a list of column names, a join expression (Column), or a list of Columns", according to the documentation. As one answer option includes a list of join expressions (transactionsDf.productId==itemsDf.itemId, transactionsDf.storeId!=itemsDf.itemId) which is unsupported according to the documentation, we can discard that answer, leaving us with two remaining candidates.
Both candidates have valid syntax, but only one of them fulfills the condition in the question "only where column storeId of DataFrame transactionsDf does not match column itemId of DataFrame itemsDf". So, this one remaining answer option has to be the correct one!
As you can see, although sometimes overwhelming at first, even more complex questions can be figured out by rigorously applying the knowledge you can gain from the documentation during the exam.
More info: pyspark.sql.DataFrame.join - PySpark 3.1.2 documentation
Static notebook | Dynamic notebook: See test 3
質問 # 75
The code block displayed below contains an error. The code block should arrange the rows of DataFrame transactionsDf using information from two columns in an ordered fashion, arranging first by column value, showing smaller numbers at the top and greater numbers at the bottom, and then by column predError, for which all values should be arranged in the inverse way of the order of items in column value. Find the error.
Code block:
transactionsDf.orderBy('value', asc_nulls_first(col('predError')))
- A. Instead of orderBy, sort should be used.
- B. Column predError should be sorted in a descending way, putting nulls last.
- C. Column predError should be sorted by desc_nulls_first() instead.
- D. Two orderBy statements with calls to the individual columns should be chained, instead of having both columns in one orderBy statement.
- E. Column value should be wrapped by the col() operator.
正解：B
解説：
Explanation
Correct code block:
transactionsDf.orderBy('value', desc_nulls_last('predError'))
Column predError should be sorted in a descending way, putting nulls last.
Correct! By default, Spark sorts ascending, putting nulls first. So, the inverse sort of the default sort is indeed desc_nulls_last.
Instead of orderBy, sort should be used.
No. DataFrame.sort() orders data per partition, it does not guarantee a global order. This is why orderBy is the more appropriate operator here.
Column value should be wrapped by the col() operator.
Incorrect. DataFrame.sort() accepts both string and Column objects.
Column predError should be sorted by desc_nulls_first() instead.
Wrong. Since Spark's default sort order matches asc_nulls_first(), nulls would have to come last when inverted.
Two orderBy statements with calls to the individual columns should be chained, instead of having both columns in one orderBy statement.
No, this would just sort the DataFrame by the very last column, but would not take information from both columns into account, as noted in the question.
More info: pyspark.sql.DataFrame.orderBy - PySpark 3.1.2 documentation, pyspark.sql.functions.desc_nulls_last - PySpark 3.1.2 documentation, sort() vs orderBy() in Spark | Towards Data Science Static notebook | Dynamic notebook: See test 3
質問 # 76
In which order should the code blocks shown below be run in order to return the number of records that are not empty in column value in the DataFrame resulting from an inner join of DataFrame transactionsDf and itemsDf on columns productId and itemId, respectively?
1. .filter(~isnull(col('value')))
2. .count()
3. transactionsDf.join(itemsDf, col("transactionsDf.productId")==col("itemsDf.itemId"))
4. transactionsDf.join(itemsDf, transactionsDf.productId==itemsDf.itemId, how='inner')
5. .filter(col('value').isnotnull())
6. .sum(col('value'))
- A. 3, 5, 2
- B. 4, 1, 2
- C. 3, 1, 6
- D. 4, 6
- E. 3, 1, 2
正解：B
解説：
Explanation
Correct code block:
transactionsDf.join(itemsDf, transactionsDf.productId==itemsDf.itemId,
how='inner').filter(~isnull(col('value'))).count()
Expressions col("transactionsDf.productId") and col("itemsDf.itemId") are invalid. col() does not accept the name of a DataFrame, only column names.
Static notebook | Dynamic notebook: See test 2
質問 # 77
Which of the following statements about stages is correct?
- A. Different stages in a job may be executed in parallel.
- B. Tasks in a stage may be executed by multiple machines at the same time.
- C. Stages may contain multiple actions, narrow, and wide transformations.
- D. Stages consist of one or more jobs.
- E. Stages ephemerally store transactions, before they are committed through actions.
正解：B
解説：
Explanation
Tasks in a stage may be executed by multiple machines at the same time.
This is correct. Within a single stage, tasks do not depend on each other. Executors on multiple machines may execute tasks belonging to the same stage on the respective partitions they are holding at the same time.
Different stages in a job may be executed in parallel.
No. Different stages in a job depend on each other and cannot be executed in parallel. The nuance is that every task in a stage may be executed in parallel by multiple machines.
For example, if a job consists of Stage A and Stage B, tasks belonging to those stages may not be executed in parallel. However, tasks from Stage A may be executed on multiple machines at the same time, with each machine running it on a different partition of the same dataset. Then, afterwards, tasks from Stage B may be executed on multiple machines at the same time.
Stages may contain multiple actions, narrow, and wide transformations.
No, stages may not contain multiple wide transformations. Wide transformations mean that shuffling is required. Shuffling typically terminates a stage though, because data needs to be exchanged across the cluster. This data exchange often causes partitions to change and rearrange, making it impossible to perform tasks in parallel on the same dataset.
Stages ephemerally store transactions, before they are committed through actions.
No, this does not make sense. Stages do not "store" any data. Transactions are not "committed" in Spark.
Stages consist of one or more jobs.
No, it is the other way around: Jobs consist of one more stages.
More info: Spark: The Definitive Guide, Chapter 15.
質問 # 78
The code block displayed below contains an error. When the code block below has executed, it should have divided DataFrame transactionsDf into 14 parts, based on columns storeId and transactionDate (in this order). Find the error.
Code block:
transactionsDf.coalesce(14, ("storeId", "transactionDate"))
- A. Operator coalesce needs to be replaced by repartition, the parentheses around the column names need to be removed, and .select() needs to be appended to the code block.
- B. Operator coalesce needs to be replaced by repartition.
- C. Operator coalesce needs to be replaced by repartition, the parentheses around the column names need to be removed, and .count() needs to be appended to the code block.
(Correct) - D. The parentheses around the column names need to be removed and .select() needs to be appended to the code block.
- E. Operator coalesce needs to be replaced by repartition and the parentheses around the column names need to be replaced by square brackets.
正解：C
解説：
Explanation
Correct code block:
transactionsDf.repartition(14, "storeId", "transactionDate").count()
Since we do not know how many partitions DataFrame transactionsDf has, we cannot safely use coalesce, since it would not make any change if the current number of partitions is smaller than 14.
So, we need to use repartition.
In the Spark documentation, the call structure for repartition is shown like this:
DataFrame.repartition(numPartitions, *cols). The * operator means that any argument after numPartitions will be interpreted as column. Therefore, the brackets need to be removed.
Finally, the question specifies that after the execution the DataFrame should be divided. So, indirectly this question is asking us to append an action to the code block. Since .select() is a transformation. the only possible choice here is .count().
More info: pyspark.sql.DataFrame.repartition - PySpark 3.1.1 documentation Static notebook | Dynamic notebook: See test 1
質問 # 79
Which is the highest level in Spark's execution hierarchy?
- A. Job
- B. Slot
- C. Task
- D. Executor
- E. Stage
正解：A
質問 # 80
The code block displayed below contains an error. The code block is intended to perform an outer join of DataFrames transactionsDf and itemsDf on columns productId and itemId, respectively.
Find the error.
Code block:
transactionsDf.join(itemsDf, [itemsDf.itemId, transactionsDf.productId], "outer")
- A. The join type needs to be appended to the join() operator, like join().outer() instead of listing it as the last argument inside the join() call.
- B. The term [itemsDf.itemId, transactionsDf.productId] should be replaced by itemsDf.itemId == transactionsDf.productId.
- C. The "outer" argument should be eliminated from the call and join should be replaced by joinOuter.
- D. The term [itemsDf.itemId, transactionsDf.productId] should be replaced by itemsDf.col("itemId") == transactionsDf.col("productId").
- E. The "outer" argument should be eliminated, since "outer" is the default join type.
正解：B
解説：
Explanation
Correct code block:
transactionsDf.join(itemsDf, itemsDf.itemId == transactionsDf.productId, "outer") Static notebook | Dynamic notebook: See test 1 (https://flrs.github.io/spark_practice_tests_code/#1/33.html ,
https://bit.ly/sparkpracticeexams_import_instructions)
質問 # 81
......
Databricks Certified Associate Developer for Apache Spark 3.0 試験は、Apache Spark 3.0の最新の機能と更新内容（適応的クエリ実行、動的パーティションプルーニング、改善されたSQLパフォーマンスなど）をカバーしています。試験は60問の多肢選択問題から構成され、90分以内に回答する必要があります。この試験の合格基準は70％です。
無料Associate-Developer-Apache-Spark別格な問題集をダウンロード：https://www.jpntest.com/shiken/Associate-Developer-Apache-Spark-mondaishu
Associate-Developer-Apache-Spark問題集で2023年最新のDatabricks試験問題：https://drive.google.com/open?id=11N_E0W5qd738ht57IUCr7f4k1fm8zmgc