最新のAssociate-Developer-Apache-Spark-3.5問題集PDFでAssociate-Developer-Apache-Spark-3.5リアルで有効な正確な問題集は85問題と解答が待ってます！
100%無料Associate-Developer-Apache-Spark-3.5試験問題集リアルDatabricks Certification問題集を試そう
質問 # 28
A data engineer is streaming data from Kafka and requires:
Minimal latency
Exactly-once processing guarantees
Which trigger mode should be used?
- A. .trigger(continuous='1 second')
- B. .trigger(processingTime='1 second')
- C. .trigger(availableNow=True)
- D. .trigger(continuous=True)
正解：B
解説：
Comprehensive and Detailed Explanation:
Exactly-once guarantees in Spark Structured Streaming require micro-batch mode (default), not continuous mode.
Continuous mode (.trigger(continuous=...)) only supports at-least-once semantics and lacks full fault- tolerance.
trigger(availableNow=True)is a batch-style trigger, not suited for low-latency streaming.
So:
Option A uses micro-batching with a tight trigger interval # minimal latency + exactly-once guarantee.
Final Answer: A
質問 # 29
A data scientist is analyzing a large dataset and has written a PySpark script that includes several transformations and actions on a DataFrame. The script ends with acollect()action to retrieve the results.
How does Apache Spark™'s execution hierarchy process the operations when the data scientist runs this script?
- A. Spark creates a single task for each transformation and action in the script, and these tasks are grouped into stages and jobs based on their dependencies.
- B. Thecollect()action triggers a job, which is divided into stages at shuffle boundaries, and each stage is split into tasks that operate on individual data partitions.
- C. The entire script is treated as a single job, which is then divided into multiple stages, and each stage is further divided into tasks based on data partitions.
- D. The script is first divided into multiple applications, then each application is split into jobs, stages, and finally tasks.
正解：B
解説：
Comprehensive and Detailed Explanation From Exact Extract:
In Apache Spark, the execution hierarchy is structured as follows:
Application: The highest-level unit, representing the user program built on Spark.
Job: Triggered by an action (e.g.,collect(),count()). Each action corresponds to a job.
Stage: A job is divided into stages based on shuffle boundaries. Each stage contains tasks that can be executed in parallel.
Task: The smallest unit of work, representing a single operation applied to a partition of the data.
When thecollect()action is invoked, Spark initiates a job. This job is then divided into stages at points where data shuffling is required (i.e., wide transformations). Each stage comprises tasks that are distributed across the cluster's executors, operating on individual data partitions.
This hierarchical execution model allows Spark to efficiently process large-scale data by parallelizing tasks and optimizing resource utilization.
質問 # 30
Given a CSV file with the content:
And the following code:
from pyspark.sql.types import *
schema = StructType([
StructField("name", StringType()),
StructField("age", IntegerType())
])
spark.read.schema(schema).csv(path).collect()
What is the resulting output?
- A. [Row(name='bambi', age=None), Row(name='alladin', age=20)]
- B. [Row(name='alladin', age=20)]
- C. The code throws an error due to a schema mismatch.
- D. [Row(name='bambi'), Row(name='alladin', age=20)]
正解：A
解説：
Comprehensive and Detailed Explanation From Exact Extract:
In Spark, when a CSV row does not match the provided schema, Spark does not raise an error by default.
Instead, it returnsnullfor fields that cannot be parsed correctly.
In the first row,"hello"cannot be cast to Integer for theagefield # Spark setsage=None In the second row,"20"is a valid integer #age=20 So the output will be:
[Row(name='bambi', age=None), Row(name='alladin', age=20)]
Final Answer: C
質問 # 31
Given a DataFramedfthat has 10 partitions, after running the code:
result = df.coalesce(20)
How many partitions will the result DataFrame have?
- A. 0
- B. 1
- C. 2
- D. Same number as the cluster executors
正解：B
解説：
Comprehensive and Detailed Explanation From Exact Extract:
The.coalesce(numPartitions)function is used to reduce the number of partitions in a DataFrame. It does not increase the number of partitions. If the specified number of partitions is greater than the current number, it will not have any effect.
From the official Spark documentation:
"coalesce() results in a narrow dependency, e.g. if you go from 1000 partitions to 100 partitions, there will not be a shuffle, instead each of the 100 new partitions will claim one or more of the current partitions." However, if you try to increase partitions using coalesce (e.g., from 10 to 20), the number of partitions remains unchanged.
Hence,df.coalesce(20)will still return a DataFrame with 10 partitions.
Reference: Apache Spark 3.5 Programming Guide # RDD and DataFrame Operations # coalesce()
質問 # 32
A developer wants to test Spark Connect with an existing Spark application.
What are the two alternative ways the developer can start a local Spark Connect server without changing their existing application code? (Choose 2 answers)
- A. Set the environment variableSPARK_REMOTE="sc://localhost"before starting the pyspark shell
- B. Execute their pyspark shell with the option--remote "sc://localhost"
- C. Execute their pyspark shell with the option--remote "https://localhost"
- D. Add.remote("sc://localhost")to their SparkSession.builder calls in their Spark code
- E. Ensure the Spark propertyspark.connect.grpc.binding.portis set to 15002 in the application code
正解：A、B
解説：
Comprehensive and Detailed Explanation From Exact Extract:
Spark Connect enables decoupling of the client and Spark driver processes, allowing remote access. Spark supports configuring the remote Spark Connect server in multiple ways:
From Databricks and Spark documentation:
Option B (--remote "sc://localhost") is a valid command-line argument for thepysparkshell to connect using Spark Connect.
Option C (settingSPARK_REMOTEenvironment variable) is also a supported method to configure the remote endpoint.
Option A is incorrect because Spark Connect uses thesc://protocol, nothttps://.
Option D requires modifying the code, which the question explicitly avoids.
Option E configures the port on the server side but doesn't start a client connection.
Final Answers: B and C
質問 # 33
A developer wants to refactor some older Spark code to leverage built-in functions introduced in Spark 3.5.0.
The existing code performs array manipulations manually. Which of the following code snippets utilizes new built-in functions in Spark 3.5.0 for array operations?
A)
B)
C)
D)
- A. result_df = prices_df \
.agg(F.count_if(F.col("spot_price") >= F.lit(min_price))) - B. result_df = prices_df \
.agg(F.min("spot_price"), F.max("spot_price")) - C. result_df = prices_df \
.withColumn("valid_price", F.when(F.col("spot_price") > F.lit(min_price), 1).otherwise(0)) - D. result_df = prices_df \
.agg(F.count("spot_price").alias("spot_price")) \
.filter(F.col("spot_price") > F.lit("min_price"))
正解：A
解説：
Comprehensive and Detailed Explanation From Exact Extract:
The correct answer isBbecause it uses the new function count_if, introduced in Spark 3.5.0, which simplifies conditional counting within aggregations.
* F.count_if(condition) counts the number of rows that meet the specified boolean condition.
* In this example, it directly counts how many times spot_price >= min_price evaluates to true, replacing the older verbose combination of when/otherwise and filtering or summing.
Official Spark 3.5.0 documentation notes the addition of count_if to simplify this kind of logic:
"Added count_if aggregate function to count only the rows where a boolean condition holds (SPARK-
43773)."
Why other options are incorrect or outdated:
* Auses a legacy-style method of adding a flag column (when().otherwise()), which is verbose compared to count_if.
* Cperforms a simple min/max aggregation-useful but unrelated to conditional array operations or the updated functionality.
* Dincorrectly applies .filter() after .agg() which will cause an error, and misuses string "min_price" rather than the variable.
Therefore,Bis the only option leveraging new functionality from Spark 3.5.0 correctly and efficiently.
質問 # 34
A data engineer is running a Spark job to process a dataset of 1 TB stored in distributed storage. The cluster has 10 nodes, each with 16 CPUs. Spark UI shows:
Low number of Active Tasks
Many tasks complete in milliseconds
Fewer tasks than available CPUs
Which approach should be used to adjust the partitioning for optimal resource allocation?
- A. Set the number of partitions to a fixed value, such as 200
- B. Set the number of partitions equal to the number of nodes in the cluster
- C. Set the number of partitions equal to the total number of CPUs in the cluster
- D. Set the number of partitions by dividing the dataset size (1 TB) by a reasonable partition size, such as
128 MB
正解：D
解説：
Comprehensive and Detailed Explanation From Exact Extract:
Spark's best practice is to estimate partition count based on data volume and a reasonable partition size - typically 128 MB to 256 MB per partition.
With 1 TB of data: 1 TB / 128 MB # ~8000 partitions
This ensures that tasks are distributed across available CPUs for parallelism and that each task processes an optimal volume of data.
Option A (equal to cores) may result in partitions that are too large.
Option B (fixed 200) is arbitrary and may underutilize the cluster.
Option C (nodes) gives too few partitions (10), limiting parallelism.
Reference: Databricks Spark Tuning Guide # Partitioning Strategy
質問 # 35
A Spark application is experiencing performance issues in client mode because the driver is resource- constrained.
How should this issue be resolved?
- A. Switch the deployment mode to local mode
- B. Increase the driver memory on the client machine
- C. Add more executor instances to the cluster
- D. Switch the deployment mode to cluster mode
正解：D
解説：
Comprehensive and Detailed Explanation From Exact Extract:
In Spark's client mode, the driver runs on the local machine that submitted the job. If that machine is resource- constrained (e.g., low memory), performance degrades.
From the Spark documentation:
"In cluster mode, the driver runs inside the cluster, benefiting from cluster resources and scalability." Option A is incorrect - executors do not help the driver directly.
Option B might help short-term but does not scale.
Option C is correct - switching to cluster mode moves the driver to the cluster.
Option D (local mode) is for development/testing, not production.
Final Answer: C
質問 # 36
Which feature of Spark Connect is considered when designing an application to enable remote interaction with the Spark cluster?
- A. It allows for remote execution of Spark jobs
- B. It provides a way to run Spark applications remotely in any programming language
- C. It can be used to interact with any remote cluster using the REST API
- D. It is primarily used for data ingestion into Spark from external sources
正解：A
解説：
Comprehensive and Detailed Explanation:
Spark Connect introduces a decoupled client-server architecture. Its key feature is enabling Spark job submission and execution from remote clients - in Python, Java, etc.
From Databricks documentation:
"Spark Connect allows remote clients to connect to a Spark cluster and execute Spark jobs without being co- located with the Spark driver." A is close, but "any language" is overstated (currently supports Python, Java, etc., not literally all).
B refers to REST, which is not Spark Connect's mechanism.
D is incorrect; Spark Connect isn't focused on ingestion.
Final Answer: C
質問 # 37
An engineer has a large ORC file located at/file/test_data.orcand wants to read only specific columns to reduce memory usage.
Which code fragment will select the columns, i.e.,col1,col2, during the reading process?
- A. spark.read.orc("/file/test_data.orc").selected("col1", "col2")
- B. spark.read.orc("/file/test_data.orc").filter("col1 = 'value' ").select("col2")
- C. spark.read.format("orc").load("/file/test_data.orc").select("col1", "col2")
- D. spark.read.format("orc").select("col1", "col2").load("/file/test_data.orc")
正解：C
解説：
Comprehensive and Detailed Explanation From Exact Extract:
The correct way to load specific columns from an ORC file is to first load the file using.load()and then apply.
select()on the resulting DataFrame. This is valid with.read.format("orc")or the shortcut.read.orc().
df = spark.read.format("orc").load("/file/test_data.orc").select("col1","col2") Why others are incorrect:
Aperforms selection after filtering, but doesn't match the intention to minimize memory at load.
Bincorrectly tries to use.select()before.load(), which is invalid.
Cuses a non-existent.selected()method.
Dcorrectly loads and then selects.
Reference:Apache Spark SQL API - ORC Format
質問 # 38
A developer is trying to join two tables,sales.purchases_fctandsales.customer_dim, using the following code:
fact_df = purch_df.join(cust_df, F.col('customer_id') == F.col('custid')) The developer has discovered that customers in thepurchases_fcttable that do not exist in thecustomer_dimtable are being dropped from the joined table.
Which change should be made to the code to stop these customer records from being dropped?
- A. fact_df = cust_df.join(purch_df, F.col('customer_id') == F.col('custid'))
- B. fact_df = purch_df.join(cust_df, F.col('cust_id') == F.col('customer_id'))
- C. fact_df = purch_df.join(cust_df, F.col('customer_id') == F.col('custid'), 'right_outer')
- D. fact_df = purch_df.join(cust_df, F.col('customer_id') == F.col('custid'), 'left')
正解：D
解説：
Comprehensive and Detailed Explanation From Exact Extract:
In Spark, the default join type is an inner join, which returns only the rows with matching keys in both DataFrames. To retain all records from the left DataFrame (purch_df) and include matching records from the right DataFrame (cust_df), a left outer join should be used.
By specifying the join type as'left', the modified code ensures that all records frompurch_dfare preserved, and matching records fromcust_dfare included. Records inpurch_dfwithout a corresponding match incust_dfwill havenullvalues for the columns fromcust_df.
This approach is consistent with standard SQL join operations and is supported in PySpark's DataFrame API.
質問 # 39
A data engineer has been asked to produce a Parquet table which is overwritten every day with the latest data.
The downstream consumer of this Parquet table has a hard requirement that the data in this table is produced with all records sorted by themarket_timefield.
Which line of Spark code will produce a Parquet table that meets these requirements?
- A. final_df \
.sortWithinPartitions("market_time") \
.write \
.format("parquet") \
.mode("overwrite") \
.saveAsTable("output.market_events") - B. final_df \
.orderBy("market_time") \
.write \
.format("parquet") \
.mode("overwrite") \
.saveAsTable("output.market_events") - C. final_df \
.sort("market_time") \
.coalesce(1) \
.write \
.format("parquet") \
.mode("overwrite") \
.saveAsTable("output.market_events") - D. final_df \
.sort("market_time") \
.write \
.format("parquet") \
.mode("overwrite") \
.saveAsTable("output.market_events")
正解：A
解説：
Comprehensive and Detailed Explanation From Exact Extract:
To ensure that data written out to disk is sorted, it is important to consider how Spark writes data when saving to Parquet tables. The methods.sort()or.orderBy()apply a global sort but do not guarantee that the sorting will persist in the final output files unless certain conditions are met (e.g. a single partition via.coalesce(1)- which is not scalable).
Instead, the proper method in distributed Spark processing to ensure rows are sorted within their respective partitions when written out is:
sortWithinPartitions("column_name")
According to Apache Spark documentation:
"sortWithinPartitions()ensures each partition is sorted by the specified columns. This is useful for downstream systems that require sorted files." This method works efficiently in distributed settings, avoids the performance bottleneck of global sorting (as in.orderBy()or.sort()), and guarantees each output partition has sorted records - which meets the requirement of consistently sorted data.
Thus:
Option A and B do not guarantee the persisted file contents are sorted.
Option C introduces a bottleneck via.coalesce(1)(single partition).
Option D correctly applies sorting within partitions and is scalable.
Reference: Databricks & Apache Spark 3.5 Documentation # DataFrame API # sortWithinPartitions()
質問 # 40
A data analyst wants to add a column date derived from a timestamp column.
Options:
- A. dates_df.withColumn("date", f.unix_timestamp("timestamp")).show()
- B. dates_df.withColumn("date", f.to_date("timestamp")).show()
- C. dates_df.withColumn("date", f.from_unixtime("timestamp")).show()
- D. dates_df.withColumn("date", f.date_format("timestamp", "yyyy-MM-dd")).show()
正解：B
解説：
f.to_date() converts a timestamp or string to a DateType.
Ideal for extracting the date component (year-month-day) from a full timestamp.
Example:
frompyspark.sql.functionsimportto_date
dates_df.withColumn("date", to_date("timestamp"))
Reference:Spark SQL Date Functions
質問 # 41
A data scientist is working on a large dataset in Apache Spark using PySpark. The data scientist has a DataFramedfwith columnsuser_id,product_id, andpurchase_amountand needs to perform some operations on this data efficiently.
Which sequence of operations results in transformations that require a shuffle followed by transformations that do not?
- A. df.groupBy("user_id").agg(sum("purchase_amount").alias("total_purchase")).repartition(10)
- B. df.withColumn("discount", df.purchase_amount * 0.1).select("discount")
- C. df.withColumn("purchase_date", current_date()).where("total_purchase > 50")
- D. df.filter(df.purchase_amount > 100).groupBy("user_id").sum("purchase_amount")
正解：A
解説：
Comprehensive and Detailed Explanation From Exact Extract:
Shuffling occurs in operations likegroupBy,reduceByKey, orjoin-which cause data to be moved across partitions. Therepartition()operation can also cause a shuffle, but in this context, it follows an aggregation.
InOption D, thegroupByfollowed byaggresults in a shuffle due to grouping across nodes.
Therepartition(10)is a partitioning transformation but does not involve a new shuffle since the data is already grouped.
This sequence - shuffle (groupBy) followed by non-shuffling (repartition) - is correct.
Option A does the opposite: thefilterdoes not cause a shuffle, butgroupBydoes - this makes it the wrong order.
質問 # 42
A data scientist at a financial services company is working with a Spark DataFrame containing transaction records. The DataFrame has millions of rows and includes columns fortransaction_id,account_number, transaction_amount, andtimestamp. Due to an issue with the source system, some transactions were accidentally recorded multiple times with identical information across all fields. The data scientist needs to remove rows with duplicates across all fields to ensure accurate financial reporting.
Which approach should the data scientist use to deduplicate the orders using PySpark?
- A. df = df.dropDuplicates(["transaction_amount"])
- B. df = df.groupBy("transaction_id").agg(F.first("account_number"), F.first("transaction_amount"), F.first ("timestamp"))
- C. df = df.dropDuplicates()
- D. df = df.filter(F.col("transaction_id").isNotNull())
正解：C
解説：
dropDuplicates() with no column list removes duplicates based on all columns.
It's the most efficient and semantically correct way to deduplicate records that are completely identical across all fields.
From the PySpark documentation:
dropDuplicates(): Return a new DataFrame with duplicate rows removed, considering all columns if none are specified.
- Source:PySpark DataFrame.dropDuplicates() API
質問 # 43
A developer is working with a pandas DataFrame containing user behavior data from a web application.
Which approach should be used for executing agroupByoperation in parallel across all workers in Apache Spark 3.5?
A)
Use the applylnPandas API
B)
C)
D)
- A. Use a regular Spark UDF:
from pyspark.sql.functions import mean
df.groupBy("user_id").agg(mean("value")).show() - B. Use a Pandas UDF:
@pandas_udf("double")
def mean_func(value: pd.Series) -> float:
return value.mean()
df.groupby("user_id").agg(mean_func(df["value"])).show() - C. Use themapInPandasAPI:
df.mapInPandas(mean_func, schema="user_id long, value double").show() - D. Use theapplyInPandasAPI:
df.groupby("user_id").applyInPandas(mean_func, schema="user_id long, value double").show()
正解：D
解説：
Comprehensive and Detailed Explanation From Exact Extract:
The correct approach to perform a parallelizedgroupByoperation across Spark worker nodes using Pandas API is viaapplyInPandas. This function enables grouped map operations using Pandas logic in a distributed Spark environment. It applies a user-defined function to each group of data represented as a Pandas DataFrame.
As per the Databricks documentation:
"applyInPandas()allows for vectorized operations on grouped data in Spark. It applies a user-defined function to each group of a DataFrame and outputs a new DataFrame. This is the recommended approach for using Pandas logic across grouped data with parallel execution." Option A is correct and achieves this parallel execution.
Option B (mapInPandas) applies to the entire DataFrame, not grouped operations.
Option C uses built-in aggregation functions, which are efficient but not customizable with Pandas logic.
Option D creates a scalar Pandas UDF which does not perform a group-wise transformation.
Therefore, to run agroupBywith parallel Pandas logic on Spark workers, Option A usingapplyInPandasis the only correct answer.
Reference: Apache Spark 3.5 Documentation # Pandas API on Spark # Grouped Map Pandas UDFs (applyInPandas)
質問 # 44
The following code fragment results in an error:
Which code fragment should be used instead?
- A.
- B.
- C.
- D.
正解：B
質問 # 45
A data engineer is working on the DataFrame:
(Referring to the table image: it has columnsId,Name,count, andtimestamp.) Which code fragment should the engineer use to extract the unique values in theNamecolumn into an alphabetically ordered list?
- A. df.select("Name").distinct().orderBy(df["Name"].desc())
- B. df.select("Name").orderBy(df["Name"].asc())
- C. df.select("Name").distinct().orderBy(df["Name"])
- D. df.select("Name").distinct()
正解：C
解説：
Comprehensive and Detailed Explanation From Exact Extract:
To extract unique values from a column and sort them alphabetically:
distinct()is required to remove duplicate values.
orderBy()is needed to sort the results alphabetically (ascending by default).
Correct code:
df.select("Name").distinct().orderBy(df["Name"])
This is directly aligned with standard DataFrame API usage in PySpark, as documented in the official Databricks Spark APIs. Option A is incorrect because it may not remove duplicates. Option C omits sorting.
Option D sorts in descending order, which doesn't meet the requirement for alphabetical (ascending) order.
質問 # 46
......
あなたを余裕でAssociate-Developer-Apache-Spark-3.5は100%試験合格率保証：https://www.jpntest.com/shiken/Associate-Developer-Apache-Spark-3.5-mondaishu
Associate-Developer-Apache-Spark-3.5問題集本日限定！無料アクセスが可能に！：https://drive.google.com/open?id=1SVlWzfTDumdh3aTC5SUGWg0802pTBRZI