[2025年最新] 高合格率な最新無料Associate-Developer-Apache-Spark-3.5試験問題集アンサーを使おう
Associate-Developer-Apache-Spark-3.5知能問題集PDF！Databricks Associate-Developer-Apache-Spark-3.5試験問セット
質問 # 46
A data engineer replaces the exact percentile() function with approx_percentile() to improve performance, but the results are drifting too far from expected values.
Which change should be made to solve the issue?
- A. Decrease the first value of the percentage parameter to increase the accuracy of the percentile ranges
- B. Increase the value of the accuracy parameter in order to increase the memory usage but also improve the accuracy
- C. Increase the last value of the percentage parameter to increase the accuracy of the percentile ranges
- D. Decrease the value of the accuracy parameter in order to decrease the memory usage but also improve the accuracy
正解：B
解説：
Comprehensive and Detailed Explanation:
The approx_percentile function in Spark is a performance-optimized alternative to percentile. It takes an optional accuracy parameter:
approx_percentile(column, percentage, accuracy)
Higher accuracy values # more precise results, but increased memory/computation.
Lower values # faster but less accurate.
From the documentation:
"Increasing the accuracy improves precision but increases memory usage." Final Answer: D
質問 # 47
An MLOps engineer is building a Pandas UDF that applies a language model that translates English strings into Spanish. The initial code is loading the model on every call to the UDF, which is hurting the performance of the data pipeline.
The initial code is:
def in_spanish_inner(df: pd.Series) -> pd.Series:
model = get_translation_model(target_lang='es')
return df.apply(model)
in_spanish = sf.pandas_udf(in_spanish_inner, StringType())
How can the MLOps engineer change this code to reduce how many times the language model is loaded?
- A. Convert the Pandas UDF to a PySpark UDF
- B. Convert the Pandas UDF from a Series # Series UDF to an Iterator[Series] # Iterator[Series] UDF
- C. Convert the Pandas UDF from a Series # Series UDF to a Series # Scalar UDF
- D. Run thein_spanish_inner()function in amapInPandas()function call
正解：B
解説：
Comprehensive and Detailed Explanation From Exact Extract:
The provided code defines a Pandas UDF of type Series-to-Series, where a new instance of the language modelis created on each call, which happens per batch. This is inefficient and results in significant overhead due to repeated model initialization.
To reduce the frequency of model loading, the engineer should convert the UDF to an iterator-based Pandas UDF (Iterator[pd.Series] -> Iterator[pd.Series]). This allows the model to be loaded once per executor and reused across multiple batches, rather than once per call.
From the official Databricks documentation:
"Iterator of Series to Iterator of Series UDFs are useful when the UDF initialization is expensive... For example, loading a ML model once per executor rather than once per row/batch."
- Databricks Official Docs: Pandas UDFs
Correct implementation looks like:
python
CopyEdit
@pandas_udf("string")
def translate_udf(batch_iter: Iterator[pd.Series]) -> Iterator[pd.Series]:
model = get_translation_model(target_lang='es')
for batch in batch_iter:
yield batch.apply(model)
This refactor ensures theget_translation_model()is invoked once per executor process, not per batch, significantly improving pipeline performance.
質問 # 48
In the code block below,aggDFcontains aggregations on a streaming DataFrame:
Which output mode at line 3 ensures that the entire result table is written to the console during each trigger execution?
- A. complete
- B. aggregate
- C. replace
- D. append
正解：A
解説：
Comprehensive and Detailed Explanation From Exact Extract:
The correct output mode for streaming aggregations that need to output the full updated results at each trigger is"complete".
From the official documentation:
"complete: The entire updated result table will be output to the sink every time there is a trigger." This is ideal for aggregations, such as counts or averages grouped by a key, where the result table changes incrementally over time.
append: only outputs newly added rows
replace and aggregate: invalid values for output mode
Reference: Spark Structured Streaming Programming Guide # Output Modes
質問 # 49
You have:
DataFrame A: 128 GB of transactions
DataFrame B: 1 GB user lookup table
Which strategy is correct for broadcasting?
- A. DataFrame B should be broadcasted because it is smaller and will eliminate the need for shuffling DataFrame A
- B. DataFrame A should be broadcasted because it is larger and will eliminate the need for shuffling DataFrame B
- C. DataFrame B should be broadcasted because it is smaller and will eliminate the need for shuffling itself
- D. DataFrame A should be broadcasted because it is smaller and will eliminate the need for shuffling itself
正解：A
解説：
Comprehensive and Detailed Explanation:
Broadcast joins work by sending the smaller DataFrame to all executors, eliminating the shuffle of the larger DataFrame.
From Spark documentation:
"Broadcast joins are efficient when one DataFrame is small enough to fit in memory. Spark avoids shuffling the larger table." DataFrame B (1 GB) fits within the default threshold and should be broadcasted.
It eliminates the need to shuffle the large DataFrame A.
Final Answer: B
質問 # 50
A Spark engineer is troubleshooting a Spark application that has been encountering out-of-memory errors during execution. By reviewing the Spark driver logs, the engineer notices multiple "GC overhead limit exceeded" messages.
Which action should the engineer take to resolve this issue?
- A. Modify the Spark configuration to disable garbage collection
- B. Cache large DataFrames to persist them in memory.
- C. Optimize the data processing logic by repartitioning the DataFrame.
- D. Increase the memory allocated to the Spark Driver.
正解：D
解説：
Comprehensive and Detailed Explanation From Exact Extract:
The message"GC overhead limit exceeded"typically indicates that the JVM is spending too much time in garbage collection with little memory recovery. This suggests that the driver or executor is under-provisioned in memory.
The most effective remedy is to increase the driver memory using:
--driver-memory 4g
This is confirmed in Spark's official troubleshooting documentation:
"If you see a lot ofGC overhead limit exceedederrors in the driver logs, it's a sign that the driver is running out of memory."
-Spark Tuning Guide
Why others are incorrect:
Amay help but does not directly address the driver memory shortage.
Bis not a valid action; GC cannot be disabled.
Dincreases memory usage, worsening the problem.
質問 # 51
A developer runs:
What is the result?
Options:
- A. It appends new partitions to an existing Parquet file.
- B. It stores all data in a single Parquet file.
- C. It creates separate directories for each unique combination of color and fruit.
- D. It throws an error if there are null values in either partition column.
正解：C
解説：
ThepartitionBy()method in Spark organizes output into subdirectories based on unique combinations of the specified columns:
e.g.
/path/to/output/color=red/fruit=apple/part-0000.parquet
/path/to/output/color=green/fruit=banana/part-0001.parquet
This improves query performance via partition pruning.
It does not consolidate into a single file.
Null values are allowed in partitions.
It does not "append" unless.mode("append")is used.
Reference:Spark Write with Partitioning
質問 # 52
A data engineer needs to persist a file-based data source to a specific location. However, by default, Spark writes to the warehouse directory (e.g., /user/hive/warehouse). To override this, the engineer must explicitly define the file path.
Which line of code ensures the data is saved to a specific location?
Options:
- A. users.write.saveAsTable("default_table", path="/some/path")
- B. users.write(path="/some/path").saveAsTable("default_table")
- C. users.write.saveAsTable("default_table").option("path", "/some/path")
- D. users.write.option("path", "/some/path").saveAsTable("default_table")
正解：D
解説：
To persist a table and specify the save path, use:
users.write.option("path","/some/path").saveAsTable("default_table")
The .option("path", ...) must be applied before calling saveAsTable.
Option A uses invalid syntax (write(path=...)).
Option B applies.option()after.saveAsTable()-which is too late.
Option D uses incorrect syntax (no path parameter in saveAsTable).
Reference:Spark SQL - Save as Table
質問 # 53
Which Spark configuration controls the number of tasks that can run in parallel on the executor?
Options:
- A. spark.driver.cores
- B. spark.task.maxFailures
- C. spark.executor.cores
- D. spark.executor.memory
正解：C
解説：
spark.executor.cores determines how many concurrent tasks an executor can run.
For example, if set to 4, each executor can run up to 4 tasks in parallel.
Other settings:
spark.task.maxFailures controls task retry logic.
spark.driver.cores is for the driver, not executors.
spark.executor.memory sets memory limits, not task concurrency.
Reference:Apache Spark Configuration
質問 # 54
The following code fragment results in an error:
Which code fragment should be used instead?
- A.
- B.
- C.
- D.
正解：B
質問 # 55
A data engineer is working ona Streaming DataFrame streaming_df with the given streaming data:
Which operation is supported with streaming_df?
- A. streaming_df.orderBy("timestamp").limit(4)
- B. streaming_df.filter(col("count") < 30).show()
- C. streaming_df.select(countDistinct("Name"))
- D. streaming_df.groupby("Id").count()
正解：D
解説：
Comprehensive and Detailed
Explanation:
In Structured Streaming, only a limited subset of operations is supported due to the nature of unbounded data.
Operations like sorting (orderBy) and global aggregation (countDistinct) require a full view of the dataset, which is not possible with streaming data unless specific watermarks or windows are defined.
Review of Each Option:
A). select(countDistinct("Name"))
Not allowed - Global aggregation like countDistinct() requires the full dataset and is not supported directly in streaming without watermark and windowing logic.
Reference: Databricks Structured Streaming Guide - Unsupported Operations.
B). groupby("Id").count()Supported - Streaming aggregations over a key (like groupBy("Id")) are supported.
Spark maintains intermediate state for each key.Reference: Databricks Docs # Aggregations in Structured Streaming (https://docs.databricks.com/structured-streaming/aggregation.html)
C). orderBy("timestamp").limit(4)Not allowed - Sorting and limiting require a full view of the stream (which is infinite), so this is unsupported in streaming DataFrames.Reference: Spark Structured Streaming - Unsupported Operations (ordering without watermark/window not allowed).
D). filter(col("count") < 30).show()Not allowed - show() is a blocking operation used for debugging batch DataFrames; it's not allowed on streaming DataFrames.Reference: Structured Streaming Programming Guide
- Output operations like show() are not supported.
Reference Extract from Official Guide:
"Operations like orderBy, limit, show, and countDistinct are not supported in Structured Streaming because they require the full dataset to compute a result. Use groupBy(...).agg(...) instead for incremental aggregations."- Databricks Structured Streaming Programming Guide
質問 # 56
Which feature of Spark Connect is considered when designing an application to enable remote interaction with the Spark cluster?
- A. It can be used to interact with any remote cluster using the REST API
- B. It is primarily used for data ingestion into Spark from external sources
- C. It provides a way to run Spark applications remotely in any programming language
- D. It allows for remote execution of Spark jobs
正解：D
解説：
Comprehensive and Detailed Explanation:
Spark Connect introduces a decoupled client-server architecture. Its key feature is enabling Spark job submission and execution from remote clients - in Python, Java, etc.
From Databricks documentation:
"Spark Connect allows remote clients to connect to a Spark cluster and execute Spark jobs without being co- located with the Spark driver." A is close, but "any language" is overstated (currently supports Python, Java, etc., not literally all).
B refers to REST, which is not Spark Connect's mechanism.
D is incorrect; Spark Connect isn't focused on ingestion.
Final Answer: C
質問 # 57
A data engineer is running a batch processing job on a Spark cluster with the following configuration:
10 worker nodes
16 CPU cores per worker node
64 GB RAM per node
The data engineer wants to allocate four executors per node, each executor using four cores.
What is the total number of CPU cores used by the application?
- A. 0
- B. 1
- C. 2
- D. 3
正解：A
解説：
Comprehensive and Detailed Explanation From Exact Extract:
If each of the 10 nodes runs 4 executors, and each executor is assigned 4 CPU cores:
Executors per node = 4
Cores per executor = 4
Total executors = 4 * 10 = 40
Total cores = 40 executors * 4 cores = 160 cores
However, Spark uses 1 core for overhead on each node when managing multiple executors. Thus, the practical allocation is:
Total usable executors = 4 executors/node × 10 nodes = 40
Total cores = 4 cores × 40 executors = 160
Answer is A - but the question asks specifically about "CPU cores used by the application," assuming all
allocated cores are usable (as Spark typically runs executors without internal core reservation unless explicitly configured).
However, if you are considering 4 executors/node × 4 cores = 16 cores per node, across 10 nodes, that's 160.
Final Answer: A
質問 # 58
A data engineer wants to write a Spark job that creates a new managed table. If the table already exists, the job should fail and not modify anything.
Which save mode and method should be used?
- A. saveAsTable with mode ErrorIfExists
- B. saveAsTable with mode Overwrite
- C. save with mode ErrorIfExists
- D. save with mode Ignore
正解：A
解説：
Comprehensive and Detailed Explanation:
The methodsaveAsTable()creates a new table and optionally fails if the table exists.
From Spark documentation:
"The mode 'ErrorIfExists' (default) will throw an error if the table already exists." Thus:
Option A is correct.
Option B (Overwrite) would overwrite existing data - not acceptable here.
Option C and D usesave(), which doesn't create a managed table with metadata in the metastore.
Final Answer: A
質問 # 59
A data engineer is building an Apache Spark™ Structured Streaming application to process a stream of JSON events in real time. The engineer wants the application to be fault-tolerant and resume processing from the last successfully processed record in case of a failure. To achieve this, the data engineer decides to implement checkpoints.
Which code snippet should the data engineer use?
- A. query = streaming_df.writeStream \
.format("console") \
.outputMode("complete") \
.start() - B. query = streaming_df.writeStream \
.format("console") \
.outputMode("append") \
.start() - C. query = streaming_df.writeStream \
.format("console") \
.option("checkpoint", "/path/to/checkpoint") \
.outputMode("append") \
.start() - D. query = streaming_df.writeStream \
.format("console") \
.outputMode("append") \
.option("checkpointLocation", "/path/to/checkpoint") \
.start()
正解：D
解説：
Comprehensive and Detailed Explanation From Exact Extract:
To enable fault tolerance and ensure that Spark can resume from the last committed offset after failure, you must configure a checkpoint location using the correct option key:"checkpointLocation".
From the official Spark Structured Streaming guide:
"To make a streaming query fault-tolerant and recoverable, a checkpoint directory must be specified using.
option("checkpointLocation", "/path/to/dir")."
Explanation of options:
Option A uses an invalid option name:"checkpoint"(should be"checkpointLocation") Option B is correct: it setscheckpointLocationproperly Option C lacks checkpointing and won't resume after failure Option D also lacks checkpointing configuration Reference: Apache Spark 3.5 Documentation # Structured Streaming # Fault Tolerance Semantics
質問 # 60
A data engineer writes the following code to join two DataFramesdf1anddf2:
df1 = spark.read.csv("sales_data.csv") # ~10 GB
df2 = spark.read.csv("product_data.csv") # ~8 MB
result = df1.join(df2, df1.product_id == df2.product_id)
Which join strategy will Spark use?
- A. Broadcast join, as df2 is smaller than the default broadcast threshold
- B. Shuffle join, because AQE is not enabled, and Spark uses a static query plan
- C. Shuffle join because no broadcast hints were provided
- D. Shuffle join, as the size difference between df1 and df2 is too large for a broadcast join to work efficiently
正解：A
解説：
Comprehensive and Detailed Explanation From Exact Extract:
The default broadcast join threshold in Spark is:
spark.sql.autoBroadcastJoinThreshold = 10MB
Sincedf2is only 8 MB (less than 10 MB), Spark will automatically apply a broadcast join without requiring explicit hints.
From the Spark documentation:
"If one side of the join is smaller than the broadcast threshold, Spark will automatically broadcast it to all executors." A is incorrect because Spark does support auto broadcast even with static plans.
B is correct: Spark will automatically broadcast df2.
C and D are incorrect because Spark's default logic handles this optimization.
Final Answer: B
質問 # 61
A data scientist of an e-commerce company is working with user data obtained from its subscriber database and has stored the data in a DataFrame df_user. Before further processing the data, the data scientist wants to create another DataFrame df_user_non_pii and store only the non-PII columns in this DataFrame. The PII columns in df_user are first_name, last_name, email, and birthdate.
Which code snippet can be used to meet this requirement?
- A. df_user_non_pii = df_user.dropfields("first_name, last_name, email, birthdate")
- B. df_user_non_pii = df_user.drop("first_name", "last_name", "email", "birthdate")
- C. df_user_non_pii = df_user.dropfields("first_name", "last_name", "email", "birthdate")
- D. df_user_non_pii = df_user.drop("first_name", "last_name", "email", "birthdate")
正解：B
解説：
Comprehensive and Detailed Explanation:
To remove specific columns from a PySpark DataFrame, the drop() method is used. This method returns a new DataFrame without the specified columns. The correct syntax for dropping multiple columns is to pass each column name as a separate argument to the drop() method.
Correct Usage:
df_user_non_pii = df_user.drop("first_name", "last_name", "email", "birthdate") This line of code will return a new DataFrame df_user_non_pii that excludes the specified PII columns.
Explanation of Options:
A).Correct. Uses the drop() method with multiple column names passed as separate arguments, which is the standard and correct usage in PySpark.
B).Although it appears similar to Option A, if the column names are not enclosed in quotes or if there's a syntax error (e.g., missing quotes or incorrect variable names), it would result in an error. However, as written, it's identical to Option A and thus also correct.
C).Incorrect. The dropfields() method is not a method of the DataFrame class in PySpark. It's used with StructType columns to drop fields from nested structures, not top-level DataFrame columns.
D).Incorrect. Passing a single string with comma-separated column names to dropfields() is not valid syntax in PySpark.
References:
PySpark Documentation:DataFrame.drop
Stack Overflow Discussion:How to delete columns in PySpark DataFrame
質問 # 62
A data analyst wants to add a column date derived from a timestamp column.
Options:
- A. dates_df.withColumn("date", f.from_unixtime("timestamp")).show()
- B. dates_df.withColumn("date", f.unix_timestamp("timestamp")).show()
- C. dates_df.withColumn("date", f.date_format("timestamp", "yyyy-MM-dd")).show()
- D. dates_df.withColumn("date", f.to_date("timestamp")).show()
正解：D
解説：
f.to_date() converts a timestamp or string to a DateType.
Ideal for extracting the date component (year-month-day) from a full timestamp.
Example:
frompyspark.sql.functionsimportto_date
dates_df.withColumn("date", to_date("timestamp"))
Reference:Spark SQL Date Functions
質問 # 63
A developer is trying to join two tables,sales.purchases_fctandsales.customer_dim, using the following code:
fact_df = purch_df.join(cust_df, F.col('customer_id') == F.col('custid')) The developer has discovered that customers in thepurchases_fcttable that do not exist in thecustomer_dimtable are being dropped from the joined table.
Which change should be made to the code to stop these customer records from being dropped?
- A. fact_df = purch_df.join(cust_df, F.col('customer_id') == F.col('custid'), 'right_outer')
- B. fact_df = purch_df.join(cust_df, F.col('customer_id') == F.col('custid'), 'left')
- C. fact_df = purch_df.join(cust_df, F.col('cust_id') == F.col('customer_id'))
- D. fact_df = cust_df.join(purch_df, F.col('customer_id') == F.col('custid'))
正解：B
解説：
Comprehensive and Detailed Explanation From Exact Extract:
In Spark, the default join type is an inner join, which returns only the rows with matching keys in both DataFrames. To retain all records from the left DataFrame (purch_df) and include matching records from the right DataFrame (cust_df), a left outer join should be used.
By specifying the join type as'left', the modified code ensures that all records frompurch_dfare preserved, and matching records fromcust_dfare included. Records inpurch_dfwithout a corresponding match incust_dfwill havenullvalues for the columns fromcust_df.
This approach is consistent with standard SQL join operations and is supported in PySpark's DataFrame API.
質問 # 64
What is the behavior for functiondate_sub(start, days)if a negative value is passed into thedaysparameter?
- A. An error message of an invalid parameter will be returned
- B. The number of days specified will be added to the start date
- C. The number of days specified will be removed from the start date
- D. The same start date will be returned
正解：B
解説：
Comprehensive and Detailed Explanation From Exact Extract:
The functiondate_sub(start, days)subtracts the number of days from the start date. If a negative number is passed, the behavior becomes a date addition.
Example:
SELECT date_sub('2024-05-01', -5)
-- Returns: 2024-05-06
So, a negative value effectively adds the absolute number of days to the date.
Reference: Spark SQL Functions # date_sub()
質問 # 65
An engineer wants to join two DataFramesdf1anddf2on the respectiveemployee_idandemp_idcolumns:
df1:employee_id INT,name STRING
df2:emp_id INT,department STRING
The engineer uses:
result = df1.join(df2, df1.employee_id == df2.emp_id, how='inner')
What is the behaviour of the code snippet?
- A. The code works as expected because the join condition explicitly matches employee_id from df1 with emp_id from df2
- B. The code fails to execute because PySpark does not support joining DataFrames with a different structure
- C. The code fails to execute because it must use on='employee_id' to specify the join column explicitly
- D. The code fails to execute because the column names employee_id and emp_id do not match automatically
正解：A
解説：
Comprehensive and Detailed Explanation From Exact Extract:
In PySpark, when performing a join between two DataFrames, the columns do not have to share the same name. You can explicitly provide a join condition by comparing specific columns from each DataFrame.
This syntax is correct and fully supported:
df1.join(df2, df1.employee_id == df2.emp_id, how='inner')
This will perform an inner join betweendf1anddf2using theemployee_idfromdf1andemp_idfromdf2.
Reference: Databricks Spark 3.5 Documentation # DataFrame API # join()
質問 # 66
Which UDF implementation calculates the length of strings in a Spark DataFrame?
- A. df.withColumn("length", spark.udf("len", StringType()))
- B. spark.udf.register("stringLength", lambda s: len(s))
- C. df.withColumn("length", udf(lambda s: len(s), StringType()))
- D. df.select(length(col("stringColumn")).alias("length"))
正解：D
解説：
Comprehensive and Detailed Explanation From Exact Extract:
Option B uses Spark's built-in SQL function length(), which is efficient and avoids the overhead of a Python UDF:
from pyspark.sql.functions import length, col
df.select(length(col("stringColumn")).alias("length"))
Explanation of other options:
Option A is incorrect syntax;spark.udfis not called this way.
Option C registers a UDF but doesn't apply it in the DataFrame transformation.
Option D is syntactically valid but uses a Python UDF which is less efficient than built-in functions.
Final Answer: B
質問 # 67
......
Databricks Associate-Developer-Apache-Spark-3.5問題集PDFを使ってベストオプションを目指そう：https://www.jpntest.com/shiken/Associate-Developer-Apache-Spark-3.5-mondaishu
2025年最新のAssociate-Developer-Apache-Spark-3.5サンプル問題は頼もしいAssociate-Developer-Apache-Spark-3.5テストエンジン：https://drive.google.com/open?id=1wfdElPHDCo4BmWnZ_F5t7OzJNVOaIO6p