Databricks-Certified-Professional-Data-Engineer問題集最新版を今すぐ試そう！[2024年11月] 試験準備には欠かせません！
有能な受験者がシミュレーション済みのDatabricks-Certified-Professional-Data-Engineer試験PDF問題を試そう
Databricksは、データエンジニアリングの分野の大手企業であり、共同データ分析と処理のためのクラウドベースのプラットフォームを提供しています。同社のプラットフォームは、フォーチュン500企業、政府機関、学術機関など、幅広い企業や組織で使用されています。 Databricksは、Databricks認定プロフェッショナルデータエンジニア認定など、専門家がプラットフォームの使用に習熟できることを実証できるように、さまざまな認定を提供しています。
質問 # 23
The Databricks workspace administrator has configured interactive clusters for each of the data engineering groups. To control costs, clusters are set to terminate after 30 minutes of inactivity. Each user should be able to execute workloads against their assigned clusters at any time of the day.
Assuming users have been added to a workspace but not granted any permissions, which of the following describes the minimal permissions a user would need to start and attach to an already configured cluster.
- A. Workspace Admin privileges, cluster creation allowed. "Can Attach To" privileges on the required cluster
- B. Cluster creation allowed. "Can Attach To" privileges on the required cluster
- C. Cluster creation allowed. "Can Restart" privileges on the required cluster
- D. "Can Manage" privileges on the required cluster
- E. "Can Restart" privileges on the required cluster
正解：B
解説：
Explanation
This is the minimal permission a user would need to start and attach to an already configured cluster. Cluster creation allowed means that the user can create new clusters or start existing clusters that are stopped. "Can Attach To" privileges on the required cluster means that the user can attach notebooks or libraries to that cluster and run commands on it. Verified References: Databricks Certified Data Engineer Professional, under
"Security & Governance" section; Databricks Documentation, under "Cluster permissions" section.
質問 # 24
A newly joined team member John Smith in the Marketing team currently has access read access to sales tables but does not have access to update the table, which of the following commands help you accomplish this?
- A. GRANT USAGE ON TABLE table_name TO [email protected]
- B. GRANT MODIFY ON TABLE table_name TO [email protected]
- C. GRANT UPDATE ON TABLE table_name TO [email protected]
- D. GRANT MODIFY TO TABLE table_name ON [email protected]
- E. GRANT UPDATE TO TABLE table_name ON [email protected]
正解：B
解説：
Explanation
The answer is GRANT MODIFY ON TABLE table_name TO [email protected]
https://docs.microsoft.com/en-us/azure/databricks/security/access-control/table-acls/object-privileges#privileges
質問 # 25
Which of the following scenarios is the best fit for AUTO LOADER?
- A. Efficiently copy data from one data lake location to another data lake location
- B. Efficiently process new data incrementally from cloud object storage
- C. Efficiently move data incrementally from one delta table to another delta table
- D. Incrementally process new data from relational databases like MySQL
- E. Incrementally process new data from streaming data sources like Kafka into delta lake
正解：B
解説：
Explanation
The answer is, Efficiently process new data incrementally from cloud object storage, AU-TO LOADER only supports ingesting files stored in a cloud object storage. Auto Loader cannot process streaming data sources like Kafka or Delta streams, use Structured streaming for these data sources.
Diagram Description automatically generated
Auto Loader and Cloud Storage Integration
Auto Loader supports a couple of ways to ingest data incrementally
1.Directory listing - List Directory and maintain the state in RocksDB, supports incremental file listing
2.File notification - Uses a trigger+queue to store the file notification which can be later used to retrieve the file, unlike Directory listing File notification can scale up to millions of files per day.
[OPTIONAL]
Auto Loader vs COPY INTO?
Auto Loader
Auto Loader incrementally and efficiently processes new data files as they arrive in cloud storage without any additional setup. Auto Loader provides a new Structured Streaming source called cloudFiles. Given an input directory path on the cloud file storage, the cloudFiles source automatically processes new files as they arrive, with the option of also processing existing files in that directory.
When to use Auto Loader instead of the COPY INTO?
*You want to load data from a file location that contains files in the order of millions or higher. Auto Loader can discover files more efficiently than the COPY INTO SQL command and can split file processing into multiple batches.
*You do not plan to load subsets of previously uploaded files. With Auto Loader, it can be more difficult to reprocess subsets of files. However, you can use the COPY INTO SQL command to reload subsets of files while an Auto Loader stream is simultaneously running.
質問 # 26
If E1 and E2 are two events, how do you represent the conditional probability given that E2 occurs given that
E1 has occurred?
- A. P(E1+E2)/P(E1)
- B. P(E2)/P(E1)
- C. P(E1)/P(E2)
- D. P(E2)/(P(E1+E2)
正解：B
質問 # 27
What is the underlying technology that makes the Auto Loader work?
- A. Delta Live Tables
- B. Live DataFames
- C. Loader
- D. DataFrames
- E. Structured Streaming
正解：E
質問 # 28
What is the purpose of the bronze layer in a Multi-hop Medallion architecture?
- A. Copy of raw data, easy to query and ingest data for downstream processes.
- B. Powers ML applications
- C. Contain aggregated data that is to be consumed into Silver
- D. Data quality checks, corrupt data quarantined
- E. Reduces data storage by compressing the data
正解：A
解説：
Explanation
The answer is, copy of raw data, easy to query and ingest data for downstream processes, Medallion Architecture - Databricks Here are the typical role of Bronze Layer in a medallion architecture.
Bronze Layer:
1. Raw copy of ingested data
2. Replaces traditional data lake
3. Provides efficient storage and querying of full, unprocessed history of data
4. No schema is applied at this layer
Exam focus: Please review the below image and understand the role of each layer(bronze, silver, gold) in medallion architecture, you will see varying questions targeting each layer and its purpose.
Sorry I had to add the watermark some people in Udemy are copying my content.
質問 # 29
A particular job seems to be performing slower and slower over time, the team thinks this started to happen when a recent production change was implemented, you were asked to take look at the job history and see if we can identify trends and root cause, where in the workspace UI can you perform this analysis?
- A. Historical job runs can only be accessed by REST API
- B. Under Workspace logs, select job logs and select the job you want to monitor to view the last 60 day historical runs
- C. Under Compute UI, select Job cluster and select the job cluster to see last 60 day his-torical runs
- D. Under jobs UI select the job cluster, under spark UI select the application job logs, then you can access last 60 day historical runs
- E. Under jobs UI select the job you are interested, under runs we can see current active runs and last 60 days historical run
正解：E
解説：
Explanation
The answer is,
Under jobs UI select the job you are interested, under runs we can see current active runs and last 60 days historical run
質問 # 30
A table is registered with the following code:
Both users and orders are Delta Lake tables. Which statement describes the results of querying recent_orders?
- A. All logic will execute at query time and return the result of joining the valid versions of the source tables at the time the query finishes.
- B. The versions of each source table will be stored in the table transaction log; query results will be saved to DBFS with each query.
- C. All logic will execute at query time and return the result of joining the valid versions of the source tables at the time the query began.
- D. All logic will execute when the table is defined and store the result of joining tables to the DBFS; this stored data will be returned when the table is queried.
- E. Results will be computed and cached when the table is defined; these cached results will incrementally update as new records are inserted into source tables.
正解：D
質問 # 31
The downstream consumers of a Delta Lake table have been complaining about data quality issues impacting performance in their applications. Specifically, they have complained that invalid latitude and longitude values in the activity_details table have been breaking their ability to use other geolocation processes.
A junior engineer has written the following code to add CHECK constraints to the Delta Lake table:
A senior engineer has confirmed the above logic is correct and the valid ranges for latitude and longitude are provided, but the code fails when executed.
Which statement explains the cause of this failure?
- A. Because another team uses this table to support a frequently running application, two-phase locking is preventing the operation from committing.
- B. The current table schema does not contain the field valid coordinates; schema evolution will need to be enabled before altering the table to add a constraint.
- C. The activity details table already exists; CHECK constraints can only be added during initial table creation.
- D. The activity details table already contains records; CHECK constraints can only be added prior to inserting values into a table.
- E. The activity details table already contains records that violate the constraints; all existing data must pass CHECK constraints in order to add them to an existing table.
正解：E
解説：
The failure is that the code to add CHECK constraints to the Delta Lake table fails when executed. The code uses ALTER TABLE ADD CONSTRAINT commands to add two CHECK constraints to a table named activity_details. The first constraint checks if the latitude value is between -90 and 90, and the second constraint checks if the longitude value is between -180 and 180. The cause of this failure is that the activity_details table already contains records that violate these constraints, meaning that they have invalid latitude or longitude values outside of these ranges. When adding CHECK constraints to an existing table, Delta Lake verifies that all existing data satisfies the constraints before adding them to the table. If any record violates the constraints, Delta Lake throws an exception and aborts the operation. Verified Reference: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "Add a CHECK constraint to an existing table" section.
https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-alter-table.html#add-constraint
質問 # 32
Which of the statements are correct about lakehouse?
- A. Lakehouse only supports end-to-end streaming workloads and Data warehouses support Batch workloads
- B. In Lakehouse Storage and compute are coupled
- C. Lakehouse only supports Machine learning workloads and Data warehouses support BI workloads
- D. Lakehouse does not support ACID
- E. Lakehouse supports schema enforcement and evolution
正解：E
解説：
Explanation
The answer is Lakehouse supports schema enforcement and evolution,
Lakehouse using Delta lake can not only enforce a schema on write which is contrary to traditional big data systems that can only enforce a schema on read, it also supports evolving schema over time with the ability to control the evolution.
For example below is the Dataframe writer API and it supports three modes of enforcement and evolution, Default: Only enforcement, no changes are allowed and any schema drift/evolution will result in failure.
Merge: Flexible, supports enforcement and evolution
* New columns are added
* Evolves nested columns
* Supports evolving data types, like Byte to Short to Integer to Bigint How to enable:
* DF.write.format("delta").option("mergeSchema", "true").saveAsTable("table_name")
* or
* spark.databricks.delta.schema.autoMerge = True ## Spark session
Overwrite: No enforcement
* Dropping columns
* Change string to integer
* Rename columns
How to enable:
* DF.write.format("delta").option("overwriteSchema", "True").saveAsTable("table_name") What Is a Lakehouse? - The Databricks Blog Graphical user interface, text, application Description automatically generated
質問 # 33
Your team has hundreds of jobs running but it is difficult to track cost of each job run, you are asked to provide a recommendation on how to monitor and track cost across various workloads
- A. Create jobs in different workspaces, so we can track the cost easily
- B. Use workspace admin reporting
- C. Use Tags, during job creation so cost can be easily tracked
- D. Use job logs to monitor and track the costs
- E. Use a single cluster for all the jobs, so cost can be easily tracked
正解：C
解説：
Explanation
The answer is Use Tags, during job creation so cost can be easily tracked Review below link for more details
https://docs.databricks.com/administration-guide/account-settings/usage-detail-tags-aws.html Here is a view how tags get propagated from pools to clusters and clusters without pools, Diagram Description automatically generated
質問 # 34
You have configured AUTO LOADER to process incoming IOT data from cloud object storage every 15 mins, recently a change was made to the notebook code to update the processing logic but the team later realized that the notebook was failing for the last 24 hours, what steps team needs to take to reprocess the data that was not loaded after the notebook was corrected?
- A. Manually re-load the data
- B. Autoloader automatically re-processes data that was not loaded
- C. Delete the checkpoint folder and run the autoloader again
- D. Move the files that were not processed to another location and manually copy the files into the ingestion path to reprocess them
- E. Enable back_fill = TRUE to reprocess the data
正解：B
解説：
Explanation
The answer is,
Autoloader automatically re-processes data that was not loaded using the checkpoint.
質問 # 35
A Delta Lake table was created with the below query:
Realizing that the original query had a typographical error, the below code was executed:
ALTER TABLE prod.sales_by_stor RENAME TO prod.sales_by_store
Which result will occur after running the second command?
- A. The table name change is recorded in the Delta transaction log.
- B. The table reference in the metastore is updated and all data files are moved.
- C. All related files and metadata are dropped and recreated in a single ACID transaction.
- D. A new Delta transaction log Is created for the renamed table.
- E. The table reference in the metastore is updated and no data is changed.
正解：E
解説：
The query uses the CREATE TABLE USING DELTA syntax to create a Delta Lake table from an existing Parquet file stored in DBFS. The query also uses the LOCATION keyword to specify the path to the Parquet file as /mnt/finance_eda_bucket/tx_sales.parquet. By using the LOCATION keyword, the query creates an external table, which is a table that is stored outside of the default warehouse directory and whose metadata is not managed by Databricks. An external table can be created from an existing directory in a cloud storage system, such as DBFS or S3, that contains data files in a supported format, such as Parquet or CSV.
The result that will occur after running the second command is that the table reference in the metastore is updated and no data is changed. The metastore is a service that stores metadata about tables, such as their schema, location, properties, and partitions. The metastore allows users to access tables using SQL commands or Spark APIs without knowing their physical location or format. When renaming an external table using the ALTER TABLE RENAME TO command, only the table reference in the metastore is updated with the new name; no data files or directories are moved or changed in the storage system. The table will still point to the same location and use the same format as before. However, if renaming a managed table, which is a table whose metadata and data are both managed by Databricks, both the table reference in the metastore and the data files in the default warehouse directory are moved and renamed accordingly. Verified References:
[Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "ALTER TABLE RENAME TO" section; Databricks Documentation, under "Metastore" section; Databricks Documentation, under "Managed and external tables" section.
質問 # 36
Which of the following Structured Streaming queries is performing a hop from a bronze table to a Silver table?
- A. 1.(spark.table("sales").agg(sum("sales"),sum("units"))
2..writeStream
3..option("checkpointLocation",checkpointPath)
4..outputMode("complete")
5..table("aggregatedSales")) - B. 1.(spark.read.load(rawSalesLocation)
2..writeStream
3..option("checkpointLocation", checkpointPath)
4..outputMode("append")
5..table("uncleanedSales") ) - C. 1.(spark.readStream.load(rawSalesLocation)
2..writeStream
3..option("checkpointLocation", checkpointPath)
4..outputMode("append")
5..table("uncleanedSales") ) - D. 1.(spark.table("sales").groupBy("store")
2..agg(sum("sales")).writeStream
3..option("checkpointLocation",checkpointPath)
4..outputMode("complete")
5..table("aggregatedSales")) - E. 1.(spark.table("sales")
2..withColumn("avgPrice", col("sales") / col("units"))
3..writeStream
4..option("checkpointLocation", checkpointPath)
5..outputMode("append")
6..table("cleanedSales"))
正解：E
解説：
Explanation
A diagram of a house Description automatically generated with low confidence
質問 # 37
Spill occurs as a result of executing various wide transformations. However, diagnosing spill requires one to proactively look for key indicators.
Where in the Spark UI are two of the primary indicators that a partition is spilling to disk?
- A. Driver's and Executor's log files
- B. Stage's detail screen and Executor's files
- C. Stage's detail screen and Query's detail screen
- D. Executor's detail screen and Executor's log files
正解：C
解説：
In Apache Spark's UI, indicators of data spilling to disk during the execution of wide transformations can be found in the Stage's detail screen and the Query's detail screen. These screens provide detailed metrics about each stage of a Spark job, including information about memory usage and spill data. If a task is spilling data to disk, it indicates that the data being processed exceeds the available memory, causing Spark to spill data to disk to free up memory. This is an important performance metric as excessive spill can significantly slow down the processing.
Reference:
Apache Spark Monitoring and Instrumentation: Spark Monitoring Guide
Spark UI Explained: Spark UI Documentation
質問 # 38
Which statement describes Delta Lake Auto Compaction?
- A. Optimized writes use logical partitions instead of directory partitions; because partition boundaries are only represented in metadata, fewer small files are written.
- B. An asynchronous job runs after the write completes to detect if files could be further compacted; if yes, an optimize job is executed toward a default of 128 MB.
- C. Before a Jobs cluster terminates, optimize is executed on all tables modified during the most recent job.
- D. An asynchronous job runs after the write completes to detect if files could be further compacted; if yes, an optimize job is executed toward a default of 1 GB.
- E. Data is queued in a messaging bus instead of committing data directly to memory; all data is committed from the messaging bus in one batch once the job is complete.
正解：B
解説：
Explanation
This is the correct answer because it describes the behavior of Delta Lake Auto Compaction, which is a feature that automatically optimizes the layout of Delta Lake tables by coalescing small files into larger ones. Auto Compaction runs as an asynchronous job after a write to a table has succeeded and checks if files within a partition can be further compacted. If yes, it runs an optimize job with a default target file size of 128 MB.
Auto Compaction only compacts files that have not been compacted previously. Verified References:
[Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "Auto Compaction for Delta Lake on Databricks" section.
"Auto compaction occurs after a write to a table has succeeded and runs synchronously on the cluster that has performed the write. Auto compaction only compacts files that haven't been compacted previously."
https://learn.microsoft.com/en-us/azure/databricks/delta/tune-file-size
質問 # 39
Which of the following data workloads will utilize a Bronze table as its destination?
- A. A job that develops a feature set for a machine learning application
- B. A job that ingests raw data from a streaming source into the Lakehouse
- C. A job that enriches data by parsing its timestamps into a human-readable format
- D. A job that queries aggregated data to publish key insights into a dashboard
- E. A job that aggregates cleaned data to create standard summary statistics
正解：B
解説：
Explanation
The answer is A job that ingests raw data from a streaming source into the Lakehouse.
The ingested data from the raw streaming data source like Kafka is first stored in the Bronze layer as first destination before it is further optimized and stored in Silver.
Medallion Architecture - Databricks
Bronze Layer:
1. Raw copy of ingested data
2. Replaces traditional data lake
3. Provides efficient storage and querying of full, unprocessed history of data
4. No schema is applied at this layer
Exam focus: Please review the below image and understand the role of each layer(bronze, silver, gold) in medallion architecture, you will see varying questions targeting each layer and its purpose.
Sorry I had to add the watermark some people in Udemy are copying my content.
Purpose of each layer in medallion architecture
質問 # 40
Incorporating unit tests into a PySpark application requires upfront attention to the design of your jobs, or a potentially significant refactoring of existing code.
Which statement describes a main benefit that offset this additional effort?
- A. Yields faster deployment and execution times
- B. Troubleshooting is easier since all steps are isolated and tested individually
- C. Validates a complete use case of your application
- D. Ensures that all steps interact correctly to achieve the desired end result
- E. Improves the quality of your data
正解：B
質問 # 41
You noticed that a team member started using an all-purpose cluster to develop a notebook and used the same all-purpose cluster to set up a job that can run every 30 mins so they can update un-derlying tables which are used in a dashboard. What would you recommend for reducing the overall cost of this approach?
- A. Enable auto termination after 30 mins
- B. Reduce the size of the cluster
- C. Change the cluster all-purpose to job cluster when scheduling the job
- D. Reduce the number of nodes and enable auto scale
- E. Change the cluster mode from all-purpose to single-mode
正解：C
解説：
Explanation
While using an all-purpose cluster is ok during development but anytime you don't need to interact with a notebook, especially for a scheduled job it is less expensive to use a job cluster. Using an all-purpose cluster can be twice as expensive as a job cluster.
Please note: The compute cost you pay the cloud provider for the same cluster type and size be-tween an all-purpose cluster and job cluster is the same the only difference is the DBU cost.
The total cost of cluster = Total cost of VM compute(Azure or AWS or GCP) + Cost per DBU The per DBU cost varies between all-purpose and Job Cluster Here is the recent cost estimate from AWS between Jobs Cluster and all-purpose Cluster, for jobs compute its
$0.15 cents per DBU v$0.55 cents per DBU for all-purpose
Graphical user interface Description automatically generated
How do I check how much the DBU cost for my cluster?
When you click on an exister cluster or when you look at the cluster details you will see this in the top right corner Graphical user interface, text, application, email Description automatically generated
質問 # 42
You are working on a email spam filtering assignment, while working on this you find there is new word e.g.
HadoopExam comes in email, and in your solutions you never come across this word before, hence probability
of this words is coming in either email could be zero. So which of the following algorithm can help you to
avoid zero probability?
- A. Logistic Regression
- B. All of the above
- C. Naive Bayes
- D. Laplace Smoothing
正解：D
解説：
Explanation
Laplace smoothing is a technique for parameter estimation which accounts for unobserved events. It is more
robust and will not fail completely when data that has never been observed in training shows up.
質問 # 43
When scheduling Structured Streaming jobs for production, which configuration automatically recovers from query failures and keeps costs low?
- A. Cluster: New Job Cluster;
Retries: None;
Maximum Concurrent Runs: 1 - B. Cluster: Existing All-Purpose Cluster;
Retries: Unlimited;
Maximum Concurrent Runs: 1 - C. Cluster: New Job Cluster;
Retries: Unlimited;
Maximum Concurrent Runs: Unlimited - D. Cluster: Existing All-Purpose Cluster;
Retries: Unlimited;
Maximum Concurrent Runs: 1 - E. Cluster: Existing All-Purpose Cluster;
Retries: None;
Maximum Concurrent Runs: 1
正解：D
解説：
The configuration that automatically recovers from query failures and keeps costs low is to use a new job cluster, set retries to unlimited, and set maximum concurrent runs to 1. This configuration has the following advantages:
A new job cluster is a cluster that is created and terminated for each job run. This means that the cluster resources are only used when the job is running, and no idle costs are incurred. This also ensures that the cluster is always in a clean state and has the latest configuration and libraries for the job1.
Setting retries to unlimited means that the job will automatically restart the query in case of any failure, such as network issues, node failures, or transient errors. This improves the reliability and availability of the streaming job, and avoids data loss or inconsistency2.
Setting maximum concurrent runs to 1 means that only one instance of the job can run at a time. This prevents multiple queries from competing for the same resources or writing to the same output location, which can cause performance degradation or data corruption3.
Therefore, this configuration is the best practice for scheduling Structured Streaming jobs for production, as it ensures that the job is resilient, efficient, and consistent.
質問 # 44
......
DataBricks認定プロフェッショナルデータエンジニア試験は、DataBricksプラットフォーム上のデータパイプラインの構築と最適化のスキルを検証したいデータ専門家向けに設計されています。この試験は、DataBrickを使用して堅牢でスケーラブルで効率的なデータパイプラインの設計、構築、および維持に必要なデータエンジニアリングの原則、ベストプラクティス、および実用的なスキルを包括的に理解することを目的としています。この認定試験は世界的に認識されており、雇用市場で競争力を提供します。
検証済み材料を使うならまずDatabricks-Certified-Professional-Data-Engineerテストエンジンを試そう：https://www.jpntest.com/shiken/Databricks-Certified-Professional-Data-Engineer-mondaishu
合格するに必要な問題集はDatabricks-Certified-Professional-Data-Engineer試験：https://drive.google.com/open?id=142bVxoXbOyOzFxVm9MkhvOR5R5_CUv8m