試験問題と解答はDatabricks-Certified-Professional-Data-Engineer学習ガイド問題を試そう！
Databricks Certified Professional Data Engineer Exam認証サンプル問題と練習試験合格させます
Databricks認定プロフェッショナルデータエンジニアの試験では、データの摂取、変換、ストレージ、処理など、幅広いトピックについて説明しています。この試験では、DataBricksツールとテクノロジーを使用して、実際の問題と課題を解決する候補者の能力をテストします。この試験に合格した候補者は、ビッグデータ処理と分析のための主要なクラウドベースのプラットフォームであるDataBricksを使用して、データパイプラインの設計、構築、管理に習熟していることを示しています。
Databricks Databricks-Certified-Professional-Data-Engineer 認定試験の出題範囲：
| トピック | 出題範囲 |
|---|---|
| トピック 1 |
|
| トピック 2 |
|
| トピック 3 |
|
| トピック 4 |
|
| トピック 5 |
|
質問 # 32
A nightly job ingests data into a Delta Lake table using the following code:
The next step in the pipeline requires a function that returns an object that can be used to manipulate new records that have not yet been processed to the next table in the pipeline.
Which code snippet completes this function definition?
def new_records():
A . return spark.readStream.table("bronze")
B . return spark.readStream.load("bronze")
C .
D .
return spark.read.option("readChangeFeed", "true").table ("bronze")
E .
- A. Option D
- B. Option A
- C. Option C
- D. Option B
- E. Option E
正解：E
解説：
https://docs.databricks.com/en/delta/delta-change-data-feed.html
質問 # 33
Which of the statements is correct when choosing between lakehouse and Datawarehouse?
- A. Traditional Data warehouses have special indexes which are optimized for Machine learning
- B. Lakehouse replaces the current dependency on data lakes and data warehouses uses an open standard storage format and supports low latency BI workloads.
- C. Traditional Data warehouses are the preferred choice if we need to support ACID, Lakehouse does not support ACID.
- D. SQL support is only available for Traditional Datawarehouse's, Lakehouses support Python and Scala
- E. Traditional Data warehouses can serve low query latency with high reliability for BI workloads
正解：B
解説：
Explanation
The lakehouse replaces the current dependency on data lakes and data warehouses for modern data companies that desire:
* Open, direct access to data stored in standard data formats.
* Indexing protocols optimized for machine learning and data science.
* Low query latency and high reliability for BI and advanced analytics.
質問 # 34
The following table consists of items found in user carts within an e-commerce website.
The following MERGE statement is used to update this table using an updates view, with schema evaluation enabled on this table.
How would the following update be handled?
- A. The update is moved to separate ''restored'' column because it is missing a column expected in the target schema.
- B. The new nested field is added to the target schema, and files underlying existing records are updated to include NULL values for the new field.
- C. The update throws an error because changes to existing columns in the target schema are not supported.
- D. The new restored field is added to the target schema, and dynamically read as NULL for existing unmatched records.
正解：B
解説：
With schema evolution enabled in Databricks Delta tables, when a new field is added to a record through a MERGE operation, Databricks automatically modifies the table schema to include the new field. In existing records where this new field is not present, Databricks will insert NULL values for that field. This ensures that the schema remains consistent across all records in the table, with the new field being present in every record, even if it is NULL for records that did not originally include it.
Reference:
Databricks documentation on schema evolution in Delta Lake: https://docs.databricks.com/delta/delta-batch.html#schema-evolution
質問 # 35
Which of the following are stored in the control pane of Databricks Architecture?
- A. Databricks Filesystem
- B. Job Clusters
- C. Databricks Web Application
- D. Delta tables
- E. All Purpose Clusters
正解：C
解説：
Explanation
The answer is Databricks Web Application
Azure Databricks architecture overview - Azure Databricks | Microsoft Docs Databricks operates most of its services out of a control plane and a data plane, please note serverless features like SQL Endpoint and DLT compute use shared compute in Control pane.
Control Plane: Stored in Databricks Cloud Account
* The control plane includes the backend services that Databricks manages in its own Azure account.
Notebook commands and many other workspace configurations are stored in the control plane and encrypted at rest.
Data Plane: Stored in Customer Cloud Account
* The data plane is managed by your Azure account and is where your data resides. This is also where data is processed. You can use Azure Databricks connectors so that your clusters can connect to external data sources outside of your Azure account to ingest data or for storage.
Timeline Description automatically generated
Bottom of Form
Top of Form
質問 # 36
The data engineering team maintains a table of aggregate statistics through batch nightly updates. This includes total sales for the previous day alongside totals and averages for a variety of time periods including the 7 previous days, year-to-date, and quarter-to-date. This table is namedstore_saies_summaryand the schema is as follows:
The tabledaily_store_salescontains all the information needed to updatestore_sales_summary. The schema for this table is:
store_id INT, sales_date DATE, total_sales FLOAT
Ifdaily_store_salesis implemented as a Type 1 table and the column might be adjusted after manual data auditing, which approach is the safest to generate accurate reports in thestore_sales_summarytable?
- A. Implement the appropriate aggregate logic as a batch read against the daily_store_sales table and use upsert logic to update results in the store_sales_summary table.
- B. Implement the appropriate aggregate logic as a batch read against the daily_store_sales table and overwrite the store_sales_summary table with each Update.
- C. Use Structured Streaming to subscribe to the change data feed for daily_store_sales and apply changes to the aggregates in the store_sales_summary table with each update.
- D. Implement the appropriate aggregate logic as a Structured Streaming read against the daily_store_sales table and use upsert logic to update results in the store_sales_summary table.
- E. Implement the appropriate aggregate logic as a batch read against the daily_store_sales table and append new rows nightly to the store_sales_summary table.
正解：C
解説：
Explanation
The daily_store_sales table contains all the information needed to update store_sales_summary. The schema of the table is:
store_id INT, sales_date DATE, total_sales FLOAT
The daily_store_sales table is implemented as a Type 1 table, which means that old values are overwritten by new values and no history is maintained. The total_sales column might be adjusted after manual data auditing, which means that the data in the table may change over time.
The safest approach to generate accurate reports in the store_sales_summary table is to use Structured Streaming to subscribe to the change data feed for daily_store_sales and apply changes to the aggregates in the store_sales_summary table with each update. Structured Streaming is a scalable and fault-tolerant stream processing engine built on Spark SQL. Structured Streaming allows processing data streams as if they were tables or DataFrames, using familiar operations such as select, filter, groupBy, or join. Structured Streaming also supports output modes that specify how to write the results of a streaming query to a sink, such as append, update, or complete. Structured Streaming can handle both streaming and batch data sources in a unified manner.
The change data feed is a feature of Delta Lake that provides structured streaming sources that can subscribe to changes made to a Delta Lake table. The change data feed captures both data changes and schema changes as ordered events that can be processed by downstream applications or services. The change data feed can be configured with different options, such as starting from a specific version or timestamp, filtering by operation type or partition values, or excluding no-op changes.
By using Structured Streaming to subscribe to the change data feed for daily_store_sales, one can capture and process any changes made to the total_sales column due to manual data auditing. By applying these changes to the aggregates in the store_sales_summary table with each update, one can ensure that the reports are always consistent and accurate with the latest data. Verified References: [Databricks Certified Data Engineer Professional], under "Spark Core" section; Databricks Documentation, under "Structured Streaming" section; Databricks Documentation, under "Delta Change Data Feed" section.
質問 # 37
A junior data engineer has been asked to develop a streaming data pipeline with a grouped aggregation using DataFrame df. The pipeline needs to calculate the average humidity and average temperature for each non-overlapping five-minute interval. Events are recorded once per minute per device.
Streaming DataFrame df has the following schema:
"device_id INT, event_time TIMESTAMP, temp FLOAT, humidity FLOAT"
Code block:
Choose the response that correctly fills in the blank within the code block to complete this task.
- A. lag("event_time", "10 minutes").alias("time")
- B. window("event_time", "5 minutes").alias("time")
- C. "event_time"
- D. window("event_time", "10 minutes").alias("time")
- E. to_interval("event_time", "5 minutes").alias("time")
正解：B
解説：
This is the correct answer because the window function is used to group streaming data by time intervals. The window function takes two arguments: a time column and a window duration. The window duration specifies how long each window is, and must be a multiple of 1 second. In this case, the window duration is "5 minutes", which means each window will cover a non-overlapping five-minute interval. The window function also returns a struct column with two fields: start and end, which represent the start and end time of each window. The alias function is used to rename the struct column as "time". Verified Reference: [Databricks Certified Data Engineer Professional], under "Structured Streaming" section; Databricks Documentation, under "WINDOW" section. https://www.databricks.com/blog/2017/05/08/event-time-aggregation-watermarking-apache-sparks-structured-streaming.html
質問 # 38
The data governance team is reviewing code used for deleting records for compliance with GDPR. They note the following logic is used to delete records from the Delta Lake table named users.
Assuming that user_id is a unique identifying key and that delete_requests contains all users that have requested deletion, which statement describes whether successfully executing the above logic guarantees that the records to be deleted are no longer accessible and why?
- A. Yes; the Delta cache immediately updates to reflect the latest data files recorded to disk.
- B. No; the Delta Lake delete command only provides ACID guarantees when combined with the merge into command.
- C. Yes; Delta Lake ACID guarantees provide assurance that the delete command succeeded fully and permanently purged these records.
- D. No; files containing deleted records may still be accessible with time travel until a vacuum command is used to remove invalidated data files.
- E. No; the Delta cache may return records from previous versions of the table until the cluster is restarted.
正解：D
解説：
The code uses the DELETE FROM command to delete records from the users table that match a condition based on a join with another table called delete_requests, which contains all users that have requested deletion.
The DELETE FROM command deletes records from a Delta Lake table by creating a new version of the table that does not contain the deleted records. However, this does not guarantee that the records to be deleted are no longer accessible, because Delta Lake supports time travel, which allows querying previous versions of the table using a timestamp or version number. Therefore, files containing deleted records may still be accessible with time travel until a vacuum command is used to remove invalidated data files from physical storage.
Verified References: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "Delete from a table" section; Databricks Documentation, under "Remove files no longer referenced by a Delta table" section.
質問 # 39
Which statement describes integration testing?
- A. Validates an application use case
- B. Validates interactions between subsystems of your application
- C. Requires an automated testing framework
- D. Validates behavior of individual elements of your application
- E. Requires manual intervention
正解：A
解説：
This is the correct answer because it describes integration testing. Integration testing is a type of testing that validates interactions between subsystems of your application, such as modules, components, or services.
Integration testing ensures that the subsystems work together as expected and produce the correct outputs or results. Integration testing can be done at different levels of granularity, such as component integration testing, system integration testing, or end-to-end testing. Integration testing can help detect errors or bugs that may not be found by unit testing, which only validates behavior of individual elements of your application. Verified References: [Databricks Certified Data Engineer Professional], under "Testing" section; Databricks Documentation, under "Integration testing" section.
質問 # 40
Kevin is the owner of the schema sales, Steve wanted to create new table in sales schema called regional_sales so Kevin grants the create table permissions to Steve. Steve creates the new table called regional_sales in sales schema, who is the owner of the table regional_sales
- A. Kevin and Smith both are owners of table
- B. By default ownership is assigned to DEFAULT_OWNER
- C. By default ownership is assigned DBO
- D. Kevin is the owner of sales schema, all the tables in the schema will be owned by Kevin
- E. Steve is the owner of the table
正解：E
解説：
Explanation
A user who creates the object becomes its owner, does not matter who is the owner of the parent object.
質問 # 41
You have noticed the Data scientist team is using the notebook versioning feature with git integra-tion, you have recommended them to switch to using Databricks Repos, which of the below reasons could be the reason the why the team needs to switch to Databricks Repos.
- A. Databricks Repos allows multiple users to make changes
- B. Databricks Repos allows merge and conflict resolution
- C. Databricks Repos has a built-in version control system
- D. Databricks Repos allow you to add comments and select the changes you want to commit.
- E. Databricks Repos automatically saves changes
正解：D
解説：
Explanation
The answer is Databricks Repos allow you to add comments and select the changes you want to commit.
質問 # 42
Which of the following technologies can be used to identify key areas of text when parsing Spark Driver log4j output?
- A. pyspsark.ml.feature
- B. C++
- C. Regex
- D. Scala Datasets
- E. Julia
正解：C
解説：
Regex, or regular expressions, are a powerful way of matching patterns in text. They can be used to identify key areas of text when parsing Spark Driver log4j output, such as the log level, the timestamp, the thread name, the class name, the method name, and the message. Regex can be applied in various languages and frameworks, such as Scala, Python, Java, Spark SQL, and Databricks notebooks. References:
* https://docs.databricks.com/notebooks/notebooks-use.html#use-regular-expressions
* https://docs.databricks.com/spark/latest/spark-sql/udf-scala.html#using-regular-expressions-in-udfs
* https://docs.databricks.com/spark/latest/sparkr/functions/regexp_extract.html
* https://docs.databricks.com/spark/latest/sparkr/functions/regexp_replace.html
質問 # 43
An upstream system is emitting change data capture (CDC) logs that are being written to a cloud object storage directory. Each record in the log indicates the change type (insert, update, or delete) and the values for each field after the change. The source table has a primary key identified by the fieldpk_id.
For auditing purposes, the data governance team wishes to maintain a full record of all values that have ever been valid in the source system. For analytical purposes, only the most recent value for each record needs to be recorded. The Databricks job to ingest these records occurs once per hour, but each individual record may have changed multiple times over the course of an hour.
Which solution meets these requirements?
- A. Use merge into to insert, update, or delete the most recent entry for each pk_id into a bronze table, then propagate all changes throughout the system.
- B. Iterate through an ordered set of changes to the table, applying each in turn; rely on Delta Lake's versioning ability to create an audit log.
- C. Create a separate history table for each pk_id resolve the current state of the table by running a union all filtering the history tables for the most recent state.
- D. Use Delta Lake's change data feed to automatically process CDC data from an external system, propagating all changes to all dependent tables in the Lakehouse.
- E. Ingest all log information into a bronze table; use merge into to insert, update, or delete the most recent entry for each pk_id into a silver table to recreate the current table state.
正解：A
解説：
This is the correct answer because it meets the requirements of maintaining a full record of all values that have ever been valid in the source system and recreating the current table state with only the most recent value for each record. The code ingests all log information into a bronze table, which preserves the raw CDC data as it is. Then, it uses merge into to perform an upsert operation on a silver table, which means it will insert new records or update or delete existing records based on the change type and the pk_id columns. This way, the silver table will always reflect the current state of the source table, while the bronze table will keep the history of all changes. Verified References: [Databricks Certified Data Engineer Professional], under "Delta Lake" section; Databricks Documentation, under "Upsert into a table using merge" section.
質問 # 44
You are currently working on a production job failure with a job set up in job clusters due to a data issue, what cluster do you need to start to investigate and analyze the data?
- A. Existing job cluster can be used to investigate the issue
- B. A Job cluster can be used to analyze the problem
- C. All-purpose cluster/ interactive cluster is the recommended way to run commands and view the data.
- D. Databricks SQL Endpoint can be used to investigate the issue
正解：C
解説：
Explanation
Answer is All-purpose cluster/ interactive cluster is the recommended way to run commands and view the data.
A job cluster can not provide a way for a user to interact with a notebook once the job is submitted, but an Interactive cluster allows to you display data, view visualizations write or edit quries, which makes it a perfect fit to investigate and analyze the data.
質問 # 45
A data architect has heard about lake's built-in versioning and time travel capabilities. For auditing purposes they have a requirement to maintain a full of all valid street addresses as they appear in the customers table.
The architect is interested in implementing a Type 1 table, overwriting existing records with new values and relying on Delta Lake time travel to support long-term auditing. A data engineer on the project feels that a Type 2 table will provide better performance and scalability.
Which piece of information is critical to this decision?
- A. Delta Lake time travel does not scale well in cost or latency to provide a long-term versioning solution.
- B. Data corruption can occur if a query fails in a partially completed state because Type 2 tables requires
- C. Delta Lake time travel cannot be used to query previous versions of these tables because Type 1 changes modify data files in place.
- D. Shallow clones can be combined with Type 1 tables to accelerate historic queries for long-term versioning.
正解：A
解説：
Setting multiple fields in a single update.
Explanation:
Delta Lake's time travel feature allows users to access previous versions of a table, providing a powerful tool for auditing and versioning. However, using time travel as a long-term versioning solution for auditing purposes can be less optimal in terms of cost and performance, especially as the volume of data and the number of versions grow. For maintaining a full history of valid street addresses as they appear in a customers table, using a Type 2 table (where each update creates a new record with versioning) might provide better scalability and performance by avoiding the overhead associated with accessing older versions of a large table. While Type 1 tables, where existing records are overwritten with new values, seem simpler and can leverage time travel for auditing, the critical piece of information is that time travel might not scale well in cost or latency for long-term versioning needs, making a Type 2 approach more viable for performance and scalability.
Reference:
Databricks Documentation on Delta Lake's Time Travel: Delta Lake Time Travel Databricks Blog on Managing Slowly Changing Dimensions in Delta Lake: Managing SCDs in Delta Lake
質問 # 46
Which REST API call can be used to review the notebooks configured to run as tasks in a multi-task job?
- A. /jobs/runs/list
- B. /jobs/runs/get-output
- C. /jobs/runs/get
- D. /jobs/list
- E. /jobs/get
正解：E
解説：
Explanation
This is the correct answer because it is the REST API call that can be used to review the notebooks configured to run as tasks in a multi-task job. The REST API is an interface that allows programmatically interacting with Databricks resources, such as clusters, jobs, notebooks, or tables. The REST API uses HTTP methods, such as GET, POST, PUT, or DELETE, to perform operations on these resources. The /jobs/get endpoint is a GET method that returns information about a job given its job ID. The information includes the job settings, such as the name, schedule, timeout, retries, email notifications, and tasks. The tasks are the units of work that a job executes. A task can be a notebook task, which runs a notebook with specified parameters; a jar task, which runs a JAR uploaded to DBFS with specified main class and arguments; or a python task, which runs a Python file uploaded to DBFS with specified parameters. A multi-task job is a job that has more than one task configured to run in a specific order or in parallel. By using the /jobs/get endpoint, one can review the notebooks configured to run as tasks in a multi-task job. Verified References: [Databricks Certified Data Engineer Professional], under "Databricks Jobs" section; Databricks Documentation, under "Get" section; Databricks Documentation, under "JobSettings" section.
質問 # 47
The security team is exploring whether or not the Databricks secrets module can be leveraged for connecting to an external database.
After testing the code with all Python variables being defined with strings, they upload the password to the secrets module and configure the correct permissions for the currently active user. They then modify their code to the following (leaving all other variables unchanged).
Which statement describes what will happen when the above code is executed?
- A. The connection to the external table will succeed; the string "redacted" will be printed.
- B. The connection to the external table will fail; the string "redacted" will be printed.
- C. An interactive input box will appear in the notebook; if the right password is provided, the connection will succeed and the password will be printed in plain text.
- D. An interactive input box will appear in the notebook; if the right password is provided, the connection will succeed and the encoded password will be saved to DBFS.
- E. The connection to the external table will succeed; the string value of password will be printed in plain text.
正解：A
解説：
This is the correct answer because the code is using the dbutils.secrets.get method to retrieve the password from the secrets module and store it in a variable. The secrets module allows users to securely store and access sensitive information such as passwords, tokens, or API keys. The connection to the external table will succeed because the password variable will contain the actual password value. However, when printing the password variable, the string "redacted" will be displayed instead of the plain text password, as a security measure to prevent exposing sensitive information in notebooks. Verified Reference: [Databricks Certified Data Engineer Professional], under "Security & Governance" section; Databricks Documentation, under "Secrets" section.
質問 # 48
......
Databricks-Certified-Professional-Data-Engineer認証問題集Databricks Certification Databricks-Certified-Professional-Data-Engineerガイド 100%有効：https://www.jpntest.com/shiken/Databricks-Certified-Professional-Data-Engineer-mondaishu
100%合格率の問題集を使ってあなたのDatabricks-Certified-Professional-Data-Engineer試験を一発合格：https://drive.google.com/open?id=159vdVUHS9E8MqH9FsTgl115Zk1bVVHxQ