お手軽に合格させる 最新Databricks Databricks-Certified-Data-Engineer-Associate問題集には102問があります
最新のDatabricks-Certified-Data-Engineer-Associate学習ガイド2024年最新の- 提供するのはテストエンジンとPDF
質問 # 19
A data engineer has a Job with multiple tasks that runs nightly. Each of the tasks runs slowly because the clusters take a long time to start.
Which of the following actions can the data engineer perform to improve the start up time for the clusters used for the Job?
- A. They can use clusters that are from a cluster pool
- B. They can use endpoints available in Databricks SQL
- C. They can use jobs clusters instead of all-purpose clusters
- D. They can configure the clusters to autoscale for larger data sizes
- E. They can configure the clusters to be single-node
正解：A
解説：
The best action that the data engineer can perform to improve the start up time for the clusters used for the Job is to use clusters that are from a cluster pool. A cluster pool is a set of idle clusters that can be used by jobs or interactive sessions. By using a cluster pool, the data engineer can avoid the cluster creation time and reduce the latency of the tasks. Cluster pools also offer cost savings and resource efficiency, as they can be shared by multiple users and jobs.
Option A is not relevant, as endpoints available in Databricks SQL are used for creating and managing SQL analytics workloads, not for improving cluster start up time.
Option B is not correct, as jobs clusters and all-purpose clusters have similar start up times. Jobs clusters are clusters that are dedicated to run a single job and are terminated when the job is completed. All-purpose clusters are clusters that can be used for multiple purposes, such as interactive sessions, notebooks, or multiple jobs. Both types of clusters can benefit from using a cluster pool.
Option C is not advisable, as configuring the clusters to be single-node will reduce the parallelism and performance of the tasks. Single-node clusters are clusters that have only one worker node and are typically used for testing or development purposes. They are not suitable for running production jobs that require high scalability and fault tolerance.
Option E is not helpful, as configuring the clusters to autoscale for larger data sizes will not affect the start up time of the clusters. Autoscaling is a feature that allows clusters to dynamically adjust the number of worker nodes based on the workload. It can help optimize the resource utilization and cost efficiency of the clusters, but it does not speed up the cluster creation process.
Reference:
Cluster Pools
Jobs
Clusters
[Databricks Data Engineer Professional Exam Guide]
質問 # 20
A data engineer needs to create a table in Databricks using data from their organization's existing SQLite database.
They run the following command:
Which of the following lines of code fills in the above blank to successfully complete the task?
- A. DELTA
- B. sqlite
- C. org.apache.spark.sql.sqlite
- D. org.apache.spark.sql.jdbc
- E. autoloader
正解：D
解説：
Explanation
CREATE TABLE new_employees_table
USING JDBC
OPTIONS (
url "<jdbc_url>",
dbtable "<table_name>",
user '<username>',
password '<password>'
) AS
SELECT * FROM employees_table_vw
https://docs.databricks.com/external-data/jdbc.html#language-sql
質問 # 21
Which of the following approaches should be used to send the Databricks Job owner an email in the case that the Job fails?
- A. Setting up an Alert in the Notebook
- B. Setting up an Alert in the Job page
- C. Manually programming in an alert system in each cell of the Notebook
- D. There is no way to notify the Job owner in the case of Job failure
- E. MLflow Model Registry Webhooks
正解：B
解説：
To send the Databricks Job owner an email in the case that the Job fails, the best approach is to set up an Alert in the Job page. This way, the Job owner can configure the email address and the notification type for the Job failure event. The other options are either not feasible, not reliable, or not relevant for this task. Manually programming an alert system in each cell of the Notebook is tedious and error-prone. Setting up an Alert in the Notebook is not possible, as Alerts are only available for Jobs and Clusters. There is a way to notify the Job owner in the case of Job failure, so option D is incorrect. MLflow Model Registry Webhooks are used for model lifecycle events, not Job events, so option E is not applicable. References:
* Add email and system notifications for job events
* Alerts
* MLflow Model Registry Webhooks
質問 # 22
A data engineer has three tables in a Delta Live Tables (DLT) pipeline. They have configured the pipeline to drop invalid records at each table. They notice that some data is being dropped due to quality concerns at some point in the DLT pipeline. They would like to determine at which table in their pipeline the data is being dropped.
Which of the following approaches can the data engineer take to identify the table that is dropping the records?
- A. They can navigate to the DLT pipeline page, click on each table, and view the data quality statistics.
- B. They can set up separate expectations for each table when developing their DLT pipeline.
- C. They can set up DLT to notify them via email when records are dropped.
- D. They cannot determine which table is dropping the records.
- E. They can navigate to the DLT pipeline page, click on the "Error" button, and review the present errors.
正解：A
解説：
One of the features of DLT is that it provides data quality metrics for each dataset in the pipeline, such as the number of records that pass or fail expectations, the number of records that are dropped, and the number of records that are written to the target. These metrics can be accessed from the DLT pipeline page, where the data engineer can click on each table and view the data quality statistics for the latest update or any previous update. This way, they can identify which table is dropping the records and why. References:
* Monitor Delta Live Tables pipelines
* Manage data quality with Delta Live Tables
質問 # 23
A data engineer wants to create a new table containing the names of customers that live in France.
They have written the following command:
A senior data engineer mentions that it is organization policy to include a table property indicating that the new table includes personally identifiable information (PII).
Which of the following lines of code fills in the above blank to successfully complete the task?
- A. There is no way to indicate whether a table contains PII.
- B. COMMENT "Contains PII"
- C. PII
- D. TBLPROPERTIES PII
- E. "COMMENT PII"
正解：B
解説：
In Databricks, when creating a table, you can add a comment to columns or the entire table to provide more information about the data it contains. In this case, since it's organization policy toindicate that the new table includes personally identifiable information (PII), option D is correct. The line of code would be added after defining the table structure and before closing with a semicolon. References: Data Engineer Associate Exam Guide, CREATE TABLE USING (Databricks SQL)
質問 # 24
A data analysis team has noticed that their Databricks SQL queries are running too slowly when connected to their always-on SQL endpoint. They claim that this issue is present when many members of the team are running small queries simultaneously. They ask the data engineering team for help. The data engineering team notices that each of the team's queries uses the same SQL endpoint.
Which of the following approaches can the data engineering team use to improve the latency of the team's queries?
- A. They can turn on the Serverless feature for the SQL endpoint and change the Spot Instance Policy to
"Reliability Optimized." - B. They can turn on the Serverless feature for the SQL endpoint.
- C. They can increase the cluster size of the SQL endpoint.
- D. They can increase the maximum bound of the SQL endpoint's scaling range.
- E. They can turn on the Auto Stop feature for the SQL endpoint.
正解：C
解説：
Explanation
When many users are running small queries simultaneously on a SQL endpoint, the database can become overloaded, causing slow query execution times. By increasing the cluster size of the SQL endpoint, the database can handle more simultaneous queries, resulting in faster query execution times.
質問 # 25
A data analyst has developed a query that runs against Delta table. They want help from the data engineering team to implement a series of tests to ensure the data returned by the query is clean. However, the data engineering team uses Python for its tests rather than SQL.
Which of the following operations could the data engineering team use to run the query and operate with the results in PySpark?
- A. There is no way to share data between PySpark and SQL.
- B. spark.delta.table
- C. spark.sql
- D. SELECT * FROM sales
- E. spark.table
正解：C
解説：
Explanation
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = spark.sql("SELECT * FROM sales")
print(df.count())
質問 # 26
A new data engineering team has been assigned to work on a project. The team will need access to database customers in order to see what tables already exist. The team has its own group team.
Which of the following commands can be used to grant the necessary permission on the entire database to the new team?
- A. GRANT CREATE ON DATABASE team TO customers;
- B. GRANT CREATE ON DATABASE customers TO team;
- C. GRANT USAGE ON DATABASE customers TO team;
- D. GRANT VIEW ON CATALOG customers TO team;
- E. GRANT USAGE ON CATALOG team TO customers;
正解：C
解説：
Explanation
The GRANT statement is used to grant privileges on a database, table, or view to a user or role. The ALL PRIVILEGES option grants all possible privileges on the specified object, such as CREATE, SELECT, MODIFY, and USAGE. The syntax of the GRANT statement is:
GRANT privilege_type ON object TO user_or_role;
Therefore, to grant full permissions on the database customers to the new data engineering team, the command should be:
GRANT ALL PRIVILEGES ON DATABASE customers TO team;
質問 # 27
A data engineer has developed a data pipeline to ingest data from a JSON source using Auto Loader, but the engineer has not provided any type inference or schema hints in their pipeline. Upon reviewing the data, the data engineer has noticed that all of the columns in the target table are of the string type despite some of the fields only including float or boolean values.
Which of the following describes why Auto Loader inferred all of the columns to be of the string type?
- A. JSON data is a text-based format
- B. Auto Loader only works with string data
- C. There was a type mismatch between the specific schema and the inferred schema
- D. All of the fields had at least one null value
- E. Auto Loader cannot infer the schema of ingested data
正解：A
解説：
JSON data is a text-based format that represents data as a collection of name-value pairs. By default, when Auto Loader infers the schema of JSON data, it treats all columns as strings. This is because JSON data can have varying data types for the same column across different files or records, and Auto Loader does not attempt to reconcile these differences. For example, a column named "age" may have integer values in some files, but string values in others. To avoid data loss or errors, Auto Loader infers the column as a string type.
However, Auto Loader also provides an option to infer more precise column types based on the sample data.
This option is called cloudFiles.inferColumnTypes and it can be set to true or false. When set to true, Auto Loader tries to infer the exact data types of the columns, such as integers, floats, booleans, or nested structures.
When set to false, Auto Loader infers all columns as strings. The default value of this option is false. References: Configure schema inference and evolution in Auto Loader, Schema inference with auto loader (non-DLT and DLT), Using and Abusing Auto Loader's Inferred Schema, Explicit path to data or a defined schema required for Auto loader.
質問 # 28
A data engineer has a Python variable table_name that they would like to use in a SQL query. They want to construct a Python code block that will run the query using table_name.
They have the following incomplete code block:
____(f"SELECT customer_id, spend FROM {table_name}")
Which of the following can be used to fill in the blank to successfully complete the task?
- A. spark.delta.sql
- B. spark.delta.table
- C. spark.sql
- D. spark.table
- E. dbutils.sql
正解：C
解説：
The spark.sql method can be used to execute SQL queries programmatically and return the result as a DataFrame. The spark.sql method accepts a string argument that contains a valid SQL statement. The data engineer can use a formatted string literal (f-string) to insert the Python variable table_name into the SQL query. The other methods are either invalid or not suitable for running SQL queries. Reference: Running SQL Queries Programmatically, Formatted string literals, spark.sql
質問 # 29
A data engineer needs to determine whether to use the built-in Databricks Notebooks versioning or version their project using Databricks Repos.
Which of the following is an advantage of using Databricks Repos over the Databricks Notebooks versioning?
- A. Databricks Repos automatically saves development progress
- B. Databricks Repos allows users to revert to previous versions of a notebook
- C. Databricks Repos provides the ability to comment on specific changes
- D. Databricks Repos supports the use of multiple branches
- E. Databricks Repos is wholly housed within the Databricks Lakehouse Platform
正解：D
解説：
Databricks Repos is a visual Git client and API in Databricks that supports common Git operations such as cloning, committing, pushing, pulling, and branch management. Databricks Notebooks versioning is a legacy feature that allows users to link notebooks to GitHub repositories and perform basic Git operations. However, Databricks Notebooks versioning does not support the use of multiple branches for development work, which is an advantage of using Databricks Repos. With Databricks Repos, users can create and manage branches for different features, experiments, or bug fixes, and merge, rebase, or resolve conflicts between them. Databricks recommends using a separate branch for each notebook and following data science and engineering code development best practices using Git for version control, collaboration, and CI/CD. Reference: Git integration with Databricks Repos - Azure Databricks | Microsoft Learn, Git version control for notebooks (legacy) | Databricks on AWS, Databricks Repos Is Now Generally Available - New 'Files' Feature in ..., Databricks Repos - What it is and how we can use it | Adatis.
質問 # 30
A data engineer and data analyst are working together on a data pipeline. The data engineer is working on the raw, bronze, and silver layers of the pipeline using Python, and the data analyst is working on the gold layer of the pipeline using SQL. The raw source of the pipeline is a streaming input. They now want to migrate their pipeline to use Delta Live Tables.
Which of the following changes will need to be made to the pipeline when migrating to Delta Live Tables?
- A. The pipeline will need to stop using the medallion-based multi-hop architecture
- B. None of these changes will need to be made
- C. The pipeline will need to be written entirely in Python
- D. The pipeline will need to be written entirely in SQL
- E. The pipeline will need to use a batch source in place of a streaming source
正解：B
解説：
Delta Live Tables is a declarative framework for building reliable, maintainable, and testable data processing pipelines. You define the transformations to perform on your data and Delta Live Tables manages task orchestration, cluster management, monitoring, data quality, and error handling. Delta Live Tables supports both SQL and Python as the languages for defining your datasets and expectations. Delta Live Tables also supports both streaming and batch sources, and can handle both append-only and upsert data patterns. Delta Live Tables follows the medallion lakehouse architecture, which consists of three layers of data: bronze, silver, and gold. Therefore, migrating to Delta Live Tables does not require any of the changes listed in the options B, C, D, or E. The data engineer and data analyst can use the same languages, sources, and architecture as before, and simply declare their datasets and expectations using Delta Live Tables syntax. Reference:
What is Delta Live Tables?
Transform data with Delta Live Tables
What is the medallion lakehouse architecture?
質問 # 31
Which of the following tools is used by Auto Loader process data incrementally?
- A. Databricks SQL
- B. Unity Catalog
- C. Checkpointing
- D. Data Explorer
- E. Spark Structured Streaming
正解：E
解説：
Auto Loader provides a Structured Streaming source called cloudFiles that can process new data files as they arrive in cloud storage without any additional setup. Auto Loader uses a scalable key-value store to track ingestion progress and ensure exactly-once semantics. Auto Loader can ingest various file formats and load them into Delta Lake tables. Auto Loader is recommended for incremental data ingestion with Delta Live Tables, which extends the functionality of Structured Streaming and allows you to write declarative Python or SQL code to deploy a production-quality data pipeline. Reference: What is Auto Loader?, What is Auto Loader? | Databricks on AWS, Solved: How does Auto Loader ingest data? - Databricks - 5629
質問 # 32
A data analyst has a series of queries in a SQL program. The data analyst wants this program to run every day.
They only want the final query in the program to run on Sundays. They ask for help from the data engineering team to complete this task.
Which of the following approaches could be used by the data engineering team to complete this task?
- A. They could only run the entire program on Sundays.
- B. They could wrap the queries using PySpark and use Python's control flow system to determine when to run the final query.
- C. They could automatically restrict access to the source table in the final query so that it is only accessible on Sundays.
- D. They could submit a feature request with Databricks to add this functionality.
- E. They could redesign the data model to separate the data used in the final query into a new table.
正解：B
解説：
This approach would allow the data engineering team to use the existing SQL program and add some logic to control the execution of the final query based on the day of the week. They could use the datetime module in Python to get the current date and check if it is a Sunday. If so, they could run the final query, otherwise they could skip it. This way, they could schedule the program to run every day without changing the data model or the source table. References: PySpark SQL Module, Python datetime Module, Databricks Jobs
質問 # 33
A data engineer has realized that the data files associated with a Delta table are incredibly small. They want to compact the small files to form larger files to improve performance.
Which of the following keywords can be used to compact the small files?
- A. VACUUM
- B. REDUCE
- C. COMPACTION
- D. OPTIMIZE
- E. REPARTITION
正解：D
解説：
The keyword that can be used to compact the small files associated with a Delta table is OPTIMIZE. The OPTIMIZE command performs file compaction on a Delta table by rewriting a set of small files into a set of larger files1. This can improve the performance of queries that scan the table by reducing the number of files that need to be read and the amount of metadata that needs to be processed1. The OPTIMIZE command can also optionally sort the data within each file by a given set of columns, which can further improve the query performance by enabling data skipping and predicate pushdown1. The OPTIMIZE command can be applied to the whole table or to a specific partition of the table1.
The other keywords are not suitable for compacting the small files associated with a Delta table. REDUCE is a keyword used in the SQL syntax for aggregating data using a user-defined function2. COMPACTION is not a valid keyword in SQL or Python. REPARTITION is a keyword used in the Python syntax for changing the number of partitions of a DataFrame or an RDD3. VACUUM is a keyword used to remove files that are no longer referenced by a Delta table and are older than a retention threshold4.
References:
* 1: OPTIMIZE | Databricks on AWS
* 2: REDUCE | Databricks on AWS
* 3: repartition | Databricks on AWS
* 4: VACUUM | Databricks on AWS
質問 # 34
......
Databricks-Certified-Data-Engineer-Associate問題集と試験テストエンジン：https://www.jpntest.com/shiken/Databricks-Certified-Data-Engineer-Associate-mondaishu
最新版を今すぐ試そうDatabricks-Certified-Data-Engineer-Associate練習テスト問題解答：https://drive.google.com/open?id=1DOrhi8bH33RHNbOJtA4BMmQjYAZ2w2sL