[2026年08月29日] Professional-Data-EngineerのPDF問題集にはあなたに不可欠なProfessional-Data-Engineer試験解答を合格に繋ぐ！
Professional-Data-EngineerのPDF解答で完璧な予見Professional-Data-Engineer練習試験問題
質問 # 16
You need to store and analyze social media postings in Google BigQuery at a rate of 10,000 messages per minute in near real-time. Initially, design the application to use streaming inserts for individual postings.
Your application also performs data aggregations right after the streaming inserts. You discover that the queries after streaming inserts do not exhibit strong consistency, and reports from the queries might miss in-flight data. How can you adjust your application design?
- A. Estimate the average latency for data availability after streaming inserts, and always run queries after waiting twice as long.
- B. Re-write the application to load accumulated data every 2 minutes.
- C. Load the original message to Google Cloud SQL, and export the table every hour to BigQuery via streaming inserts.
- D. Convert the streaming insert code to batch load for individual messages.
正解：A
解説：
The data is first comes to buffer and then written to Storage. If we are running queries in buffer we will face above mentioned issues. If we wait for the bigquery to write the data to storage then we won't face the issue. So We need to wait till it's written to storage.
質問 # 17
You are migrating your data warehouse to Google Cloud and decommissioning your on-premises data center. Because this is a priority for your company, you know that bandwidth will be made available for the initial data load to the cloud. The files being transferred are not large in number, but each file is 90 GB Additionally, you want your transactional systems to continually update the warehouse on Google Cloud in real time. What tools should you use to migrate the data and ensure that it continues to write to your warehouse?
- A. Storage Transfer Service for the migration, Pub/Sub and Cloud Data Fusion for the real-time updates
- B. gsutil for the migration; Pub/Sub and Dataflow for the real-time updates
- C. BigQuery Data Transfer Service for the migration, Pub/Sub and Dataproc for the real-time updates
- D. gsutil for both the migration and the real-time updates
正解：B
解説：
Use Gsutil when there is enough bandwidth to meet your project deadline for less than 1 TB of data.
Storage Transfer Service is for much larger volumes for migration. Moreover, Cloud Data Fusion and Dataproc are not ideal for real-time updates. BigQuery Data Transfer Service does not support all on-prem sources.
Reference:
https://cloud.google.com/architecture/migration-to-google-cloud-transferring-your-large-datasets#gsutil_for_smaller_transfers_of_on-premises_data
質問 # 18
You need to create a SQL pipeline. The pipeline runs an aggregate SOL transformation on a BigQuery table every two hours and appends the result to another existing BigQuery table. You need to configure the pipeline to retry if errors occur. You want the pipeline to send an email notification after three consecutive failures.
What should you do?
- A. Use the BigQueryUpsertTableOperator in Cloud Composer, set the retry parameter to three, and set the email_on_failure parameter to true.
- B. Create a BigQuery scheduled query to run the SOL transformation with schedule options that repeats every two hours, and enable email notifications.
- C. Use the BigQuerylnsertJobOperator in Cloud Composer, set the retry parameter to three, and set the email_on_failure parameter to true.
- D. Create a BigQuery scheduled query to run the SQL transformation with schedule options that repeats every two hours, and enable notification to Pub/Sub topic. Use Pub/Sub and Cloud Functions to send an email after three tailed executions.
正解：D
解説：
To create a robust and resilient SQL pipeline in BigQuery that handles retries and failure notifications, consider the following:
* BigQuery Scheduled Queries: This feature allows you to schedule recurring queries in BigQuery. It is a straightforward way to run SQL transformations on a regular basis without requiring extensive setup.
* Error Handling and Retries: While BigQuery Scheduled Queries can run at specified intervals, they don't natively support complex retry logic or failure notifications directly. This is where additional Google Cloud services like Pub/Sub and Cloud Functions come into play.
* Pub/Sub for Notifications: By configuring a BigQuery scheduled query to publish messages to a Pub
/Sub topic upon failure, you can create a decoupled and scalable notification system.
* Cloud Functions: Cloud Functions can subscribe to the Pub/Sub topic and implement logic to count consecutive failures. After detecting three consecutive failures, the Cloud Function can then send an email notification using a service like SendGrid or Gmail API.
Implementation Steps:
* Set up a BigQuery Scheduled Query:
* Create a scheduled query in BigQuery to run your SQL transformation every two hours.
* Configure the scheduled query to publish a notification to a Pub/Sub topic in case of a failure.
* Create a Pub/Sub Topic:
* Create a Pub/Sub topic that will receive messages from the scheduled query.
* Develop a Cloud Function:
* Write a Cloud Function that subscribes to the Pub/Sub topic.
* Implement logic in the Cloud Function to track failure messages. If three consecutive failure messages are detected, the function sends an email notification.
Reference Links:
* BigQuery Scheduled Queries
* Pub/Sub Documentation
* Cloud Functions Documentation
* SendGrid Email API
* Gmail API
質問 # 19
Your company is performing data preprocessing for a learning algorithm in Google Cloud Dataflow. Numerous data logs are being are being generated during this step, and the team wants to analyze them. Due to the dynamic nature of the campaign, the data is growing exponentially every hour.
The data scientists have written the following code to read the data for a new key features in the logs.
BigQueryIO.Read
.named("ReadLogData")
.from("clouddataflow-readonly:samples.log_data")
You want to improve the performance of this data read. What should you do?
- A. Call a transform that returns TableRowobjects, where each element in the PCollectionrepresents a single row in the table.
- B. Use .fromQueryoperation to read specific fields from the table.
- C. Use of both the Google BigQuery TableSchemaand TableFieldSchemaclasses.
- D. Specify the TableReferenceobject in the code.
正解：A
質問 # 20
When a Cloud Bigtable node fails, ____ is lost.
- A. the time dimension
- B. all data
- C. the last transaction
- D. no data
正解：D
解説：
Explanation
A Cloud Bigtable table is sharded into blocks of contiguous rows, called tablets, to help balance the workload of queries. Tablets are stored on Colossus, Google's file system, in SSTable format. Each tablet is associated with a specific Cloud Bigtable node.
Data is never stored in Cloud Bigtable nodes themselves; each node has pointers to a set of tablets that are stored on Colossus. As a result:
Rebalancing tablets from one node to another is very fast, because the actual data is not copied. Cloud Bigtable simply updates the pointers for each node.
Recovery from the failure of a Cloud Bigtable node is very fast, because only metadata needs to be migrated to the replacement node.
When a Cloud Bigtable node fails, no data is lost
Reference: https://cloud.google.com/bigtable/docs/overview
質問 # 21
As your organization expands its usage of GCP, many teams have started to create their own projects.
Projects are further multiplied to accommodate different stages of deployments and target audiences. Each project requires unique access control configurations. The central IT team needs to have access to all projects.
Furthermore, data from Cloud Storage buckets and BigQuery datasets must be shared for use in other projects in an ad hoc way. You want to simplify access control management by minimizing the number of policies.
Which two steps should you take? (Choose two.)
- A. Introduce resource hierarchy to leverage access control policy inheritance.
- B. Create distinct groups for various teams, and specify groups in Cloud IAM policies.
- C. Use Cloud Deployment Manager to automate access provision.
- D. For each Cloud Storage bucket or BigQuery dataset, decide which projects need access. Find all the active members who have access to these projects, and create a Cloud IAM policy to grant access to all these users.
- E. Only use service accounts when sharing data for Cloud Storage buckets and BigQuery datasets.
正解：B、C
質問 # 22
You work for a shipping company that uses handheld scanners to read shipping labels. Your company has strict data privacy standards that require scanners to only transmit recipients' personally identifiable information (PII) to analytics systems, which violates user privacy rules. You want to quickly build a scalable solution using cloud-native managed services to prevent exposure of PII to the analytics systems.
What should you do?
- A. Use Stackdriver logging to analyze the data passed through the total pipeline to identify transactions that may contain sensitive information.
- B. Install a third-party data validation tool on Compute Engine virtual machines to check the incoming data for sensitive information.
- C. Build a Cloud Function that reads the topics and makes a call to the Cloud Data Loss Prevention API.
Use the tagging and confidence levels to either pass or quarantine the data in a bucket for review. - D. Create an authorized view in BigQuery to restrict access to tables with sensitive data.
正解：D
質問 # 23
Your globally distributed auction application allows users to bid on items. Occasionally, users place identical bids at nearly identical times, and different application servers process those bids. Each bid event contains the item, amount, user, and timestamp. You want to collate those bid events into a single location in real time to determine which user bid first. What should you do?
- A. Create a file on a shared file and have the application servers write all bid events to that file. Process the file with Apache Hadoop to identify which user bid first.
- B. Have each application server write the bid events to Google Cloud Pub/Sub as they occur. Use a pull subscription to pull the bid events using Google Cloud Dataflow. Give the bid for each item to the user in the bid event that is processed first.
- C. Have each application server write the bid events to Cloud Pub/Sub as they occur. Push the events from Cloud Pub/Sub to a custom endpoint that writes the bid event information into Cloud SQL.
- D. Set up a MySQL database for each application server to write bid events into. Periodically query each of those distributed MySQL databases and update a master MySQL database with bid event information.
正解：D
質問 # 24
Suppose you have a dataset of images that are each labeled as to whether or not they contain a human face. To create a neural network that recognizes human faces in images using this labeled dataset, what approach would likely be the most effective?
- A. Use deep learning by creating a neural network with multiple hidden layers to automatically detect features of faces.
- B. Use feature engineering to add features for eyes, noses, and mouths to the input data.
- C. Build a neural network with an input layer of pixels, a hidden layer, and an output layer with two categories.
- D. Use K-means Clustering to detect faces in the pixels.
正解：A
解説：
Traditional machine learning relies on shallow nets, composed of one input and one output layer, and at most one hidden layer in between. More than three layers (including input and output) qualifies as "deep" learning. So deep is a strictly defined, technical term that means more than one hidden layer.
In deep-learning networks, each layer of nodes trains on a distinct set of features based on the previous layer's output. The further you advance into the neural net, the more complex the features your nodes can recognize, since they aggregate and recombine features from the previous layer.
A neural network with only one hidden layer would be unable to automatically recognize high-level features of faces, such as eyes, because it wouldn't be able to "build" these features using previous hidden layers that detect low-level features, such as lines. Feature engineering is difficult to perform on raw image data.
K-means Clustering is an unsupervised learning method used to categorize unlabeled data.
Reference: https://deeplearning4j.org/neuralnet-overview
質問 # 25
You are working on a sensitive project involving private user dat
- A. Grant the consultant the Cloud Dataflow Developer role on the project.
- B. Grant the consultant the Viewer role on the project.
- C. Create a service account and allow the consultant to log on with it.
- D. Create an anonymized sample of the data for the consultant to work with in a different project.
- E. You have set up a project on Google Cloud Platform to house your work internally. An external consultant is going to assist with coding a complex transformation in a Google Cloud Dataflow pipeline for your project. How should you maintain users' privacy?
正解：A
質問 # 26
Your company built a TensorFlow neutral-network model with a large number of neurons and layers. The
model fits well for the training data. However, when tested against new data, it performs poorly. What
method can you employ to address this?
- A. Threading
- B. Dimensionality Reduction
- C. Serialization
- D. Dropout Methods
正解：D
解説：
Explanation/Reference:
Reference: https://medium.com/mlreview/a-simple-deep-learning-model-for-stock-price-prediction-using-
tensorflow-30505541d877
質問 # 27
You have a BigQuery dataset named "customers". All tables will be tagged by using a Data Catalog tag template named "gdpr". The template contains one mandatory field,
"has_sensitive_data", with a boolean value. All employees must be able to do a simple search and find tables in the dataset that have either true or false in the "has_sensitive_data' field.
However, only the Human Resources (HR) group should be able to see the data inside the tables for which "has_sensitive data" is true. You give the all employees group the bigquery.metadataViewer and bigquery.connectionUser roles on the dataset. You want to minimize configuration overhead. What should you do next?
- A. Create the "gdpr" tag template with public visibility. Assign the datacatalog.tagTemplateViewer role on this tag to the all employees group, and assign the bigquery.dataViewer role to the HR group on the tables that contain sensitive data.
- B. Create the "gdpr" tag template with private visibility. Assign the bigquery.dataViewer role to the HR group on the tables that contain sensitive data.
- C. Create the "gdpr" tag template with public visibility. Assign the bigquery.dataViewer role to the HR group on the tables that contain sensitive data.
- D. Create the "gdpr" tag template with private visibility. Assign the datacatalog.tagTemplateViewer role on this tag to the all employees group, and assign the bigquery.dataViewer role to the HR group on the tables that contain sensitive data.
正解：C
質問 # 28
You operate a logistics company, and you want to improve event delivery reliability for vehicle-based sensors. You operate small data centers around the world to capture these events, but leased lines that provide connectivity from your event collection infrastructure to your event processing infrastructure are unreliable, with unpredictable latency. You want to address this issue in the most cost-effective way. What should you do?
- A. Establish a Cloud Interconnect between all remote data centers and Google.
- B. Have the data acquisition devices publish data to Cloud Pub/Sub.
- C. Deploy small Kafka clusters in your data centers to buffer events.
- D. Write a Cloud Dataflow pipeline that aggregates all data in session windows.
正解：C
質問 # 29
Which of the following are feature engineering techniques? (Select 2 answers)
- A. Bucketization of a continuous feature
- B. Hidden feature layers
- C. Crossed feature columns
- D. Feature prioritization
正解：A、C
解説：
Selecting and crafting the right set of feature columns is key to learning an effective model. Bucketization is a process of dividing the entire range of a continuous feature into a set of consecutive bins/buckets, and then converting the original numerical feature into a bucket ID (as a categorical feature) depending on which bucket that value falls into. Using each base feature column separately may not be enough to explain the data. To learn the differences between different feature combinations, we can add crossed feature columns to the model.
Reference:
https://www.tensorflow.org/tutorials/wide#selecting_and_engineering_features_for_the_model
質問 # 30
The _________ for Cloud Bigtable makes it possible to use Cloud Bigtable in a Cloud Dataflow pipeline.
- A. BiqQuery API
- B. Cloud Dataflow connector
- C. DataFlow SDK
- D. BigQuery Data Transfer Service
正解：B
解説：
The Cloud Dataflow connector for Cloud Bigtable makes it possible to use Cloud Bigtable in a Cloud Dataflow pipeline. You can use the connector for both batch and streaming operations.
Reference:
https://cloud.google.com/bigtable/docs/dataflow-hbase
質問 # 31
You need to choose a database to store time series CPU and memory usage for millions of computers. You need to store this data in one-second interval samples. Analysts will be performing real-time, ad hoc analytics against the database. You want to avoid being charged for every query executed and ensure that the schema design will allow for future growth of the dataset. Which database and data model should you choose?
- A. Create a table in BigQuery, and append the new samples for CPU and memory to the table
- B. Create a wide table in Cloud Bigtable with a row key that combines the computer identifier with the sample time at each minute, and combine the values for each second as column data.
- C. Create a wide table in BigQuery, create a column for the sample value at each second, and update the row with the interval for each second
- D. Create a narrow table in Cloud Bigtable with a row key that combines the Computer Engine computer identifier with the sample time at each second
正解：B
質問 # 32
A shipping company has live package-tracking data that is sent to an Apache Kafka stream in real time. This is then loaded into BigQuery. Analysts in your company want to query the tracking data in BigQuery to analyze geospatial trends in the lifecycle of a package. The table was originally created with ingest-date partitioning. Over time, the query processing time has increased. You need to copy all the data to a new clustered table. What should you do?
- A. Implement clustering in BigQuery on the package-tracking ID column.
- B. Re-create the table using data partitioning on the package delivery date.
- C. Implement clustering in BigQuery on the ingest date column.
- D. Tier older data onto Cloud Storage files and create a BigQuery table using Cloud Storage as an external data source.
正解：A
質問 # 33
You are developing a fraud detection model using BigQuery ML. You have a raw transaction dataset and need to create new features such as the average_transaction_amount_last_24_hours and time_since_last_transaction. These features require aggregation and time-window calculations on the existing data. The goal is to ensure that these features are consistently applied during both model training and prediction without manual intervention. You need to prepare these features efficiently for your model. What should you do?
- A. Implement a Cloud Run function that triggers on new transactions, calculates the features, and inserts them into a feature store before model serving.
- B. Use the TRANSFORM clause within the CREATE MODEL statement, leveraging SQL functions for aggregations and time-based calculations.
- C. Export the BigQuery data to Cloud Storage, perform feature engineering using a custom Python script in a Dataflow job, and then re-import the engineered features into BigQuery.
- D. Create a separate BigQuery table containing pre-computed features using complex SQL queries and join this table with the raw data during model training and serving.
正解：D
解説：
Pre-computing features in a dedicated BigQuery table ensures consistency between training and prediction, supports complex aggregations and time-window calculations efficiently, and allows BigQuery ML to join these features with raw data without manual intervention.
質問 # 34
You want to archive data in Cloud Storage. Because some data is very sensitive, you want to use the "Trust No One" (TNO) approach to encrypt your data to prevent the cloud provider staff from decrypting your dat
a. What should you do?
- A. Use gcloud kms keys create to create a symmetric key. Then use gcloud kms encrypt to encrypt each archival file with the key. Use gsutil cp to upload each encrypted file to the Cloud Storage bucket. Manually destroy the key previously used for encryption, and rotate the key once and rotate the key once.
- B. Use gcloud kms keys create to create a symmetric key. Then use gcloud kms encrypt to encrypt each archival file with the key and unique additional authenticated data (AAD). Use gsutil cp to upload each encrypted file to the Cloud Storage bucket, and keep the AAD outside of Google Cloud.
- C. Specify customer-supplied encryption key (CSEK) in the .boto configuration file. Use gsutil cp to upload each archival file to the Cloud Storage bucket. Save the CSEK in a different project that only the security team can access.
- D. Specify customer-supplied encryption key (CSEK) in the .boto configuration file. Use gsutil cp to upload each archival file to the Cloud Storage bucket. Save the CSEK in Cloud Memorystore as permanent storage of the secret.
正解：A
質問 # 35
You use a dataset in BigQuery for analysis. You want to provide third-party companies with access to the same dataset. You need to keep the costs of data sharing low and ensure that the data is current. What should you do?
- A. Use Cloud Scheduler to export the data on a regular basis to Cloud Storage, and provide third-party companies with access to the bucket.
- B. Use Analytics Hub to control data access, and provide third party companies with access to the dataset
- C. Create a separate dataset in BigQuery that contains the relevant data to share, and provide third-party companies with access to the new dataset.
- D. Create a Dataflow job that reads the data in frequent time intervals and writes it to the relevant BigQuery dataset or Cloud Storage bucket for third-party companies to use.
正解：B
解説：
Analytics Hub is a service that allows you to securely share and discover data assets across your organization and with external partners. You can use Analytics Hub to create and manage data assets, such as BigQuery datasets, views, and queries, and control who can access them. You can also browse and use data assets that others have shared with you. By using Analytics Hub, you can keep the costs of data sharing low and ensure that the data is current, as the data assets are not copied or moved, but rather referenced from their original sources.
質問 # 36
You are designing a data mesh on Google Cloud by using Dataplex to manage data in BigQuery and Cloud Storage. You want to simplify data asset permissions. You are creating a customer virtual lake with two user groups:
* Data engineers, which require lull data lake access
* Analytic users, which require access to curated data
You need to assign access rights to these two groups. What should you do?
- A. 1. Grant the bigquery.dataownex role on BigQuery datasets and the storage.objectcreator role on Cloud Storage buckets to data engineers.
2. Grant the bigquery.dataViewer role on BigQuery datasets and the storage.objectViewer role on Cloud Storage buckets to analytic users. - B. 1. Grant the dataplex.dataOwner role to the data engineer group on the customer data lake.
2. Grant the dataplex.dataReader role to the analytic user group on the customer curated zone. - C. 1. Grant the bigquery.dataViewer role on BigQuery datasets and the storage.objectviewer role on Cloud Storage buckets to data engineers.
2. Grant the bigquery.dataOwner role on BigQuery datasets and the storage.objectEditor role on Cloud Storage buckets to analytic users. - D. 1. Grant the dataplex.dataReader role to the data engineer group on the customer data lake.
2. Grant the dataplex.dataOwner to the analytic user group on the customer curated zone.
正解：B
解説：
When designing a data mesh on Google Cloud using Dataplex to manage data in BigQuery and Cloud Storage, it is essential to simplify data asset permissions while ensuring that each user group has the appropriate access levels. Here's why option A is the best choice:
* Data Engineer Group:
* Data engineers require full access to the data lake to manage and operate data assets comprehensively. Granting the dataplex.dataOwner role to the data engineer group on the customer data lake ensures they have the necessary permissions to create, modify, and delete data assets within the lake.
* Analytic User Group:
* Analytic users need access to curated data but do not require full control over all data assets.
Granting the dataplex.dataReader role to the analytic user group on the customer curated zone provides read-only access to the curated data, enabling them to analyze the data without the ability to modify or delete it.
Steps to Implement:
* Grant Data Engineer Permissions:
* Assign the dataplex.dataOwner role to the data engineer group on the customer data lake to ensure full access and management capabilities.
* Grant Analytic User Permissions:
* Assign the dataplex.dataReader role to the analytic user group on the customer curated zone to provide read-only access to curated data.
Reference Links:
* Dataplex IAM Roles and Permissions
* Managing Access in Dataplex
質問 # 37
You are building a new data pipeline to share data between two different types of applications: jobs generators and job runners. Your solution must scale to accommodate increases in usage and must accommodate the addition of new applications without negatively affecting the performance of existing ones.
What should you do?
- A. Use a Cloud Pub/Sub topic to publish jobs, and use subscriptions to execute them
- B. Create a table on Cloud SQL, and insert and delete rows with the job information
- C. Create an API using App Engine to receive and send messages to the applications
- D. Create a table on Cloud Spanner, and insert and delete rows with the job information
正解：C
解説：
References:
質問 # 38
You created an analytics environment on Google Cloud so that your data scientist team can explore data without impacting the on-premises Apache Hadoop solution. The data in the on-premises Hadoop Distributed File System (HDFS) cluster is in Optimized Row Columnar (ORC) formatted files with multiple columns of Hive partitioning. The data scientist team needs to be able to explore the data in a similar way as they used the on-premises HDFS cluster with SQL on the Hive query engine. You need to choose the most cost-effective storage and processing solution. What should you do?
- A. Copy the ORC files on Cloud Storage, then deploy a Dataproc cluster for the data scientist team.
- B. Copy the ORC files on Cloud Storage, then create external BigQuery tables for the data scientist team.
- C. Import the ORC files lo Bigtable tables for the data scientist team.
- D. Import the ORC files to BigOuery tables for the data scientist team.
正解：B
解説：
The requirements are:
* Explore ORC formatted files with Hive partitioning.
* Mimic the SQL on Hive query engine experience.
* Cost-effective storage and processing.
* Avoid impacting the on-premises Hadoop solution.
Let's analyze the options:
* Option A (Import to Bigtable):Bigtable is a NoSQL database, not suited for SQL-based exploration of ORC files or Hive-style partitioning directly. This would require significant data transformation and a different query paradigm. Not cost-effective for this use case.
* Option B (Import to BigQuery native tables):Importing data into BigQuery native storage is an option. BigQuery can load ORC files. This provides excellent query performance. However, it involves an ETL step (importing) and storage costs for the datawithin BigQuery, which might be higher than keeping it in its original format on Cloud Storage if query patterns are exploratory and not extremely frequent on all data.
* Option C (Copy to Cloud Storage, deploy Dataproc):Dataproc allows you to run Hadoop/Spark (and thus Hive) clusters on Google Cloud. This would provide a very similar experience ("SQL on the Hive query engine"). However, running a persistent Dataproc cluster incurs compute costs for the cluster nodes, even when not actively querying. While ephemeral clusters are possible, it adds operational overhead for exploratory queries. Storage on Cloud Storage is cost-effective.
* Option D (Copy to Cloud Storage, create external BigQuery tables):This is often the most cost- effective and straightforward solution for this scenario.
* Cost-effective Storage:Cloud Storage is a low-cost option for storing files like ORC.
* SQL Interface:BigQuery provides a familiar SQL interface.
* External Tables:BigQuery can query data directly from Cloud Storage (including ORC files) using external tables. This avoids the need to load data into BigQuery's managed storage, saving on storage costs and ETL effort.
* Hive Partitioning:BigQuery external tables support Hive partitioning layouts. When you define the external table, you can specify the partitioning scheme, and BigQuery will use partition pruning to scan only relevant partitions, improving performance and reducing costs for queries that filter on partition keys. This directly mimics the Hive experience.
* Processing Cost:You only pay for the data scanned by BigQuery queries, which aligns with exploratory analysis.
Comparing D with B: External tables are generally more cost-effective for storage and initial setup if the data is already in ORC and an ETL process into BigQuery native storage is to be avoided. Query performance might be slightly less than native tables but is often sufficient for exploration, especially with partitioning.
Comparing D with C: BigQuery external tables are serverless, meaning no cluster to manage or pay for when idle. Dataproc requires managing and paying for a cluster. For exploration, the serverless nature of BigQuery is usually more cost-effective.
Therefore, copying ORC files to Cloud Storage and using BigQuery external tables is the most cost-effective solution that meets all requirements.
Reference:
Google Cloud Documentation: BigQuery > External data sources > Querying Cloud Storage data. "You can query data in Cloud Storage by using external tables or federated queries. External tables are tables that read data directly from files in Cloud Storage." Google Cloud Documentation: BigQuery > External data sources > Supported formats and compression types. ORC is a supported format.
Google Cloud Documentation: BigQuery > Creating and using tables > Creating external tables. "External tables let you query data stored in Cloud Storage as if it were a standardBigQuery table. You can use external tables to query data in various formats, including... ORC..." Google Cloud Documentation: BigQuery > Creating and using tables > Querying partitioned external tables.
"You can create an external table that is partitioned on Hive partitioning keys. When you query a Hive partitioned external table, BigQuery performs partition pruning to skip reading unnecessary partitions." This directly addresses the "Hive partitioning" and "explore data in a similar way" requirements.
Google Cloud Blog: "Choosing the right data processing option on GCP: BigQuery vs. Dataproc" (and similar articles) often highlight BigQuery external tables as a cost-effective way to query data in place on Cloud Storage, especially for data lake scenarios.
質問 # 39
......
Professional-Data-Engineerリアル試験問題と正確なGoogle Certified Professional Data Engineer ExamのPDF解答：https://www.jpntest.com/shiken/Professional-Data-Engineer-mondaishu
リアルGoogle試験の素晴らしい練習問題集でProfessional-Data-Engineer試験：https://drive.google.com/open?id=1Sylf1Qa_gevQpQ5fP1oSauz0FegDlx2w