DATA-ENG-PRO sample questions

Question 1

A development team is using Databricks Asset Bundles (DABs) to manage their project deployments. They need to define different compute resources and job parameters for their `development` and `production` environments. For example, the development environment should use a small, single-node cluster, while production requires a large auto-scaling cluster. Which section of the `databricks.yml` file is used to define these environment-specific configurations?

Accepted Answer

The `targets` section

Answer

The `resources` section

Answer

The `workspace` section

Answer

The `artifacts` section

Question 2

When configuring an Auto Loader stream, what is the primary purpose of the `.option("cloudFiles.schemaLocation", ...)` setting?

Accepted Answer

To specify the location for storing the inferred schema and tracking progress.

Answer

To define the cloud storage path where the target Delta table will be written.

Answer

To provide a fixed, user-defined schema for the data being ingested.

Answer

To set the location of the checkpoint directory for the Structured Streaming query.

Question 3

A data engineer needs to calculate a 7-day moving average of sales for each product in a large sales dataset. The dataset is ordered by `product_id` and `sale_date`. Which type of Spark function is best suited for this calculation? ```mermaid graph LR A[Input Data] --> B{Partition by product_id}; B --> C{Order by sale_date}; C --> D{Apply Window Function 7-day moving average}; D --> E[Output Data]; ```

Accepted Answer

A window function with `PARTITION BY product_id` and an appropriate frame definition.

Answer

A standard aggregation function with `groupBy('product_id')`.

Answer

A User-Defined Function (UDF) that iterates through a collected list of sales.

Answer

A self-join on the sales table where the join condition matches `product_id` and the date is within the 7-day range.

Question 4

A data engineering team has migrated a large, partitioned Delta table to use Liquid Clustering instead. The table is clustered by `user_id` and `event_timestamp`. After the migration, the team notices that some queries filtering by `user_id` are not showing the expected performance improvements. What is a critical operational step that must be performed regularly on a liquid-clustered table to ensure the data layout is continuously optimized and query performance is maintained?

Accepted Answer

Run the `OPTIMIZE` command on the table.

Answer

Run `ANALYZE TABLE ... COMPUTE STATISTICS` to update the table stats.

Answer

Run the `VACUUM` command to remove old data files.

Answer

Re-create the table with a `CLUSTER BY` clause every night.

Question 5

A data engineer needs to programmatically trigger a Databricks job, pass custom parameters to it, and then monitor its status until completion. Which Databricks tool is most suitable for this automation task from an external system?

Accepted Answer

The Databricks Jobs REST API.

Answer

The Databricks SQL Connector for Python.

Answer

The Databricks CLI within a shell script.

Answer

Using `%run` magic command within a scheduled notebook.

Question 6

A data architect is designing a petabyte-scale fact table for an e-commerce platform. The table will store transaction data and will be queried by `product_id`, `customer_id`, and `transaction_date`. The `customer_id` column has very high cardinality (over 100 million unique values), while `transaction_date` is used for filtering in 90% of queries, often with range scans. The primary goals are to optimize query performance for a wide variety of analytical queries and to minimize ongoing data layout maintenance. Which data layout strategy is most effective for these requirements?

Accepted Answer

Implement Liquid Clustering on (`customer_id`, `product_id`, `transaction_date`).

Answer

Partition by `transaction_date` and apply Z-Ordering on `customer_id` and `product_id`.

Answer

Partition the table by `customer_id` to ensure data locality for customer-specific queries.

Answer

Create two separate materialized views: one partitioned by `transaction_date` and another partitioned by `customer_id`.

Question 7

A financial institution is using Unity Catalog to govern access to a transactions table containing sensitive customer data. The following access rules must be enforced:

Analysts in the EU_Analysts group should only see transactions from European countries.
For all analysts, the customer_full_name column must be masked, showing only the first initial and last name.
Auditors in the Compliance group must see all data, unmasked and unfiltered.

Which TWO actions are required to implement this security model directly in Unity Catalog? (Select TWO)

Accepted Answer

Create a row filter function using `CASE` statements and the `is_member()` function to filter data by country based on group membership.

Accepted Answer

Create and apply a column mask function to the `customer_full_name` column that returns the full name only if the user is a member of the 'Compliance' group.

Answer

Create separate dynamic views for each region and grant access to the respective groups.

Answer

Implement the logic in a Delta Live Tables pipeline to create separate, secured silver tables for each analyst group.

Answer

Grant the `SELECT` privilege on the table to 'EU_Analysts' and deny access to all other columns except the masked one.

Question 8

A developer is defining a Databricks Asset Bundle (DAB) to deploy a multi-task job. The job includes a task that runs a Python script packaged as a wheel file.

Accepted Answer

True

Answer

False

Question 9

**Company Background:** Streamlytics, a real-time analytics provider, processes IoT sensor data. They have a Spark Structured Streaming job that joins a stream of sensor readings (`sensor_stream`) with a static dimension table of sensor metadata (`sensor_metadata`). The job performs an aggregation to calculate the average temperature per sensor type every 5 minutes. **Current Architecture:** The streaming job uses a 10-minute watermark on the event timestamp to handle late-arriving data. The state store is located on cloud object storage, and the pipeline runs on an auto-scaling job cluster. This setup has been running efficiently for several months. **The Problem:** After a new set of legacy devices were onboarded, the pipeline's performance has severely degraded. Micro-batch processing times have inflated from 30 seconds to over 15 minutes, causing the stream to fall behind. The state store size has grown to several terabytes. An investigation reveals that the legacy devices occasionally send data that is several days late. These records are critical and must eventually be processed for auditing purposes. **Business Requirement:** The engineering lead has tasked you with resolving the performance degradation and the massive state size. The solution must not lose the late-arriving data and should avoid a significant increase in steady-state compute costs.

Accepted Answer

Rearchitect the pipeline to write all raw data to a Delta table. The main streaming job will process data with a tight watermark. A separate, nightly batch job will process data older than the watermark from the raw table and merge it into the final aggregated table.

Answer

Increase the watermark to 7 days to accommodate all late data and provision a larger cluster.

Answer

Enable RocksDB-based state management on instances with local SSDs to improve state access speed.

Answer

Split the input stream into two separate streams. Process the normal data with a 10-minute watermark in one stream, and the legacy data with a 7-day watermark in another, then union the results.

Question 10

A data engineering team has designed a multi-task job in Databricks Workflows to orchestrate their daily ETL process. The job has a linear dependency chain for the main data processing and an independent task for auditing that can run at any time after the start. An engineer needs to add a new task, generate_reports, which must run only after both the validate_data and process_data tasks have successfully completed.

Which task dependency configuration correctly implements this requirement?

graph TD A[start_job] --> B[ingest_data]; B --> C[validate_data]; C --> D[process_data]; D --> E[load_warehouse]; A --> F[run_audits];

Accepted Answer

Set `generate_reports` to depend on both `validate_data` and `process_data`.

Answer

Set `generate_reports` to depend on `process_data`.

Answer

Create two separate jobs: one for the main pipeline and one for `generate_reports` triggered via the API upon completion of `process_data`.

Answer

Set `generate_reports` to run in parallel with `load_warehouse`, with both depending on `process_data`.

DATA-ENG-PRO Data Engineer Professional Practice Questions

Create an account to continue.

Create an account to continue.

Related exams