Data Engineer Associate Practice Questions

Wide transformations, like groupBy, join, or repartition, require data to be redistributed or 'shuffled' among the worker nodes across the network. A single stage with many tasks performing such an operation indicates a large shuffle. This is a common and significant performance bottleneck in Spark because network I/O is much slower than memory access. The solution often involves optimizing the query to reduce the amount of data being shuffled.

File notification mode is the most scalable and efficient method for discovering new files in large storage locations. It subscribes to file event notifications from the cloud provider (e.g., S3 Event Notifications, Azure Event Grid). Instead of repeatedly listing all files in a directory (which is slow and expensive with millions of files), it processes new files as events are received. This significantly reduces latency and cost for continuous processing.

This is the correct approach. It uses a PySpark Window function, which is the standard way to perform calculations over a sliding range of rows. partitionBy("product_id") ensures the calculation restarts for each product. orderBy("sale_date") sorts the data correctly. rowsBetween(-6, Window.currentRow) defines the 7-day window (the current row plus the previous 6 rows). Finally, avg("sales_amount").over(windowSpec) applies the average aggregation over this defined window.

Databricks Asset Bundles (DAB) are the intended solution for this problem. They allow developers to define all project artifacts—notebooks, jobs, libraries, cluster definitions, and environment-specific configurations—in a declarative YAML file (databricks.yml). This bundle can be checked into Git and deployed as a single, consistent unit using the Databricks CLI, which is ideal for CI/CD automation.

This is a primary benefit. With serverless compute, Databricks manages the cluster sizing, optimization, and patching, providing a 'hands-off' experience that reduces operational overhead.

Serverless compute uses a warm pool of resources managed by Databricks, which significantly reduces the startup latency for jobs compared to traditional clusters that need to provision new VMs from the cloud provider for each run.

Databricks Connect requires that the major and minor Python versions of the local client environment match the version on the Databricks cluster exactly. In this case, the local version is 3.11.5 while the cluster version is 3.11.2. Even though the major version (3) and minor version (11) match, the strict requirement often extends to the patch version for full compatibility, but the major/minor mismatch is the key principle. An expired PAT would result in an authentication error, not a Py4JError. A firewall issue would likely cause a timeout. An incorrect cluster ID would result in a 'not found' error.

The targets section is specifically designed to manage environment-specific configurations. Each target can have its own workspace URL, root path, and variable definitions, allowing the same bundle to be deployed to different environments with the correct settings.

Variables, defined within each target, allow for parameterization of the bundle's resources. This is how you would specify a different cluster policy ID for production versus development, or reference a different secret scope for database credentials in each environment.

The rescue mode for cloudFiles.schemaEvolutionMode is designed for this exact scenario. It instructs Auto Loader to infer the schema and, if a data type change is detected (like numeric to string), it adds the new column to the _rescue column. For schema evolution, the correct option is to enable schema evolution on the write stream using .option("mergeSchema", "true") and handle rescued data. However, among the given choices, rescue mode is the feature specifically designed to handle columns with mixed data types by rescuing them, which aligns with the observed behavior. While mergeSchema is also needed, schemaEvolutionMode rescue directly addresses the problem of incompatible data types appearing in a column.

Unity Catalog operates on a default-deny model. To meet the requirement, you should avoid granting any broad permissions at the catalog or schema level. The correct approach is to grant the necessary, specific privileges (SELECT to read, MODIFY might be needed for the job's operations) directly to the service principal on the target Bronze tables. This ensures that only that principal can access the sensitive data, enforcing the principle of least privilege.

This statement is true. The primary purpose of cluster pools is to reduce cluster start and auto-scaling times by maintaining a set of idle, ready-to-use instances. When a job requests a cluster attached to a pool, it gets its nodes from this warm pool instead of requesting new instances from the cloud provider, which significantly shortens the startup latency.