Apache-Spark-Developer Associate Developer for Apache Spark Practice Questions

The primary advantage of the Pandas API on Spark is that it provides a familiar pandas-like API that operates on Spark DataFrames. This allows data scientists and engineers to use their existing pandas knowledge to process datasets that are too large to fit in the memory of a single machine by distributing the computation across a Spark cluster.

This statement is true. The Trigger.Once (or trigger(once=True)) setting configures the streaming query to run as a single micro-batch. It will identify and process all new data that has arrived since the last time it ran, update the state and sink, and then shut down the query. This is a common pattern for converting a streaming job into an incremental batch job that can be scheduled by an external orchestrator like Databricks Jobs or Airflow.

Spark Connect is designed specifically for this use case. It provides a decoupled client-server architecture, allowing various clients (IDEs, notebooks, applications) to connect remotely to a Spark cluster through a stable API. This centralizes cluster management and allows developers to work from their preferred environments without needing to configure complex cluster access on each client machine.

The distinct() method returns a new DataFrame containing only the unique rows from the original DataFrame. It is an alias for dropDuplicates() when no columns are specified. This operation is a wide transformation as it requires a shuffle to identify and remove duplicate rows across all partitions.

A broadcast variable is the correct choice for distributing a large, read-only dataset to all worker nodes. The driver serializes the variable and sends it to each executor only once, where it is cached in memory. Tasks on that executor can then access the data locally without causing repeated network transfers. Accumulators are used for aggregating results back to the driver. Caching the DataFrame would work, but broadcast is specifically designed for this 'side data' distribution pattern and is generally more efficient for joins. A UDF is for custom logic, not data distribution.

Spark creates a new stage at each shuffle boundary. Wide transformations, such as groupBy(), join(), or repartition(), require data to be redistributed (shuffled) across the network between executors. This redistribution marks the end of one stage and the beginning of another. Narrow transformations can be pipelined within a single stage. Actions trigger the execution of a job, which consists of one or more stages, but the action itself doesn't define the stage boundary; the shuffle does.

For partition pruning to be effective, the filter conditions must be applied directly to the partition columns. This allows Spark's Catalyst optimizer to read the file system metadata and skip reading the data files in partitions that do not match the filter. Applying functions to the partition columns (like concat or to_date) can prevent the optimizer from pushing down the predicate, forcing a full table scan. The correct approach is to filter directly on the raw partition columns.

Update mode is specifically designed for use cases involving aggregations. In this mode, only the rows that were updated in the result table since the last trigger will be written to the sink. This is perfect for outputting running counts where only the users with new events in the current micro-batch will have their counts updated in the output. Complete mode would rewrite the entire result table, and Append mode is not supported for aggregations without a watermark.