DataX Practice Questions

The Area Under the Receiver Operating Characteristic Curve (AUC) represents a model's ability to discriminate between positive and negative classes. A higher AUC value (closer to 1.0) indicates a better model. It can be interpreted as the probability that the model will rank a randomly chosen positive instance higher than a randomly chosen negative one. Therefore, Model A (AUC=0.85) is superior to Model B (AUC=0.79) in this regard.

The Traveling Salesman Problem (TSP) is a well-known algorithmic problem in the field of optimization and graph theory. It asks for the shortest possible route that visits each city in a given list exactly once and returns to the origin city. This directly matches the scenario described for the logistics company.

A data lake is a centralized repository designed to store, process, and secure large amounts of structured, semi-structured, and unstructured data. Its key characteristic is the 'schema-on-read' approach, where data is ingested in its raw format without a predefined schema. This perfectly fits the requirement of handling diverse data types from various sources without upfront structuring.

Pruning involves removing sections of the tree that are non-critical and provide little predictive power. This reduces the model's complexity and its tendency to memorize the training data, thereby improving its ability to generalize to new data.

A Random Forest is an ensemble method that builds multiple decision trees on different subsets of the data and features, then averages their predictions. This process, known as bagging, is highly effective at reducing the variance of individual trees and mitigating overfitting.

The validation set is used to evaluate the model's performance during development, specifically for tuning hyperparameters (like learning rate, tree depth, etc.) and comparing different model types. This prevents 'data leakage' into the test set, which must remain unseen until the final evaluation to provide an unbiased estimate of the model's performance on new data.

Recall, also known as Sensitivity or True Positive Rate, measures the proportion of actual positives that were correctly identified. In this scenario, the cost of missing a potential failure (a False Negative) is very high. Therefore, the primary goal is to maximize the number of true failures caught by the model, which is precisely what Recall measures. Accuracy would be misleadingly high due to the class imbalance. Precision focuses on the proportion of positive predictions that are actually correct, which is less critical here than catching all potential failures.

Analysis of Variance (ANOVA) is used to compare the means of three or more independent groups to determine if there is a statistically significant difference between them. A T-test is used for comparing the means of only two groups. A Chi-squared test is used for categorical data, not continuous data like test scores. Pearson correlation measures the linear relationship between two continuous variables, not differences in means across groups.

Docker is used to create containers, which package the model, its dependencies, and the API server into a single, isolated, and portable unit. This addresses the isolation and versioning requirement.

Kubernetes is a container orchestration platform that manages containerized applications (like those created with Docker) at scale. It handles auto-scaling, load balancing, and self-healing, which is essential for a high-traffic application.

This statement accurately describes the process of transfer learning. A model pre-trained on a large, general dataset (like ImageNet) has already learned useful features. This knowledge is transferred by using its weights as a starting point for training on a new, smaller, and more specific dataset, which is a highly effective technique when data is limited.

This solution is the most comprehensive. It uses a managed Spark service (e.g., AWS EMR, Databricks, GCP Dataproc) to handle the massive scale of batch training, which aligns with the team's need to minimize infrastructure overhead. It correctly uses the ALS algorithm, which is designed for large-scale collaborative filtering. It explicitly addresses the cold-start problem with a hybrid approach (content-based model). Finally, it uses a low-latency NoSQL database (like DynamoDB or Cassandra) for serving, which is a best practice for real-time recommendation systems.