Certified Associate Developer for Apache Spark 3.0 Dumps in PDF and Online

QUESTION: 6

Which of the following is the deepest level in Spark's execution hierarchy?

Job
Task
Executor
Slot
Stage

Answer(s): B

Explanation:

The hierarchy is, from top to bottom: Job, Stage, Task.
Executors and slots facilitate the execution of tasks, but they are not directly part of the hierarchy. Executors are launched by the driver on worker nodes for the purpose of running a specific Spark application. Slots help Spark parallelize work. An executor can have multiple slots which enable it to process multiple tasks in parallel.

Reveal Solution Next Question

QUESTION: 7

Which of the following statements about garbage collection in Spark is incorrect?

Garbage collection information can be accessed in the Spark UI's stage detail view.
Optimizing garbage collection performance in Spark may limit caching ability.
Manually persisting RDDs in Spark prevents them from being garbage collected.
In Spark, using the G1 garbage collector is an alternative to using the default Parallel garbage collector.
Serialized caching is a strategy to increase the performance of garbage collection.

Answer(s): C

Explanation:

Manually persisting RDDs in Spark prevents them from being garbage collected.
This statement is incorrect, and thus the correct answer to the question. Spark's garbage collector will remove even persisted objects, albeit in an "LRU" fashion. LRU stands for least recently used.
So, during a garbage collection run, the objects that were used the longest time ago will be garbage collected first.
See the linked StackOverflow post below for more information.
Serialized caching is a strategy to increase the performance of garbage collection.
This statement is correct. The more Java objects Spark needs to collect during garbage collection, the longer it takes. Storing a collection of many Java objects, such as a DataFrame with a complex schema, through serialization as a single byte array thus increases performance. This means that garbage collection takes less time on a serialized DataFrame than an unserialized DataFrame.
Optimizing garbage collection performance in Spark may limit caching ability.
This statement is correct. A full garbage collection run slows down a Spark application. When taking about "tuning" garbage collection, we mean reducing the amount or duration of these slowdowns.
A full garbage collection run is triggered when the Old generation of the Java heap space is almost full. (If you are unfamiliar with this concept, check out the link to the Garbage Collection Tuning docs below.) Thus, one measure to avoid triggering a garbage collection run is to prevent the Old generation share of the heap space to be almost full.
To achieve this, one may decrease its size. Objects with sizes greater than the Old generation space will then be discarded instead of cached (stored) in the space and helping it to be "almost full".
This will decrease the number of full garbage collection runs, increasing overall performance. Inevitably, however, objects will need to be recomputed when they are needed. So, this mechanism only works when a Spark application needs to reuse cached data as little as possible.
Garbage collection information can be accessed in the Spark UI's stage detail view.
This statement is correct. The task table in the Spark UI's stage detail view has a "GC Time" column, indicating the garbage collection time needed per task.
In Spark, using the G1 garbage collector is an alternative to using the default Parallel garbage collector.
This statement is correct. The G1 garbage collector, also known as garbage first garbage collector, is an alternative to the default Parallel garbage collector.
While the default Parallel garbage collector divides the heap into a few static regions, the G1 garbage collector divides the heap into many small regions that are created dynamically. The G1 garbage collector has certain advantages over the Parallel garbage collector which improve performance particularly for Spark workloads that require high throughput and low latency.
The G1 garbage collector is not enabled by default, and you need to explicitly pass an argument to Spark to enable it. For more information about the two garbage collectors, check out the Databricks article linked below.

Reveal Solution Next Question

QUESTION: 8

Which of the following describes characteristics of the Dataset API?

The Dataset API does not support unstructured data.
In Python, the Dataset API mainly resembles Pandas' DataFrame API.
In Python, the Dataset API's schema is constructed via type hints.
The Dataset API is available in Scala, but it is not available in Python.
The Dataset API does not provide compile-time type safety.

Answer(s): D

Explanation:

The Dataset API is available in Scala, but it is not available in Python.
Correct. The Dataset API uses fixed typing and is typically used for object-oriented programming. It is available when Spark is used with the Scala programming language, but not for Python. In Python, you use the DataFrame API, which is based on the Dataset API. The Dataset API does not provide compile-time type safety.
No – in fact, depending on the use case, the type safety that the Dataset API provides is an advantage.
The Dataset API does not support unstructured data.
Wrong, the Dataset API supports structured and unstructured data. In Python, the Dataset API's schema is constructed via type hints.
No, this is not applicable since the Dataset API is not available in Python. In Python, the Dataset API mainly resembles Pandas' DataFrame API. The Dataset API does not exist in Python, only in Scala and Java.

Reveal Solution Next Question

QUESTION: 9

Which of the following describes the difference between client and cluster execution modes?

In cluster mode, the driver runs on the worker nodes, while the client mode runs the driver on the client machine.
In cluster mode, the driver runs on the edge node, while the client mode runs the driver in a worker node.
In cluster mode, each node will launch its own executor, while in client mode, executors will exclusively run on the client machine.
In client mode, the cluster manager runs on the same host as the driver, while in cluster mode, the cluster manager runs on a separate node.
In cluster mode, the driver runs on the master node, while in client mode, the driver runs on a virtual machine in the cloud.

Answer(s): A

Explanation:

In cluster mode, the driver runs on the master node, while in client mode, the driver runs on a virtual machine in the cloud.
This is wrong, since execution modes do not specify whether workloads are run in the cloud or on- premise.
In cluster mode, each node will launch its own executor, while in client mode, executors will exclusively run on the client machine.
Wrong, since in both cases executors run on worker nodes.
In cluster mode, the driver runs on the edge node, while the client mode runs the driver in a worker node.
Wrong – in cluster mode, the driver runs on a worker node. In client mode, the driver runs on the client machine.
In client mode, the cluster manager runs on the same host as the driver, while in cluster mode, the cluster manager runs on a separate node.
No. In both modes, the cluster manager is typically on a separate node – not on the same host as the driver. It only runs on the same host as the driver in local execution mode.
More info: Learning Spark, 2nd Edition, Chapter 1, and Spark: The Definitive Guide, Chapter 15. ()

Reveal Solution Next Question

QUESTION: 10

Which of the following statements about executors is correct, assuming that one can consider each of the JVMs working as executors as a pool of task execution slots?

Slot is another name for executor.
There must be less executors than tasks.
An executor runs on a single core.
There must be more slots than tasks.
Tasks run in parallel via slots.

Answer(s): E

Explanation:

Tasks run in parallel via slots.
Correct. Given the assumption, an executor then has one or more "slots", defined by the equation spark.executor.cores / spark.task.cpus. With the executor's resources divided into slots, each task takes up a slot and multiple tasks can be executed in parallel. Slot is another name for executor.
No, a slot is part of an executor. An executor runs on a single core.
No, an executor can occupy multiple cores. This is set by the spark.executor.cores option. There must be more slots than tasks.
No. Slots just process tasks. One could imagine a scenario where there was just a single slot for multiple tasks, processing one task at a time. Granted – this is the opposite of what Spark should be used for, which is distributed data processing over multiple cores and machines, performing many tasks in parallel.
There must be less executors than tasks. No, there is no such requirement.
More info: Spark Architecture | Distributed Systems Architecture (https://bit.ly/3x4MZZt)

Reveal Solution Next Question

Databricks Certified Associate Developer for Apache Spark Databricks Certified Associate Developer for Apache Spark 3.0 Dumps in PDF

Free Databricks Databricks Certified Associate Developer for Apache Spark 3.0 Real Questions (page: 2)

QUESTION: 6

Explanation:

QUESTION: 7

Explanation:

QUESTION: 8

Explanation:

QUESTION: 9

Explanation:

QUESTION: 10

Explanation: