Certified Data Engineer Professional Dumps in PDF and Online

QUESTION: 17

A production workload incrementally applies updates from an external Change Data Capture feed to a Delta Lake table as an always-on Structured Stream job. When data was initially migrated for this table, OPTIMIZE was executed and most data files were resized to 1 GB. Auto Optimize and Auto Compaction were both turned on for the streaming production job. Recent review of data files shows that most data files are under 64 MB, although each partition in the table contains at least 1 GB of data and the total table size is over 10 TB.

Which of the following likely explains these smaller file sizes?

Databricks has autotuned to a smaller target file size to reduce duration of MERGE operations
Z-order indices calculated on the table are preventing file compaction
Bloom filter indices calculated on the table are preventing file compaction
Databricks has autotuned to a smaller target file size based on the overall size of data in the table
Databricks has autotuned to a smaller target file size based on the amount of data in each partition

Answer(s): A

Reveal Solution Next Question

QUESTION: 18

Which statement regarding stream-static joins and static Delta tables is correct?

Each microbatch of a stream-static join will use the most recent version of the static Delta table as of each microbatch.
Each microbatch of a stream-static join will use the most recent version of the static Delta table as of the job's initialization.
The checkpoint directory will be used to track state information for the unique keys present in the join.
Stream-static joins cannot use static Delta tables because of consistency issues.
The checkpoint directory will be used to track updates to the static Delta table.

Answer(s): A

Reveal Solution Next Question

QUESTION: 19

A junior data engineer has been asked to develop a streaming data pipeline with a grouped aggregation using DataFrame df. The pipeline needs to calculate the average humidity and average temperature for each non- overlapping five-minute interval. Events are recorded once per minute per device.

Streaming DataFrame df has the following schema:

"device_id INT, event_time TIMESTAMP, temp FLOAT, humidity FLOAT"

Code block:

Choose the response that correctly fills in the blank within the code block to complete this task.

to_interval("event_time", "5 minutes").alias("time")
window("event_time", "5 minutes").alias("time")
"event_time"
window("event_time", "10 minutes").alias("time")
lag("event_time", "10 minutes").alias("time")

Answer(s): B

Reveal Solution Next Question

QUESTION: 20

A data architect has designed a system in which two Structured Streaming jobs will concurrently write to a single bronze Delta table. Each job is subscribing to a different topic from an Apache Kafka source, but they will write data with the same schema. To keep the directory structure simple, a data engineer has decided to nest a checkpoint directory to be shared by both streams.

The proposed directory structure is displayed below:

Which statement describes whether this checkpoint directory structure is valid for the given scenario and why?

No; Delta Lake manages streaming checkpoints in the transaction log.
Yes; both of the streams can share a single checkpoint directory.
No; only one stream can write to a Delta Lake table.
Yes; Delta Lake supports infinite concurrent writers.
No; each of the streams needs to have its own checkpoint directory.

Answer(s): E

Reveal Solution Next Question

QUESTION: 21

A Structured Streaming job deployed to production has been experiencing delays during peak hours of the day. At present, during normal execution, each microbatch of data is processed in less than 3 seconds. During peak hours of the day, execution time for each microbatch becomes very inconsistent, sometimes exceeding 30 seconds. The streaming write is currently configured with a trigger interval of 10 seconds.

Holding all other variables constant and assuming records need to be processed in less than 10 seconds, which adjustment will meet the requirement?

Decrease the trigger interval to 5 seconds; triggering batches more frequently allows idle executors to begin processing the next batch while longer running tasks from previous batches finish.
Increase the trigger interval to 30 seconds; setting the trigger interval near the maximum execution time observed for each batch is always best practice to ensure no records are dropped.
The trigger interval cannot be modified without modifying the checkpoint directory; to maintain the current stream state, increase the number of shuffle partitions to maximize parallelism.
Use the trigger once option and configure a Databricks job to execute the query every 10 seconds; this ensures all backlogged records are processed with each batch.
Decrease the trigger interval to 5 seconds; triggering batches more frequently may prevent records from backing up and large batches from causing spill.

Answer(s): E

Reveal Solution Next Question

QUESTION: 22

Which statement describes Delta Lake Auto Compaction?

An asynchronous job runs after the write completes to detect if files could be further compacted; if yes, an OPTIMIZE job is executed toward a default of 1 GB.
Before a Jobs cluster terminates, OPTIMIZE is executed on all tables modified during the most recent job.
Optimized writes use logical partitions instead of directory partitions; because partition boundaries are only represented in metadata, fewer small files are written.
Data is queued in a messaging bus instead of committing data directly to memory; all data is committed from the messaging bus in one batch once the job is complete.
An asynchronous job runs after the write completes to detect if files could be further compacted; if yes, an OPTIMIZE job is executed toward a default of 128 MB.

Answer(s): E

Reveal Solution Next Question

QUESTION: 23

Which statement characterizes the general programming model used by Spark Structured Streaming?

Structured Streaming leverages the parallel processing of GPUs to achieve highly parallel data throughput.
Structured Streaming is implemented as a messaging bus and is derived from Apache Kafka.
Structured Streaming uses specialized hardware and I/O streams to achieve sub-second latency for data transfer.
Structured Streaming models new data arriving in a data stream as new rows appended to an unbounded table.
Structured Streaming relies on a distributed network of nodes that hold incremental state values for cached stages.

Answer(s): D

Reveal Solution Next Question

QUESTION: 24

Which configuration parameter directly affects the size of a spark-partition upon ingestion of data into Spark?

spark.sql.files.maxPartitionBytes
spark.sql.autoBroadcastJoinThreshold
spark.sql.files.openCostInBytes
spark.sql.adaptive.coalescePartitions.minPartitionNum
spark.sql.adaptive.advisoryPartitionSizeInBytes

Answer(s): A

Reveal Solution Next Question

Databricks Certified Data Engineer Professional Certified Data Engineer Professional Dumps in PDF

Free Databricks Certified Data Engineer Professional Real Questions (page: 3)

QUESTION: 17

QUESTION: 18

QUESTION: 19

QUESTION: 20

QUESTION: 21

QUESTION: 22

QUESTION: 23

QUESTION: 24