Skip to main content

types

ApplicationAttempt​

id : string​

ID of YARN application attempt

amContainerId : string​

ID of YARN Application Master container

ApplicationInfo​

id : string​

ID of YARN application

applicationAttempts : ApplicationAttempt​

YARN application attempts

HiveJob​

properties : string​

A mapping of property names to values, used to configure Hive.

continueOnFailure : bool​

Whether to continue executing queries if a query fails.

scriptVariables : string​

Mapping of query variable names to values.

jarFileUris : string​

Jar file URIs to add to the CLASSPATHs of the Hive driver and tasks.

One of queryType​

  • queryFileUri : string​

    URI of the script that contains Hive queries.

  • queryList : QueryList​

Job​

Data Proc job.

Status​

  • STATUS_UNSPECIFIED

  • PROVISIONING

    Job created in metadb and is waiting agent to acquire.

  • PENDING

    Job acquired by agent and is waiting for execution.

  • RUNNING

    Job is running.

  • ERROR

    Job failed.

  • DONE

    Job finished.

  • CANCELLED

    Job cancelled.

  • CANCELLING

    Job is waiting for cancellation.

id : string​

Unique ID of the Data Proc job. This ID is assigned by MDB in the process of creating Data Proc job.

clusterId : string​

ID of the Data Proc cluster.

createdAt : google.protobuf.Timestamp​

The time when the Data Proc job was created.

startedAt : google.protobuf.Timestamp​

The time when the Data Proc job was started.

finishedAt : google.protobuf.Timestamp​

The time when the Data Proc job was finished.

name : string​

Name of the Data Proc job.

createdBy : string​

The id of the user who created the job

status : Status​

Status.

One of jobSpec​

Job specification.

  • mapreduceJob : MapreduceJob​
  • sparkJob : SparkJob​
  • pysparkJob : PysparkJob​
  • hiveJob : HiveJob​
applicationInfo : ApplicationInfo​

Attributes of YARN application.

MapreduceJob​

args : string​

Optional arguments to the driver.

jarFileUris : string​

URIs of file to run.

fileUris : string​

URIs of files to be copied to the working directory of Data Proc drivers and distributed tasks.

archiveUris : string​

URIs of archives to be extracted in the working directory of Data Proc drivers and tasks.

properties : string​

A mapping of property names to values, used to configure Data Proc.

One of driver​

  • mainJarFileUri : string​

    The HCFS URI of the jar file containing the main class.

  • mainClass : string​

    The name of the driver's main class.

PysparkJob​

args : string​

Optional arguments to the driver.

jarFileUris : string​

Jar file URIs to add to the CLASSPATHs of the Data Proc driver and tasks.

fileUris : string​

URIs of files to be copied to the working directory of Data Proc drivers and distributed tasks.

archiveUris : string​

URIs of archives to be extracted in the working directory of Data Proc drivers and tasks.

properties : string​

A mapping of property names to values, used to configure Data Proc.

mainPythonFileUri : string​

URI of the main Python file to use as the driver. Must be a .py file.

pythonFileUris : string​

URIs of Python files to pass to the PySpark framework.

packages : string​

List of maven coordinates of jars to include on the driver and executor classpaths.

repositories : string​

List of additional remote repositories to search for the maven coordinates given with --packages.

excludePackages : string​

List of groupId:artifactId, to exclude while resolving the dependencies provided in --packages to avoid dependency conflicts.

QueryList​

queries : string​

SparkJob​

args : string​

Optional arguments to the driver.

jarFileUris : string​

Jar file URIs to add to the CLASSPATHs of the Data Proc driver and tasks.

fileUris : string​

URIs of files to be copied to the working directory of Data Proc drivers and distributed tasks.

archiveUris : string​

URIs of archives to be extracted in the working directory of Data Proc drivers and tasks.

properties : string​

A mapping of property names to values, used to configure Data Proc.

mainJarFileUri : string​

The HCFS URI of the jar file containing the main class.

mainClass : string​

The name of the driver's main class.

packages : string​

List of maven coordinates of jars to include on the driver and executor classpaths.

repositories : string​

List of additional remote repositories to search for the maven coordinates given with --packages.

excludePackages : string​

List of groupId:artifactId, to exclude while resolving the dependencies provided in --packages to avoid dependency conflicts.

SupportJob​

Data Proc support job.

Status​

  • STATUS_UNSPECIFIED

  • PROVISIONING

    Job created in metadb and is waiting agent to acquire.

  • PENDING

    Job acquired by agent and is waiting for execution.

  • RUNNING

    Job is running.

  • ERROR

    Job failed.

  • DONE

    Job finished.

  • CANCELLED

    Job cancelled.

  • CANCELLING

    Job is waiting for cancellation.

id : string​

Required. Unique ID of the Data Proc job. This ID is assigned by MDB in the process of creating Data Proc job.

clusterId : string​

Required. Unique ID of the Data Proc cluster.

createdAt : google.protobuf.Timestamp​

The time when the Data Proc job was created.

startedAt : google.protobuf.Timestamp​

The time when the Data Proc job was started.

finishedAt : google.protobuf.Timestamp​

The time when the Data Proc job was finished.

status : Status​

Status.

cmd : string​

Command.

timeout : int64​

Execution timeout in seconds.

createdBy : string​

The id of the user who created the job