Always notify when a process completes

Notify when a process fails

Notify when a process goes from succeeding to failing and vica versa

Notify when going from failing to succeeded and on each failure

Never notify

The process should wait until the currently running instance finishes

The currently running process should be killed

The process has tasks currently executing

All of the process's tasks succeeded on its last run

At least one of the process's tasks failed on its last run

The task has been submitted

The task is waiting on compute resources

The task is currently executing or awaiting backoff

The task has irrevocably failed

The task has succeeded without serious errors

Name of docker image including registry URL if needed

Tag on docker image

Command to pass to Docker container

ARN of an IAM role, see http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html

environment variables to be passed to container

Override default job queue, eg: for priority queue or GPU instances queue

The minimum amount of time (in seconds) that must pass between executions of the process

The EMR cluster on which to launch the job (aka step)

NAme to be assigned to the Job

the AWS Region

The spark job's Main.class to run

Example: org.apache.spark.examples.SparkPi

the s3 path to the jar of the job to be run

Example: s3://my-emr-source-bucket/my-emr-jar.jar

options that will be sent to spark as --conf key=value, e.g. --conf spark.driver.extraJavaOptions=-Denvironment=integration

Example: ["spark.driver.extraJavaOptions=-Denvironment=integration"]

command line arguments to be passe to the job's main class

Example: ["--executor-memory", "1g"]

AWS EMR periodically (currently every 5 minutes) zips up all the logs and upload them into S3. This does not work well with sundial's live logs. Make use of S3 Logs to visualise in real time job's log on sundial's live logs panel

will trigger a s3 dist cp job for every entry in this array, can be used to load data from s3 into HDFS. requires Hadoop as application on the cluster

will trigger a s3 dist cp job for every entry in this array, can be used to save data from HDFS to s3. requires Hadoop as application on the cluster

optional EBS volume in GB to attach to the emr instance, for simplicity only one volume for the given size and type gp2 will be created.

The EMR cluster id

Example: j-2CRR69WTM7N31

Uniquely identifies the log message to prevent duplication

Uniquely identifies the metadata entry to prevent duplication

To be set in case the EMR cannot be launched in the default VPC

role to be assigned to the service, the default one should cover most of the cases

role to be assigned to the ec2 instances composing the EMR cluster. The default one should cover most of the cases

whether or not to show the new cluster in the list of clusters. For more details see emr docs

If true, ignore schedule and only start process if triggered manually

The schedule that the process runs on; if not specified, the process will only run when triggered manually

Source folder or file to copy. It can be either a s3 or HDFS location

Example: s3:///my-bucket-root - hdfs://tmp/my-temp-folder

Source folder or file to copy. It can be either a s3 or HDFS location

Example: s3:///my-bucket-root - hdfs://tmp/my-temp-folder

Internal bookkeeping metadata used for task scheduling (e.g. ECS task ID and cluster name)

The canonical name for this task used by other tasks to identify this task

The tasks that must have completed prior to this one beginning

The execution time (for a single attempt) after which the system will kill the task

If true, the task must explicitly update its status with Sundial in order to succeed.