Compact job server with API, CLI, Web UI and a Sidekiq heart.


Welcome to the Jobly Documentation

Jobly is a sidekiq-based job server with a command line interface, web API and a web dashboard.

What's in the Box

  • Command line interface - for starting the server, starting the worker, and for running jobs.
  • Web API - for executing jobs.
  • Web Dashboard - including job progress and status.
  • Remote Syslog Support - send output to Papertrail or remote syslog.
  • Slack Notifications - report job progress to Slack.

Screencast

Screenshot

Interactive Demo

For an interactive demo, visit the Jobly scenario on Katacoda.

Keyboard Shortcuts for this Documentation Site

ShortcutAction
sFocus the search field
Up / DownNavigate in search results
Right / LeftNavigate to next / previous page

Installation

Jobly is packaged as a Ruby gem, install it by running:

$ gem install jobly

Alternatively, you can download the the official Jobly docker image.

Prerequisites

  • Ruby 2.5 or higher
  • Redis server

Usage

Quick Start

These steps are similar (but not identical) to the steps demonstrated in the What is Jobly screencast.

Step 1: Setup a Jobly workspace

# Create a new workspace (directory)
$ jobly init myjobs

# Make sure dependencies are installed
$ cd myjobs
$ bundle install

# Start the web server and the worker
$ foreman start

# Alternative:
# You may also start the server and worker manually in two separate screens
$ jobly server
$ jobly worker

Step 2: Verify the server is working

Go to http://localhost:3000 - you should see the Sidekiq dashboard

Step 3: Execute a job through the API

Go to http://localhost:3000/do/Hello - you should receive a JSON encoded acknowledgement.

Go back to the dashboard and view the job's status.

Step 4: Execute a job through the CLI

Form the same workspace directory, run:

# Execute the job directly (without the server)
$ jobly run Ping

# Send the job through the Web API server
$ jobly send Ping

# Send the job through the Web API server, with parameters
$ jobly send Hello name:Arnold

Finally, view the http://localhost:3000/statuses page to ensure your jobs were executed.

Server Side

Dashboard and API Server

To start the web server:

  1. Run jobly server from the command line.
  2. Open http://localhost:3000 in your browser.

This will start a server with two primary entry points: Web Dashboard and Jobly API.

Web Dashboard

This dashboard is available at the root of the web server. It is a sidekiq web dashboard, with additional plugins for job progress and status.

The Jobly API Server

This is a simple web API server that allows you to execute jobs remotely by calling /do/JobName?param=value.

See Also: Running Jobs with the API

Worker

To start processing jobs:

  1. Run jobly worker.
  2. That's it.

See Also: Worker Configuration

Client Side

Running Jobs with the CLI

There are three ways to run a job from the command line:

Run the job locally, without going through any of the background job processing chain:

$ jobly run JobName param:value

Run the job locally, but wait for a worker to process it.

$ jobly run --later JobName param:value

Send a job through the API (either localhost or remote).

$ jobly send JobName param:value

Running Jobs with the API

The API supports running jobs either by GET or by POST in the following URL structure:

http://localhost:3000/do/JobName

Example

$ curl localhost:3000/do/Build

Or, if your job requires parameters, provide them through the query string:

$ curl localhost:3000/do/Build?deploy=no

Response

On success (200 OK)

{
  "status": "received",
  "job": "Build",
  "params": {
    "deploy":"no"
  }
}

On failure (404 Not Found)

{
  "status": "error",
  "message": "No such job",
  "job": "BuildAndRun",
  "params": {}
}

Building Jobs

In order to build a jobs workspace, start in an empty folder and create a ./jobs sub folder inside it. All your job classes go in this folder. The name of this folder is configurable.

See Also: Configuration

All the Ruby files in the ./jobs folder will be loaded by all of Jobly's commands. It is recommended to only keep Job classes in this folder. If you need to load additional code, you can use the ./app folder.

See Also: Loading Additional Code

Your job classes should simply inherit from Jobly::Job, like this:

class Hello < Jobly::Job
  def execute
    # Your code here
  end
end

Scaffolding

Jobly comes with a simple scaffolding command to get you started quickly with a proper directory structure for a jobs workspace.

Run:

$ jobly init myjobs

in order to create a jobs workspace inside the myjobs directory, or:

$ jobly init --help

for additional options.

The Job Class

A job class is a simple Ruby class inheriting from Jobly::Job.

The only requirement is that your class implements an execute method that optionally accepts keyword arguments (recommended), or a hash.

Example:

class Hello < Jobly::Job
  def execute(name: 'bob')
    puts "Hello #{name}"
    logger.info "said hello to #{name}"
  end
end

Note that these classes are simply Jobly-flavored sidekiq jobs, with these key differences:

  • You need to implement execute instead of perform

  • Job arguments are defined as keyword arguments, instead of positional

    arguments.

Job Options

The Jobly::Job class supports these class-level options:

KeyDefaultPurpose
queuedefaultset the name of the queue for this job.
retries5number of times to retry on failure.
backtrace5number of backtrace lines to show in case of failures. Can be true, false or a number of lines to save.
solofalseSpecify that this task should only be executed once. If the same job is executed while this job is running, it will be skipped. See Solo Jobs for more information.

For example:

class Deploy < Jobly::Job
  solo
  queue 'critical'
  backtrace 10
  retries 3

  def execute
    puts "Deploying"
  end
end

See Also: Solo Jobs

Job Callbacks

The Jobly::Job class supports these callback methods:

MethodDescription
beforeExecutes before the job starts
on_skipExecutes if skip_job was called from the before block
on_successExecutes after the job finishes, and only if it succeeds
on_failureExecutes after the job finishes, and only if it fails
afterExecutes after the job finishes, regardless of success or failure

Each callback method can either be a block or a symbol that points to a local method. When using a block, you will have the params variable available, with all the parameters sent to the job.

For example:

class Greet < Jobly::Job
  before do
    logger.info "Starting with #{params[:message]}"
  end

  after :reboot_computer

  def execute(message: "Hello")
    puts message
  end

  def reboot_computer
    system "reboot"
  end
end

In order to conditionally skip a job from its before block, you can call skip_job. This will avoid running the job, and will execute the on_skip action and the after action, if present.

Solo Jobs

Jobs can be defined so that they only have one instance running at any given time, by using the solo method.

class Test < Jobly::Job
  solo

  def execute
    # Job code here
  end
end

Solo Jobs (also known as Unique Jobs) will register a lock key in redis before execution, which will prevent subsequent jobs with the same fingerprint from running until the lock is released. The lock will be automatically removed when the job completes (even if it fails), or after a one hour expiration period.

When a job is skipped due to locking, it will execute the on_skip callback.

See Also: Job Callbacks

Job Fingerprint

By default, the job's fingerprint is built of its class name and its parameters. You can specify a different fingerprint in one of two ways:

Option 1 - Use the 'key' property:

class Test < Jobly::Job
  solo key: 'my-test-job'

  def execute
    # Job code here
  end
end

Option 2: Define a 'solo_key' method in your job:

class Test < Jobly::Job
  solo

  def execute(region: 'eu')
    # Job code here
  end

  def solo_key
    "Test:#{params[:region]}"
  end
end

Tip: The #solo_key method can use any of the job's parameters by accessing the params hash.

Lock Expiration Safeguard

By default, the locks are set to expire after one hour. This is designed as a safeguard to avoid dead locks. Under most circumstances, you do not need to worry about it, since locks are automatically released on job completion, but in case you wish to change the default expiration period, you can use the expire option:

class Test < Jobly::Job
  solo expire: 30.minutes

  def execute(region: 'eu')
    # Job code here
  end

  def solo_key
    "Test:#{param[:region]}"
  end
end

You can use seconds, minutes, hours and days.

Note: Expiration timer starts to count down from the moment the job is queued, and not from the moment it is executed.

See Also: Solo Job Example

Isolated Jobs

Jobs can be defined so that they are executed in their own temporary directory by using the isolated method.

class Test < Jobly::Job
  isolated

  def execute
    # Job code here - will be executed in a new temporary directory
  end
end

Storing Arbitrary Data with Redis

Jobly comes bundled with Sidekiq::Status which has a built in ability allowing you to store and use arbitrary data in its Redis database.

# jobs/hello.rb
class Hello < Jobly::Job
  def execute(name: 'bob')
    # Store arbitrary data
    store vino: 'veritas'

    # Retrieve data.
    vino = retrieve :vino
  end
end

Note that:

  1. Values are always stored as a String, so you will need to convert them to other types when appropriate.
  2. When retrieve does not find the key, nil is returned.
  3. Stored keys are available across jobs.

Running Shell Commands

Although you can use any Ruby method to execute shell and system commands from within your jobs, Jobly comes bundled with TTY::Command which is made available to your jobs by using the #shell method.

# jobs/shell.rb
class Shell < Jobly::Job
  def execute
    shell.run "docker pull ubuntu"
  end
end

Using this method is recommended, since it automatically sends the STDOUT and STDERR of the command to the configured logging device, and it will raise an exception on failure, which will fail the job.

Dry run

When the JOBLY_SHELL_DRY_RUN environment variable is set, the shell.run helper will not run the commands, and instead, only print them to the log.

Accessing the shell helper from other classes

To include the shell helper in other classes (non Jobly::Job), you can include the Jobly::Shell module (or the more inclusive Jobly::Helpers module).

# app/git.rb
class Git
  include Jobly::Shell

  def clone(repo)
    shell.exec "git clone #{repo}"
  end
end

Loading Additional Code

In case your jobs require additional functionality, you may create the ./app folder as a sibling to the ./jobs folder (configurable).

  • All ruby files in this folder (and sub folders) will be automatically loaded and available to your jobs. Note that files will be required in alphabetical order.
  • In case there is an app/boot.rb file, it will be loaded first.

Access job helpers from custom code

Custom code that does not inherit from Jobly::Job will not be able to access some job features, such as logging and slack notifications.

To enable such access, you can include Jobly::Helpers in your custom code:

# app/git.rb
class Git
  include Jobly::Helpers

  def pull(repo)
    logger.info "pulling #{repo}..."
  end
end

The Jobly::Helpers module is a shortcut that includes several helpers. You can include them separately if you prefer:

include Jobly::Logging
include Jobly::Slack
include Jobly::Shell
include Jobly::Settings

Logging, Notifications and Status

Logging

Logging from within jobs

All your jobs have access to a standard Ruby logger.

# jobs/hello.rb
class Hello < Jobly::Job
  def execute(name: 'bob')
    logger.info "said hello to #{name}"
  end
end

Logging from other classes

In case you need access to the logger from other classes (that do not inherit from Jobly::Job), you can include the Jobly::Logging module (or the more inclusive Jobly::Helpers module).

# app/git.rb
class Git
  include Jobly::Logging

  def pull(repo)
    logger.info "pulling #{repo}..."
  end
end

Configuring the logger

By default, output is sent to STDOUT but you can provide your own logger and log to a file or to syslog by using the config.log or the config.logger option in ./config/jobly.rb

# config/jobly.rb
Jobly.configure do |config|
  config.log = '/var/log/jobly.log'
  
  # same as:
  # config.log = Logger.new '/var/log/jobly.log'

  config.log_level = 'warn'
end

The config.log option controls how logging is handled.

  1. If left empty (nil), the web server and workers will NOT log anywhere, and your jobs will log to STDOUT whenever you use logger.
  2. Setting it to config.log = 'stdout' will also instruct the web server and worker to send their logging to STDOUT.
  3. Setting it to a filename, will log to a file.
  4. Setting it to a syslog connection string will log to a remote syslog server.

Logging to syslog

Set config.log to a syslog connection string in the following format:

syslog://system:[email protected]:port

Omitting any of the options will fall back to a sensible default.

See Also: Syslog Example

Separate log files for each job class

If Jobly.log contains %s in the file path, it will be replaced with the slug of the job, and will create separate log files for each job class.

# config/jobly.rb
Jobly.configure do |config|
  config.log = 'logs/%s.log'
end

Automatic syslog tagging

The same %s replacement principle applies when using a syslog connection string. This is intended to allow tagging of syslog messages with the job name.

# config/jobly.rb
Jobly.configure do |config|
  config.log = 'syslog://jobserver:%[email protected]:514'
end

Bring your own logger

The config.log option can also accept any Logger instance, in case you wish to provide a custom logger.

Reporting Job Progress

Jobly comes bundled with Sidekiq::Status which provides you with a live status dashboard, and a way to report progress in your jobs.

In order to report progress from your job, use the total and at methods. In addition, you can store arbitrary data related to your job with store and retrieve.

# jobs/hello.rb
class Hello < Jobly::Job
  def execute(name: 'bob')
    # How many steps this job has (default is 100)
    total 100 

    # 5/100 = 5% completion
    at 5, "Initializing"
  end
end

Sending Slack Notifications

Jobly comes bundled with Slack::Notifier which is made available to your jobs using the #slack method.

# jobs/greet.job
class Greet < Job
  # Optional settings (defaults to #debug and Jobly)
  # Can also be set in the configuration
  slack_channel '#debug'
  slack_user 'Greeter'

  def execute(name: 'bob')
    # Quick message
    slack.ping "Hello #{name}"

    # Message with attachment
    slack.post attachments: { text: "Good text", color: "good" }
  end
end

Sending notifications from other classes

To include the slack helper in other classes (non Jobly::Job), you can include the Jobly::Slack module (or the more inclusive Jobly::Helpers module).

# app/git.rb
class Git
  include Jobly::Slack

  def pull(repo)
    slack.ping "pulling #{repo}"
  end
end

See Also: Slack Notifications Example

Configuration

Jobly Configuration

Configuring Jobly can be done by one of two methods:

  1. Setting environment variables.
  2. Adding a config/jobly.rb to your jobs workspace.

Environment variable reference

Environment VariableDefault
JOBLY_ENVIRONMENTdevelopment
JOBLY_API_URLhttp://localhost:3000/do
JOBLY_APP_PATHapp
JOBLY_CONFIG_PATHconfig
JOBLY_REDIS_URLredis://localhost:6379/0
JOBLY_STATUS_EXPIRATION30
JOBLY_JOBS_NAMESPACEunset
JOBLY_LOGunset
JOBLY_LOG_LEVELinfo
JOBLY_SLACK_WEBHOOKunset
JOBLY_SLACK_CHANNEL#general
JOBLY_SLACK_USERJobly
JOBLY_AUTHunset
JOBLY_SHELL_DRY_RUNunset

Jobly configuration file

The below configuration file shows all the available options. Everything is optional and has defaults. This file is loaded on boot by the server, worker and jobly CLI.

# config/jobly.rb
Jobly.configure do |config|
  # environment: development | production
  # Sets the environment for the API Sinatra server and for Sidekiq.
  # Default: ENV['JOBLY_ENVIRONMENT'] || 'development'
  config.environment = 'production'

  # api_url: url
  # Sets the URL to the API server.
  # Default: ENV['JOBLY_API_URL'] || 'http://localhost:3000/do'
  config.api_url = 'http://localhost:3000/do'

  # app_path: relative or absolute path
  # Sets the path to the optional folder that may contain support files for
  # the jobs. All ruby files in this folder are loaded first.
  # Default: ENV['JOBLY_APP_PATH'] || 'app'
  config.app_path = 'app'

  # jobs_path: relative or absolute path
  # Sets the path to the jobs folder. All ruby files in this folder are loaded
  # automatically by all of Jobly's components.
  # Default: ENV['JOBLY_JOBS_PATH'] || 'jobs'
  config.jobs_path = 'jobs'

  # config_path: relative or absolute path
  # Sets the path to the config directory. This is where Jobly looks for this
  # jobly.rb configuration file, so it should be set via the environment
  # variable if you wish to change it.
  # Default: ENV['JOBLY_CONFIG_PATH'] || 'config'
  config.config_path = 'config'

  # redis_url: url
  # Sets the full URL to the redis database
  # Default: ENV['JOBLY_REDIS_URL'] || 'redis://localhost:6379/0'
  config.redis_url = 'redis://localhost:6379/0'

  # status_expiration: minutes
  # Sets the number of minutes that completed jobs are kept in the Statuses
  # tab.
  # Default: ENV['JOBLY_STATUS_EXPIRATION'] || 30
  config.status_expiration = 5

  # jobs_namespace: string
  # Sets the namespace (module name) that your jobs are defined in. For 
  # example, if your Job classes are in fact defined as `MyJobs::SomeJob` 
  # then this option should be set to `MyJobs`.
  # Default: ENV['JOBLY_JOBS_NAMESPACE'] || nil
  config.jobs_namespace = nil

  # log: string or logger instance
  # Sets the logger that will be used by the servers and worker. This logger
  # is also available to your jobs with the `logger` method.
  # This can be one of these strings:
  # - stdout
  # - syslog://system:[email protected]:port
  # - filename
  # Or your own Logger instance.
  # Default: ENV['JOBLY_LOG'] || nil (log to STDOUT)
  config.log = 'jobly.log'

  # log_level: string
  # Sets the logger level.
  # This can be one of these strings:
  # - debug
  # - info
  # - warn
  # - error
  # - fatal
  # Default: ENV['JOBLY_LOG_level'] || 'info'
  config.log_level = 'info'

  # slack_webhook: url
  # To use the built in slack notifier, set this slack
  # webhook.
  # Default: ENV['JOBLY_SLACK_WEBHOOK']
  config.slack_webhook = "https://hooks.slack.com/..."
  
  # slack_channel: string
  # Sets the default slack channel.
  # This can be overwritten on a per-job basis.
  # Default: ENV['JOBLY_SLACK_CHANNEL'] || '#general'
  config.slack_channel = '#debug'
  
  # slack_user: string
  # Sets the default slack user name to display.
  # This can be overwritten on a per-job basis.
  # Default: ENV['JOBLY_SLACK_USER'] || 'Jobly'
  config.slack_user = 'Botly'
  
  # mounts: hash
  # Mount additional rack apps to the web server.
  config.mounts = { "/my" => MyApp }
  
  # auth: user:password string
  # Set basic authentication for the server and client.
  config.auth = "admin:secret"

  # shell_dry_run: boolean
  # When true, calls to shell.run will print the command to the screen 
  # instead of executing.
  # Default: ENV['JOBLY_SHELL_DRY_RUN'] || false
  config.shell_dry_run = false

end

Accessing options from your code

In case you need to access any of these settings from your code, simply use the Jobly.option_name syntax or Jobly.settings to get a hash of all settings.

# job/show_options.rb
class ShowOptions < Jobly::Job
  def execute
    pp Jobly.options
    pp Jobly.slack_user
    pp Jobly.root
  end
end

Additional read only options

These options can also be accessed from anywhere in your code, as read only:

Jobly.root              # the root path of the workspace
Jobly.full_app_path     # the full path to Jobly.app_path
Jobly.full_jobs_path    # the full path to Jobly.jobs_path
Jobly.full_config_path  # the full path to Jobly.config_path

Worker Configuration

The worker options can be configured by command line options or by specifying a worker configuration file when running it.

The configuration file can be used to:

  • Specify default values for the worker (instead of specifying them inside the jobs themselves).
  • Specify additional advanced sidekiq configuration options.

Tip: The worker configuration file is in fact a sidekiq configuration file. Refer to the sidekiq documentation if you need more advanced options.

Sample Worker Configuration File

# config/my-worker.yml
---
:concurrency: 2
:queues:
- critical
- default
- mundane

Place your YAML config file inside the config folder, and use the jobly worker --config my-worker to use it.

Note: The filename specified with the --config flag must be placed inside the config folder. You do not need to specify the folder name or the .yml extension.

See Also: Worker Config Example

Custom Settings

Jobly comes bundled with Sting, which provides you with a settings helper that lets you use custom YAML files for configuring your jobs.

Settings files

  1. Place settings.yml in the config folder.
  2. Optionally, place additional settings.<environment>.yml (for example settings.production.yml) files for each of your environments. These files will be loaded based on the JOBLY_ENVIRONMENT configuration.

ERB

Before the YAML files are loaded, they are evaluated with ERB, so you can use Ruby code in them (for example, to load environment variables):

# config/settings.yml
host: localhost
port: <%= ENV['PORT'] || 3000 %>

Extended YAML

Your settings*.yml files can use the extends key to load and merge additional YAML files:

# config/settings.yml
extends: some-other-file.yml
host: localhost
  • The .yml extension is optional.
  • You may extend either a single file or an array of files.

This functionality is provided by ExtendedYAML.

Using settings in your jobs

Your jobs will have access to a settings object:

# jobs/push.rb
class Push < Jobly::Job
  def execute
    puts settings.host
  end
end

Using settings in other classes

To have the settings object available to other (non Jobly::Job) classes, include the Jobly::Settings module (or the more inclusive Jobly::Helpers module).

# app/docker.rb
class Docker
  include Jobly::Settings
  
  def pull
    puts settings.docker_registry
  end
end

See Also: Settings Example

Workspace Info

You can place a file named info.md in your config folder with markdown-formatted information about your jobs workspace.

This file will be pretty-printed when running jobly info.

You can use this file as a template:

# Workspace Name

Describe your jobs here using markdown.

# JobName

Describe the job

## Arguments:

*branch* (required)
Argument description here

*environment* (default: `production`)
Argument description here




Examples

The examples in this section are also available on GitHub.

Each example demonstrates a specific feature or aspect of Jobly.

Basic Example

Code

# jobs/hello.rb
class Hello < Jobly::Job
  def execute(name: 'bob')
    puts "Hello #{name}"
    logger.info "said hello to #{name}"
  end
end

Commands to Try

cd examples/basic

# Run the job locally and immediately
jobly run Hello

# Start the worker
jobly worker

# From another terminal, send the job to the worker
jobly run --later Hello name:Jimmy

# Start the API / Dashboard server then visit http://localhost:3000
jobly server

# From another terminal, send the job from the command line to the API
jobly send Hello

# Schedule a job by accessing the API:
open http://localhost:3000/do/Hello?name=Guest

Job Callbacks Example

This example illustrates the use of before, after, on_skip, on_success, and on_failure blocks.

Code

# jobs/actions.rb
class Actions < Jobly::Job
  before { puts "before" }
  after  { puts "after" }

  on_success { puts "on_success" }
  on_failure { puts "on_failure" }

  after :after_party

  def execute(fail: false)
    puts "execute 1/2"
    raise "RAISED" if fail
    puts "execute 2/2"
  end

  def after_party
    puts "after_party"
  end
end
# jobs/filter.rb
class Filter < Jobly::Job
  before do
    puts "before"
    skip_job
  end

  after do
    puts "after"
  end

  on_skip    { puts "on_skip" }
  on_success { puts "on_success" }
  on_failure { puts "on_failure" }

  def execute
    puts "execute"
  end
end

Commands to Try

cd examples/before-after-actions

# Run the job in its successful form
jobly run Actions

# Run the job in its failed form
jobly run Actions fail:yes

# Run the filter job (will not execute the job body)
jobly run Filter

Docker Compose Example

This example demonstrates how to use Jobly inside a docker container and with docker compose.

Code

# Dockerfile
FROM dannyben/alpine-ruby
RUN gem install jobly
WORKDIR /app
COPY app .
# docker-compose.yml
version: '3'

services:
  redis:
    image: redis

  bash: &default
    build: .
    image: temp/jobly
    depends_on: [redis]
    volumes:
    - ./app:/app
    environment:
      JOBLY_REDIS_URL: redis://redis:6379/0
      JOBLY_API_URL: http://web:3000/do

  web:
    <<: *default
    ports: ["3000:3000"]
    command: jobly server

  worker:
    <<: *default
    command: jobly worker

  send:
    <<: *default
    depends_on: [web]
    entrypoint: jobly send
    command: Hello
# app/jobs/hello.rb
class Hello < Jobly::Job
  def execute(name: 'bob')
    puts "Hello #{name}"
    logger.info "said hello to #{name}"
  end
end

Commands to Try

cd examples/docker-compose

# Build the images
docker-compose build

# Start the server and the worker
docker-compose up web worker

# Visit your server
open http://localhost:3000

# Send a test job
docker-compose run send

# ... or with parameters
docker-compose run send Hello name:Bobby

Job in Job Example

This example demonstrates how to call a job from inside another job.

Two approaches are possible:

  1. Executing secondary jobs synchronously, as part of the parent job.
  2. Spawning the secondary jobs to be executed later by the worker, as separate jobs.

Code

# jobs/run_sync.rb

# This job will call the other jobs synchronously and display progress
# in the dashboard

class RunSync < Jobly::Job
  def execute(count: 10)
    total count

    count.times do |i|
      at i
      puts "Executing Job ##{i}"
      SubJob.run number: i
    end

    at count, "Done"
  end
end
# jobs/run_async.rb

# This job will spawn child threads for the sub jobs.
# Each of the spawned jobs will have its own progress bar in the dashboard.

class RunAsync < Jobly::Job
  def execute(count: 10)
    count.times do |i|
      SubJob.run_later number: i
    end
  end
end
# jobs/sub_job.rb
class SubJob < Jobly::Job
  def execute(number:)
    logger.info "Executing... something... #{number}"
    sleep rand(1..5)
  end
end

Commands to Try

cd examples/job-in-job

# Start the server and worker
foreman start

# From another terminal, send the jobs to the worker
jobly send RunSync count:20
jobly send RunAsync count:10

Namespaces Example

This example illustrates how to define jobs that are namespaced in modules.

It illustrates:

  1. How all jobs can be nested inside a specific module that is used as the default namespace for all jobs - see the config folder.
  2. How additional namespacing is done by putting some jobs under a nested module and (optionally, by convention) putting them inside a subfolder - see the jobs folder.

Code

# config/jobly.rb
Jobly.configure do |config|
  config.jobs_namespace = 'Jobs'
end
# jobs/hello.rb

# Global namespace for all jobs
module Jobs
  class Hello < Jobly::Job
    def execute(name: 'bob')
      puts "Hello #{name}"
      logger.info "said hello to #{name}"
    end
  end
end
# jobs/inner/hi.rb
module Jobs
  # A nested job, can be executed with Inner::Hi or Inner/Hi
  module Inner
    class Hi < Jobly::Job
      def execute(name: 'bob')
        puts "Hi #{name}, this is #{self.class.name}"
        logger.info "said hello to #{name} from #{self.class.name}"
      end
    end
  end
end

Commands to Try

cd examples/namespaces

# Run the namespaced job that is found in the root namespace `Jobs`
jobly run Hello

# Run the nested job that is found in the root namespace `Jobs::Inner`
jobly run Inner::Hi

Options and Errors Example

This example illustrates setting the job options: backtrace, queue and retries.

Code

# jobs/deploy.rb
class Deploy < Jobly::Job
  # This job runs on the 'critical' queue
  queue 'critical'

  # On failure, show 5 backtrace lines in the web UI
  backtrace 5

  # On failure, retry once only
  retries 1

  def execute(env: 'stage')
    puts "Deploying to #{env}"
    raise "Something went wrong..."
  end
end

Commands to Try

cd examples/options-and-errors

# Start the server and the worker
foreman start

# Or one by one
jobly server
jobly worker --queue critical

# From another terminal, send the job to the worker
jobly send Deploy

# View the failed job and its backtrace in the Retries tab
open http://localhost:3000/retries

Slack Notifications Example

This example illustrates how to send slack notifications from your jobs. Note that for this to work, you need to set up a web hook in your Slack configuration and then make it available to Jobly by the slack_webhook option or the JOBLY_SLACK_WEBHOOK environment variable.

Code

# jobs/greet.rb
class Greet < Jobly::Job
  # Optional settings (defaults to #general and Jobly)
  slack_channel '#debug'
  slack_user 'Greeter'

  def execute(name: 'bob')
    # Quick message
    slack.ping "Hello #{name}"

    # Message with attachment
    slack.post attachments: { text: "Good text", color: "good" }
  end
end

Commands to Try

cd examples/slack-notifications

# Set your slack webhook in an environment variable
export JOBLY_SLACK_WEBHOOK=https://hooks.slack.com/services/...

# Run the job
jobly run Greet

Syslog Example

This example illustrates how to configure Jobly to send all logger events to a local or remote syslog.

Enabling syslog UDP connection

Note that for this example to work, your syslog server must be configured to accept UDP connections.

Edit /etc/rsyslog.conf and uncomment these lines:

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

Code

# config/jobly.rb
Jobly.configure do |config|
  # Log to local syslog and use the system name `jobly` and tag `demo`
  config.log = "syslog://jobly:[email protected]:514"
  config.log_level = 'warn'
end
# jobs/hello.rb
class Hello < Jobly::Job
  def execute(name: 'bob')
    puts "Hello #{name}"
    logger.info "said hello to #{name}"
  end
end

Commands to Try

cd examples/syslog

# Install additional gem dependencies
bundle install

# Start watching the syslog
sudo tail -f /var/log/syslog

# Start both the server and the worker
foreman start

# Send jobs from the command line to the API
jobly send Hello

Worker Config Example

This example makes use of worker configuration file in order to tell the worker to process two queues (critical and mundane) instead of the default queue.

Note that the same can also be achieved through the command line, without using a config file.

Code

# config/worker.yml
:concurrency: 2
:queues:
- critical
- default
- mundane
# jobs/critical.rb
class Critical < Jobly::Job
  options queue: 'critical'

  def execute
    logger.info "Running a critical job"
  end
end
# jobs/mundane.rb
class Mundane < Jobly::Job
  options queue: 'mundane'

  def execute
    logger.info "Running a mundane job"
  end
end
# Procfile
web: jobly server
worker: jobly worker --config worker

Commands to Try

cd examples/worker-config

# Start the server and worker
# Note that the `Procfile` is defined to run `jobly worker --config worker`
# which points the worker to the `config/worker.yml` configuration file
foreman start

# From another terminal, send the jobs to the worker
jobly send Critical
jobly send Mundane

Solo Job Example

This example demonstrates how to ensure only one instance of the job is executed at any given time.

Code

# jobs/solo.rb
class Solo < Jobly::Job
  solo

  on_skip do
    at 70, "Skipped"
    logger.info "Job was skipped"
  end

  def execute(name: 'bob')
    total 100
    at 10, "Initializing"
    sleep 20
    at 100, "Done"
  end
end

Commands to Try

Before running these commands, it is recommended you open the Statuses tab in the web dashboard.

cd examples/solo

# Start the server and worker
foreman start

# From another terminal, send the Solo job twice
jobly send Solo
jobly send Solo

# Finally, while the job is still locked, send it again with different 
# parameters (and therefore, a different fingerprint).
jobly send Solo name:Bart

Settings Example

This example illustrates how to use the settings helper to load custom settings into your jobs.

Code

# jobs/show_settings.rb
class ShowSettings < Jobly::Job
  def execute
    puts "Loading settings for #{Jobly.environment}"

    # Show all settings hash
    pp settings.settings

    # Access a value
    puts settings.host

    # Access a deeper value or an array value
    puts settings.apps[0]
    puts settings.login['user']
  end
end
# config/settings.yml
host: example.com
port: 80

apps:
- /apps/todo-list
- /apps/greeter

login:
  user: admin
  password: <%= ENV['MY_SECRET'] =>
# config/settings.development.yml
host: localhost
port: 3000

Commands to Try

cd examples/settings

# Run the job
jobly run ShowSettings

Data Store Example

This example demonstrates how to store and retrieve arbitrary data on redis.

Code

# jobs/build.rb
class Build < Jobly::Job
  def execute
    # Get "build_number" from redis, or default to 0
    build_number = retrieve(:build_number).to_i || 0

    # Increase build number and store it back on redis
    build_number += 1
    store build_number: build_number.to_s

    # Use the value
    puts "Building #{build_number}"
    logger.info "Built #{build_number}"
  end
end

Commands to Try

cd examples/storing-data-in-redis

# Run the command several times to get a new build number
jobly run Build
jobly run Build

Custom Rack Mount Example

This example demonstrates how to amend the web server so that it also serves any custom rack app.

Code

# config/jobly.rb
require_relative '../app/my_server'

Jobly.configure do |config|
  config.mounts = { "/my" => MyServer }
end
# app/my_server.rb
require "sinatra/base"

class MyServer < Sinatra::Base
  get '/' do
    "Hello from MyServer!"
  end
end

Commands to Try

cd examples/custom-rack-mount

# Start the server
jobly server

# Call your custom app
curl http://localhost:3000/my

Basic Auth Example

Commands to Try

cd examples/basic

# Start the server with basic auth enabled
JOBLY_AUTH=user:pass jobly server

# Send a job without authentication
# This operation will fail
jobly send Hello

# Send an authenticated job request
# This operation will succeed
JOBLY_AUTH=user:pass jobly send Hello

GitHub Integration Example

This example demonstrates how to integrate Jobly with GitHub for:

  • Handling incoming webhooks from GitHub
  • Sending pull request status updates

These features are not built into Jobly, but can be easily implemented by adding the Loadrunner gem.

Code

# jobs/build.rb
require 'loadrunner/status'

class Build < Jobly::Job
  def execute(repo:, commit: nil)
    logger.info "Building #{repo}"

    # Send status update to GitHub
    if commit
      Loadrunner::Status.update repo: "YourGithubUser/#{repo}", 
        sha: commit, 
        state: :pending,  # :pending :success :failure :error
        context: "Jobly Demo",
        description: "Build in progress",
        url: "http://example.com"

    end
  end
end

# hooks/global
#!/usr/bin/env ruby

# Handle any incoming GitHub hook

require 'jobly/boot'

commit = ENV['LOADRUNNER_COMMIT']
repo = ENV['LOADRUNNER_REPO']

# Execute a Jobly job

Build.run_later repo: repo, commit: commit

# config/jobly.rb
require 'loadrunner/server'

Jobly.configure do |config|
  # Mount the Loadrunner server under Jobly's /github endpoint
  config.mounts = { "/github" => Loadrunner::Server }
end

Commands to Try

cd examples/github
bundle install

# Start the server
foreman start

# Simulate a webhook using loadrunner command line interface
loadrunner event localhost:3000/github repo push