Fireworq is a lightweight, high-performance job queue system with the
following abilities.

• Portability - It is available from ANY programming language
  which can talk HTTP. It works with a single binary without external
  dependencies.

• Reliability - It is built on top of RDBMS (MySQL), so that jobs
  won't be lost even if the job queue process dies. You can apply an
  ordinary replication scheme to the underlying DB for the reliability
  of the DB itself.

• Availability - It supports primary/backup nodes. Only one node
  becomes primary simultaneously and the others become backup nodes.
  A backup node will automatically be active when the primary node
  dies.

• Scalability - It always works with a single dispatcher per queue
  which can concurrently dispatch jobs to workers via HTTP.
  Scalability of workers themselves should be maintained by a load
  balancer in the ordinary way. This means that adding a worker will
  never harm performance of grabbing jobs from a queue.

• Flexibility - It supports the following features.

  • Multiple queues - You can define multiple queues and use them
    in different ways: for example, one for a low priority queue for a
    limited number of high latency workers and another one for a high
    priority queue for a large number of low latency workers.
  • Delayed jobs - You can specify a delay for each job, which
    will make the job dispatched after the delay.
  • Job retrying - You can specify the maximum number of retries
    for each job.

• Maintainability - It can be managed on a Web UI. It also provides metrics suitable for monitoring.

• Getting Started
• Using the API
• Inspecting Running Queues
• Configuration
• Other Topics
  • Full List of API Endpoints
  • Full List of Configurations
  • Make It Production-Ready
  • Developer Guide
• License

Getting Started

Run the following commands and you will get the whole system working
all at once. Make sure you have Docker installed before running
these commands.

$ git clone https://github.com/fireworq/fireworq
$ cd fireworq
$ script/docker/compose up

When Fireworq gets ready, it will listen on localhost:8080 (on the
host machine). Specify FIREWORQ_PORT environment variable if you
want Fireworq to listen on a different port.

$ FIREWORQ_PORT=1234 script/docker/compose up

Pressing Ctrl+C will gracefully shut it down.

Behind HTTP proxy

If you are behind HTTP proxy, script/docker/compose up will fail.
Add following configuration to script/docker/docker-compose.yml:

 services:
   fireworq:
+    build:
+      args:
+        - http_proxy=http://.../
+        - https_proxy=http://.../
     environment:

Using the API

Preparing a Worker

First of all, you need a Web server which does an actual work for a
job. We call it a 'worker'.

A worker must accept a POST request with a body, which is typically
a JSON value, and respond a JSON result. For example, if you have a
worker at localhost:3000, it must handle a request like the
following.

POST /work HTTP/1.1
Host: localhost:3000

{"id":12345}

HTTP/1.1 200 OK

{"status":"success","message":"It's working!"}

The response JSON must have status field, which describes whether
the job succeeded. It must be one of the following values., Value, Meaning, :--------------------, :---------------------------------------, "success", The job succeeded., "failure", The job failed and it can be retried., "permanent-failure", The job failed and it cannot be retried., Any other values are regarded as "failure". The HTTP status code is
always ignored.

Enqueuing a Job to Fireworq

Let's make the job asynchronous using Fireworq. All you have to do is
to make a POST request to Fireworq with a worker URL and a job
payload. If you have
a docker-composed Fireworq instance and your docker
host IP (from the container's point of view) is 172.17.0.1, then
requesting something like the following will enqueue exactly the same
job in the previous example.

$ curl -XPOST -d '{"url":"http://172.17.0.1:3000/work","payload":{"id":12345}}' http://localhost:8080/job/foo

When Fireworq gets ready to grab this job, it will POST the
payload to the url. When the job is completed on the worker, the
log output of Fireworq should say something like this.

fireworq_1, {"level":"info","time":1507128673123,"tag":"fireworq.dev","action":"complete","queue":"default","category":"foo","id":2,"status":"completed","created_at":1507128673025,"elapsed":98,"url":"http://172.17.0.1:3000/work","payload":"{\"id\":12345}","next_try":1507128673025,"retry_count":0,"retry_delay":0,"fail_count":0,"timeout":0,"message":"It's working!"}

Further Reading

See the full list of API endpoints for the details of the
API.

Inspecting Running Queues

There is only a set of API endpoints provided by Fireworq itself to
inspect running queues. They are useful for machine monitoring but
not intended for human use.

Instead, use Fireworqonsole, a powerful Web UI which enables
monitoring stats of queues, inspecting running or failed jobs and
defining queues and routings.

Configuration

You can configure Fireworq by providing environment variables on
starting a daemon. There are many of them but we only describe
important ones here. See the full list for the
other variables.

• FIREWORQ_MYSQL_DSN

  Specifies a data source name for the job queue and the repository
  database in a form
  user:password@tcp(mysql_host:mysql_port)/database?options.
  This is for a manual setup and is mandatory for it.

• FIREWORQ_QUEUE_DEFAULT

  Specifies the name of a default queue. A job whose category is
  not defined via the routing API will be delivered
  to this queue. If no default queue name is specified, pushing a job
  with an unknown category will fail for a
  manual setup. A docker-composed instance
  uses default as a default value.

  If you already have a queue with the specified name in the job queue
  database, that one is used. Or otherwise a new queue is created
  automatically.

• FIREWORQ_QUEUE_DEFAULT_POLLING_INTERVAL

  Specifies the default interval, in milliseconds, at which Fireworq
  checks the arrival of new jobs, used when polling_interval in the
  queue API is omitted. The default value is 200.

• FIREWORQ_QUEUE_DEFAULT_MAX_WORKERS

  Specifies the default maximum number of jobs that are processed
  simultaneously in a queue, used when max_workers in the
  queue API is omitted. The default value is 20.

Other Topics

• Full List of API Endpoints
  • Queue Management
  • Routing Management
  • Job Management
• Full List of Configurations
  • Common Variables/Arguments
  • Variables/Arguments only Applicable to Manual Setup
  • Variables only Applicable to a Docker-composed Instance
• Make It Production-Ready
  • Using a Release Build (Manual setup)
  • Preparing a Backup Instance
  • Graceful Shutdown/Restart
  • Logging
  • Monitoring
• Developer Guide

License

• Copyright (c) 2017 The Fireworq Authors. All rights reserved.
• Fireworq is licensed under the Apache License, Version 2.0. See
  LICENSE for the full license text.