Xbus

Build Status License

Xbus provides high-level application messaging on top of NATS.

Requirements

Golang

This is a Go project. We build and test xbus with go 1.11.x

Postgresql

Xbus uses a postgresql database 9.5+ to serialize configuration and events.

Build

Build as a regular Go package:

make devtools
govendor sync
make install

Running

3 binaries are created:

  • xbusd: Main Xbus program.
  • xbusctl: Xbus administration program (ctl = “control”); remotely connects to xbusd.
  • xbus-client: Administration of actors (programs that can connect to Xbus).

Each program may be run with --help to see available commands / flags / etc.

Warning: xbusd <xbusd> can produce a LOT of logs when placed under load. The server will crash if it is unable to write to stdio/stderr. Init systems that implement rate limitation for logs (such as systemd) may cause xbusd <xbus> to crash if the server exceeds this limit.

Updating

  • The migration directory contains scripts one can use to migrate Xbus databases from one version to the other.

Versions

3.0.5 (2019-10-17)

  • service ravencapture:
    • improve the message that is sent to sentry with a process fails. The new message will contain the pipeline and emiter names, and if available the first error log message by the actor that failed. This helps sentry grouping entries in a more usable way, and the issue labels will give information immediately.
    • Include all the process logs and not only the ERROR logs. It helps analysing process that failed with no ERROR message.
  • xbus-http webhooks: add settings to define the http codes that needs reattempts and the ones that means a success.

3.0.4 (2019-08-22)

  • xbusd:
    • control.Process.Purge: improved performances
  • xbusctl:
    • fix ps purge failure with too many processes

3.0.3 (2019-06-25)

  • metrics: fix database metrics
  • ps purge: change how the max purge jobs count is calculated, resulting in better parallelisation when the storage process has more database connections.

3.0.2 (2019-07-24)

  • logstorage: fix storing non-utf8 log messages. This bug could cause crashes in the director service, leading to zombie processes.

  • xbusd: disable completely the raven capture if no SentryDSN is set.

  • xbusctl pipeline save don’t overwrite the source file anymore.

  • metrics:

    • Add gauges for the workers and consumers online status.
    • Add database connections metrics
    • Export most gnatsd ‘Varz’ indicators as prometheus metrics

3.0.1 (2019-06-19)

  • actor deletion is no longer blocked if some logs refer to it
  • envelope purge is no longer blocked if some logs refer to it
  • account deletion: related actors are also deleted after user confirmation
  • small documentation changes
  • pipelines (and process): consumers can be matched by name or id and not only by roles.
  • log an error when an actor is marked offline after not responding
  • xbusd-enterprise & xbusd-lts: fix an excessive CPU usage.
  • xbusctl:
    • actor list now display the name of the account of each actor
    • account: add a ‘export’ command to recreate a client configuration file.
    • ps export: fix exporting huge list of process (10th of thousands)
  • xbus-http: add the ‘renew-cert’ command
  • ‘renew-cert’ command:
    • print an error if reset-cert was not applied to the account prior to the cert renewal.

3.0.0 (2019-03-26)

  • service envelopestorage: raise slow emitter detection delay from 15 to 30 seconds.
  • sentry: report failed Process to sentry
  • xbusd: new setting “sentry.environment”

3.0.0-rc.6 (2019-02-15)

  • API:

    • ProcessState.Envelope now returns a single Process
    • ProcessState.Envelope subject: emitter is now a method param

    Those changes reflects the fact that a single envelope should never trigger several _concurrent_ processes.

3.0.0-rc.5 (2019-02-01)

Warning

All the NATS subject of the API are now prefixed with “xbus.default” instead of just “xbus”, which breaks the API. Xbusd, xbus-http, xbusctl and _all_ the clients must be upgraded at the same time.

It also impacts hand-written clients that need to adjust the subject they use.

  • xbusctl: Fix ‘xbusctl pm’ timeout on big lists.
  • xbusd: Fix pipeline saving to forbid non-draft pipeline update.
  • xbusd: Fix a crash after 1000 process on one actor through xbus-http /processrequest (#159).
  • xbusd --version now ignores completly the configuration and the ‘certs’ directory (#160).

3.0.0-rc.4 (2019-01-22)

  • xbusd cert generate: improve –host-defs documentation
  • xbus-fullenv: extend variable substitution to client fullenv-specific settings
  • xbusd: fix pipeline matching by actor/roles (#155)
  • xbusd: fix service ‘control’ stall if a ps export is canceled (#157)
  • xbusd: service ‘control’ process functions use a worker pool, enabling parallel executions of ‘xbusctl ps’ commands.
  • xbusd: fix help of the ‘–config’ flag (#115)
  • xbus-http: implements auto-reconnection of webhooks (#153)
  • xbusctl: ps export: add a “–quiet” flag to disable extra logging (#144)

3.0.0-rc.3 (2019-01-07)

  • database: remove actor.config size limit.
  • xbusd: adjust control.Process.Query batch size to avoid too big fragments (see #142).
  • xbus-http: fix webhooks response handling (see #139, #140 and #141)

3.0.0-rc.2 (2018-12-27)

  • xbus-fullenv: add a ‘–var’ flag to do variable substitution in the configuration file.
  • documentation: Add a ‘migration’ section
  • Enforce client protocol version compatibility: a client must advertise its protocol version when connecting, and will fail to authenticate if the version is not present or is not compatible with the server protocol version.
  • Add link options for producing licensed binaries.
  • Remove some ci files
  • Add json support to the WhoAmI service

3.0.0-rc.1 (2018-12-14)

  • fix the 3.0.0-beta.14 migration script
  • The client configuration are not rewritten at ‘register’ anymore. The TLS entries are written to one or several separate files instead.
  • The client ‘actors’ configuration entry now accepts actor names as keys instead of integer values
  • http: Fix an issue with enveloppe/process polling returning the same value several times (see #133)
  • update most of the dependencies (see #134)
  • xbusd: The default ‘cert-dir’ is now relative to the configuration file instead of the current directory.

3.0.0-beta.16 (2018-12-03)

  • Base the xbus-fullenv docker image on busybox

  • Fix the certificate renewal tests

  • Detect stalled envelope emissions and cancel the process accordingly

  • xbus-fullenv run:

    • Add the --auto-startup flag
    • Add the --batch flag

3.0.0-beta.15 (2018-11-19)

  • Renamed ‘graph’ to ‘pipeline’ in CLI and API
  • xbusctl
    • Fix configuration by environment variable
    • ps export: Add --export-jobs flag
    • ps: Add --pipeline flag to filter by pipeline

3.0.0-beta.14 (2018-11-13)

Note

This is the last version for which a SQL script has to be applied manually. Once done, run xbusd migrate to finalize the database migration.

  • [ci] Upgrade go to 1.11

  • xbusd migrate: new command that takes care of updating the database.

  • Add certificate renewal tooling:

    • (xbusctl|xbusd) account reset-cert
    • (xbus-client|xbusctl) renew-cert

3.0.0-beta.13 (2018-10-30)

  • xbusd:
    • Implements process cancel (#27)
  • xbusctl:
    • ps display the graph name & version, and the emitter name instead of their ids
    • Fix a crash on ‘ps pause’ and other ‘ps’ subcommands (#124)
    • Add the command ps cancel for canceling processes (#27)
  • doc:
    • Embed a basic documentation of the API

3.0.0-beta.12 (2018-10-22)

Note

This version is NOT protocol compatible with former versions of xbus. xbusd and the clients must all be upgraded simultaneously.

  • features:
    • Store configuration entries on the actor definitions in the server. This allow external tools like xbus-http to remain stateless apps by storing a few configuration entries on the actors.
  • xbusd:
    • Implements the new Process.ExportStream api endpoint
    • Implements the new Storage.ActorLeaving api endpoint
    • Implements the new Online/Unresponsive properties of Actor
    • As part of the actor protocol refactoring: - Implements the new Director service - Implements the new EnvelopeStorage service - Drop the emission micro-service
    • Gracefully handle broker overflow (#41)
    • A gateway account can now login for itself, getting access to the actor definitions.
    • A gateway account can now login as an actor without extra credentials.
    • service register: Invalid account names are now rejected (#118)
  • xbusctl:
    • register now gives the actual xbusd command to run to complete the registration
    • ps export
      • is not size-limited anymore.
      • added --export-envelope to enable envelope export
      • added --export-envelope-size-limit to limit envelope export size
    • ps pause|resume now takes the same filtering options as the other ps commands.
    • Add the account get-apikey and account set-apikey commands (#101)
    • Add the envelope get command (#96)
    • graph query is now graph list for consistency purpose.
  • xbus-client:
    • register now gives the actual xbusctl command to run to complete the registration
  • Merge xbus-http into the xbus project. The http gateway is becoming an official part of xbus.
  • Introduce a new tool, Xbus Fullenv. It is meant to set up and pilot test environments of xbus from a single configuration file.

3.0.0-beta.11 (2018-04-13)

Note

This version is NOT protocol compatible with former versions of xbus. xbusd and the clients must all be upgraded simultaneously.

  • Reimplement all the NATS calls and handlers with nrpc and the xbus-api proto files.
  • Expose prometheus metrics of the API
  • Implement ‘xbusctl graph get’
  • When –config is used, log an error if the configuration file is not found.
  • Fix: In some case (especially when xbus-http is involved), the emission service would deadlock itself.
  • Fix: Server crashed at startup if sentry was enabled.
  • Fix: Purges would timeout with big volumes of process or envelopes.
  • Optimization: Add indexes to item and log_entry tables. This fix performances issues with big envelopes.
  • Update gnatsd to 1.0.4
  • Switch to go 1.8
  • Cleanups
  • ci: provides automatically built packages and docker files
  • Fix: A ‘active’ graph that is invalid could not be deactivated.

3.0.0-beta.10 (2017-10-19)

  • Fix 3.0.0-beta.9 migration script performances on medium to large databases.
  • Fix an issue when too many inputs or ouputs were defined on a graph node
  • Fix a deadlock in the emission service that can happen with big envelopes sent in many small envelopes with the right timing.
  • Improve process state management:
    • State update is now centralized in a dedicated routine, avoiding some possible race conditions.
    • If an error occurs during state update, the process status is now set in ‘error’.
    • When a process status is set to ‘error’, the reason is added to the process state.
  • xbusd:
    • Add a sentry.dsn setting to send errors to sentry
  • xbus-client:
    • Add a sentry.dsn setting

Database

  • Table graph_node:
    • Change column inputs type to TEXT
    • Change column outpus type to TEXT
  • Table process_log_entry:
    • Add column processing_error (BOOLEAN)
  • Table process_state:
    • Add column status_reason (TEXT)

3.0.0-beta.9 (2017-09-19)

  • solve issue #64 (connection errors when too many client connect at the same time)
  • switch to gnatsd 1.0.2
  • Implement the process ‘export’ and ‘purge’ APIs
  • Implement the new process filtering API
  • Track a few the process change timestamps
  • Implement the envelope and logs purge APIs
  • xbusd:
    • add ‘–gnatsd-http[s]-(host/port)’ flags for enabling gnatsd monitoring
    • cert generate: add a ‘–expire’ flag for certificate validity range
    • account accept: add a ‘–cert-expire’ flag for the account cert validity range
  • xbusctl:
    • ‘actor reject’ now accepts the same arguments as ‘actor accept’
    • Add the ‘ps export’ and ‘ps purge’ commands
    • Use the new process filtering API to improve the cli
    • Add the ‘envelope purge’ command
    • Add the ‘logs purge’ command
    • account accept: add a ‘–cert-expire’ flag for the account cert validity range

Database

  • Table process_state: - Add field status_changed - Add field nodes_state_changed - Add field pm_changed
  • Tables emission, process_log_entry, process_state, route: Add ON DELETE CASCADE to *_process_id_fkey foreign keys
  • Table route_target: Add ON DELETE CASCADE to route_target_route_id_fkey

3.0.0-beta.8 (2017-06-03)

  • Add a postmortem API, service and CLI. They allow to inspect and handle failed processes.
  • Add database connection limit parameter: database.max-connections
  • Add APIs and command line tools (:prog:`xbusctl pm <xbusctl>`) for handling processes with errors or warning after they ended.
  • Add full environment tests, allowing some tests not possible with fullstack_tests, like testing the cli tools, testing random server crashes…
  • Update the graph API implementation, following go-xbus/api changes
  • Make envelope reception more robuts (Issue #39). Some inconsistent states are not possible anymore.
  • Handle gateway authentication with api keys.
  • Fix all the deadlocks we found on heavy load.
  • graph validation: Now identify invalid emitter/consumer input/output, and sourcematch-less emitters.
  • xbusd:
    • ‘account accept/reject’ now takes id or name in addition to fingerprints
  • xbusctl:
    • Add ‘actor delete’
    • Fix ‘actor list’ filter handling
    • ‘actor accept/reject’ now takes a name or id
    • Add ‘account delete’
    • Refactor ‘graph’ commands. The graph source file is now never updated by the command, and the name, version and status are updated by command line actions and arguments, not in the source file.
  • [doc] Document the “processing end” message in the connector API doc.
  • [doc] Fix minor issues in the connector API doc.

Database

  • Table process_state: - Add field pm_level varchar(255) - Add field pm_status varchar(255) - Add field pm_comment varchar(255)

Notice

  • A backwards incompatible change has been introduced in the configuration: database becomes database.dsn, the cli flag database becomes database-dns. Refer to the sample file etc/xbusd_sample.yaml.

3.0.0-beta.7 (2017-03-09)

  • Add support for fragmented envelopes, which implies a lot of code cleaning and a few refactoring.
  • Add a route cache
  • Reduce broker/emission coupling

Database

  • Now requires Postgresql >= 9.5
  • New Table: route
  • New Table: route_target

3.0.0-beta.6 (2017-02-03)

  • [doc] Update Quickstart to match latest go-xbus demo.relay settings
  • [auth] Remove debug log when a client is authenticated
  • Switch to go-xbus 3.0.0-beta.6
  • Fix a host / port parsing issue that was preventing the nats-host setting from being set to 0.0.0.0.

3.0.0-beta.5 (2017-01-26)

  • Switch to go-xbus 3.0.0-beta.5
  • Fix: graph validation error message for invalid edges were wrong (‘origin’ was switched with ‘destination’.
  • Doc: Add a fully fonctionnal graph to the Quickstart section, making it a complete and running example of a bus.

API

  • New: Graph nodes how have a actors attribute, which references actors by name or id. The actorids attribute is deprecated.
  • Fix: In case no graph matches a new envelope, a ack error is returned to the emitter.

Database

  • New: Table graph_node: add field actors varchar(255)

CLI

  • xbusd: Allow custom hostdefs for server CA
  • xbus-client: Embed demo actors from go-xbus/demo

3.0.0-beta.4 (2017-01-20)

  • Build releases binaries from the right changeset.
  • Extract reusable parts of this program to go-xbus.
  • Switch to go-xbus 3.0.0-beta.4

3.0.0-beta.3 (2017-01-13)

API

  • API Break: Add the “ActorProcessingEnd” entry point. It must be called by consumers to signal processing success or failure, and by other actors for processing errors. It adds support for long-running processing by consumers without hitting transfert timeout.
  • API Beak: Rename ActorType to ActorKind, and all ‘Type’ fields of type ActorType to ‘Kind’.
  • API Addition: Add a logging API for actors.

Database

  • Provides a SQL migration script, that does: - actor table: Drop unique index (name / type). - actor table: Add a unique index (name). - actor table: Rename type to kind. - emission table: event_positions type changed to TEXT (was VARCHAR(255)) - New log_entry table that stores log messages.

Other

  • Gnatsd: switch to upstream version (which merged our TLS auth related patch)
  • Better test coverage
  • More graph examples in the graph doc
  • Graph validation: Check for invalid or duplicate node ids
  • xbus-client: Better behavior when configuration file is missing
  • make xbus-client/cmd extendable
  • client: Add apis to easily start a group of actors from a configuration file

3.0.0-beta.2 (2016-12-22)

  • Graph validation. When pushing a graph, a list of validation errors (if any) will be returned and displayed. A non-valid graph cannot be published, preventing many common mistakes that would be otherwise hard to detect.
  • Optionnally capture SQL logs (see xbusd serve options –debug-sql-queries and –debug-sql-bindings)
  • Embed detailed version information in the binaries (try –version and –full-version flags on the binaries)
  • Upgrade gnatsd embedded version
  • Build static binaries
  • Embed documentation in the distribution package
  • Documentation fixes
  • A few logging level changes

3.0.0-beta.1 (2016-12-12)

Initial release. It features:

  • Handling of simple envelopes
  • Graphs
  • Process
  • Account/actors management
  • …