Fossil

JSON API: General Conventions
Login

JSON API: General Conventions

(⬑JSON API Index)

Jump to:

JSON Property Naming

Since most JSON usage conventionally happens in JavaScript environments, this API has (without an explicit decision ever having been made) adopted the ubiquitous JavaScript convention of camelCaseWithStartingLower for naming properties in JSON objects.

HTTP GET Requests

Many (if not most) requests can be made via simple GET requests, e.g. we could use any of the following patterns for a hypothetical JSON-format timeline:

The API settled on the /json/... convention, primarily because it simplifies dispatching and argument-handling logic compared to the /[.../]foo.json approach. Using /json/... allows us to unify that logic for all JSON sub-commands, for both CLI and HTTP modes.

HTTP Post Requests

Certain requests, mainly things like editing checkin messages and committing new files entirely, require POST data. This is fundamentally very simple to do - clients post plain/unencoded JSON using a common wrapper envelope which contains the request-specific data to submit as well as some request-independent information (like authentication data).

POST Request Envelope

POST requests are sent to the same URL as their GET counterpart (if any, else their own path), and are sent as plain-text/unencoded JSON wrapped in a common request envelope with the following properties:

The API allows most of those (normally all but the payload) to come in as either GET parameters or properties of the top-level POSTed request JSON envelope, with GET taking priority over POST. (Reminder to self: we could potentially also use values from cookies. Fossil currently only uses 1 cookie (the login token), and i'd prefer to keep it that way.)

POST requests without such an envelope will be rejected, generating a Fossil/JSON error response (as opposed to an HTTP error response). GET requests, by definition, never have an envelope.

POSTed client requests must send a Content-Type header of either application/json , application/javascript, or text/plain, or the JSON-handling code will never see the POST data. The POST handler optimistically assumes that type text/plain "might be JSON", since application/json is a newer convention which many existing clients do not use (as of the time these docs were written, back in 2011).

POST Envelope vs. POST.payload

When this document refers to POST data, it is referring to top-level object in JSON-format POSTed input data. When we say POST.payload we refer to the "payload" property of the POST data. While fossil's core handles form-urlencoded POST data, if such data is sent in then parsing it as JSON cannot succeed, which means that (at worst) the JSON-mode bits will "not see" any POST data. Data POSTed to the JSON API must be sent non-form-urlencoded (i.e. as plain text).

Framework-level configuration options are always set via the top-level POST envelope object or GET parameters. Request-specific options are set either in POST.payload or GET parameters (though the former is required in some cases). Here is an example which demonstrates the possibly not-so-obvious difference between the two types of options (framework vs. request-specific):

{
"requestId":"my request", // standard envelope property (optional)
"command": "timeline/wiki", // also standard
"indent":2, // output indention is a framework-level option
"payload":{ // *anything* in the payload is request-specific
   "limit":1
}
}

When a given parameter is set in two places, e.g. GET and POST, or POST-from-a-file and CLI parameters, which one takes precedence depends on the concrete command handler (and may be unspecified). Most will give precedence to CLI and GET parameters, but POSTed values are technically preferred for non-string data because no additional "type guessing" or string-to-whatever conversion has to be made (GET/CLI parameters are always strings, even if they look like a number or boolean).

Request Parameter Data Types

When parameters are sent in the form of GET or CLI arguments, they are inherently strings. When they come in from JSON they keep their full type (boolean, number, etc.). All parameters in this API specify what type (or types) they must (or may) be. For strings, there is no internal conversion/interpretation needed for GET- or CLI-provided parameters, but for other types we sometimes have to convert strings to other atomic types. This section describes how those string-to-whatever conversions behave.

No higher-level constructs, e.g. JSON arrays or objects, are accepted in string form. Such parameters must be set in the POST envelope or payload, as specified by the specific API.

This API does not currently use any floating-point parameters, but does return floating-point results in a couple places.

For integer parameters we use a conventional string-to-int algorithm and assume base 10 (analog to atoi(3)). The API may err on the side of usability when given technically invalid values. e.g. "123abc" will likely be interpreted as the integer 123. No APIs currently rely on integer parameters with more than 32 bits (signedness is call-dependent but few, if any, use negative values).

Boolean parameters are a bit schizophrenic...

In CLI mode, boolean flags do not have a value, per se, and thus require no string-to-bool conversion. e.g. fossil foo -aBoolOpt -non-bool-opt value.

Those which arrive as strings via GET parameters treat any of the following as true: a string starting with a character in the set [1-9tT]. All other string values are considered to be false for this purpose.

Those which are part of the POST data are normally (but not always - it depends on the exact context) evaluated as the equivalent of JavaScript booleans. e.g. if we have POST.envelope.foo="f", and evaluate it as a JSON boolean (as opposed to a string-to-bool conversion), the result will be true because the underlying JSON API follows JavaScript semantics for any-type-to-bool conversions. As long as clients always send "proper" booleans in their POST data, the difference between GET/CLI-provided booleans should never concern them.

TODO: consider changing the GET-value-to-bool semantics to match the JS semantics, for consistency (within the JSON API at least, but that might cause inconsistencies vis-a-vis the HTML interface).

Response Envelope

Every response comes in the form of a HTTP response or (in CLI mode) JSON sent to stdout. The body of the response is a JSON object following a common envelope format. The envelope has the following properties:

HTTP Response Headers

The Content-Type HTTP header of a response will be either application/json, application/javascript, or text/plain, depending on whether or not we are in JSONP mode or (failing that) the contents of the "Accept" header sent in the request. The response type will be text/plain if it cannot figure out what to do. The response's Content-Type header may contain additional metadata, e.g. it might look like: application/json; charset=utf-8

Apropos UTF-8: note that JSON is, by definition, Unicode and recommends UTF-8 encoding (which is what we use). That means if your console cannot handle UTF-8 then using this API in CLI mode might (depending on the content) render garbage on your screen.

CLI vs. HTTP Mode

CLI (command-line interface) and HTTP modes (CGI and standalone server) are consolidated in the same implementations and behave essentially identically, with only minor exceptions.

An HTTP path of /json/foo translates to the CLI command fossil json foo. CLI mode takes options in the fossil-convention forms (e.g. --foo 3 or -f 3) whereas HTTP mode takes them via GET/POST data (e.g. ?foo=1). (Note that per long-standing fossil convention CLI parameters taking a value do not use an equal sign before the value!)

For example:

Some commands may only work in one mode or the other (for various reasons). In CLI mode the user automatically has full setup/admin access.

In HTTP mode, request-specific options can also be specified in the POST.payload data, and doing so actually has an advantage over specifying them as URL parameters: posting JSON data retains the full type information of the values, whereas GET-style parameters are always strings and must be explicitly type-checked/converted (which may produce unpredictable results when given invalid input). That said, oftentimes it is more convenient to pass the options via URL parameters, rather than generate the request envelope and payload required by POST requests, and the JSON API makes some extra effort to treat GET-style parameters type-equivalent to their POST counterparts. If a property appears in both GET and POST.payload, GET-style parameters typically take precedence over POST.payload by long-standing convention (=="PHP does it this way by default").

(That is, however, subject to eventual reversal because of the stronger type safety provided by POSTed JSON. Philosophically speaking, though, GET should take precedence, in the same way that CLI-provided options conventionally override app-configuration-level options.)

One notable functional difference between CLI and HTTP modes is that in CLI mode error responses might be accompanied by a non-0 exit status (they "should" always be, but there might be cases where that does not yet happen) whereas in HTTP mode we always try to exit with code 0 to avoid generating an HTTP 500 ("internal server error"), which could keep the JSON response from being delivered. The JSON code only intentionally allows an HTTP 500 when there is a serious internal error like allocation or assertion failure. HTTP clients are expected to catch errors by evaluating the response object, not the HTTP result code.

Simulating POSTed data

We have a mechanism to feed request data to CLI mode via files (simulating POSTed data), as demonstrated in this example:

$ cat in.json
{ "command": "timeline/wiki", "indent":2, "payload":{"limit":1}}
$ fossil json --json-input in.json # use filename - for stdin

The above is equivalent to:

$ echo '{"indent":2, "payload":{"limit":1}}' \
 | fossil json timeline wiki --json-input -

Note that the "command" JSON parameter is only checked when no json subcommand is provided on the CLI or via the HTTP request path. Thus we cannot pass the CLI args "json timeline" in conjunction with a "command" string of "wiki" this way.

HOWEVER...

Much of the existing JSON code was written before the --json-input option was possible. Because of this, there might be some "misinteractions" when providing request-specific options via both CLI options and simulated POST data. Those cases will eventually be ironed out (with CLI options taking precedence). Until then, when "POSTing" data in CLI mode, for consistent/predictible results always provide any options via the JSON request data, not CLI arguments. That said, there "should not" be any blatant incompatibilities, but some routines will prefer POST.payload over CLI/GET arguments, so there are some minor inconsistencies across various commands with regards to which source (POST/GET/CLI) takes precedence for a given option. The precedence "should always be the same," but currently cannot be due to core fossil implementation details (the internal consolidation of GET/CLI/POST vars into a single set).

Indentation/Formatting of JSON Output

CLI mode accepts the --indent|-I # option to set the indention level and HTTP mode accepts indent=# as a GET/POST parameter. The semantics of the indention level are derived from the underlying JSON library and have the following meanings: 0 (zero) or less disables all superfluous indentation (this is the default in HTTP mode). A value of 1 uses 1 hard TAB character (ASCII 0x09) per level of indention (the default in CLI mode). Values greater than 1 use that many whitespaces (ASCII 32d) per level of indention. e.g. a value of 7 uses 7 spaces per level of indention. There is no way to specify one whitespace per level, but if you really want one whitespace instead of one tab (same data size) you can filter the output to globally replace ASCII 9dec (TAB) with ASCII 32dec (space). Because JSON string values never contain hard tabs (they are represented by \t) there is no chance that such a global replacement will corrupt JSON string contents - only the formatting will be affected.

Potential TODO: because extraneous indention "could potentially" be used as a form DoS, the option might be subject to later removed from HTTP mode (in CLI it's fine).

In HTTP mode no trailing newline is added to the output, whereas in CLI mode one is normally appended (exception: in JSONP mode no newline is appended, to (rather pedantically and arbitraily) allow the client to add a semicolon at the end if he likes). There is currently no option to control the newline behaviour, but the underlying JSON code supports this option, so adding it to this API is just a matter of adding the CLI/HTTP args for it.

Pedantic note: internally the indention level is stored as a single byte, so giving large indention values will cause harmless numeric overflow (with only cosmetic effects), meaning, e.g., 257 will overflow to the value 1.

Potential TODO: consider changing cson's indention mechanism to use a signed number, using negative values for tabs and positive for whitespace count (or the other way around). This would require more doc changes than code changes :/.

JSONP

The API supports JSONP-style output. The caller specifies the callback name and the JSON response will be wrapped in a function call to that name. For HTTP mode pass the jsonp=string option (via GET or POST envelope) and for CLI use --jsonp string.

For example, if we pass the JSONP name myCallback then a response will look like:

myCallback({...response...})

Note that fossil does not evaluate the callback name itself, other than to verify that it is-a string, so "garbage in, garbage out," and all that. (Remember that CLI and GET parameters are always strings, even if they look like numbers.)

API Result Codes

Result codes are strings which tell the client whether or not a given API call succeeded or failed, and if it failed perhaps some hint as to why it failed.

The result code is available via the resultCode property of every error response envelope. Since having a result code value for success responses is somewhat redundant, success responses contain no resultCode property. In practice this simplifies error checking on the client side.

The codes are strings in the form FOSSIL-####, where #### is a 4-digit integral number, left-padded with zeros. The numbers follow these conventions:

The tentative list of result codes is shown in the following table. These numbers/ranges are "nearly arbitrarily" chosen except for the "special" value 0000.

Maintenance reminder: these codes are defined in src/json_detail.h (enum FossilJsonCodes) and assigned default resultText values in src/json.c:json_err_cstr(). Changes there need to be reflected here (and vice versa). Also, we have assertions in place to ensure that C-side codes are in the range 1000-9999, so do not just go blindly change the numeric ranges used by the enum.

FOSSIL-0###: Non-error Category

FOSSIL-1000: Generic Errors Category

FOSSIL-2000: Authentication/Access Error Category

ONLY FOR TESTING purposes should the remaning 210X sub-codes be enabled (they are potentially security-relevant, in that the client knows which part of the request was valid/invalid):

FOSSIL-3000: Usage Error Category

FOSSIL-4000: Database-related Error Category

Special-case DB-related errors...

Some of those error codes are of course "too detailed" for the client to do anything with (e.g.. 4001-4004), but their intention is to make it easier for Fossil developers to (A) track down problems and (B) support clients who report problems. If a client reports, "I get a FOSSIL-4000, how can I fix it?" then the developers/support personnel can't say much unless they know if it's a 4001, 4002, 4003, 4004, or 4101 (in which case they can probably zero in on the problem fairly quickly, since they know which API call triggered it and they know (from the error code) the general source of the problem).

Why Standard/Immutable Result Codes are Important