Fossil

<h1>Installing Fossil</h1>

<p>This page describes how to build and install Fossil.  The
whole process is designed to be very easy.</p>

<h2>0.0 Using A Precompiled Binary</h2>

<p>You can skip steps 1.0 and 2.0 below by downloading
a <a href="http://www.fossil-scm.org/download.html">precompiled binary</a>
appropriate for your platform.  If you use a precompiled binary
jump immediate to step 3.0.</p>

<h2>1.0 Obtaining The Source Code</h2>

<p>Fossil is self-hosting, so you can obtain a ZIP archive containing
a snapshot of the latest version directly from fossil's own fossil

<p>In the next section, when we say things like "use the <b>help</b>
command" we mean to use the command name "help" as the first
token after the name of the fossil executable, as shown above.</p>

<h2>4.0 Workflow</h2>

<img src="concept2.gif" align="right" hspace="10">

<ol>
<li><p>
Establish a local repository using either the <b>new</b> command
to start a new project, or the <b>clone</b> command to make a clone
of a repository for an existing project. 
</p></li>

<nowiki>
<h1 align="center">
Fossil Delta Encoding Algorithm
</h1>

<p>A key component for the efficient storage of multiple revisions of
a file in fossil repositories is the use of delta-compression, i.e. to
store only the changes between revisions instead of the whole
file.</p>

<p>This document describes the encoding algorithm used by Fossil to
generate deltas. It is targeted at developers working on either
<a href="index.wiki">fossil</a> itself, or on tools compatible with
it. The exact format of the generated byte-sequences, while in general
not necessary to understand encoder operation, can be found in the
companion specification titled "<a href="delta_format.wiki">Fossil
Delta Format</a>".
</p>

<p>The entire algorithm is inspired
by <a href="http://samba.anu.edu.au/rsync/">rsync</a>.</p>

<a name="argresparam"></a><h2>1.0 Arguments, Results, and Parameters</h2>

<p>The encoder takes two byte-sequences as input, the "original", and
the "target", and returns a single byte-sequence containing the
"delta" which transforms the original into the target upon its
application.</p>

<p>Note that the data of a "byte-sequence" includes its length,
i.e. the number of bytes contained in the sequence.</p>

<p>The algorithm has one parameter named "NHASH", the size of the
"sliding window" for the "rolling hash", in bytes. These two terms are
explained in the next section. The value of this parameter has to be a
power of two for the algorithm to work. For Fossil the value of this
parameter is set to "16".</p>

<a name="operation"></a><h2>2.0 Operation</h2>

<p>The algorithm is split into three phases which generate
the <a href="delta_format.wiki#header">header</a>,
<a href="delta_format.wiki#slist">segment list</a>,
and <a href="delta_format.wiki#trailer">trailer</a> of the delta, per
its general <a href="delta_format.wiki#structure">structure</a>.</p>

<p>The two phases generating header and trailer are not covered here
as their implementation trivially follows directly from the
specification of the <a href="delta_format.wiki">delta format</a>.</p>

<p>This leaves the segment-list. Its generation is done in two phases,
a pre-processing step operating on the "original" byte-sequence,
followed by the processing of the "target" byte-sequence using the
information gathered by the first step.</p>

<a name="preprocessing"></a><h3>2.1 Preprocessing the original</h3>

<p>A major part of the processing of the "target" is to find a range
in the "original" which contains the same content as found at the
current location in the "target".</p>

<p>A naive approach to this would be to search the whole "original"
for such content. This however is very inefficient as it would search

computed.
</li>
<li>A hashtable is filled, mapping from the hashes of the chunks to
the list of chunk locations having this hash.
</li>
</ol>

<a name="processing"></a><h3>2.1 Processing the target</h3>

<p>This, the main phase of the encoder, processes the target in a loop
from beginning to end. The state of the encoder is captured by two
locations, the "base" and the "slide". "base" points to the first byte
of the target for which no delta output has been generated yet, and
"slide" is the location of the window used to look in the "origin" for
commonalities. This window is NHASH bytes long.</p>

<p>Initially both "base" and "slide" point to the beginning of the
"target". In each iteration of the loop the encoder decides whether to
<ul>
<li>emit a single instruction
to <a href="delta_format.wiki#copyrange">copy a range</a>, or
</li>
<li>emit two instructions, first
to <a href="delta_format.wiki#insertlit">insert a literal</a>, then
to <a href="delta_format.wiki#copyrange">copy a range</a>, or
</li>
<li>move the window forward one byte.
</li>
</ul>
</p>

<img src="encode10.gif" align="right" hspace="10">

</p>

<p>If the processing loop left bytes unencoded, i.e. "base" not
exactly at the end of the "target", as is possible for both end
conditions, then one last insert instruction is emitted to put these
bytes into the delta.<p>

<a name="exceptions"></a><h2>3.0 Exceptions</h2>

<p>If the "original" is at most NHASH bytes long no compression of
changes is possible, and the segment-list of the delta consists of a
single literal which contains the entire "target".</p>

<p>This is actually equivalent to the second end condition of the
processing loop described in the previous section, just checked before
actually entering the loop.</p>

<a name="rollhash"></a><h2>4.0 The rolling hash</h2>

<p>The rolling hash described below and used to compute content
signatures was chosen not only for good hashing properties, but also
to enable the easy (incremental) recalculation of its value for a
sliding window, i.e. where the oldest byte is removed from the window
and a new byte is shifted in.<p>

<a name="rhdef"></a><h3>4.1 Definition</h3>

<p>Assuming an array Z of NHASH bytes (indexing starting at 0) the
hash V is computed via</p>

<p align=center><table><tr><td>
<p><img src="encode1.gif" align="center"></p>
<p><img src="encode2.gif" align="center"></p>
<p><img src="encode3.gif" align="center"></p>
</td></tr></table></p>

where A and B are unsigned 16-bit integers (hence the <u>mod</u>), and
V is a 32-bit unsigned integer with B as MSB, A as LSB.

<a name="rhincr"></a><h3>4.2 Incremental recalculation</h3>

<p>Assuming an array Z of NHASH bytes (indexing starting at 0) with
hash V (and components A and B), the dropped
byte <img src="encode4.gif" align="center">, and the new byte
<img src="encode5.gif" align="center"> , the new hash can
be computed incrementally via: </p>

<nowiki>
<h1 align="center">
Fossil Delta Format
</h1>

<p>Fossil achieves efficient storage and low-bandwidth synchronization
through the use of delta-compression.  Instead of storing
or transmitting the complete content of an artifact, fossil stores or
transmits only the changes relative to a related artifact.
</p>

<p>This document describes the delta-encoding format used by fossil.
The intended audience is developers working on either
<a href="index.wiki">fossil</a> itself, or on tools compatible with
fossil.</p>

<p>Note that the delta-encoding is not a fundamental element of the
state of a fossil repository.  A state of a fossil repository is
defined by the uncompressed and undeltaed content of all artifacts.
The fact the artifacts
are stored on disk using this delta-encoding format is merely an
optimization.  One could, in theory, create an entirely new and
compatible implementation of fossil that used a different delta-encoding
or did no delta-encoding at all.  However, experience has shown that
the delta-encoding described here is both efficient to compute and
results in very small deltas, so its continued use is recommended.</p>

<a name="structure"></a><h2>1.0 Structure</h2>
<img src="delta1.gif" align="left" hspace="10">

<p>A delta consists of three parts, a "header", a "trailer", and a
"segment-list" between them.</p>

<p>Both header and trailer provide information about the target
helping the decoder, and the segment-list describes how the target can
be constructed from the original.</p>

<a name="header"></a><h3>1.1 Header</h3>
<img src="delta6.gif" align="left" hspace="10">

<p>The header consists of a single number followed by a newline
character (ASCII 0x0a). The number is the length of the target in
bytes.</p>

<p>This means that, given a delta, the decoder can compute the size of
the target (and allocate any necessary memory based on that) by simply
reading the first line of the delta and decoding the number found
there. In other words, before it has to decode everything else.</p>

<a name="trailer"></a><h3>1.2 Trailer</h3>
<img src="delta5.gif" align="left" hspace="10">

<p>The trailer consists of a single number followed by a semicolon (ASCII
0x3b). This number is a checksum of the target and can be used by a
decoder to verify that the delta applied correctly, reconstructing the
target from the original.</p>

<p>The checksum is computed by treating the target as a series of
32-bit integer numbers (MSB first), and summing these up, modulo
2^32-1. A target whose length is not a multiple of 4 is padded with
0-bytes (ASCII 0x00) at the end.</p>

<p>By putting this information at the end of the delta a decoder has
it available immediately after the target has been reconstructed
fully.</p>

<a name="slist"></a><h3>1.3 Segment-List</h3>
<img src="delta2.gif" align="left" hspace="10">

<p>The segment-list of a delta describes how to create the target from
the original by a combination of inserting literal byte-sequences and
copying ranges of bytes from the original. This is there the
compression takes place, by encoding the large common parts of
original and target in small copy instructions.</p>

<p>The target is constructed from beginning to end, with the data
generated by each instruction appended after the data of all previous
instructions, with no gaps.</p>

<a name="insertlit"></a><h4>1.3.1 Insert Literal</h4>

<p>A literal is specified by two elements, the size of the literal in
bytes, and the bytes of the literal itself.</p>

<img src="delta4.gif" align="left" hspace="10">
<p>The length is written first, followed by a colon character (ASCII
0x3a), followed by the bytes of the literal.</p>

<a name="copyrange"></a><h4>1.3.2 Copy Range</h4>

<p>A range to copy is specified by two numbers, the offset of the
first byte in the original to copy, and the size of the range, in
bytes. The size zero is special, its usage indicates that the range
extends to the end of the original.</p>

<img src="delta3.gif" align="left" hspace="10">
<p>The length is written first, followed by an "at" character (ASCII
0x40), then the offset, followed by a comma (ASCII 0x2c).</p>

<a name="intcoding"></a><h2>2.0 Encoding of integers</h2>

<p>
The format currently handles only 32 bit integer numbers. They are
written base-64 encoded, MSB first, and without leading
"0"-characters, except if they are significant (i.e. 0 => "0").
</p>

<p>
The base-64 coding is described in
<a href="http://www.ietf.org/rfc/rfc3548.txt">RFC 3548</a>.
</p>

<a name="examples"></a><h2>3.0 Examples</h2>

<a name="examplesint"></a><h3>3.1 Integer encoding</h3>

<table border=1>
<tr>
<th>Value</th>
<th>Encoding</th>
</tr>
<tr>
<td>0</td>
<td>0</td>
</tr>
<tr>
<td>6246</td>
<td>1Xb</td>
</tr>
<tr>
<td>-1101438770</td>
<td>2zMM3E</td>
</tr>
</table>

<a name="examplesdelta"></a><h3>3.2 Delta encoding</h3>

<p>An example of a delta using the specified encoding is:</p>

<table border=1><tr><td><pre>
1Xb
4E@0,2:thFN@4C,6:scenda1B@Jd,6:scenda5x@Kt,6:pieces79@Qt,F: Example: eskil~E@Y0,2zMM3E;</pre>
</td></tr></table>

 
  *  Ticketing interface (expand this bullet)

</pre></td></tr></table>



<a name="notes"></a><h2>Notes</h2>

<ul>
<li>Pure text files generate a pure text delta.
</li>
<li>Binary files generate a delta that may contain some binary data.
</li>

<li>The delta encoding does not attempt to compress the content
It was considered to be much
more sensible to do compression using a separate general-purpose
compression library, like <a href="http://www.zlib.net">zlib</a>.

</li>
</ul>

shared.</p></li>

<li><p>Every baseline has a special file at the top-level
named "manifest" which is an index of all other files in
that baseline.  The manifest is automatically created and
maintained by the system.</p></li>

<li><p>The <a href="fileformat.wiki">file formats</a>
used by Fossil are all very simple so that with access
to the original content files, one can easily reconstruct
the content of a baseline without the need for any
special tools or software.</p></li>

<nowiki>
<h1 align="center">Fossil Quick Start</h1>

<p>This is a guide to get you started using fossil quickly
and painlessly.</p>

<h2>Installing</h2><blockquote>

    <p>Fossil is a single self-contained C program.  You need to
    either download a 
    <a href="http://www.fossil-scm.org/download.html">precompiled binary</a>
    or <a href="build.wiki">build it yourself</a> from sources.
    Install fossil by putting the fossil binary
    someplace on your PATH environment variable.</p>
    
    </blockquote>
    <h2>Cloning A Existing Repository</h2>
    <blockquote>

    fossil test-commands
    </b></blockquote>

    <p>Explore and have fun!</p>


</blockquote>

<nowiki>
<h1 align="center">
Fossil Repository Integrity Self-Checks
</h1>

<p>
Even though fossil is a relatively new project and still contains
many bugs, it is designed with features to give it a high level
of integrity so that you can have confidence that you will not
lose your files.  This note describes the defensive measures that
fossil uses to help prevent file loss due to bugs.
</p>

<p><i>Follow-up as of 2007-11-24:</i>
<i>Reiterated on 2008-05-16:</i>
Fossil has been hosting itself and many other projects for
months now.  Many bugs have been encountered.  But, thanks in large
part to the defensive measures described here, no data has been
lost.  The integrity checks are doing their job well.</p>

<h2>Atomic Check-ins With Rollback</h2>

<p>
The fossil repository is an
<a href="http://www.sqlite.org/">SQLite version 3</a> database file.  
SQLite is very mature and stable and has been in wide-spread use for many
years, so we are confident it will not cause repository
corruption.  SQLite
databases do not corrupt even if a program or system crash or power
failure occurs in the middle of the update.  If some kind of crash
does occur in the middle of a change, then all the changes are rolled
back the next time that the database is accessed.
</p>