Fossil

Check-in [3dcf69a7]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Bring the sync protocol documentation more up-to-date. Fix typos in the command-line documentation for "push" and "sync".
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: 3dcf69a7410b4066bc555bbb343a39c97b58158d
User & Date: drh 2016-08-04 00:42:20
Context
2016-08-04
11:34
Fix a typo in the sync protocol document. check-in: 12cbe2bd user: drh tags: trunk
00:42
Bring the sync protocol documentation more up-to-date. Fix typos in the command-line documentation for "push" and "sync". check-in: 3dcf69a7 user: drh tags: trunk
2016-08-03
14:04
Fix a minor preprocessor directive typo. check-in: a5c08fde user: jan tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to src/sync.c.

231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
...
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
** Options:
**
**   -B|--httpauth USER:PASS    Credentials for the simple HTTP auth protocol,
**                              if required by the remote website
**   --ipv4                     Use only IPv4, not IPv6
**   --once                     Do not remember URL for subsequent syncs
**   --proxy PROXY              Use the specified HTTP proxy
**   --private                  Pull private branches too
**   -R|--repository REPO       Repository to pull into
**   --ssl-identity FILE        Local SSL credentials, if requested by remote
**   --ssh-command SSH          Use SSH as the "ssh" command
**   -v|--verbose               Additional (debugging) output
**   --verily                   Exchange extra information with the remote
**                              to ensure no content is overlooked
**
................................................................................
** Options:
**
**   -B|--httpauth USER:PASS    Credentials for the simple HTTP auth protocol,
**                              if required by the remote website
**   --ipv4                     Use only IPv4, not IPv6
**   --once                     Do not remember URL for subsequent syncs
**   --proxy PROXY              Use the specified HTTP proxy
**   --private                  Pull private branches too
**   -R|--repository REPO       Repository to pull into
**   --ssl-identity FILE        Local SSL credentials, if requested by remote
**   --ssh-command SSH          Use SSH as the "ssh" command
**   -v|--verbose               Additional (debugging) output
**   --verily                   Exchange extra information with the remote
**                              to ensure no content is overlooked
**







|







 







|







231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
...
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
** Options:
**
**   -B|--httpauth USER:PASS    Credentials for the simple HTTP auth protocol,
**                              if required by the remote website
**   --ipv4                     Use only IPv4, not IPv6
**   --once                     Do not remember URL for subsequent syncs
**   --proxy PROXY              Use the specified HTTP proxy
**   --private                  Push private branches too
**   -R|--repository REPO       Repository to pull into
**   --ssl-identity FILE        Local SSL credentials, if requested by remote
**   --ssh-command SSH          Use SSH as the "ssh" command
**   -v|--verbose               Additional (debugging) output
**   --verily                   Exchange extra information with the remote
**                              to ensure no content is overlooked
**
................................................................................
** Options:
**
**   -B|--httpauth USER:PASS    Credentials for the simple HTTP auth protocol,
**                              if required by the remote website
**   --ipv4                     Use only IPv4, not IPv6
**   --once                     Do not remember URL for subsequent syncs
**   --proxy PROXY              Use the specified HTTP proxy
**   --private                  Sync private branches too
**   -R|--repository REPO       Repository to pull into
**   --ssl-identity FILE        Local SSL credentials, if requested by remote
**   --ssh-command SSH          Use SSH as the "ssh" command
**   -v|--verbose               Additional (debugging) output
**   --verily                   Exchange extra information with the remote
**                              to ensure no content is overlooked
**

Changes to www/server.wiki.

4
5
6
7
8
9
10
11
12



13
14
15
16
17
18
19
peers.  A Fossil server also works well as a complete website for a project.
For example, the complete [https://www.fossil-scm.org/] website, including the
page you are now reading (but excepting the 
[https://www.fossil-scm.org/download.html|download page]),
is just a Fossil server displaying the content of the 
self-hosting repository for Fossil.</p>
<p>This article is a guide for setting up your own Fossil server.
<p>See the [./aboutcgi.wiki|How CGI Works In Fossil] document for background
information on the underlying technology.</p></blockquote>



<h2>Overview</h2><blockquote>
There are basically four ways to set up a Fossil server:
<ol>
<li>A stand-alone server
<li>Using inetd or xinetd or stunnel
<li>CGI
<li>SCGI (a.k.a. SimpleCGI)







|
|
>
>
>







4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
peers.  A Fossil server also works well as a complete website for a project.
For example, the complete [https://www.fossil-scm.org/] website, including the
page you are now reading (but excepting the 
[https://www.fossil-scm.org/download.html|download page]),
is just a Fossil server displaying the content of the 
self-hosting repository for Fossil.</p>
<p>This article is a guide for setting up your own Fossil server.
<p>See "[./aboutcgi.wiki|How CGI Works In Fossil]" for background
information on the underlying CGI technology.
See "[./sync.wiki|The Fossil Sync Protocol]" for information on the
wire protocol used for client/server communication.</p>
</blockquote>
<h2>Overview</h2><blockquote>
There are basically four ways to set up a Fossil server:
<ol>
<li>A stand-alone server
<li>Using inetd or xinetd or stunnel
<li>CGI
<li>SCGI (a.k.a. SimpleCGI)

Changes to www/sync.wiki.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
..
24
25
26
27
28
29
30
31
32
33
34
35

36
37
38
39
40
41
42
43
44
45
46
47
48


49
50
51
52
53
54
55
56
57









58
59
60
61
62
63
64
...
231
232
233
234
235
236
237
















238
239
240
241
242
243
244
...
253
254
255
256
257
258
259


260
261
262
263
264
265
266
...
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333




334
335
336
337
338
339
340
...
475
476
477
478
479
480
481







































482
483
484
485
486
487
488
489
...
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
...
663
664
665
666
667
668
669

670
671



672
673
674
675
676
677
678

679
680
681
682
683
684
685
686
687
<title>The Fossil Sync Protocol</title>

<p>Fossil supports commands <b>push</b>, <b>pull</b>, and <b>sync</b>
for transferring information from one repository to another.  The
command is run on the client repository.  A URL for the server repository
is specified as part of the command.  This document describes what happens
behind the scenes in order to synchronize the information on the two
repositories.</p>

<h2>1.0 Overview</h2>

<p>The global state of a fossil repository consists of an unordered
collection of artifacts.  Each artifact is identified by its SHA1 hash
expressed as a 40-character lower-case hexadecimal string.
Synchronization is simply the process of sharing artifacts between
servers so that all servers have copies of all artifacts.  Because
artifacts are unordered, the order in which artifacts are received
at a server is inconsequential.  It is assumed that the SHA1 hashes
of artifacts are unique - that every artifact has a different SHA1 hash.
To a first approximation, synchronization proceeds by sharing lists
SHA1 hashes of available artifacts, then sharing those artifacts that
are not found on one side or the other of the connection.  In practice,
................................................................................
SHA1 hashes for this many artifacts can be large.  So optimizations are
employed that usually reduce the number of SHA1 hashes that need to be
shared to a few hundred.</p>

<p>Each repository also has local state.  The local state determines
the web-page formatting preferences, authorized users, ticket formats,
and similar information that varies from one repository to another.
The local state is not transferred by the <b>push</b>, <b>pull</b>,
and <b>sync</b> command, though some local state is transferred during
a <b>clone</b> in order to initialize the local state of the new
repository.  The <b>configuration push</b> and <b>configuration pull</b>
commands can be used to send or receive local state.</p>



<h2>2.0 Transport</h2>

<p>All communication between client and server is via HTTP requests.
The server is listening for incoming HTTP requests.  The client
issues one or more HTTP requests and receives replies for each
request.</p>

<p>The server might be running as an independent server
using the <b>server</b> command, or it might be launched from
inetd or xinetd using the <b>http</b> command.  Or the server might
be launched from CGI.  The details of how the server is configured


to "listen" for incoming HTTP requests is immaterial.  The important
point is that the server is listening for requests and the client
is the issuer of the requests.</p>

<p>A single push, pull, or sync might involve multiple HTTP requests.
The client maintains state between all requests.  But on the server
side, each request is independent.  The server does not preserve
any information about the client from one request to the next.</p>










<h3>2.1 Server Identification</h3>

<p>The server is identified by a URL argument that accompanies the
push, pull, or sync command on the client.  (As a convenience to
users, the URL can be omitted on the client command and the same URL
from the most recent push, pull, or sync will be reused.  This saves
typing in the common case where the client does multiple syncs to
................................................................................
delta and the second argument is the ID of another artifact that is the
source of the delta and the third argument is the original size of the
delta artifact.</p>

<p>Unlike file cards, cfile cards are only sent in one direction during a
clone from server to client for clone protocol version "3" or greater.</p>

















<h3>3.4 Push and Pull Cards</h3>

<p>Among the first cards in a client-to-server message are
the push and pull cards.  The push card tells the server that
the client is pushing content.  The pull card tells the server
that the client wants to pull content.  In the event of a sync,
both cards are sent.  The format is as follows:</p>
................................................................................
of the software project that the client repository contains.
The projectcode for the client and server must match in order
for the transaction to proceed.</p>

<p>The server will also send a push card back to the client
during a clone.  This is how the client determines what project
code to put in the new repository it is constructing.</p>



<h3>3.5 Clone Cards</h3>

<p>A clone card works like a pull card in that it is sent from
client to server in order to tell the server that the client
wants to pull content.  The clone card comes in two formats.  Older
clients use the no-argument format and newer clients use the
................................................................................
<h3>3.6 Igot Cards</h3>

<p>An igot card can be sent from either client to server or from
server to client in order to indicate that the sender holds a copy
of a particular artifact.  The format is:</p>

<blockquote>
<b>igot</b> <i>artifact-id</i>
</blockquote>

<p>The argument of the igot card is the ID of the artifact that
the sender possesses.
The receiver of an igot card will typically check to see if
it also holds the same artifact and if not it will request the artifact
using a gimme card in either the reply or in the next message.</p>





<h3>3.7 Gimme Cards</h3>

<p>A gimme card is sent from either client to server or from server
to client.  The gimme card asks the receiver to send a particular
artifact back to the sender.  The format of a gimme card is this:</p>

<blockquote>
................................................................................
<p>The server will only accept a config card if the user has
"Admin" privilege.  A client will only accept a config card if
it had sent a corresponding reqconfig card in its request.

<p>The content of the configuration item is used to overwrite the
corresponding configuration data in the receiver.








































<h3>3.11 Error Cards</h3>

<p>If the server discovers anything wrong with a request, it generates
an error card in its reply.  When the client sees the error card,
it displays an error message to the user and aborts the sync
operation.  An error card looks like this:</p>

<blockquote>
................................................................................
A space (ASCII 0x20) is represented as "\s" (ASCII 0x5C, 0x73).  A
newline (ASCII 0x0a) is "\n" (ASCII 0x6C, x6E).  A backslash
(ASCII 0x5C) is represented as two backslashes "\\".  Apart from
space and newline, no other whitespace characters nor any
unprintable characters are allowed in
the error message.</p>

<h3>3.12 Comment Cards</h3>

<p>Any card that begins with "#" (ASCII 0x23) is a comment card and
is silently ignored.</p>

<h3>3.13 Unknown Cards</h3>

<p>If either the client or the server sees a card that is not
described above, then it generates an error and aborts.</p>

<h2>4.0 Phantoms And Clusters</h2>

<p>When a repository knows that an artifact exists and knows the ID of
................................................................................
    card per line.
<li>Card formats are:
    <ul>
    <li> <b>login</b> <i>userid nonce signature</i>
    <li> <b>push</b> <i>servercode projectcode</i>
    <li> <b>pull</b> <i>servercode projectcode</i>
    <li> <b>clone</b>

    <li> <b>file</b> <i>artifact-id size</i> <b>\n</b> <i>content</i>
    <li> <b>file</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i>



    <li> <b>igot</b> <i>artifact-id</i>
    <li> <b>gimme</b> <i>artifact-id</i>
    <li> <b>cookie</b>  <i>cookie-text</i>
    <li> <b>reqconfig</b> <i>parameter-name</i>
    <li> <b>config</b> <i>parameter-name size</i> <b>\n</b> <i>content</i>
    <li> <b>#</b> <i>arbitrary-text...</i>
    <li> <b>error</b> <i>error-message</i>

    </ul>
<li>Phantoms are artifacts that a repository knows exist but does not possess.
<li>Clusters are artifacts that contain IDs of other artifacts.
<li>Clusters are created automatically on the server during a pull.
<li>Repositories keep track of all artifacts that are not named in any
cluster and send igot messages for those artifacts.
<li>Repositories keep track of all the phantoms they hold and send
gimme messages for those artifacts.
</ol>


|
<
<
<
<
|






|







 







|
|
|
|
|
>












|
>
>
|
|
|






>
>
>
>
>
>
>
>
>







 







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







 







>
>







 







|


|





>
>
>
>







 







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
|







 







<
<
<
<
<
|







 







>


>
>
>
|




|

>









1
2
3




4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
..
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
...
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
...
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
...
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
...
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
...
564
565
566
567
568
569
570





571
572
573
574
575
576
577
578
...
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
<title>The Fossil Sync Protocol</title>

<p>This document describes the wire protocol used to synchronize 




content between two Fossil repositories.</p>

<h2>1.0 Overview</h2>

<p>The global state of a fossil repository consists of an unordered
collection of artifacts.  Each artifact is identified by its SHA1 hash
expressed as a 40-character lower-case hexadecimal string.
Synchronization is the process of sharing artifacts between
servers so that all servers have copies of all artifacts.  Because
artifacts are unordered, the order in which artifacts are received
at a server is inconsequential.  It is assumed that the SHA1 hashes
of artifacts are unique - that every artifact has a different SHA1 hash.
To a first approximation, synchronization proceeds by sharing lists
SHA1 hashes of available artifacts, then sharing those artifacts that
are not found on one side or the other of the connection.  In practice,
................................................................................
SHA1 hashes for this many artifacts can be large.  So optimizations are
employed that usually reduce the number of SHA1 hashes that need to be
shared to a few hundred.</p>

<p>Each repository also has local state.  The local state determines
the web-page formatting preferences, authorized users, ticket formats,
and similar information that varies from one repository to another.
The local state is not using transferred during a sync.  Except,	
some local state is transferred during a [/help?cmd=clone|clone]
in order to initialize the local state of the new repository.  And
the [/help?cmd=configuration|config push] and
[/help?cmd=configuration|config pull]
commands can be an administrator to sync local state.</p>


<h2>2.0 Transport</h2>

<p>All communication between client and server is via HTTP requests.
The server is listening for incoming HTTP requests.  The client
issues one or more HTTP requests and receives replies for each
request.</p>

<p>The server might be running as an independent server
using the <b>server</b> command, or it might be launched from
inetd or xinetd using the <b>http</b> command.  Or the server might
be launched from CGI.  
(See "[./server.wiki|How To Configure A Fossil Server]" for details.)
The specifics of how the server listens
for incoming HTTP requests is immaterial to this protocol.
The important point is that the server is listening for requests and 
the client is the issuer of the requests.</p>

<p>A single push, pull, or sync might involve multiple HTTP requests.
The client maintains state between all requests.  But on the server
side, each request is independent.  The server does not preserve
any information about the client from one request to the next.</p>

<h4>2.0.1 Encrypted Transport</h4>

<p>In the current implementation of Fossil, the server only
understands HTTP requests.  The client can send either 
clear-text HTTP requested or or encrypted HTTPS requests.  But when
HTTPS requests are sent, they first must be decrypted by a webserver
or proxy before being passed to the Fossil server.  This limitation
may be relaxed in a future release.</p>

<h3>2.1 Server Identification</h3>

<p>The server is identified by a URL argument that accompanies the
push, pull, or sync command on the client.  (As a convenience to
users, the URL can be omitted on the client command and the same URL
from the most recent push, pull, or sync will be reused.  This saves
typing in the common case where the client does multiple syncs to
................................................................................
delta and the second argument is the ID of another artifact that is the
source of the delta and the third argument is the original size of the
delta artifact.</p>

<p>Unlike file cards, cfile cards are only sent in one direction during a
clone from server to client for clone protocol version "3" or greater.</p>

<h4>3.3.3 Private artifacts</h4>

<p>"Private" content consist of artifacts that are not normally synced.
However, private content will be synced when the
the [/help?cmd=sync|fossil sync] command includes the "--private" option.
</p>

<p>Private content is marked by a "private" card:

<blockquote>
<b>private</b>
</blockquote>

<p>The private card has no arguments and must directly precede a
file card that contains the private content.</p>

<h3>3.4 Push and Pull Cards</h3>

<p>Among the first cards in a client-to-server message are
the push and pull cards.  The push card tells the server that
the client is pushing content.  The pull card tells the server
that the client wants to pull content.  In the event of a sync,
both cards are sent.  The format is as follows:</p>
................................................................................
of the software project that the client repository contains.
The projectcode for the client and server must match in order
for the transaction to proceed.</p>

<p>The server will also send a push card back to the client
during a clone.  This is how the client determines what project
code to put in the new repository it is constructing.</p>

<p>The <i>servercode</i> argument is currently unused.

<h3>3.5 Clone Cards</h3>

<p>A clone card works like a pull card in that it is sent from
client to server in order to tell the server that the client
wants to pull content.  The clone card comes in two formats.  Older
clients use the no-argument format and newer clients use the
................................................................................
<h3>3.6 Igot Cards</h3>

<p>An igot card can be sent from either client to server or from
server to client in order to indicate that the sender holds a copy
of a particular artifact.  The format is:</p>

<blockquote>
<b>igot</b> <i>artifact-id</i> ?<i>flag</i>?
</blockquote>

<p>The first argument of the igot card is the ID of the artifact that
the sender possesses.
The receiver of an igot card will typically check to see if
it also holds the same artifact and if not it will request the artifact
using a gimme card in either the reply or in the next message.</p>

<p>If the second argument exists and is "1", then the artifact
identified by the first argument is private on the sender and should
be ignored unless a "--private" [/help?cmd=sync|sync] is occurring.

<h3>3.7 Gimme Cards</h3>

<p>A gimme card is sent from either client to server or from server
to client.  The gimme card asks the receiver to send a particular
artifact back to the sender.  The format of a gimme card is this:</p>

<blockquote>
................................................................................
<p>The server will only accept a config card if the user has
"Admin" privilege.  A client will only accept a config card if
it had sent a corresponding reqconfig card in its request.

<p>The content of the configuration item is used to overwrite the
corresponding configuration data in the receiver.

<h3>3.11 Pragma Cards</h3>

<p>The client may try to influence the behavior of the server by
issuing a pragma card:

<blockquote>
<b>pragma</i> <i>name value...</i>
</blockquote>

<p>The "pragma" card has at least one argument which is the pragma name.
The pragma name defines what the pragma does.
A pragma might have zero or more "value" arguments
depending on the pragma name.

<p>New pragma names may be added to the protocol from time to time
in order to enhance the capabilities of Fossil.
Unknown pragmas are silently ignored, for backwards compatibility.

<p>The following are the known pragma names as of 2016-08-03:

<ol>
<li><p><b>send-private</b>
<p>The send-private pragma instructs the server to send all of its
private artifacts to the client.  The server will only obey this
request if the user has the "x" or "Private" privilege.

<li><p><b>send-catalog</b>
<p>The send-catalog pragma instructs the server to transmit igot
cards for every known artifact.  This can help the client and server
to get back in synchronization after a prior protocol error.  The
"--verily" option to the [/help?cmd=sync|fossil sync] command causes
the send-catalog pragma to be transmitted.</p>
</ol>

<h3>3.12 Comment Cards</h3>

<p>Any card that begins with "#" (ASCII 0x23) is a comment card and
is silently ignored.</p>

<h3>3.13 Error Cards</h3>

<p>If the server discovers anything wrong with a request, it generates
an error card in its reply.  When the client sees the error card,
it displays an error message to the user and aborts the sync
operation.  An error card looks like this:</p>

<blockquote>
................................................................................
A space (ASCII 0x20) is represented as "\s" (ASCII 0x5C, 0x73).  A
newline (ASCII 0x0a) is "\n" (ASCII 0x6C, x6E).  A backslash
(ASCII 0x5C) is represented as two backslashes "\\".  Apart from
space and newline, no other whitespace characters nor any
unprintable characters are allowed in
the error message.</p>






<h3>3.14 Unknown Cards</h3>

<p>If either the client or the server sees a card that is not
described above, then it generates an error and aborts.</p>

<h2>4.0 Phantoms And Clusters</h2>

<p>When a repository knows that an artifact exists and knows the ID of
................................................................................
    card per line.
<li>Card formats are:
    <ul>
    <li> <b>login</b> <i>userid nonce signature</i>
    <li> <b>push</b> <i>servercode projectcode</i>
    <li> <b>pull</b> <i>servercode projectcode</i>
    <li> <b>clone</b>
    <li> <b>clone_seqno</b> <i>sequence-number</i>
    <li> <b>file</b> <i>artifact-id size</i> <b>\n</b> <i>content</i>
    <li> <b>file</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i>
    <li> <b>cfile</b> <i>artifact-id size</i> <b>\n</b> <i>content</i>
    <li> <b>cfile</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i>
    <li> <b>private</b>
    <li> <b>igot</b> <i>artifact-id</i> ?<i>flag</i>?
    <li> <b>gimme</b> <i>artifact-id</i>
    <li> <b>cookie</b>  <i>cookie-text</i>
    <li> <b>reqconfig</b> <i>parameter-name</i>
    <li> <b>config</b> <i>parameter-name size</i> <b>\n</b> <i>content</i>
    <li> <b>pragma</b> <i>name</i> <i>value...</i>
    <li> <b>error</b> <i>error-message</i>
    <li> <b>#</b> <i>arbitrary-text...</i>
    </ul>
<li>Phantoms are artifacts that a repository knows exist but does not possess.
<li>Clusters are artifacts that contain IDs of other artifacts.
<li>Clusters are created automatically on the server during a pull.
<li>Repositories keep track of all artifacts that are not named in any
cluster and send igot messages for those artifacts.
<li>Repositories keep track of all the phantoms they hold and send
gimme messages for those artifacts.
</ol>