Fossil

Check-in [d7809493]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Update the sync protocol documentation to describe the unversioned file sync.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | unversioned-files
Files: files | file ages | folders
SHA1: d78094932026b0c3d161507d0e5b20c16b3a5d12
User & Date: drh 2016-08-17 14:40:18
Context
2016-08-17
15:36
Show the number and total size of unversioned files on the /stat page. Add a link to /uvlist from /sitemap. Closed-Leaf check-in: 83c7571f user: drh tags: unversioned-files
14:40
Update the sync protocol documentation to describe the unversioned file sync. check-in: d7809493 user: drh tags: unversioned-files
13:37
Limit the amount of unversioned content sent with each HTTP request. Also: fix up source code lines in xfer.c to be less than 80 characters long. check-in: faa73ae0 user: drh tags: unversioned-files
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to www/sync.wiki.

165
166
167
168
169
170
171
172
173
174

175
176



177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
...
255
256
257
258
259
260
261


































262
263
264
265
266
267
268
...
357
358
359
360
361
362
363
































364
365
366
367
368
369
370
...
372
373
374
375
376
377
378
















379
380
381
382
383
384
385
...
537
538
539
540
541
542
543







































544
545
546
547
548
549
550
...
711
712
713
714
715
716
717










































718
719
720
721
722
723
724
...
732
733
734
735
736
737
738

739
740

741

742
743
744
745
746
747
748
checks out, then the client is granted all privileges of the
specified user.</p>

<p>Privileges are cumulative.  There can be multiple successful
login cards.  The session privileges are the bit-wise OR of the
privileges of each individual login.</p>

<h3>3.3 File and Compressed File Cards</h3>

<p>Artifacts are transferred using either "file" cards, or "cfile" cards.

(The name "file" card comes from the fact that most artifacts correspond to
files, and "cfile" is just a compressed file.)



</p>

<h4>3.3.1 File Cards</h4>

<p>For sync protocols, artifacts are transferred using "file"
cards.  File cards come in two different formats depending
on whether the artifact is sent directly or as a delta from some
other artifact.</p>

<blockquote>
<b>file</b> <i>artifact-id size</i> <b>\n</b> <i>content</i><br>
<b>file</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i>
</blockquote>

<p>File cards are different from all other cards in that they
followed by in-line "payload" data.  The content of the artifact
or the artifact delta consists of the first <i>size</i> bytes of the
x-fossil content that immediately follow the newline that
terminates the file card.  Only file and cfile cards have this characteristic.
</p>

<p>The first argument of a file card is the ID of the artifact that
................................................................................
<blockquote>
<b>private</b>
</blockquote>

<p>The private card has no arguments and must directly precede a
file card that contains the private content.</p>



































<h3>3.4 Push and Pull Cards</h3>

<p>Among the first cards in a client-to-server message are
the push and pull cards.  The push card tells the server that
the client is pushing content.  The pull card tells the server
that the client wants to pull content.  In the event of a sync,
both cards are sent.  The format is as follows:</p>
................................................................................
it also holds the same artifact and if not it will request the artifact
using a gimme card in either the reply or in the next message.</p>

<p>If the second argument exists and is "1", then the artifact
identified by the first argument is private on the sender and should
be ignored unless a "--private" [/help?cmd=sync|sync] is occurring.

































<h3>3.7 Gimme Cards</h3>

<p>A gimme card is sent from either client to server or from server
to client.  The gimme card asks the receiver to send a particular
artifact back to the sender.  The format of a gimme card is this:</p>

<blockquote>
................................................................................
</blockquote>

<p>The argument to the gimme card is the ID of the artifact that
the sender wants.  The receiver will typically respond to a
gimme card by sending a file card in its reply or in the next
message.</p>

















<h3>3.8 Cookie Cards</h3>

<p>A cookie card can be used by a server to record a small amount
of state information on a client.  The server sends a cookie to the
client.  The client sends the same cookie back to the server on
its next request.  The cookie card has a single argument which
is its payload.</p>
................................................................................

<li><p><b>send-catalog</b>
<p>The send-catalog pragma instructs the server to transmit igot
cards for every known artifact.  This can help the client and server
to get back in synchronization after a prior protocol error.  The
"--verily" option to the [/help?cmd=sync|fossil sync] command causes
the send-catalog pragma to be transmitted.</p>







































</ol>

<h3>3.12 Comment Cards</h3>

<p>Any card that begins with "#" (ASCII 0x23) is a comment card and
is silently ignored.</p>

................................................................................

<p>A sync is just a pull and a push that happen at the same time.
The first three steps of a pull are combined with the first five steps
of a push.  Steps (4) through (7) of a pull are combined with steps
(5) through (8) of a push.  And steps (8) through (10) of a pull
are combined with step (9) of a push.</p>











































<h2>6.0 Summary</h2>

<p>Here are the key points of the synchronization protocol:</p>

<ol>
<li>The client sends one or more PUSH HTTP requests to the server.
    The request and reply content type is "application/x-fossil".
................................................................................
    <li> <b>pull</b> <i>servercode projectcode</i>
    <li> <b>clone</b>
    <li> <b>clone_seqno</b> <i>sequence-number</i>
    <li> <b>file</b> <i>artifact-id size</i> <b>\n</b> <i>content</i>
    <li> <b>file</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i>
    <li> <b>cfile</b> <i>artifact-id size</i> <b>\n</b> <i>content</i>
    <li> <b>cfile</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i>

    <li> <b>private</b>
    <li> <b>igot</b> <i>artifact-id</i> ?<i>flag</i>?

    <li> <b>gimme</b> <i>artifact-id</i>

    <li> <b>cookie</b>  <i>cookie-text</i>
    <li> <b>reqconfig</b> <i>parameter-name</i>
    <li> <b>config</b> <i>parameter-name size</i> <b>\n</b> <i>content</i>
    <li> <b>pragma</b> <i>name</i> <i>value...</i>
    <li> <b>error</b> <i>error-message</i>
    <li> <b>#</b> <i>arbitrary-text...</i>
    </ul>







|

|
>
|
<
>
>
>


|











|







 







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







 







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







 







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







 







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







 







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







 







>


>

>







165
166
167
168
169
170
171
172
173
174
175
176

177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
...
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
...
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
...
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
...
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
...
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
...
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
checks out, then the client is granted all privileges of the
specified user.</p>

<p>Privileges are cumulative.  There can be multiple successful
login cards.  The session privileges are the bit-wise OR of the
privileges of each individual login.</p>

<h3>3.3 File Cards</h3>

<p>Artifacts are transferred using either "file" cards, or "cfile"
or "uvfile" cards.
The name "file" card comes from the fact that most artifacts correspond to

files that are under version control.
The "cfile" name is an abbreviation for "compressed file".
The "uvfile" name is an abbreviation for "unversioned file".
</p>

<h4>3.3.1 Ordinary File Cards</h4>

<p>For sync protocols, artifacts are transferred using "file"
cards.  File cards come in two different formats depending
on whether the artifact is sent directly or as a delta from some
other artifact.</p>

<blockquote>
<b>file</b> <i>artifact-id size</i> <b>\n</b> <i>content</i><br>
<b>file</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i>
</blockquote>

<p>File cards are different from most other cards in that they are
followed by in-line "payload" data.  The content of the artifact
or the artifact delta consists of the first <i>size</i> bytes of the
x-fossil content that immediately follow the newline that
terminates the file card.  Only file and cfile cards have this characteristic.
</p>

<p>The first argument of a file card is the ID of the artifact that
................................................................................
<blockquote>
<b>private</b>
</blockquote>

<p>The private card has no arguments and must directly precede a
file card that contains the private content.</p>

<h4>3.3.4 Unversioned File Cards</h4>

<p>Unversioned content is sent in both directions (client to server and
server to client) using "uvfile" cards in the following format:

<blockquote>
<b>uvfile</b> <i>name mtime hash size flags</i> <b>\n</b> <i>content</i>
</blockquote>

<p>The <i>name</i> field is the name of the unversioned file.  The
<i>mtime</i> is the last modification time of the file in seconds
since 1970.  The <i>hash</i> field is the SHA1 hash of the content
for the unversioned file, or "<b>-</b>" for deleted content.
The <i>size</i> field is the (uncompressed) size of the content
in bytes.  The <i>flags</i> field is an integer which is interpreted
as an array of bits.  The 0x0004 bit of <i>flags</i> indicates that
the <i>content</i> is to be omitted.  The content might be omitted if
it is too large to transmit, or if the send merely wants to update the
modification time of the file without changing the files content.
The <i>content</i> is the (uncompressed) content of the file.

<p>The receiver should only accept the uvfile card if the hash and
size match the content and if the mtime is newer than any existing
instance of the same file held by the receiver.  The sender will not
normally transmit a uvfile card unless all these constraints are true,
but the receiver should double-check.

<p>A server should only accept uvfile cards if the login user has
the "y" write-unversioned permission.

<p>Servers send uvfile cards in response to uvgimme cards received from
the client.  Clients send uvfile cards when they determine that the server
needs the content based on uvigot cards previously received from the server.

<h3>3.4 Push and Pull Cards</h3>

<p>Among the first cards in a client-to-server message are
the push and pull cards.  The push card tells the server that
the client is pushing content.  The pull card tells the server
that the client wants to pull content.  In the event of a sync,
both cards are sent.  The format is as follows:</p>
................................................................................
it also holds the same artifact and if not it will request the artifact
using a gimme card in either the reply or in the next message.</p>

<p>If the second argument exists and is "1", then the artifact
identified by the first argument is private on the sender and should
be ignored unless a "--private" [/help?cmd=sync|sync] is occurring.

<h4>3.6.1 Unversioned Igot Cards</h4>

<p>Zero or more "uvigot" cards are sent from client to server when
synchronizing unversioned content.  The format of a uvigot card is
as follows:

<blockquote>
<b>uvigot</b> <i>name mtime hash size</i>
</blockquote>

<p>The <i>name</i> argument is the name of an unversioned file.
The <i>mtime</i> is the last modification time of the unversioned file
in seconds since 1970.  
The <i>hash</i> is the SHA1 hash of the unversioned file content, or
"<b>-</b>" if the file has been deleted.
The <i>size</i> is the uncompressed size of the file in bytes.

<p>When the server sees a "pragma uv-hash" card for which the hash
does not match, it sends uvigot cards for every unversioned file that it
holds.  The client will use this information to figure out which
unversioned files need to be synchronized.
The server might also send a uvigot card when it receives a uvgimme card
but its reply message size is already oversized and hence unable to hold
the usual uvfile reply.

<p>When a client receives a "uvigot" card, it checks to see if the the
file needs to be transfered from client to server or from server to client.
If a client-to-server transmission is needed, the client schedules that
transfer to occur on a subsequent HTTP request.  If a server-to-client
transfer is needed, then the client sends a "uvgimme" card back to the
server to request the file content.

<h3>3.7 Gimme Cards</h3>

<p>A gimme card is sent from either client to server or from server
to client.  The gimme card asks the receiver to send a particular
artifact back to the sender.  The format of a gimme card is this:</p>

<blockquote>
................................................................................
</blockquote>

<p>The argument to the gimme card is the ID of the artifact that
the sender wants.  The receiver will typically respond to a
gimme card by sending a file card in its reply or in the next
message.</p>

<h4>3.7.1 Unversioned Gimme Cards</h4>

<p>Sync synchronizing unversioned content, the client may send "uvgimme"
cards to the server.  A uvgimme card requests that the server send
unversioned content to the client.  The format of a uvgimme card is
as follows:

<blockquote>
<b>uvgimme</b> <i>name</i>
</blockquote>

<p>The <i>name</i> is the name of the unversioned file found on the
server that the client would like to have.  When a server sees a
uvgimme card, it normally responses with a uvfile card, though it might
also send anoter uvigot card if the HTTP reply is already oversized.

<h3>3.8 Cookie Cards</h3>

<p>A cookie card can be used by a server to record a small amount
of state information on a client.  The server sends a cookie to the
client.  The client sends the same cookie back to the server on
its next request.  The cookie card has a single argument which
is its payload.</p>
................................................................................

<li><p><b>send-catalog</b>
<p>The send-catalog pragma instructs the server to transmit igot
cards for every known artifact.  This can help the client and server
to get back in synchronization after a prior protocol error.  The
"--verily" option to the [/help?cmd=sync|fossil sync] command causes
the send-catalog pragma to be transmitted.</p>

<li><p><b>uv-hash</b> <i>HASH</i>
<p>The uv-hash pragma is sent from client to server to provoke a
synchronization of unversioned content.  The <i>HASH</i> is a SHA1
hash of the names, modification times, and individual hashes of all
unversioned files on the client.  If the unversioned content hash
from the client does not match the unversioned content hash on the
server, then the server will reply with either a "pragma uv-push-ok"
or "pragma uv-pull-only" card followed by one "uvigot" card for
each unversioned file currently held on the server.  The collection
of "uvigot" cards sent in response to a "uv-hash" pragma is called
the "unversioned catalog".  The client will used the unversioned
catalog to figure out which files (if any) need to be synchronized
between client and server and send appropriate "uvfile" or "uvgimme"
cards on the next HTTP request.</p>

<p>If a client sends a uv-hash pragma and does not receive back
either a uv-pull-only or uv-push-ok pragma, that means that the
content on the server exactly matches the content on the client and
no further synchronization is required.

<li><p><b>uv-pull-only</b></i>
<p>A server sends the uv-pull-only pragma to the client in response
to a uv-hash pragma with a mismatched content hash argument.  This
pragma indicates that there are differences in unversioned content
between the client and server but that content can only be transfered
from server to client.  The server is unwilling to accept content from
the client because the client login lacks the "write-unversioned"
permission.</p>

<li><p><b>uv-push-ok</b></i>
<p>A server sends the uv-push-ok pragma to the client in response
to a uv-hash pragma with a mismatched content hash argument.  This
pragma indicates that there are differences in unversioned content
between the client and server and that content can only be transfered
in either direction.  The server is willing to accept content from
the client because the client login has the "write-unversioned"
permission.</p>

</ol>

<h3>3.12 Comment Cards</h3>

<p>Any card that begins with "#" (ASCII 0x23) is a comment card and
is silently ignored.</p>

................................................................................

<p>A sync is just a pull and a push that happen at the same time.
The first three steps of a pull are combined with the first five steps
of a push.  Steps (4) through (7) of a pull are combined with steps
(5) through (8) of a push.  And steps (8) through (10) of a pull
are combined with step (9) of a push.</p>

<h3>5.4 Unversioned File Sync</h3>

<p>"Unversioned files" are files held in the repository
where only the most recent version of the file is kept rather than
the entire change history.  Unversioned files are intended to be
used to store ephemeral content, such as compiled binaries of the
most recent release.

<p>Unversioned files are identified by name and timestamp (mtime).
Only the most recent version of each file (the version with
the largest mtime value) is retained.

<p>Unversioned files are synchronized using the
[/help?cmd=unversioned|fossil unversioned sync] command.

<p>A schematic of an unversioned file synchronization is as follows:

<ol>
<li>The client sends a "pragma uv-hash" card to the server.  The argument
    to the uv-hash pragma is a hash of all filesnames, mtimes, and
    content hashes for the unversioned files held by the client.
    <hr>
<li>If the unversion content hash from the client matches the unversioned
    content hash on the server, then nothing needs to be done and the
    server no-ops.  But if the hashes are different, then the server
    replies with either a uv-pull-only or a uv-push-ok pragma followed by
    uvigot cards for all unversioned files held on the server.
    <hr>
<li>The client examines the uvigot cards received from the server and
    determines which unversioned files need to be exchanged in order
    to bring the client and server into synchronization.  The client
    then sends appropriate "uvgimme" or "uvfile" cards back to the
    server.
    <hr>
<li>The server updates its unversioned file store with received "uvfile"
    cards and answers "uvgimme" cards with "uvfile" cards in its reply.
</ol>

<p>The last two steps might be repeated multiple
times if there is more unversioned content to be transferred than will
fit comfortably in a single HTTP request.

<h2>6.0 Summary</h2>

<p>Here are the key points of the synchronization protocol:</p>

<ol>
<li>The client sends one or more PUSH HTTP requests to the server.
    The request and reply content type is "application/x-fossil".
................................................................................
    <li> <b>pull</b> <i>servercode projectcode</i>
    <li> <b>clone</b>
    <li> <b>clone_seqno</b> <i>sequence-number</i>
    <li> <b>file</b> <i>artifact-id size</i> <b>\n</b> <i>content</i>
    <li> <b>file</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i>
    <li> <b>cfile</b> <i>artifact-id size</i> <b>\n</b> <i>content</i>
    <li> <b>cfile</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i>
    <li> <b>uvfile</b> <i>name mtime hash size flags</i> <b>\n</b> <i>content</i>
    <li> <b>private</b>
    <li> <b>igot</b> <i>artifact-id</i> ?<i>flag</i>?
    <li> <b>uvigot</b> <i>name mtime hash size</i>
    <li> <b>gimme</b> <i>artifact-id</i>
    <li> <b>uvgimme</b> <i>name</i>
    <li> <b>cookie</b>  <i>cookie-text</i>
    <li> <b>reqconfig</b> <i>parameter-name</i>
    <li> <b>config</b> <i>parameter-name size</i> <b>\n</b> <i>content</i>
    <li> <b>pragma</b> <i>name</i> <i>value...</i>
    <li> <b>error</b> <i>error-message</i>
    <li> <b>#</b> <i>arbitrary-text...</i>
    </ul>