Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
Comment: | Update the sync protocol documentation to describe the unversioned file sync. |
---|---|
Downloads: | Tarball | ZIP archive | SQL archive |
Timelines: | family | ancestors | descendants | both | unversioned-files |
Files: | files | file ages | folders |
SHA1: |
d78094932026b0c3d161507d0e5b20c1 |
User & Date: | drh 2016-08-17 14:40:18 |
Context
2016-08-17
| ||
15:36 | Show the number and total size of unversioned files on the /stat page. Add a link to /uvlist from /sitemap. Closed-Leaf check-in: 83c7571f user: drh tags: unversioned-files | |
14:40 | Update the sync protocol documentation to describe the unversioned file sync. check-in: d7809493 user: drh tags: unversioned-files | |
13:37 | Limit the amount of unversioned content sent with each HTTP request. Also: fix up source code lines in xfer.c to be less than 80 characters long. check-in: faa73ae0 user: drh tags: unversioned-files | |
Changes
Changes to www/sync.wiki.
165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 ... 255 256 257 258 259 260 261 262 263 264 265 266 267 268 ... 357 358 359 360 361 362 363 364 365 366 367 368 369 370 ... 372 373 374 375 376 377 378 379 380 381 382 383 384 385 ... 537 538 539 540 541 542 543 544 545 546 547 548 549 550 ... 711 712 713 714 715 716 717 718 719 720 721 722 723 724 ... 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 |
checks out, then the client is granted all privileges of the specified user.</p> <p>Privileges are cumulative. There can be multiple successful login cards. The session privileges are the bit-wise OR of the privileges of each individual login.</p> <h3>3.3 File and Compressed File Cards</h3> <p>Artifacts are transferred using either "file" cards, or "cfile" cards. (The name "file" card comes from the fact that most artifacts correspond to files, and "cfile" is just a compressed file.) </p> <h4>3.3.1 File Cards</h4> <p>For sync protocols, artifacts are transferred using "file" cards. File cards come in two different formats depending on whether the artifact is sent directly or as a delta from some other artifact.</p> <blockquote> <b>file</b> <i>artifact-id size</i> <b>\n</b> <i>content</i><br> <b>file</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i> </blockquote> <p>File cards are different from all other cards in that they followed by in-line "payload" data. The content of the artifact or the artifact delta consists of the first <i>size</i> bytes of the x-fossil content that immediately follow the newline that terminates the file card. Only file and cfile cards have this characteristic. </p> <p>The first argument of a file card is the ID of the artifact that ................................................................................ <blockquote> <b>private</b> </blockquote> <p>The private card has no arguments and must directly precede a file card that contains the private content.</p> <h3>3.4 Push and Pull Cards</h3> <p>Among the first cards in a client-to-server message are the push and pull cards. The push card tells the server that the client is pushing content. The pull card tells the server that the client wants to pull content. In the event of a sync, both cards are sent. The format is as follows:</p> ................................................................................ it also holds the same artifact and if not it will request the artifact using a gimme card in either the reply or in the next message.</p> <p>If the second argument exists and is "1", then the artifact identified by the first argument is private on the sender and should be ignored unless a "--private" [/help?cmd=sync|sync] is occurring. <h3>3.7 Gimme Cards</h3> <p>A gimme card is sent from either client to server or from server to client. The gimme card asks the receiver to send a particular artifact back to the sender. The format of a gimme card is this:</p> <blockquote> ................................................................................ </blockquote> <p>The argument to the gimme card is the ID of the artifact that the sender wants. The receiver will typically respond to a gimme card by sending a file card in its reply or in the next message.</p> <h3>3.8 Cookie Cards</h3> <p>A cookie card can be used by a server to record a small amount of state information on a client. The server sends a cookie to the client. The client sends the same cookie back to the server on its next request. The cookie card has a single argument which is its payload.</p> ................................................................................ <li><p><b>send-catalog</b> <p>The send-catalog pragma instructs the server to transmit igot cards for every known artifact. This can help the client and server to get back in synchronization after a prior protocol error. The "--verily" option to the [/help?cmd=sync|fossil sync] command causes the send-catalog pragma to be transmitted.</p> </ol> <h3>3.12 Comment Cards</h3> <p>Any card that begins with "#" (ASCII 0x23) is a comment card and is silently ignored.</p> ................................................................................ <p>A sync is just a pull and a push that happen at the same time. The first three steps of a pull are combined with the first five steps of a push. Steps (4) through (7) of a pull are combined with steps (5) through (8) of a push. And steps (8) through (10) of a pull are combined with step (9) of a push.</p> <h2>6.0 Summary</h2> <p>Here are the key points of the synchronization protocol:</p> <ol> <li>The client sends one or more PUSH HTTP requests to the server. The request and reply content type is "application/x-fossil". ................................................................................ <li> <b>pull</b> <i>servercode projectcode</i> <li> <b>clone</b> <li> <b>clone_seqno</b> <i>sequence-number</i> <li> <b>file</b> <i>artifact-id size</i> <b>\n</b> <i>content</i> <li> <b>file</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i> <li> <b>cfile</b> <i>artifact-id size</i> <b>\n</b> <i>content</i> <li> <b>cfile</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i> <li> <b>private</b> <li> <b>igot</b> <i>artifact-id</i> ?<i>flag</i>? <li> <b>gimme</b> <i>artifact-id</i> <li> <b>cookie</b> <i>cookie-text</i> <li> <b>reqconfig</b> <i>parameter-name</i> <li> <b>config</b> <i>parameter-name size</i> <b>\n</b> <i>content</i> <li> <b>pragma</b> <i>name</i> <i>value...</i> <li> <b>error</b> <i>error-message</i> <li> <b>#</b> <i>arbitrary-text...</i> </ul> |
| | > | < > > > | | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > |
165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 ... 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 ... 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 ... 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 ... 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 ... 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 ... 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 |
checks out, then the client is granted all privileges of the specified user.</p> <p>Privileges are cumulative. There can be multiple successful login cards. The session privileges are the bit-wise OR of the privileges of each individual login.</p> <h3>3.3 File Cards</h3> <p>Artifacts are transferred using either "file" cards, or "cfile" or "uvfile" cards. The name "file" card comes from the fact that most artifacts correspond to files that are under version control. The "cfile" name is an abbreviation for "compressed file". The "uvfile" name is an abbreviation for "unversioned file". </p> <h4>3.3.1 Ordinary File Cards</h4> <p>For sync protocols, artifacts are transferred using "file" cards. File cards come in two different formats depending on whether the artifact is sent directly or as a delta from some other artifact.</p> <blockquote> <b>file</b> <i>artifact-id size</i> <b>\n</b> <i>content</i><br> <b>file</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i> </blockquote> <p>File cards are different from most other cards in that they are followed by in-line "payload" data. The content of the artifact or the artifact delta consists of the first <i>size</i> bytes of the x-fossil content that immediately follow the newline that terminates the file card. Only file and cfile cards have this characteristic. </p> <p>The first argument of a file card is the ID of the artifact that ................................................................................ <blockquote> <b>private</b> </blockquote> <p>The private card has no arguments and must directly precede a file card that contains the private content.</p> <h4>3.3.4 Unversioned File Cards</h4> <p>Unversioned content is sent in both directions (client to server and server to client) using "uvfile" cards in the following format: <blockquote> <b>uvfile</b> <i>name mtime hash size flags</i> <b>\n</b> <i>content</i> </blockquote> <p>The <i>name</i> field is the name of the unversioned file. The <i>mtime</i> is the last modification time of the file in seconds since 1970. The <i>hash</i> field is the SHA1 hash of the content for the unversioned file, or "<b>-</b>" for deleted content. The <i>size</i> field is the (uncompressed) size of the content in bytes. The <i>flags</i> field is an integer which is interpreted as an array of bits. The 0x0004 bit of <i>flags</i> indicates that the <i>content</i> is to be omitted. The content might be omitted if it is too large to transmit, or if the send merely wants to update the modification time of the file without changing the files content. The <i>content</i> is the (uncompressed) content of the file. <p>The receiver should only accept the uvfile card if the hash and size match the content and if the mtime is newer than any existing instance of the same file held by the receiver. The sender will not normally transmit a uvfile card unless all these constraints are true, but the receiver should double-check. <p>A server should only accept uvfile cards if the login user has the "y" write-unversioned permission. <p>Servers send uvfile cards in response to uvgimme cards received from the client. Clients send uvfile cards when they determine that the server needs the content based on uvigot cards previously received from the server. <h3>3.4 Push and Pull Cards</h3> <p>Among the first cards in a client-to-server message are the push and pull cards. The push card tells the server that the client is pushing content. The pull card tells the server that the client wants to pull content. In the event of a sync, both cards are sent. The format is as follows:</p> ................................................................................ it also holds the same artifact and if not it will request the artifact using a gimme card in either the reply or in the next message.</p> <p>If the second argument exists and is "1", then the artifact identified by the first argument is private on the sender and should be ignored unless a "--private" [/help?cmd=sync|sync] is occurring. <h4>3.6.1 Unversioned Igot Cards</h4> <p>Zero or more "uvigot" cards are sent from client to server when synchronizing unversioned content. The format of a uvigot card is as follows: <blockquote> <b>uvigot</b> <i>name mtime hash size</i> </blockquote> <p>The <i>name</i> argument is the name of an unversioned file. The <i>mtime</i> is the last modification time of the unversioned file in seconds since 1970. The <i>hash</i> is the SHA1 hash of the unversioned file content, or "<b>-</b>" if the file has been deleted. The <i>size</i> is the uncompressed size of the file in bytes. <p>When the server sees a "pragma uv-hash" card for which the hash does not match, it sends uvigot cards for every unversioned file that it holds. The client will use this information to figure out which unversioned files need to be synchronized. The server might also send a uvigot card when it receives a uvgimme card but its reply message size is already oversized and hence unable to hold the usual uvfile reply. <p>When a client receives a "uvigot" card, it checks to see if the the file needs to be transfered from client to server or from server to client. If a client-to-server transmission is needed, the client schedules that transfer to occur on a subsequent HTTP request. If a server-to-client transfer is needed, then the client sends a "uvgimme" card back to the server to request the file content. <h3>3.7 Gimme Cards</h3> <p>A gimme card is sent from either client to server or from server to client. The gimme card asks the receiver to send a particular artifact back to the sender. The format of a gimme card is this:</p> <blockquote> ................................................................................ </blockquote> <p>The argument to the gimme card is the ID of the artifact that the sender wants. The receiver will typically respond to a gimme card by sending a file card in its reply or in the next message.</p> <h4>3.7.1 Unversioned Gimme Cards</h4> <p>Sync synchronizing unversioned content, the client may send "uvgimme" cards to the server. A uvgimme card requests that the server send unversioned content to the client. The format of a uvgimme card is as follows: <blockquote> <b>uvgimme</b> <i>name</i> </blockquote> <p>The <i>name</i> is the name of the unversioned file found on the server that the client would like to have. When a server sees a uvgimme card, it normally responses with a uvfile card, though it might also send anoter uvigot card if the HTTP reply is already oversized. <h3>3.8 Cookie Cards</h3> <p>A cookie card can be used by a server to record a small amount of state information on a client. The server sends a cookie to the client. The client sends the same cookie back to the server on its next request. The cookie card has a single argument which is its payload.</p> ................................................................................ <li><p><b>send-catalog</b> <p>The send-catalog pragma instructs the server to transmit igot cards for every known artifact. This can help the client and server to get back in synchronization after a prior protocol error. The "--verily" option to the [/help?cmd=sync|fossil sync] command causes the send-catalog pragma to be transmitted.</p> <li><p><b>uv-hash</b> <i>HASH</i> <p>The uv-hash pragma is sent from client to server to provoke a synchronization of unversioned content. The <i>HASH</i> is a SHA1 hash of the names, modification times, and individual hashes of all unversioned files on the client. If the unversioned content hash from the client does not match the unversioned content hash on the server, then the server will reply with either a "pragma uv-push-ok" or "pragma uv-pull-only" card followed by one "uvigot" card for each unversioned file currently held on the server. The collection of "uvigot" cards sent in response to a "uv-hash" pragma is called the "unversioned catalog". The client will used the unversioned catalog to figure out which files (if any) need to be synchronized between client and server and send appropriate "uvfile" or "uvgimme" cards on the next HTTP request.</p> <p>If a client sends a uv-hash pragma and does not receive back either a uv-pull-only or uv-push-ok pragma, that means that the content on the server exactly matches the content on the client and no further synchronization is required. <li><p><b>uv-pull-only</b></i> <p>A server sends the uv-pull-only pragma to the client in response to a uv-hash pragma with a mismatched content hash argument. This pragma indicates that there are differences in unversioned content between the client and server but that content can only be transfered from server to client. The server is unwilling to accept content from the client because the client login lacks the "write-unversioned" permission.</p> <li><p><b>uv-push-ok</b></i> <p>A server sends the uv-push-ok pragma to the client in response to a uv-hash pragma with a mismatched content hash argument. This pragma indicates that there are differences in unversioned content between the client and server and that content can only be transfered in either direction. The server is willing to accept content from the client because the client login has the "write-unversioned" permission.</p> </ol> <h3>3.12 Comment Cards</h3> <p>Any card that begins with "#" (ASCII 0x23) is a comment card and is silently ignored.</p> ................................................................................ <p>A sync is just a pull and a push that happen at the same time. The first three steps of a pull are combined with the first five steps of a push. Steps (4) through (7) of a pull are combined with steps (5) through (8) of a push. And steps (8) through (10) of a pull are combined with step (9) of a push.</p> <h3>5.4 Unversioned File Sync</h3> <p>"Unversioned files" are files held in the repository where only the most recent version of the file is kept rather than the entire change history. Unversioned files are intended to be used to store ephemeral content, such as compiled binaries of the most recent release. <p>Unversioned files are identified by name and timestamp (mtime). Only the most recent version of each file (the version with the largest mtime value) is retained. <p>Unversioned files are synchronized using the [/help?cmd=unversioned|fossil unversioned sync] command. <p>A schematic of an unversioned file synchronization is as follows: <ol> <li>The client sends a "pragma uv-hash" card to the server. The argument to the uv-hash pragma is a hash of all filesnames, mtimes, and content hashes for the unversioned files held by the client. <hr> <li>If the unversion content hash from the client matches the unversioned content hash on the server, then nothing needs to be done and the server no-ops. But if the hashes are different, then the server replies with either a uv-pull-only or a uv-push-ok pragma followed by uvigot cards for all unversioned files held on the server. <hr> <li>The client examines the uvigot cards received from the server and determines which unversioned files need to be exchanged in order to bring the client and server into synchronization. The client then sends appropriate "uvgimme" or "uvfile" cards back to the server. <hr> <li>The server updates its unversioned file store with received "uvfile" cards and answers "uvgimme" cards with "uvfile" cards in its reply. </ol> <p>The last two steps might be repeated multiple times if there is more unversioned content to be transferred than will fit comfortably in a single HTTP request. <h2>6.0 Summary</h2> <p>Here are the key points of the synchronization protocol:</p> <ol> <li>The client sends one or more PUSH HTTP requests to the server. The request and reply content type is "application/x-fossil". ................................................................................ <li> <b>pull</b> <i>servercode projectcode</i> <li> <b>clone</b> <li> <b>clone_seqno</b> <i>sequence-number</i> <li> <b>file</b> <i>artifact-id size</i> <b>\n</b> <i>content</i> <li> <b>file</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i> <li> <b>cfile</b> <i>artifact-id size</i> <b>\n</b> <i>content</i> <li> <b>cfile</b> <i>artifact-id delta-artifact-id size</i> <b>\n</b> <i>content</i> <li> <b>uvfile</b> <i>name mtime hash size flags</i> <b>\n</b> <i>content</i> <li> <b>private</b> <li> <b>igot</b> <i>artifact-id</i> ?<i>flag</i>? <li> <b>uvigot</b> <i>name mtime hash size</i> <li> <b>gimme</b> <i>artifact-id</i> <li> <b>uvgimme</b> <i>name</i> <li> <b>cookie</b> <i>cookie-text</i> <li> <b>reqconfig</b> <i>parameter-name</i> <li> <b>config</b> <i>parameter-name size</i> <b>\n</b> <i>content</i> <li> <b>pragma</b> <i>name</i> <i>value...</i> <li> <b>error</b> <i>error-message</i> <li> <b>#</b> <i>arbitrary-text...</i> </ul> |