Fossil

View Ticket
Login

View Ticket

Ticket Hash: 4013b0a81a66de9f7b5914e2eb56b4ce7c3213a3
Title: Timewarps cause Git export to malfunction
Status: Open Type: Code_Defect
Severity: Important Priority:
Subsystem: Resolution: Open
Last Modified: 2014-11-04 01:07:23
Version Found In: 1.25
User Comments:
drh added on 2013-05-23 23:20:52: (text/x-fossil-wiki)
An attempt to run "fossil export" on a repository that includes timewarps
generates a get-fast-export file that contains out-of-order records.  Git
requires that records be in check-in order, not in chronological order.

An example timewarp can be seen at [http://www.sqlite.org/docsrc/timeline?c=1ac9233c15&n=7].  Timewarps are caused when developers do check-ins
from a machine where the system clock is set incorrectly.  You can get a list
of all timewarps in a repository by visiting the test_timewarps URI.  
For example: [http://www.sqlite.org/docsrc/test_timewarps].

A work-around it to repair the timewarps by editing the check-in time of
the commit.

nobody added on 2014-11-03 22:40:05: (text/x-fossil-wiki)
Any chance of getting this fixed?

nobody (claiming to be tim) added on 2014-11-03 22:49:27: (text/x-fossil-wiki)
$ fossil export --git ../sqlite.fossil | git fast-import
fatal: mark :60713 not declared
fast-import: dumping crash report to .git/fast_import_crash_3740

To replicate, using latest version of sqlite

nobody (claiming to be tim) added on 2014-11-03 22:52:38: (text/x-fossil-wiki)
  commit refs/heads/trunk
  mark :60705
  committer drh <drh> 1257341402 +0000
  data 83
  from :60691
  M 100644 :60706 src/main.c
  M 100644 :60698 src/sqliteInt.h

  commit refs/heads/shunning_error
  mark :60721
  committer drh <drh> 1257360677 +0000
  data 24
* from :60713

Active Branch LRU
-----------------
    active_branches = 4 cur, 5 max

  pos  clock name

nobody added on 2014-11-04 00:07:28:
  db_prepare(&q,
    "SELECT strftime('%%s',mtime), objid, coalesce(ecomment,comment),"
    "       coalesce(euser,user),"
    "       (SELECT value FROM tagxref WHERE rid=objid AND tagid=%d)"
    "  FROM event"
    " WHERE type='ci' AND NOT EXISTS (SELECT 1 FROM oldcommit WHERE objid=rid)"
    " ORDER BY mtime ASC",
    TAG_BRANCH
  );

export.c has the db call above. In investigating the sqlite3 database for sqlite, objid 30360 is of type 'ci' but the from, 30356 is not of type ci. Conversion is 2* + 1, this corresponds to 60721 and 60713. The latter is not included in the output (shunned?) and so git freaks. What to do about this?

nobody added on 2014-11-04 00:10:07:
60713 is a tag (and excluded from the commit and yet tied to 60720 as a plink).

sqlite> SELECT * FROM event WHERE objid IN (30360, 30356) ORDER BY mtime;
g|2455140.26131944|30356|||||drh||Edit [eb7a544fe49d1626bacecfe53ddc03fe082e3243|eb7a544fe4]: Change background color to "#ffc0d0". Edit check-in comment. Timestamp 2009-11-04 13:30:02.||
ci|2455140.28561343|30360|||#c0c0c0||drh||Fix documentation typos.||2455140.28561343
sqlite> SELECT * FROM plink WHERE cid = 30360;
30356|30360|1|2455140.28561343

nobody added on 2014-11-04 00:14:31:
db_prepare(&q3,
      "SELECT pid FROM plink"
      " WHERE cid=%d AND isprim"
      "   AND pid IN (SELECT objid FROM event)",
      ckinId
    );

I wonder if this query for the "from" should also include a check for type 'ci'

nobody added on 2014-11-04 00:43:02: (text/x-fossil-wiki)
I made the change and ended up running into my first "real" timewarp. 

<pre>
  commit refs/heads/trunk
  mark :70907
  committer drh <drh> 1285705604 +0000
  data 94
* from :70901

sqlite> SELECT * FROM event WHERE objid = (70907-1)/2;
ci|2455468.35189815|35453|||||drh||Simplify the test that determines if the name of a new table collides with
a prior index name.||2455468.35189815
sqlite> SELECT * FROM event WHERE objid = (70901-1)/2;
ci|2455468.80332176|35450|||||dan||Fix some problems that can occur if a trigger has the same name as another database object.||2455468.80332176
sqlite> SELECT * FROM plink WHERE cid = 35453;
35450|35453|1|2455468.35189815
</pre>

Thinking I had some good ideas, I changed the sort from mtime to objid and ran into the inverse timewarp where the plink actually goes backwards.

<pre>
  commit refs/heads/trunk
  mark :91
  committer drh <drh> 959622265 +0000
  data 11
* from :93

SQLite version 3.8.7 2014-10-17 11:24:17
Enter ".help" for usage hints.
sqlite> SELECT * FROM event WHERE objid = (93-1)/2;
ci|2451694.10138889|46|||||drh||initial check-in of the new version (CVS 1)||2451694.10138889
sqlite> SELECT * FROM event WHERE objid = (91-1)/2;
ci|2451694.23917824|45|||||drh||:-) (CVS 2)||2451694.23917824
sqlite> SELECT * FROM plink WHERE cid = 45;
46|45|1|2451694.23917824
sqlite>
</pre>

nobody added on 2014-11-04 00:50:32: (text/x-fossil-wiki)
This makes me want to cry a little bit.

<pre>
sqlite> SELECT COUNT(1) FROM plink WHERE pid > cid;
1065
sqlite> SELECT COUNT(1) FROM plink WHERE cid > pid;;
13533
</pre>

Legacy?

nobody (claiming to be tim) added on 2014-11-04 00:59:14: (text/x-fossil-wiki)
I know I'm just a nobody around here... but if there is anyone out there to advise or console it would be much appreciated. Upon further analysis, the correct sort order is NOT objid or mtime, but something derived from the plink table, so that no "from" reference comes up before it is declared. Probably this is not an issue in more modern fossil repositories. Wonder if the right solution is a "repair" of sqlite.fossil rather than trying to build something fancy in export.c.

nobody added on 2014-11-04 01:07:23: (text/x-fossil-wiki)
Maybe not as bad as I thought

<pre>
sqlite> SELECT l.*, ep.mtime, ec.mtime FROM plink l LEFT JOIN event ep ON pid = ep.objid LEFT JOIN event ec ON cid = ec.objid WHERE ep.mtime > ec.mtime LIMIT 10;
35460|35462|1|2455469.26833333|2455469.46484954|2455469.26833333
35450|35453|1|2455468.35189815|2455468.80332176|2455468.35189815
sqlite>
</pre>

Did old timewarps get fixed?