Fossil

Check-in [faf57d74]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Modified the changeset class to move handling of the changeset lists to fully after their creation and storage. This is item (3) in cvsfossil.txt. The results do not satisfy however. During the creation of each changeset memory usage is (fractonally) lower, however at the end, after all changesets haven been loaded memory usage is consistently higher. The reason for that is not known. I am saving this for possible future evolution and usage, but will not pursue this further right now. The gains seem to be too small compared to the overall loss. InitializeBreakstate is likely a better target, despite its complexity.
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1:faf57d7462b47d78ff41924e14497a313d0147c7
User & Date: aku 2008-02-20 06:03:48
Context
2008-02-23
06:37
Merged bugfix [b3d61d7829] into this semi-abandoned branch just in case we will work on it again. Do it now instead of forgetting it later. Closed-Leaf check-in: 383c10f0 user: aku tags: trunk
2008-02-20
06:03
Modified the changeset class to move handling of the changeset lists to fully after their creation and storage. This is item (3) in cvsfossil.txt. The results do not satisfy however. During the creation of each changeset memory usage is (fractonally) lower, however at the end, after all changesets haven been loaded memory usage is consistently higher. The reason for that is not known. I am saving this for possible future evolution and usage, but will not pursue this further right now. The gains seem to be too small compared to the overall loss. InitializeBreakstate is likely a better target, despite its complexity. check-in: faf57d74 user: aku tags: trunk
2008-02-17
02:06
Reworked the basic structure of pass InitCSets to keep memory consumption down. Now incremental creates, breaks, saves, and releases changesets, instead of piling them on before saving all at the end. Memory tracking confirms that this changes the accumulating mountain into a near-constant usage, with the expected spikes from the breaking. check-in: f46458d5 user: aku tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Changes to tools/cvs2fossil/lib/c2f_prev.tcl.

48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63














64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
...
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
...
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
...
776
777
778
779
780
781
782

783
784
785
786
787
788
789
....
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
	set mytypeobj   ::vc::fossil::import::cvs::project::rev::${cstype}
	set mysrcid	$srcid
	set myitems     $items
	set mypos       {} ; # Commit location is not known yet.

	# Keep track of the generated changesets and of the inverse
	# mapping from items to them.
	lappend mychangesets   $self
	lappend mytchangesets($cstype) $self
	set     myidmap($myid) $self
	foreach iid $items {
	    set key [list $cstype $iid]
	    set myitemmap($key) $self
	    lappend mytitems $key
	    log write 8 csets {MAP+ item <$key> $self = [$self str]}
	}














	return
    }

    destructor {
	# The main thing is to keep track of the itemmap and remove
	# the object from it. The lists of changesets (mychangesets,
	# mytchangesets) are not maintained (= reduced), for the
	# moment. We may be able to get rid of this entirely, at least
	# for (de)construction and pass InitCSets.

	foreach iid $myitems {
	    set key [list $mytype $iid]
	    unset myitemmap($key)
	    log write 8 csets {MAP- item <$key> $self = [$self str]}
	}
	return
    }

    method str {} {
	set str    "<"
................................................................................
	# (*) We clear out the associated part of the myitemmap
	# in-memory index in preparation for new data. A simple unset
	# is enough, we have no symbol changesets at this time, and
	# thus never more than one reference in the list.

	foreach iid $myitems {
	    set key [list $mytype $iid]
	    unset myitemmap($key)
	    log write 8 csets {MAP- item <$key> $self = [$self str]}
	}

	# Create changesets for the fragments, reusing the current one
	# for the first fragment. We sort them in order to allow
	# checking for gaps and nice messages.

................................................................................
	# (*) above. Persistence does not matter here, none of the
	# changesets has been saved to the persistent state yet.

	set myitems  [lrange $myitems  0 $firste]
	set mytitems [lrange $mytitems 0 $firste]
	foreach iid $myitems {
	    set key [list $mytype $iid]
	    set myitemmap($key) $self
	    log write 8 csets {MAP+ item <$key> $self = [$self str]}
	}

	return 1
    }

    method persist {} {
................................................................................
	    log progress 2 csets $n {}
	    set r [$type %AUTO% [$repository projectof $pid] $cstype $srcid [state run {
		SELECT C.iid
		FROM   csitem C
		WHERE  C.cid = $id
		ORDER BY C.pos
	    }] $id]

	    incr n
	}
	return
    }

    typemethod loadcounter {} {
	# Initialize the counter from the state
................................................................................
					   # can be used by only one
					   # changeset.
    typevariable myidmap   -array {} ; # Map from changeset id to
				       # changeset.

    typemethod all    {}    { return $mychangesets }
    typemethod of     {cid} { return $myidmap($cid) }
    typemethod ofitem {iid} { return $myitemmap($iid) }

    typemethod rev    {}    { return $mytchangesets(rev) }
    typemethod sym    {}    { return [concat \
					  ${mytchangesets(sym::branch)} \
					  ${mytchangesets(sym::tag)}] }

    # # ## ### ##### ######## #############







|
|



|



>
>
>
>
>
>
>
>
>
>
>
>
>
>












|







 







|







 







|







 







>







 







|







48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
...
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
...
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
...
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
....
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
	set mytypeobj   ::vc::fossil::import::cvs::project::rev::${cstype}
	set mysrcid	$srcid
	set myitems     $items
	set mypos       {} ; # Commit location is not known yet.

	# Keep track of the generated changesets and of the inverse
	# mapping from items to them.
	#lappend mychangesets   $self
	#lappend mytchangesets($cstype) $self
	set     myidmap($myid) $self
	foreach iid $items {
	    set key [list $cstype $iid]
	    #set myitemmap($key) $self
	    lappend mytitems $key
	    log write 8 csets {MAP+ item <$key> $self = [$self str]}
	}
	return
    }

    method loadmaps {} {
	# Keep track of the generated changesets and of the inverse
	# mapping from items to them.
	lappend mychangesets   $self
	lappend mytchangesets($mytype) $self
	set     myidmap($myid) $self
	foreach iid $myitems {
	    set key [list $mytype $iid]
	    set myitemmap($key) $self
	    #log write 8 csets {MAP+ item <$key> $self = [$self str]}
	}
	return
    }

    destructor {
	# The main thing is to keep track of the itemmap and remove
	# the object from it. The lists of changesets (mychangesets,
	# mytchangesets) are not maintained (= reduced), for the
	# moment. We may be able to get rid of this entirely, at least
	# for (de)construction and pass InitCSets.

	foreach iid $myitems {
	    set key [list $mytype $iid]
	    #unset myitemmap($key)
	    log write 8 csets {MAP- item <$key> $self = [$self str]}
	}
	return
    }

    method str {} {
	set str    "<"
................................................................................
	# (*) We clear out the associated part of the myitemmap
	# in-memory index in preparation for new data. A simple unset
	# is enough, we have no symbol changesets at this time, and
	# thus never more than one reference in the list.

	foreach iid $myitems {
	    set key [list $mytype $iid]
	    #unset myitemmap($key)
	    log write 8 csets {MAP- item <$key> $self = [$self str]}
	}

	# Create changesets for the fragments, reusing the current one
	# for the first fragment. We sort them in order to allow
	# checking for gaps and nice messages.

................................................................................
	# (*) above. Persistence does not matter here, none of the
	# changesets has been saved to the persistent state yet.

	set myitems  [lrange $myitems  0 $firste]
	set mytitems [lrange $mytitems 0 $firste]
	foreach iid $myitems {
	    set key [list $mytype $iid]
	    #set myitemmap($key) $self
	    log write 8 csets {MAP+ item <$key> $self = [$self str]}
	}

	return 1
    }

    method persist {} {
................................................................................
	    log progress 2 csets $n {}
	    set r [$type %AUTO% [$repository projectof $pid] $cstype $srcid [state run {
		SELECT C.iid
		FROM   csitem C
		WHERE  C.cid = $id
		ORDER BY C.pos
	    }] $id]
	    $r loadmaps
	    incr n
	}
	return
    }

    typemethod loadcounter {} {
	# Initialize the counter from the state
................................................................................
					   # can be used by only one
					   # changeset.
    typevariable myidmap   -array {} ; # Map from changeset id to
				       # changeset.

    typemethod all    {}    { return $mychangesets }
    typemethod of     {cid} { return $myidmap($cid) }
    #typemethod ofitem {iid} { return $myitemmap($iid) }

    typemethod rev    {}    { return $mytchangesets(rev) }
    typemethod sym    {}    { return [concat \
					  ${mytchangesets(sym::branch)} \
					  ${mytchangesets(sym::tag)}] }

    # # ## ### ##### ######## #############