Fossil

Check-in [e1dbf318]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Reworked the revision import to use the new state tracking system instead of the simple array. Moved some log outputs. Added a file listing the known problems to prevent me from forgetting stuff as it piles up :/
Downloads: Tarball | ZIP archive | SQL archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA1: e1dbf3186dd40762f79a22957096dfa1a498875a
User & Date: aku 2008-02-04 06:05:11
Context
2008-02-04
06:26
Added some musings to one of the situations to deal with. check-in: 812c91bb user: aku tags: trunk
06:05
Reworked the revision import to use the new state tracking system instead of the simple array. Moved some log outputs. Added a file listing the known problems to prevent me from forgetting stuff as it piles up :/ check-in: e1dbf318 user: aku tags: trunk
06:02
Moved the really large log output during revision import (log message, and command) to very high log levels. check-in: e7138d7f user: aku tags: trunk
Changes
Hide Diffs Unified Diffs Ignore Whitespace Patch

Added cvs2fossil.txt.

















































































































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

Known problems and areas to work on
===================================

*	Currently not properly tracking when a file is removed on some
	branch (detectable by a 'dead' revision (optype)) during the
	import of changesets.

*	Not yet able to handle the specification of multiple projects
	for one CVS repository. I.e. I can, for example, import all of
	tcllib, or a single subproject of tcllib, like tklib, but not
	multiple sub-projects in one go.

*	An internal error thrown when trying to import tcllib of
	tcllib shows that I am apparently not properly handling the
	possibility of more than one symbol used to create a
	vendor-branch with.

	In tcllib most files (18) have 'tcllib-vendor-branch' as the
	name of their vendor branch, done in 2000, however two files
	use the name 'vendor' instead, they were done in 2003. Each
	set of files corresponds a single changeset.

	This causes the code importing the changesets to flip out when
	the second changeset tries to create ':trunk:' and finds it
	already existing (both changesets are the last trunk-changeset
	on the vendor branch :) )

	Not sure yet if I should try to abort this at the beginning,
	i.e. CVS integrity failure, force the user to manually edit
	the RCS archives to bring the symbol used for the vendor
	branch into sync. Or if I should allow the import to let this
	slide by, by simply assuming that all such second changesets
	should not try to create the :trunk: if it exists.

*	An internal error thrown when trying to import bwidget of
	tcllib shows that there have to be some situation I am not
	handling correctly in the cycle-breaker and sorting passes.

	It tries to import a changeset on the
	'scriptics-sc-2-0-beta-branch' line of development (X), which
	has no commits yet. So it goes to the parent LOD to get the
	state we are spawning from. This parent is
	'scriptics-sc-1-1-branch' (Y). And is has no changesets
	committed to it yet. That should not be possible, the ordering
	constraints should have put all changesets for Y before the
	changesets of X, and Y had to have at least one changeset,
	from which the branch could be spawned.

	This need deep diving into the various linkages to understand
	what is happening, or not happening, depending.

	Note: The code I had before more fully tracking the workspace
	      state of the various lods wrongly slid over this problem
	      without erroring out.

Changes to tools/cvs2fossil/lib/c2f_file.tcl.

273
274
275
276
277
278
279


280
281
282
283
284
285
286
	return
    }

    # # ## ### ##### ######## #############
    ## Pass XII (Import).

    method pushto {repository} {


	set ws [$repository workspace]
	struct::list assign [$self Expand $ws] filemap revmap
	# filemap = dict (path -> uuid)
	# revmap  = dict (path -> rid)

	array set idmap [$repository importfiles $filemap]








>
>







273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
	return
    }

    # # ## ### ##### ######## #############
    ## Pass XII (Import).

    method pushto {repository} {
	log write 2 file {Importing file "$mypath"}

	set ws [$repository workspace]
	struct::list assign [$self Expand $ws] filemap revmap
	# filemap = dict (path -> uuid)
	# revmap  = dict (path -> rid)

	array set idmap [$repository importfiles $filemap]

Changes to tools/cvs2fossil/lib/c2f_pimport.tcl.

20
21
22
23
24
25
26

27
28
29
30
31
32
33
..
82
83
84
85
86
87
88

89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104


105
106
107
108
109
110
111
...
138
139
140
141
142
143
144

145
146
147
148
149
150
151
152
153
154

package require Tcl 8.4                                   ; # Required runtime.
package require snit                                      ; # OO system.
package require vc::tools::log                            ; # User feedback.
package require vc::fossil::import::cvs::repository       ; # Repository management.
package require vc::fossil::import::cvs::state            ; # State storage.
package require vc::fossil::import::cvs::fossil           ; # Access to fossil repositories.


# # ## ### ##### ######## ############# #####################
## Register the pass with the management

vc::fossil::import::cvs::pass define \
    Import \
    {Import the changesets and file revisions into fossil repositories} \
................................................................................
	# Pass manager interface. Executed to perform the
	# functionality of the pass.

	foreach project [repository projects] {
	    log write 1 import {Importing project "[$project base]"}

	    set fossil [fossil %AUTO%]


	    state transaction {
		# Layer I: Files and their revisions
		foreach file [$project files] {
		    set path [$file path]
		    log write 2 import {Importing file "$path"}
		    $file pushto $fossil
		}
		# Layer II: Changesets
		array set rstate {}
		foreach {revision date} [$project revisionsinorder] {
		    log write 2 import {Importing revision [$revision str]}
		    $revision pushto rstate $fossil $date
		}
		unset rstate
	    }



	    # At last copy the temporary repository file to its final
	    # destination and release the associated memory.

	    $fossil finalize [$project base].fsl
	}

................................................................................

namespace eval ::vc::fossil::import::cvs::pass {
    namespace export import
    namespace eval import {
	namespace import ::vc::fossil::import::cvs::repository
	namespace import ::vc::fossil::import::cvs::state
	namespace import ::vc::fossil::import::cvs::fossil

	namespace import ::vc::tools::log
	log register import
    }
}

# # ## ### ##### ######## ############# #####################
## Ready

package provide vc::fossil::import::cvs::pass::import 1.0
return







>







 







>




<
<



<

<
|



>
>







 







>










20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
..
83
84
85
86
87
88
89
90
91
92
93
94


95
96
97

98

99
100
101
102
103
104
105
106
107
108
109
110
111
...
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155

package require Tcl 8.4                                   ; # Required runtime.
package require snit                                      ; # OO system.
package require vc::tools::log                            ; # User feedback.
package require vc::fossil::import::cvs::repository       ; # Repository management.
package require vc::fossil::import::cvs::state            ; # State storage.
package require vc::fossil::import::cvs::fossil           ; # Access to fossil repositories.
package require vc::fossil::import::cvs::ristate          ; # Import state (revisions)

# # ## ### ##### ######## ############# #####################
## Register the pass with the management

vc::fossil::import::cvs::pass define \
    Import \
    {Import the changesets and file revisions into fossil repositories} \
................................................................................
	# Pass manager interface. Executed to perform the
	# functionality of the pass.

	foreach project [repository projects] {
	    log write 1 import {Importing project "[$project base]"}

	    set fossil [fossil %AUTO%]
	    set rstate [ristate %AUTO%]

	    state transaction {
		# Layer I: Files and their revisions
		foreach file [$project files] {


		    $file pushto $fossil
		}
		# Layer II: Changesets

		foreach {revision date} [$project revisionsinorder] {

		    $revision pushto $fossil $date $rstate
		}
		unset rstate
	    }

	    $rstate destroy

	    # At last copy the temporary repository file to its final
	    # destination and release the associated memory.

	    $fossil finalize [$project base].fsl
	}

................................................................................

namespace eval ::vc::fossil::import::cvs::pass {
    namespace export import
    namespace eval import {
	namespace import ::vc::fossil::import::cvs::repository
	namespace import ::vc::fossil::import::cvs::state
	namespace import ::vc::fossil::import::cvs::fossil
	namespace import ::vc::fossil::import::cvs::ristate
	namespace import ::vc::tools::log
	log register import
    }
}

# # ## ### ##### ######## ############# #####################
## Ready

package provide vc::fossil::import::cvs::pass::import 1.0
return

Changes to tools/cvs2fossil/lib/c2f_prev.tcl.

387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
...
406
407
408
409
410
411
412


413
414
415
416
417








418
419
420
421









422
423
424
425
426
427
428
...
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468

469
470
471
472
473
474
475
476
477
478


479
480
481
482


483
484
485
486
487
488
489
490
491
492
493


494





495
496
497
498
499
500
501
502
503
504
505
506
507
508
509

510
511
512
513
514
515
516




517
518
519
520
521
522
523
...
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
	log write 0 csets $ftr

	if {!$kill} return
	trouble internal "[$self str] depends on itself"
	return
    }

    method pushto {sv repository date} {
	upvar 1 $sv state

	# Generate and import the manifest for this changeset.
	#
	# Data needed:
	# - Commit message               (-- mysrcid -> repository meta)
	# - User doing the commit        (s.a.)
	#
	# - Timestamp of when committed  (command argument)
................................................................................
	#   will use the empty base revision as parent.
	#
	# - List of the file revisions in the changeset.

	struct::list assign [$myproject getmeta $mysrcid] __ branch user message
	struct::list assign $branch __ lodname



	# Perform the import. As part of that we determine the parent
	# we need, and convert the list of items in the changeset into
	# uuids and printable data.

	set uuid [Updatestate state $lodname \








		      [$repository importrevision [$self str] \
			   $user $message $date \
			   [Getparent state $lodname $myproject $myitems] \
			   [Getrevisioninfo $myitems]]]










	# Remember the whole changeset / uuid mapping, for the tags.

	state run {
	    INSERT INTO csuuid (cid,   uuid)
	    VALUES             ($myid, $uuid)
	}
................................................................................
	    AND    F.fid = R.fid     -- get file of revision
	}]] {
	    lappend revisions $frid $path $fname/$revnr
	}
	return $revisions
    }

    proc Getparent {sv lodname project items} {
	upvar 1 $sv state

	struct::list assign [Getisdefault $items] isdefault lastdefaultontrunk

	log write 8 csets {LOD    '$lodname'}
	log write 8 csets { def?  $isdefault}
	log write 8 csets { last? $lastdefaultontrunk}

	foreach k [lsort [array names state]] {
	    log write 8 csets {    $k = $state($k)}
	}

	# See (a) below, we have to remember if the changeset is last
	# on vendor branch also belonging to trunk even if we find a
	# parent in the state. The caller will later (after import)
	# make us the first trunk changeset in the state (See (**)).

	if {$lastdefaultontrunk} {
	    set state(:vendor:last:) .
	}


	# The state array holds for each line-of-development (LOD) the
	# last committed changeset belonging to that LOD.

	# (*) Standard handling if in-LOD changesets. If the LOD of
	#     the current changeset exists in the state (= has been
	#     committed to) then the stored changeset is the parent we
	#     are looking for.

	if {[info exists state($lodname)]} {
	    return $state($lodname)


	}

	# If the LOD is not yet known the current changeset can either
	# be


	# (a) the root of a vendor branch,
	# (b) the root of the trunk LOD, or
	# (c) the first changeset in a new LOD which was spawned from
	#     an existing LOD.

	if {$isdefault} {
	    # In case of (a) the changeset has no parent, signaled by
	    # the empty string. We do remember if the changeset is
	    # last on the vendor branch still belonging to trunk, for
	    # the trunk root.
	    return {}


	}






	if {$lodname eq ":trunk:"} {
	    # This is case (b), and we also can be sure that there is
	    # no vendor branch changeset which could be our
	    # parent. That was already dealt with through the
	    # :vendor:last: signal and code in the caller (setting
	    # such a changeset up as parent in the state, causing the
	    # standard LOD handler at (*) to kick in. So, no parent
	    # here at all.
	    return {}
	}

	# Case (c). We find the parent LOD of our LOD and take the
	# last changeset committed to that as our parent. If that
	# doesn't exist we have an error on our hands.


	set plodname [[[$project getsymbol $lodname] parent] name]

	log write 8 csets {pLOD   '$plodname'}

	if {[info exists state($plodname)]} {
	    return $state($plodname)




	}

	trouble internal {Unable to determine changeset parent}
	return
    }

    proc Getisdefault {revisions} {
................................................................................

	# TODO/CHECK: look for changesets where isdefault/dbchild is
	# ambigous.

	return [list $def [expr {$last ne ""}]]
    }

    proc Updatestate {sv lodname uuid} {
	upvar 1 $sv state

	# Remember the imported changeset in the state, under our
	# LOD. (**) And if the :vendor:last: signal is present then
	# the revision is also the actual root of the :trunk:, so
	# remember it as such.

	set state($lodname) $uuid
	if {[info exists state(:vendor:last:)]} {
	    unset state(:vendor:last:)
	    set state(:trunk:) $uuid
	}

	return $uuid
    }

    typemethod split {cset args} {
	# As part of the creation of the new changesets specified in
	# ARGS as sets of items, all subsets of CSET's item set, CSET
	# will be dropped from all databases, in and out of memory,
	# and then destroyed.
	#
	# Note: The item lists found in args are tagged items. They







|
<
<







 







>
>




|
>
>
>
>
>
>
>
>
|
|
<
<
>
>
>
>
>
>
>
>
>







 







|
<

<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
>
|
|



|
|

<
<
>
>


<
<
>
>
|
|



|
<
<
<
<
<
>
>
|
>
>
>
>
>

<
<
<
<
<
<
<
<
|


|
<
<
>





|
|
>
>
>
>







 







<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<
<







387
388
389
390
391
392
393
394


395
396
397
398
399
400
401
...
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427


428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
...
455
456
457
458
459
460
461
462

463



















464
465
466
467
468
469
470
471
472


473
474
475
476


477
478
479
480
481
482
483
484





485
486
487
488
489
490
491
492
493








494
495
496
497


498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
...
525
526
527
528
529
530
531

















532
533
534
535
536
537
538
	log write 0 csets $ftr

	if {!$kill} return
	trouble internal "[$self str] depends on itself"
	return
    }

    method pushto {repository date rstate} {


	# Generate and import the manifest for this changeset.
	#
	# Data needed:
	# - Commit message               (-- mysrcid -> repository meta)
	# - User doing the commit        (s.a.)
	#
	# - Timestamp of when committed  (command argument)
................................................................................
	#   will use the empty base revision as parent.
	#
	# - List of the file revisions in the changeset.

	struct::list assign [$myproject getmeta $mysrcid] __ branch user message
	struct::list assign $branch __ lodname

	log write 2 csets {Importing revision [$self str] on $lodname}

	# Perform the import. As part of that we determine the parent
	# we need, and convert the list of items in the changeset into
	# uuids and printable data.

	struct::list assign [Getisdefault $myitems] isdefault lastdefaultontrunk

	log write 8 csets {LOD    '$lodname'}
	log write 8 csets { def?  $isdefault}
	log write 8 csets { last? $lastdefaultontrunk}

	set lws  [Getworkspace    $rstate $lodname $myproject $isdefault]
	$lws add [Getrevisioninfo $myitems]

	set uuid [$repository importrevision [$self str] \
		      $user $message $date \


		      [$lws getid] [$lws get]]

	# Remember the imported changeset in the state, under our
	# LOD. And if it is the last trunk changeset on the vendor
	# branch then the revision is also the actual root of the
	# :trunk:, so we remember it as such in the state.

	$lws defid $uuid
	if {$lastdefaultontrunk} { $rstate new :trunk: [$lws name] }

	# Remember the whole changeset / uuid mapping, for the tags.

	state run {
	    INSERT INTO csuuid (cid,   uuid)
	    VALUES             ($myid, $uuid)
	}
................................................................................
	    AND    F.fid = R.fid     -- get file of revision
	}]] {
	    lappend revisions $frid $path $fname/$revnr
	}
	return $revisions
    }

    proc Getworkspace {rstate lodname project isdefault} {





















	# The state object holds the workspace state of each known
	# line-of-development (LOD), up to the last committed
	# changeset belonging to that LOD.

	# (*) Standard handling if in-LOD changesets. If the LOD of
	#     the current changeset exists in the state (= has been
	#     committed to) then this it has the workspace we are
	#     looking for.



	if {[$rstate has $lodname]} {
	    return [$rstate get $lodname]
	}



	# If the LOD is however not yet known, then the current
	# changeset can be either of
	# (a) root of a vendor branch,
	# (b) root of the trunk LOD, or
	# (c) the first changeset in a new LOD which was spawned from
	#     an existing LOD.

	if {$isdefault || ($lodname eq ":trunk:")} {





	    # For both (a) and (b) we have to create a new workspace
	    # for the lod, and it doesn't inherit from anything.

	    # Note that case (b) may never occur. See the variable
	    # 'lastdefaultontrunk' in the caller (method pushto). This
	    # flag can the generation of the workspace for the :trunk:
	    # LOD as well, making it inherit the state of the last
	    # trunk-changeset on the vendor-branch.









	    return [$rstate new $lodname]
	}

	# Case (c). We find the parent LOD of our LOD and let the new


	# workspace inherit from the parent's workspace.

	set plodname [[[$project getsymbol $lodname] parent] name]

	log write 8 csets {pLOD   '$plodname'}

	if {[$rstate has $plodname]} {
	    return [$rstate new $lodname $plodname]
	}

	foreach k [lsort [$rstate names]] {
	    log write 8 csets {    $k = [[$rstate get $k] getid]}
	}

	trouble internal {Unable to determine changeset parent}
	return
    }

    proc Getisdefault {revisions} {
................................................................................

	# TODO/CHECK: look for changesets where isdefault/dbchild is
	# ambigous.

	return [list $def [expr {$last ne ""}]]
    }


















    typemethod split {cset args} {
	# As part of the creation of the new changesets specified in
	# ARGS as sets of items, all subsets of CSET's item set, CSET
	# will be dropped from all databases, in and out of memory,
	# and then destroyed.
	#
	# Note: The item lists found in args are tagged items. They