|Title:||Alternative to reconstruct for large imports|
|Last Modified:||2010-12-21 13:52:10|
|Version Found In:||aab38ef02f|
The current "fossil reconstruct" interface works well for small scale operation, but having one file per artifact (even across multiple directories) brings a lot of overhead.
Attached is a patch to implement a variant called reconstruct-sql, which takes the artifacts from a sqlite3 database. This makes it a lot easier to deal with, especially if the processing is already done in a higher level language. Compressing the artifacts helps to cut down required disk space a lot, too.
Related questions would be if explicitly tagging artifacts as manifests or including the hash or temporal order would be used to speed up the import. A lot of time is currently still spend in content_put. I haven't analyzed yet whether it is the work required for delta processing or whether it is the plain issue of having very large manifests (e.g. a working copy of pkgsrc is 60k files).
drh added on 2010-10-06 00:06:41:
Or, how about an --incremental option to "fossil reconstruct". With --incremental, you don't have to populate the inport directory with all of your files all at once. Start with some subset, import them, clear out the directory and refill it with new files, import again, repeat until done.
anonymous added on 2010-10-06 09:40:24:
anonymous claiming to be Joerg Sonnenberger added on 2010-11-05 14:09:37:
anonymous claiming to be Joerg Sonnenberger added on 2010-11-05 15:28:52:
anonymous claiming to be Joerg Sonnenberger added on 2010-12-21 13:52:10:
- rebuild.diff [download] added by anonymous on 2010-10-05 15:40:31. [details]