I2P anonymous protocol want to use Fossil
(1) By ToB30rI2P (AnonymousMaybe) on 2018-10-17 10:58:57 [link] [source]
Hi There, Im from I2P community and our developers using Monotone to develop the famous anonymous protocol I2P (or the invisible internet project). Sadly monotone project died since 4 years ago, so we want to have alternative. our conditions to make the development software to work well (specially with I2P) , to have these criteria: - supports resume - key signed checkins - revocation of keys/trust function implementation - easy migration of mtn history into new system does Fossil support these options? if no, is it on the road map? if no, then why not? Thank You!
(2) By Stephan Beal (stephan) on 2018-10-17 13:11:28 in reply to 1 [link] [source]
Resume: fossil has no feature by that name. What is it?
Keys: fossil supports PGP-signed checkins but it doesn't provide any mechanism for verifying or revoking those keys. It simply supports signing of commit entries, and at all other times ignores the key wrapping signed commit entries (applying no logic to the key itself).
Migration: if monotone supports exporting using git's format then fossil could import it that way, otherwise you're out of luck unless you want to write a script which checks out each version from monotone and re-imports it into fossil.
(3) By Stephan Beal (stephan) on 2018-10-17 13:19:06 in reply to 2 [link] [source]
To answer the bottom part of your question: "does Fossil support these options? if no, is it on the road map? if no, then why not?"
Fossil has no road map, per se. Either someone is interested in a problem and starts implementing it, or not. There was a detailed discussion about signing of commits some years ago (back in 2011 or 2012, IIRC), but i don't recall what the consensus was. Unless someone takes an interest in the problem and implements it, it won't ever be supported.
As far as the lack of monotone migration goes: in the nearly 8 years i've followed fossil, i don't recall anyone ever bringing up monotone before (so direct-to/from-monotone export/import tool is highly unlikely). git's export format has, for better or worse, long since become the common denominator for converting between SCM systems.
(7) By ddumitriu on 2018-10-17 14:29:48 in reply to 3 [link] [source]
The documentation on Monotone's website has an entry for mtn git_export.
(4) By Warren Young (wyoung) on 2018-10-17 13:22:06 in reply to 2 [link] [source]
Resume: fossil has no feature by that name. What is it?
It means that Monotone can continue an interrupted synchronization attempt without re-downloading the parts it previously transferred.
Fossil currently doesn't have this because synchronizations are implemented using SQLite transactions: the whole transaction completes or none of it does. That's not to say that Fossil couldn't stash a partial sync somewhere and try to resume it; it just explains why it doesn't bother to do that at the moment.
My initial thought on implementing that is that it would have to store both hash values ("from" and "to") involved in the sync before attempting to resume, and throw away the saved partial sync data if on the second attempt the implicit "to" hash has changed. (e.g. Someone committed to trunk in the meantime.)
Is this really an important feature? How often are syncs interrupted in your world, AnonymousMaybe, and how much time is saved by resuming instead of restarting? In my world, about the only time a sync is interrupted is because I hit Ctrl-C, and most syncs are brief, because I sync often.
(5) By anonymous on 2018-10-17 13:51:06 in reply to 4 [link] [source]
I believe this is wrong. Unless the code has changed, Fossil does a commit at the end of each round-trip during a sync operation. If successful, it does not get rolled back. This means that if the sync operation is interrupted part-way through, one side will have an incomplete set of artifacts, however, the next time a sync operation is performed the missing artifacts will be synchronized without the need for the previous artifacts to
(6) By anonymous on 2018-10-17 14:09:37 in reply to 4 [link] [source]
Somehow my previous comment was incomplete---probably user error on my part. I meant to say that Fossil does have partial sync support already and that if a sync operation fails only those artifacts that were not committed successfully will need to be synchronized. If this is what is meant by Resume then Fossil has already met that requirement. Also, regarding how often sync operations are interrupted, if the I2P protocol is anything like other anonyous networks, interruptions can be a regular occurrence.
(9.1) By Warren Young (wyoung) on 2018-10-17 18:01:52 edited from 9.0 in reply to 6 [link] [source]
If this is what is meant by Resume then Fossil has already met that requirement.
That's my understanding, which is based on only brief mentions in the Monotone NEWS file and the I2P project's Monotone Guide.
Fossil does have partial sync support already
It does indeed appear that Fossil closes the transaction on receiving each block of data from the remote repository. (That's the last line in the "while (go)" loop that wraps the bulk of the code in the Fossil xfer loop, and there are no continue statements in that loop, so it happens on each iteration.)
When I said that the whole transaction completes or none of it does, I was thinking about checkins, not syncs.
(10) By anonymous on 2018-10-18 19:05:29 in reply to 9.1 [link] [source]
Just to be clear, this partial sync has nothing to do with cloning. A clone operation is still all-or-nothing as far as I know.
(11) By Richard Hipp (drh) on 2018-10-18 20:20:14 in reply to 10 [link] [source]
Incorrect. A clone is basically just a big sync, though there are some added optimizations that make it run faster.
Are you having problems? There could be a bug, perhaps, in one of the clone-sync optimizations that prevent it from restarting automatically.
(13) By anonymous on 2018-10-19 19:32:33 in reply to 11 [link] [source]
Perhaps I have been misunderstood; let me try to clarify as I believe we are using different meanings for "resume" or "restart". My definition is that if the fossil process exits before the "sync operation" completes only the uncommitted artifacts will have to be transmitted. Consider the following: What happens if I press ctrl-c after 3 round-trips of the clone (aka big sync) operation? Will I be left with a partially cloned repository that I can then simply complete with "fossil pull"? What happens if the network drops after 3 round-trips of a clone (aka big sync) operation? Will I end up with a partially cloned repository that I can bring up to date with "fossil pull"? In either case, the answer is no, and I will have to get all artifacts from the beginning. This is not true for sync as it commits the artifacts once per round-trip into the repository, so any interruption leaves me with a working repository but with missing artifacts, however, the next time I sync, those artifacts will then be sent. Demo: The first question: $ ./fossil clone http://localhost:8080/ clone.fossil ^Cund-trips: 3 Artifacts sent: 0 received: 25455 So this went 3 round-trips and produced a file called clone.fossil before being interrupted, now let's see what can be done with it: $ ./fossil pull -R clone.fossil SQLITE_NOTICE(539): recovered 14 pages from /tmp/clone.fossil-journal incorrect repository schema version: current repository schema version is "" but need versions between "2011-04-25 19:50" and "2015-01-24". run "fossil rebuild" to fix this problem $ ./fossil rebuild -R clone.fossil 100.0% complete... $ ./fossil pull -R clone.fossil Usage: ./fossil pull URL $ ./fossil pull -R clone.fossil http://localhost:8080/ Cannot figure out who you are! Consider using the --user command line option, setting your USER environment variable, or setting a default user with "fossil user default USER". cannot determine user $ ./fossil pull -R clone.fossil --user someone http://localhost:8080/ no such user: someone Even if I could get a user, there isn't anything here anyway: $ echo "SELECT count(*) FROM blob;" | ./fossil sql -R clone.fossil 0 Now, let's look at the second question where the fossil server drops off the network while cloning: $ ./fossil clone http://localhost:8080/ clone.fossil Round-trips: 3 Artifacts sent: 0 received: 25455 response truncated: got 3937540 bytes of 5000379 Clone done, sent: 1042 received: 19353451 ip: 127.0.0.1 server returned an error - clone aborted $ ./fossil pull -R clone.fossil repository does not exist or is in an unreadable directory: clone.fossil $ ls -l clone.fossil ls: clone.fossil: No such file or directory So, when cloning, if anything interrupts the clone, I will have to clone everything again as I do not end up with a Fossil repository that is partially complete and simply needs more artifacts. Under what conditions, then, will a clone "restart automatically"? However, with an already cloned repository, if I initiate a fossil pull/push, and either of the above mentioned situations happens, I will not have to pull/push those things that were already received, only those things that are missing. For example, here the server has 304 artifacts that my client doesn't know about, and part way through the pull, I use ctrl-c to interrupt the pull. Then I run pull again and instead of getting all artifacts that were previously received, I only get the remaining few that were missed: $ ./fossil pull -R clone.fossil Pull from http://localhost:8080/ ^Cund-trips: 4 Artifacts sent: 0 received: 275 $ ./fossil pull -R clone.fossil Pull from http://localhost:8080/ Round-trips: 1 Artifacts sent: 0 received: 29 Pull done, sent: 1406 received: 382624 ip: 127.0.0.1 Same is true for a push, if it is interrupted, or the connection drops, it will send just those missing artifacts: $ ./fossil sync Sync with http://cloner@localhost:8080/ Round-trips: 17 Artifacts sent: 113 received: 1 cannot connect to host localhost:8080 Sync done, sent: 1618293 received: 122102 ip: 127.0.0.1 $ ./fossil sync Sync with http://cloner@localhost:8080/ Round-trips: 13 Artifacts sent: 72 received: 1 Sync done, sent: 997145 received: 50918 ip: 127.0.0.1 Does this clarify? Maybe there is a bug here? If clone is supposed to "restart" in the face of a network hiccup, then this might be a bug. How many times should it retry? What about the other case? Even if clone retries multiple times to sync missing content before the Fossil process exits it still is not the same as sync unless clone can resume *after* the Fossil process has exited (for whatever reason) because sync can do so.
(8) By Richard Hipp (drh) on 2018-10-17 16:05:51 in reply to 1 [source]
I did not realize that Monotone had died. That is very sad. In an abstract sense, Fossil is a spin-off of Monotone. Many of the core ideas of Monotone are copied by Fossil, though Fossil is a completely fresh implementation, without any use or or reference to the monotone code.
For the first part of its life, SQLite was version-controlled using CVS and a wrapper project called CVSTrac to provide bug tracking and a sensible web interface. I recognized that this needed to change, and I searched for an alternative Version Control System (VCS). I looked at monotone. But at the time Monotone would not work as a CGI script - it needed to run as its own server on its own TCP port. That was a show-stopper for me, as I needed to integrate the VCS function into a larger website. I wrote to the Monotone project asking about the ability to support Monotone sync over HTTP, and they besically said that was a bad idea and that I should abandon. If Monotone had had the ability to sync over HTTP, I probably have just used Monotone for managing the SQLite source code. But because Monotone did not support sync over HTTP, I ended up writing Fossil to manage SQLite.
All that said, Monotone pioneered the idea that a "checkin" would be identified by a cryptographic hash of its content, which in turn became the key idea behind Fossil and Git and several other VCSes. So it is sad to hear that Monotone is no longer being maintained.
As for your other questions, I think others have answered them well. I will try to summarize:
"supports resume" - Don't know exactly what this means. The sync mechanism in Fossil is built around HTTP requests. A single sync might consist of multiple HTTP requests and responses. If they is the case, each request/response is independing of all the others, as it ought to be. Maybe this is what you mean by "resume".
"key signed checkins" - Checkins can be PGP clear-signed. I use this on the TH3 repository (a proprietary project). It works.
"revocation of keys" - Though PGP clear-signing of checkins is possible, I know of nobody other than myself who uses it. Nobody else seems to think it is important. As a consequence, not much infrastructure has grown up around it. In particular, there is currently no support for key management. However, the protocol is flexible enough that key management could be added, in a completely backwards compatible way, by a motivated developer.
"migration of monotone to Fossil" - If monotone can export to the git-fast-export protocol, then Fossil should be able import it. On the other hand, git-fast-export is not a well-defined protocol. Most knowledge of git-fast-export must be determined by experiment. So it is possible that Fossil might have issue importing from a git-fast-export file generated by Monotone. If that is a problem, we can probably fix it.
(12) By andygoth on 2018-10-19 15:41:56 in reply to 1 [link] [source]
Recently I wrote a post with a number of rambling ideas initially regarding private messages in the forum, then branching out into what it would take to implement that, then exploring other possibilities arising from that necessary infrastructure. In the process, I cover "key signed checkins" and "revocation of keys/trust function implementation". I'd be interested to get your comments, though please post in that thread rather than this one.