Fossil Forum

Seeking a cradle-to-grave development platform
Login

Seeking a cradle-to-grave development platform

Seeking a cradle-to-grave development platform

(1) By Richard Hipp (drh) on 2019-07-09 13:37:45 [link] [source]

Konstantin Ryabitsev as a recent blog post that includes the following text (emphasis added):

[U]sing Git..b obviously introduces both a single point of failure and a single point of trust. Git repositories may be decentralized, but commits are merely the final product of a lot of developer back-and-forth that ends up walled-in inside the beautiful Git..b garden. You can export your project from Git..b, but very few people bother to do so, and almost nobody does it on a regular basis.

If a maintainer steps away and all development moves to a different fork, the project loses part of its history that is not committed to git, because all its issues, CI test results, pull requests and conversations are now split between the old fork and the new fork. If the original developer has a personal crisis and purges their original repository, that part of the project history is now forever gone, even if the code remains.

Furthermore, if you've been around for a while, you've seen beautiful gardens come and go. Before Github there was Sourceforge, which at some point poisoned its beautiful wells by bundling adware with binary downloads. Google Code has come and gone, like most Google things do. Github has seen a significant exodus of projects to Gitlab after it got acquired by Microsoft, and there's certainly no guarantee that Gitlab won't be acquired by some other $TechGiant looking to spruce up its open-source community image.

...

I think it's way past due time for us to come up with a solution that would offer decentralized, self-archiving, fully attestable, “cradle-to-grave” development platform that covers all aspects of project development and not just the code. It must move us away from mailing lists, but avoid introducing single points of trust, authority, and failure.

It sounds to me like Mr. Ryabitsev is looking for Fossil. Perhaps one or more of the readers in this forum could contact him and let him know about it. I'm guessing that Konstantin is a Git partisan and is unlikely to switch to Fossil for version control, but maybe he could at least steal some ideas from Fossil that would help him devise his own solution.

(2) By Stephan Beal (stephan) on 2019-07-09 13:56:57 in reply to 1 [link] [source]

Perhaps one or more of the readers in this forum could contact him and let him know about it.

For those looking for how to contact him, the details are near the bottom of the post:

I realize that there is no way to leave a comment here, but you can reach out to me via either floss.social or by emailing me at mricon@kernel.org.

(3) By Warren Young (wyoung) on 2019-07-09 17:14:18 in reply to 1 [source]

(Hint: "Git..b" is "GitHub." Apparently that's a dirty word to the blog author. It took me a fair bit of time to figure that out. You're welcome. :) )

It sounds to me like Mr. Ryabitsev is looking for Fossil.

There's a whole lot in that blog post that Fossil doesn't do:

  1. Automatically seek out local repos given only a prefix of the project code or a QR encoding of that project code. "Local" means not just LAN but also PAN. We don't have to invent protocols for this. We can just reuse Zeroconf or similar.

  2. Phone app, serving both as client and server.

  3. A decentralized method to seek out other repositories. In our case, that would have to include not just other Fossil repositories but also GitHub and other sources, since we can't expect the rest of the world to provide Fossil repos for everything we'd want to reference from our own Fossil repos.

    The blog post is implicitly talking about the Scuttlebutt P2P protocol for this, not about a centralized package discovery service a la CPAN, npm, or NuGet. ("...the protocol's decentralized, mesh-like P2P replication...") So, ChiselApp won't solve this problem for us.

    Implicit in this might be a need for Git-style subrepos, so that such dependencies can be automatically resolved through cloning. How else does your local Fossil client notice that libsnafu has had 2 updates since the last version you built locally?

    And implicit in that is that you should be able to have multiple local Fossil repos that each independently reference a common third-party dependency, so you don't need to clone multiple times. If both your Fossil project and mine depend on OpenSSL, and I clone your Fossil project onto my system, Fossil shouldn't re-clone the OpenSSL repo, it should just reuse the one I've already got. Notice that we don't even have to depend on the same major version of OpenSSL, if the OpenSSL repo is a full clone: we just each check out a different branch, if necessary.

  4. Commit feedback. (The Tested-by and Signed-off-by stuff in the blog post.) I proposed an alternative mechanism for Fossil back during the initial discussions around Fossil Forums. I called it reply to checkin. It's only simple if you have the forum and the code in the same repo, but perhaps a distributed version of the same idea could also be made to work.

    I once also proposed a feature turning Fossil into a generic user-focused web database engine, so you could create custom structured data apps atop Fossil. Now imagine if you could use that to make structured comments on a commit. Instead of a free-form text reply to the commit, you could say you want to add a Tested-By reply, which would record your verdict from a pull-down list of choices similar to the default Resolution set for a Fossil ticket, plus your user login — see IdM stuff below — and any comments you have on it.

    This then gets you the occasionally wished-for mechanism for code review. You could have a bot that would automatically merge a branch down to its parent when the number of Tested-by comments on a commit reach some critical number, perhaps gated by user capabilities, so that it would ignore "Tested by bozo-the-clown."

    That bot could also take the resolution value into account: if a branch needs 2 "OK" resolutions to be committed but there are two "OKs" from trusted parties and one "Bad" from another, the branch isn't merged until the user who marked it "Bad" overrides it with an "OK," or until a superuser marks it "OK," which could be interpreted as "automatic 2 OKs and ignore all Bad."

    The blog article makes a distinction between "tested-by" and "signed-off-by." I have no strong feeling on that. The core idea is that you need a set of permissions and votes to make automatic decisions about commits/branches. The details we can argue about separately.

  5. "Mail clients routinely mangle structured data." Implicit in that observation is that this vision of a better DVCS include a method for untrusted users to send patches/changesets/bundles/etc. to the project without needing a user login. Ideas for this in Fossil have come up many times before. I think my push requests proposal summarizes it well.

    Keep in mind that not all projects rely on signed contributor agreements! Some projects are quite willing to accept drive-by patches as-is.

  6. "...sending any other kind of structured data is the wild west. Bot reports, automated issue notifications, etc, attempt to present data as both human- and machine-readable and largely fail at both." That's the structured Fossil database apps feature again. A CI system should be able to post results to a Fossil repo, for example, and a Fossil wiki article should be able to reference those results, reporting that 32 of 34 branches currently build on all platforms, and all 34 branches build on 3 of 4 test platforms, with links back to the CI output for the failure cases.

  7. "PGP signatures in email are mostly treated like noise." Isn't that Fossil's answer to the attestation problem? I don't see that Fossil solves the PGP key distribution problem.

  8. "Git..b and others can, too, become community bridges..." Okay, here the blog author starts inhaling powerful organic molecules and reasoning from the visions they produce in his mind. He's completely ignoring the driving forces that lead to platform lock-in.

    The only way we're going to see GitHub start feeding patches into a future distributed network of repositories is when (we may hope!) something comes along and turns it into the next Sourceforge, so they have to join the new ecosystem to survive. For now, it's the other way around: everyone else has to feed into GitHub if they want to be highly visible projects. (Thus fossil git export!)

    I'm including this point not to stomp on the blog author's dreams, but to sweep his Plan A aside so we can come up with a workable Plan B. I think we need to go back to point #5: some method to accept outside contributions into the project without requiring that the provider install Fossil, sign a contributor agreement, get a login on the central Fossil repo, etc.

    One such option would be to implement fossil git import in a way that allows the GitHub mirror to accept PRs that make their way back into the central Fossil project.

    Ultimately, I think you need several such methods. Not everyone will want to use GitHub and PRs, either, so you also need a method to accept plain old patches. And then there's the one who is willing to install Fossil, but not get a user login on the central repo, so you want a method for her Fossil to send yours a bundle, automatically, per my push requests proposal.


It's not explicitly discussed in the blog post, but I think you'd also eventually want the ability to delegate identity to a third party, so you don't need to rely on a single centralized Fossil repo to provide that. You only need to introduce a small change to the blog post's story to show the need: instead of cloning the repo from one smartphone to another, someone comes up to you at a conference and sends you a commit from his smartphone. Should the Fossil server on your smartphone accept it? There are mathematically robust ways of coming to a trustworthy "Yes" answer.

That "third party" could be a true third party IdM, or it might be a Fossil instance acting as an IdM, which gets included in syncs: "I cloned from 1.2.3.4, so I trust it as my IdM. If you want to push to me, you have to send me a token I can verify with the IdM service running on 1.2.3.4." That sort of transitive trust doesn't exist with Fossil today.

To solve that, you need to get into zero-knowledge proofs and such. And there you need a much better mathematician than me. :(


This blog post also focuses on the problems of the Linux kernel. My recollection is that the attempts to get the full history of the NetBSD project into Fossil failed on practical grounds, since opening the repo took far too long. (Many hours.)

I'd expect the Linux kernel to be an even harder problem. Yes, yes, I know, the Linux kernel isn't a whole OS, whereas NetBSD is, but the Linux kernel probably gets more commits per day than the NetBSD project does, hence more history, hence bigger repo.

Fossil's not taking on such projects until it solves that problem.


We've got a lot of work to do. :)

(4) By anonymous on 2019-07-09 18:23:07 in reply to 3 [link] [source]

I suspect "Git..b" is regex style shorthand for "Github" or "Gitlab" or "Gitzzb" or whatever...