Ahead of Self-hosted LiveSync 1.0

Hello, everyone. vrtmrz (a.k.a., Kiminobu) here.

There is something I think I should tell you ahead of Self-hosted LiveSync 1.0.

I felt that I needed to talk about this seriously, so parts of this post have ended up a little formal. I hope you will bear with me until the end.

And for everyone (especially Japanese readers) wondering what happened to the three-part series I started last time: do not worry. I promise I will finish it.

The major review leading up to 1.0 began when the Community Directory Review pointed out a large number of problems. The score was, honestly, spectacularly bad.

Until now, I had committed generated type definitions to the repository because submodules were not included in the review, and added ESLint exclusions for the CLI. At the time, both were practical workarounds that allowed the build and review to succeed. As those workarounds accumulated, however, I realised that rather than solving the underlying structural problems, they were actually making them harder to see.

The enormous size of this release does not mean that this is how I intend to handle releases from now on. If anything, I think it shows just how many problems I had put off dealing with.

I often described some of them as 'inherently difficult', but I have seriously reflected on the fact that this was not a particularly admirable attitude. In effect, I was asking all of you to bear the burden of that complexity. While actually running the tests, I found myself thinking, 'Oh, this might be completely broken', more times than I would like to admit.

I also rewrote the troubleshooting documentation from the ground up. Someone pointed out, not unfairly, that parts of it simply were not true.

And yes: I have made extensive use of AI for this work, including investigation, regression-test development, verification, and the fixes themselves. This was possible because, through OpenAI's generosity, I was given an opportunity to try Codex for Open Source on this project.

That said, I do not treat AI output itself as evidence of quality. As before, I remain responsible for deciding what to change, reviewing it, and publishing it -- or at least, that is what I intend.

I have deliberately kept the commit history visible. I have also kept change authority and publication operations behind clearly separated gates, while combining unit tests, integration tests, end-to-end tests using a real Obsidian instance, and verification on physical devices. Incidentally, if you are interested in the specific instructions I use, you can find them in my dotfiles.

What I find reassuring about working with AI is that I can ask it to investigate overlooked assumptions and past decisions as many times as necessary. As I mentioned earlier, that process uncovered a remarkable number of documentation defects, confusing setup paths, obsolete workarounds, and discrepancies between the implementation and its explanation.

Honestly, it was bad.

I have also substantially reconsidered reusability.

I opened Issue #1 in livesync-commonlib at almost exactly this time of year in 2022.
Reading the English I wrote back then makes me feel that I have grown quite a lot.

Since then, people have occasionally asked whether Commonlib would be published as an npm package. At the time, I thought it was 'too niche to publish globally'. As the number of projects and use cases grew, however, I began to wonder whether that reasoning still held water.

It was not something I could address immediately, either. I ended up dragging my feet on it for a very long time, but with Self-hosted LiveSync 1.0, Commonlib has finally been published as an npm package -- four years later.

There were several reasons why building Self-hosted LiveSync and its related projects was difficult. Treating Commonlib as a submodule, and then reaching across that boundary to use its internal files directly, was one of the larger problems.

This time, I made that boundary explicit and moved ownership of the translation catalogue to Self-hosted LiveSync. This has improved both stability and independence. I also reversed a few dependency relationships, which means translation contributions can now be accepted in the single LiveSync repository instead of being spread across several places. Adding entirely new messages is a separate matter, but I hope this will make translation pull requests much less fragmented.

Another major change is Fancy Kit.

In addition to Self-hosted LiveSync, I develop several other Obsidian plug-ins, including TagFolder and Differential ZIP Backup. Each had similar UI components, dialogues, testing helpers, and integration code for Obsidian. They were also receiving very similar findings from the Community Directory Review.

At first, I addressed those findings separately in each project. Eventually, though, I realised that continuing to fix everything independently would only cause us to repeat the same problems.

As part of this work, I therefore organised the parts with genuinely reusable boundaries into Fancy Kit and published them as npm packages. I did not try to make everything common. I limited it to the parts whose responsibilities could be clearly explained and tested across multiple plug-ins. Well, but, the one and only value of this is that it has been combat-proven. That’s important too, isn’t it?

I am hoping that some of you may find them useful as well.

Oh, and automated testing has expanded considerably.

For some time, Self-hosted LiveSync has had end-to-end tests which launch a real Obsidian session. Existing Obsidian automation tools did not fit some of this plug-in's rather niche requirements very well: replacing the plug-in, restarting Obsidian, or synchronising across multiple Vaults. So I had been building those capabilities little by little.

This time, I reorganised that work into a testing foundation which can be run continuously. Once I started using it in earnest, I discovered that it was missing not just a few features, but a great many of them.

The tests now cover not only CouchDB, but also other synchronisation methods such as Object Storage and P2P. They include scenario-based tests for Setup URIs, conflict resolution -- which has been an ongoing concern for a while -- and, as far as automation can reasonably cover it, frequently discussed features such as Hidden File Sync.

The test tools can also capture screenshots, and I am now using them to build the documentation. This was partly a side effect of the work, but having screenshots also made it much easier for me to review what had happened afterwards.

Of course, simply having more tests is not automatically better. Tests have costs of their own.

From here, I need to make it clearer what each test actually guarantees, and remove or reorganise tests which no longer serve a purpose or which only exercise states that cannot occur in practice. I think the current suite is deliberately excessive because, this time, the priority was to verify as much as possible.

So please do not be alarmed. I genuinely do think I went a little too far.

I believe this work has repaid a substantial portion of the technical debt that had been accumulating for a long time. There will, of course, still be many problems. But we now have a foundation which should make future changes smaller, easier to understand, and easier to verify.

I also intend to reduce the number of breaking changes compared with the past.

For everyone who currently has a pull request open, I may need to ask you to review the source again and, in some cases, adjust your contribution. I am sorry that I have already kept your valuable work waiting for so long.

As always, I look forward to your issues, pull requests, translations, testing, and feedback.