Stability of releases & Beta channel

[gbcatrinoiu]

Hey everyone,

First of all, thank you for always being quick to release fixes! However, there’s a concern many of us frontend developers working in e-commerce agencies have been discussing over the past few weeks, if not months.

Unfortunately, the quality of recent releases seems to have declined, causing significant issues in production stores and, perhaps even more critically, impacting the theme development process. We’ve encountered instances where the tool breaks down every few minutes, JSON templates are lost or overwritten, sometimes unrelated errors to the actual cause, and the issues only get resolved through hotfix releases (which we appreciate, by the way!).

Since this is the only tool available for developing and deploying themes, it’s essential to have a stable version that functions reliably. Would it be possible to establish a beta channel where new features can be pre-released and tested? I’d be more than happy to use the beta channel as my primary option but still have the ability to switch to a stable version in the event of a major bug.

Looking forward to hearing your thoughts on this. Thanks!

[lesterdefreitas]

Seconded! Thanks @gbcatrinoiu for proposing this.

[karreiro]

Hey @gbcatrinoiu,

Thank you for this kind and honest feedback ❤️

there’s a concern many of us frontend developers working in e-commerce agencies have been discussing over the past few weeks, if not months.

We totally understand this. Shopify CLI for themes was not in the state we'd liked in the last few months.

Having a Ruby process being invoked by the TypeScript CLI was something we needed to do in the past to keep Themes and Apps being distributed in the same package. While that architecture allowed both CLIs to be distributed together, it introduced some fragile points exposed by the challenging setup that developers were facing and other issues (that seem unrelated, but are not).

In Shopify CLI 3.67, we fixed that issue with the new TypeScript implementations for all theme commands, and we're glad that won't be a problem in future releases.

Unfortunately, the quality of recent releases seems to have declined, causing significant issues in production stores and, perhaps even more critically, impacting the theme development process. We’ve encountered instances where the tool breaks down every few minutes, JSON templates are lost or overwritten, sometimes unrelated errors to the actual cause, and the issues only get resolved through hotfix releases (which we appreciate, by the way!).

With that previous piece of context in mind, we may notice the recent issues have a different root cause than the unstable development experience theme developers had in the last few months.

We have the entire Themes CLI running in TypeScript now and some edge cases emerged from that and have been addressed (issues coming from how regexes work differently between Ruby and TypeScript, development server backward-compatibility, etc).

Before releasing 3.67, we knew that friction wouldn't be ideal, so to mitigate that, we released in advance a dev preview version of the CLI. We asked for some feedback in advance via email and our Partners Slack. We recognize we could have done a better job in getting more attention from theme developers in that process. Still, that's a process that looks very much like the "Beta channel" this issue proposes.

We also released the new implementation with a --legacy flag as an alternative to allow theme developers to run the former version of the CLI without the inconvenience of a downgrade.

Would it be possible to establish a beta channel where new features can be pre-released and tested? I’d be more than happy to use the beta channel as my primary option but still have the ability to switch to a stable version in the event of a major bug.

We learned from the dev preview process that collecting feedback about beta releases didn't work so well. Instead of doing that, we're adopting the practice of sharing custom builds with reporters when we fix issues that are large or have a less straightforward nature.

This helps maintainers and theme developers on validating fixes frequently. Also, reporters are generally more involved with issues, so that might be a bit more effective than the process of sharing an entire beta build (as we did during the dev preview).

Looking forward to hearing your thoughts on this. Thanks!

Thanks again @gbcatrinoiu, and @lesterdefreitas, for sharing these thoughts. I hope this message provides some clarity about where we were, where we are now, and the future releases.

If you ever notice any issue with Shopify CLI, please report them here, and we'll look into a fix or enhancement as soon as possible.

We had a lot of moving parts in the last few weeks, but we expect to have our tooling in a much better stage now that we've been able to address those core infrastructure issues that now are part of the past of Themes CLI.

[rcasimmons]

@karreiro

What about if we opened up the platform for third-party tools so developers have more choice on the amount of tools?

I get a first-party tool is the best case scenario but having more tools to interact with themes etc. would be much better in cases where developers can't use the first-party option.

I'm still on the opinion that we should have stable, beta and alpha channels like @gbcatrinoiu / @lesterdefreitas stated - something that's industry standard usually.

It just doesn't strike confidence that we as developers have to report these issues in an already closed ecosystem.

[mgmanzella]

@rcasimmons @gbcatrinoiu @lesterdefreitas

👋 really appreciate bringing feedback here, to reiterate what @karreiro shared above:

Before releasing 3.67, we knew that friction wouldn't be ideal, so to mitigate that, we released in advance a dev preview version of the CLI.

^ Had an open beta to try out the Typescript implementation of theme commands

We also released the new implementation with a --legacy flag as an alternative to allow theme developers to run the former version of the CLI without the inconvenience of a downgrade.

^ Made the Typescript implementation the default for theme commands with the option to opt out with a --legacy flag

So, we did in fact have a beta channel. Knowing this, we would love your feedback on how we could've done this better (visibility wise we probably didn't do enough to promote the open beta).

We’ll also be having a twitter spaces AMA for Liquid Devs on 10/16 at 12 PM ET, if you want to bring those thoughts or more ideas about Liquid DX 🙏
https://x.com/ShopifyDevs/status/1844408172007784701

What about if we opened up the platform for third-party tools so developers have more choice on the amount of tools?

@rcasimmons definitely bring this to the AMA tomorrow if you're free

[github-actions]

This issue seems inactive. If it's still relevant, please add a comment saying so. Otherwise, take no action.
→ If there's no activity within a week, then a bot will automatically close this.
Thanks for helping to improve Shopify's dev tooling and experience.

P.S. You can learn more about why we stale issues here.

[karreiro]

Thanks again for these comments, @gbcatrinoiu @rcasimmons!

We've been heavily relying on the use of snapshots [1] [2] [3] as a strategy to shorten the feedback loop with CLI users and have more stable releases.

In the future, when larger changes are introduced, we're going to establish a temporary Beta channel as suggested, so we can globally validate and test the new implementation with CLI users. It should be similar to the one we did with the dev preview; however, we're going to invest even more in communication, so we can engage more developers there (we understand it wasn't the case in the 3.67 release).

I'm closing this issue for now. Also, I'd like to publicly thank you, @isaacroldan, for adding the /snapit into the repository -- as I mentioned, that was instrumental in shortening our feedback loop with theme developers.