Checking GPS 🧭

[ashfame]

Our current direction is to build a browser extension and collaborate with the user to make sense of the HTML markup of their site and accordingly import those blobs of data into WordPress playground that’s running in the browser so that they can have a real-time preview right away.

Since we are parsing HTML in this approach it does not matter which platform the site is on and small sites can be liberated without much effort, at least that’s the idea. Markup context learnt from processing a single blog post for example is applied across all blog post pages as we crawl the website, until we need to prompt the user again for importing regular pages, navigation links etc.

At the end of this interaction with the user, WordPress playground takes over with the options of either exporting the site content or in future, directly streaming to a web host of choice to go live with an actual web host running WordPress on the server.

Classic problems with migrations + Solution (Our USP?)

Post migration, user almost always discover bits of data that doesn’t look right. Maybe it wasn’t parsed correctly or the support to handle some data wasn’t present when the migration was executed.

Examples of data that might need fixing:

• Gallery of images (imported as bunch of inserted images one by one).
• Nested levels of navigation links (lost nested context).
• Individual user ratings of a product didn’t get carried over, the final calculated one did.
• Precision of datetime lost and just date retained for published/modified timestamps.

We thought it would be interesting to offer the user a UI, within WordPress, even long after they have done the migration, which would give them some transparency on what the raw parsed data was and whether that data can be processed better now. This can come in the form of our newer version of native transformer logic or even a third party plugin that’s more fitting to transform that data better. This allows the user to choose re-transformation of the raw data once again, overwriting the live data hence improving the quality of data.

There is no time window within which user must act. This also incentives third parties to build better transformers for making the user opt them. And make it possible for users to submit fixtures for broken transformations as they themselves discover it. Based on how well this we could execute the idea, this could very well become the USP of the data-liberation tool.

Hybrid approach

While parsing of HTML approach has its benefits and a potential to work really well for small websites, let’s not forget about platforms which do offer exports for the site content. They are already providing structured data to reliably parse easily at scale and automate a lot of the migration work. Already built importers can also be leveraged in this hybrid approach where we process a lot of data automatically and then collaborate with user on just the needed bits, if at all.

So, most likely a matured version of migration tool would be based on a hybrid approach.

We would still be relying on the same UI/tool, that we are building here to process exported archives to give a head start and preview in the Playground.

Who are the users of this tool?

A good chunk, if not the biggest, of the WordPress ecosystem are users who runs a one-man show without any in-house team or an organization behind them. This tool would be very fitting to serve this segment of users. But what about the likes of folks/organizations who run serious businesses with in-house teams? Think WooCommerce store with 10K+ SKUs or Media publishing houses with tons of articles and comments.

Would they in all seriousness consider the reliability of a browser based tool, when their developers would be inclined to write code before they think? This would be a hard sell, though not impossible after having a proven track record down the road in a few years.

Deciding which segment we are going to serve or serve first is crucial to make design choices today.

Two UIs Concern + Premature implementation?

Paulo (@psrpinto) highlighted the concern around building 2 UIs when it’s already hard to build a good UI, just a single one. First UI that is the browser extension and Second UI that is shipped in the imported WordPress site itself. That discussion is happening on Github.

Important point to highlight is that this tool doesn’t need to be a browser extension, it can be a desktop app as well. In fact, we might choose it to be a desktop app since running a custom browser also alleviate some of the browser restrictions that we would run into, trying to fetch data from cross-origin domains.

Paulo expressed a very valid question of whether we should even be supporting the feature of re-running transformations this early on? And if/when, this should be built as a single codebase for this tool, with multiple targets for the UI. It can be browser extension, desktop app or a screen within WordPress itself. I mostly agree, considering we don’t have an entire team behind this idea until I start thinking about the responsibilities involved for both the UIs, because of which I am not sure.

UI Responsibilities

First UI has the responsibility of liberating the data from their existing website/platform. Second UI has the responsibility of re-running transformations to improve the quality of the data. How the second UI would look could be totally different from the first UI. A good analogy is visiting a doctor (Ops) and then going home with medicines (Post Ops).

We also don’t have clear picture of how the second UI will end up looking. Would this be a dashboard of sorts, that allows bulk processing? Would this be a meta widget along with the Gutenberg editor on edit screens? Or would this be a sidebar on top of the frontend of the website? And how would the data be even presented with options when the live data could have evolved further & simple overwrite is not going to cut it?

Hence, I think the first UI should not dictate the layout/experience of second UI, considering it’s much more harder to build the second UI. It appears to be a completely different UI with diff responsibility hence different codebase powering it. The only thing that would be shared across both the UIs are transformers, which hold the transformation logic of different blobs of data.

It appears to me that we would be trying to merge two different UIs (or Apps?) when it just needs to share a transformer package that will do the transformations.

We do want to strive for maintaining a good UX, and a single UI for both UIs tries to deliver that using a consistent theme (familiarity). But I believe that comes from second UI being more relatable to the WordPress UI, rather than the first UI.

Where should transformation logic reside?

Though an implementation detail but I thought would be worth mentioning since this has overlap with the realm of architecting the tool. It can very well exist on both JS or PHP and developed using tests, while leaving UI interactions only for the actual users. Initially, I thought JS would be faster to process a lot of imports but then it’s difficult to say so, considering PHP is also running under WebAssembly (Wasm) that might give it an edge.

The idea is also for users to submit fixtures with few clicks to Github, that they found are not transforming well. This would make for a quick feedback loop. I am also inclined to suggest PHP might be beneficial to attract contributions & how convoluted JS tooling can be

Considering second UI and how natively it should ties to WordPress UI (should we choose to), also demands transformations to mainly exist on PHP side.

[psrpinto]

Thanks for writing this post @ashfame, it documents many of the topics we have been discussing in the past days, and it makes sense to have this information in written form, so it can be discussed.

While parsing of HTML approach has its benefits and a potential to work really well for small websites, let’s not forget about platforms which do offer exports for the site content.

Completely agree with this. Our app should prioritize using structured data when available, and fallback to the HTML approach when there is no API we can call, or export we can parse.

this tool doesn’t need to be a browser extension, it can be a desktop app as well.

Agree, I think we could keep building it as a browser extension for now (as that's what we currently have working), and re-evaluate later whether we should instead provide it as a Desktop app, or whether we should provide both an extension and a Desktop app.

Additionally, we can also decide to embed it in a WordPress plugin, so that users can, for example, re-run transformations (more on this below) directly at their new site.

Who are the users of this tool?

• without any in-house team or an organization behind them
• organizations who run serious businesses with in-house teams

Deciding which segment we are going to serve or serve first is crucial to make design choices today.

I agree it's important that we identify our target users. I think organizations who have in-house teams would probably not be the target of our app.

However, I think our app should be built in a way that can scale to any size of site. I think we should prioritize small sites first, and address any usability or performance issues. Once the app is stable and addresses the needs of small sites, we can make it scale to larger sites.

One or two UIs

First UI has the responsibility of liberating the data from their existing website/platform. Second UI has the responsibility of re-running transformations to improve the quality of the data.
How the second UI would look could be totally different from the first UI.

Let's assume that indeed the UI for re-running transformations would have very different requirements than the UI for doing the initial import. (I'm not convinced that this would be the case, but let's proceed with that assumption for now, more on this later.)

The way I would address this would be to have dedicated screens that implement the needs of re-running transformations. In this scenario, there are screens that implement the flows for doing the initial imports, and there are screens that implement the flows for re-running transformations.

The screens for re-running transformations can be completely different than the ones for doing the initial import. But all screens would share a common look and feel, and behave in a consistent way.

I think this is what makes the most sense for the user: the app looks, feels and behaves in a consistent way regardless of that kind of operation the user is doing.

concern around building 2 UIs when it’s already hard to build a good UI, just a single one.

I reiterate this concern, for the reasons mentioned above (consistent look and feel throughout the whole flow) but also because it would require double the development and maintenance efforts, especially if the 2 UIs are built using different technologies.

Re-running transformations

Post migration, user almost always discover bits of data that doesn’t look right. Maybe it wasn’t parsed correctly or the support to handle some data wasn’t present when the migration was executed.

whether we should even be supporting the feature of re-running transformations this early on?

I'm still of the opinion that we shouldn't be focusing on this feature yet, because this is something that happens once the user has the new site running, and we haven't yet built the app that allows them to get to that stage.

However, we should not close any doors by making early architecture decisions that would prevent this feature from being implemented. Additionally, since this is also related to the one-or-two-UIs topic, I would like to discuss this further here.

The way I understand what re-running transformations means is: "imported content is wrong, it needs to become right". This process would be done in an automated fashion, where all blog posts (for example) that have been imported will get processed again so that the content that was wrong becomes right.

The user can encounter wrong content at two distinct stages:

1. While they are doing the import, i.e. while the site is still only in Playground
2. Later, when they already have migrated the site to its new permanent location

Since the user will have a need to fix the wrong content already when they are doing the import, the feature of re-running transformations must also be present in the UI that they are using to do the import.

Thus why I mentioned earlier that I'm convinced that the UI needs for pre and post import are very similar. It would be the same UI, pointed at a different backend.

If we don't want the user to need to go back to the browser extension (or Desktop app) after they migrated to their new site, the app we're building can be embedded in the WordPress plugin so they can run it directly on the new site. For example, there's a page in their admin that just loads the exact same app they used to do the import, but without the "side panes" (source site and preview). However, I think we should focus on building the browser extension first, and consider this possibility later.

[ashfame]

However, I think our app should be built in a way that can scale to any size of site. I think we should prioritize small sites first, and address any usability or performance issues. Once the app is stable and addresses the needs of small sites, we can make it scale to larger sites.

Not that I don't agree but what I meant to say was, if our target users are small to medium content sites only, then just parsing HTML would be enough, since single approach would work for everyone. No need to worry about consuming exported archives from different platforms.

Big sites and e-commerce stores are surely not ok just liberating the content available on frontend side + have better tooling to export data out, so could be they don't end up using this tool at all.

Paulo: The screens for re-running transformations can be completely different than the ones for doing the initial import. But all screens would share a common look and feel, and behave in a consistent way.

This is exactly what I reference in my original post:

Ashish: We do want to strive for maintaining a good UX, and a single UI for both UIs tries to deliver that using a consistent theme (familiarity). But I believe that comes from second UI being more relatable to the WordPress UI, rather than the first UI.

The reason why I believe the second UI would be more native/relatable to WordPress UI rather a dedicated screen in the liberation tool is when user finds some data that is wrong or need fixing, we prompt them where they would go for editing it i.e. WordPress admin screen itself, that we can transform this better now, should you choose it and the option of doing it for all the imported content, once the fix on a particular piece of content looks good. It would be quite limiting to have a dedicated page or screen, without which user would not even realize "a fix" is available and locating that piece of data within that dedicated page or screen.

I think, it would also be easier to attract contributions if submitting fixtures or patching transformations would just be in the context of a WordPress plugin, rather than the liberation tool itself?

A single UI gives up the benefits that I describe above + I am not convinced it would mean decreased dev+maintenance effort from our side, since now the tool must interact with a live WordPress install rather than just the playground, which we would also have to support, with more code on the WordPress side for that communication.

[akirk]

Post migration, user almost always discover bits of data that doesn’t look right. Maybe it wasn’t parsed correctly or the support to handle some data wasn’t present when the migration was executed.

Thank you, @ashfame for bringing those good examples. Also in the other comment your wrote:

The reason why I believe the second UI would be more native/relatable to WordPress UI rather a dedicated screen in the liberation tool is when user finds some data that is wrong or need fixing, we prompt them where they would go for editing it i.e. WordPress admin screen itself, that we can transform this better now, should you choose it and the option of doing it for all the imported content, once the fix on a particular piece of content looks good. It would be quite limiting to have a dedicated page or screen, without which user would not even realize "a fix" is available and locating that piece of data within that dedicated page or screen.

I think, it would also be easier to attract contributions if submitting fixtures or patching transformations would just be in the context of a WordPress plugin, rather than the liberation tool itself?

I had already started writing this response but overall, I agree with @ashfame:

Two UIs vs. Fixing Things

I am hearing from this discussion doubts about building two UIs that do similar things, and this like comes from the fact that we haven't yet painted a strong picture on how the user flow would work. What @psrpinto described in the Project scope discussion and we all agreed on, is a bit of a golden path. So let's talk about what the user would do if things go wrong (and, as Ashish described we better assume that this is the default case, there is always something to improve).

I believe that there are two schools of liberating sites:

1. A site migration is a chance to update the site's design and layout, so moving the data (and associated media) in a structured way is the most important thing.
2. A pixel-perfect site migration.

While it will be interesting to preserve as much of the layout as possible, this can be worked on as a second stage, building on migrating all the data. So we'll focus on the school no 1.

Typical Flow

(Somewhat recapping the Project Scope discussion)

The user...

1. goes to the site to liberate.
2. opens the Try WordPress extension and start importing.
3. then helps the extension with importing by answering its questions about where structural elements are (e.g. navigation, page title, product price etc.)
4. now browses their new WordPress site and see how well things went.
5. identifies areas that have not been imported well and fixes them.
6. migrates their website out of Playground to a hoster.

Let's focus on items number 3 and 5 here.

Helping the Extension with Importing

This is browser extension UI with a view of their website, and the extension (2 panes), potentially also their WordPress visible (3 panes), for example, for viewing previews.

The UI is meant for an interactive experience, where the user can be asked questions, and they respond and move the import process further. They can also lead the extension to further content types by navigating there and resuming the import process from there.

Identifying Content that has not been Imported Correctly

This would happen in WordPress. They would browse their site and see things that don't work well. We need to guide them how to fix it, maybe through a button in the master bar.

I believe that we need to help the user identify how these problematic things need to be fixed:

1. Small things can be fixed using tools provided by WordPress, for example with the block editor: applying new patterns, moving images. We should guide the user to using these tools.
2. Big things that require re-parsing. See below.

Generally, where would progratically fixing something be faster than doing it manually.

Fixing Content UI

I belive that the Data Liberation extension should give an overview page of what was imported. It would be good if we could say for each data fragment where it came from and what we turned it into.

Practically, these fragments are custom post types. For each post, we can offer some buttons:

1. Reparse: This would be when the code has been updated. This would result in updating (or delete + a new insert) the target pieces.
2. Report: this would open a Github issue that contains the post type data so that we can identify the problems in code and fix it, reusing the provided content as fixture.
3. Inspect: this would give a detailed view of the content that was parsed from. Maybe the user can extract more content from there without having to visit the source site anymore.

Thus, I don't see the UIs on the extension being similar to the one in WordPress. We want to teach people to use WordPress to fix things and just offer to reparse where it makes sense.

We need to remember that the extension is just a helper to transition to WordPress. Initially, I had proposed for that UI to already be inside WordPress.

There should only be one pass at extracting data. From then on, everything should be done inside WordPress. It is a capable tool for doing so.