Rewriting the internals of Leon

[svenjacobs]

I'm currently rewriting the internals of Leon. Instead of sanitizers based on regular expressions, the idea is that Leon creates an internal representation of an address that contains the domain, path and query parameters. Sanitizers can then declare which parameters should be removed. This change will also allow the UI to show all parameters so that users can manually select or deselect parameters for removal. Hence Leon will also work with "unknown" addresses.

Until this refactoring is completed, I will not add any new sanitizers because they need to be rewritten anyway.

[serrq]

Okay. For Christmas is ready the new Leon?

[TPS]

Can we use import java.net.URL or import android.net.Uri url parser instead?

https://github.com/ethauvin/urlencoder#why-not-simply-use-javaneturlencoder denotes some errors in those, also.

[svenjacobs]

Can we use import java.net.URL or import android.net.Uri url parser instead?

https://github.com/ethauvin/urlencoder denotes some errors in those, also.

Thanks for the link. That's an interesting alternative for URLDecoder.

Regarding Url.kt, the intention was to have a class that splits an URL into its components so that these components can than be used by cleaning rules.

[rsb-23]

Maybe there's a confusion, l am not referring to Url encoder/decoder.
Hence rephrasing my query :

• java.net.URL splits an url into its components. (what Url.kt is intended)
• Since java.net.URLDecoder is already used in project, doesn't it mean java.net.URL is also compatible to Kotlin Multiplatform?
• Shared Url encoder project does not mention any issue with the url parser.

[TPS]

• Shared Url encoder project does not mention any issue with the url parser.

@rsb-23 Maybe I misunderstand your details, but here's what the relevant section at the above repo link says (emphasis mine):

For example, a simple search query such as:

val u = URLEncoder.encode("foo +bar", StandardCharsets.UTF_8)  

would be encoded as:

foo+%2Bbar  

Trying to decode it with Spring, for example:

UriUtils.decode(u,   StandardCharsets.UTF_8)  

would return:

foo++bar  

Unfortunately, decoding with Uri.decode on Android, decodeURI in Javascript, etc. would yield the exact same result.

[rsb-23]

@TPS I had understood that the UrlEncoder.encode is not consistent with decode logic.

But my query is about parser.

Parser and encoder are different functionality, please don't mix the two.

[farOverNinethousand]

Hi Sven,
I'm currently writing a Telegram Bot which will have similar functionality to your application (if something like this already exists please let me know but I didn't find anything nice and open source).
While a native app like yours is more powerful, I will still work on releasing said bot.
During my research about similar tools I found yours to be nearly the only actively developed project of that kind.

Now I'm asking myself:
Will you design the new link-sanatizer-system in a way that it can be programmatically used in other applications?
I'm asking as this could save double-work and I'd love to add URL-sanatizers to my bot which will also be usable in other applications :)

Thanks for your time!

[TPS]

Yes, I think if we aim for a general format or even a standard, JSON would be more appropriate.

Here's perhaps a devil's-advocate perspective to consider: why invent another new format when the current adblock syntax might do fine? It's very mature, virtually universal, extensively extensible, & gives immediate access to a vast amount of data that's a superset of what you're looking for.

[svenjacobs]

Yes, I think if we aim for a general format or even a standard, JSON would be more appropriate.

Here's perhaps a devil's-advocate perspective to consider: why invent another new format when the current adblock syntax might do fine? It's very mature, virtually universal, extensively extensible, & gives immediate access to a vast amount of data that's a superset of what you're looking for.

Yes sure, sounds good. I'm not yet familiar with the AdBlock/EasyList syntax, which is documented here. Since the format was mainly created to block requests and not remove parameters from URLs, it might lack some features? Removing parameters is supported but what about some more special cases where we rewrite an URL (see AmazonProductSanitizer)? Of course we could argue that Leon loses some features but therefor is able to parse popular block lists?

[farOverNinethousand]

@svenjacobs
Why would Leon lose features?
You could support both, those rules and your own rules.
It shouldn't take much more work.

[svenjacobs]

Hm, I'll have to think about it. I would prefer if we go one route: Let's say import all rules from adblock lists and get rid of custom managed sanitizers 🤷🏼

[TPS]

Removing parameters is supported but what about some more special cases where we rewrite an URL (see AmazonProductSanitizer)? Of course we could argue that Leon loses some features but therefor is able to parse popular block lists?

See $replace & $rewrite on that same writeup. This is already added via commonly supported extensions from the base AdBlock syntax.