v6.0.0: big revbank.products syntax change

Rationale in UPGRADING.md

It's a big change technically, but converting the format won't be hard
for admins.

There's a compatibility mode with loud warnings in case the file isn't
converted.

UPGRADING.md

@@ -1,3 +1,79 @@
+# When upgrading, always:
+
+1. Make sure nobody is using RevBank.
+2. Make a backup of your RevBank data and code repo(s).
+3. Read this file :)
+
+# (2024-01-20) RevBank 6.0.0
+
+Note that the changes to `revbank.products` do NOT apply to `revbank.market`
+and other files.
+
+## Update your `revbank.products` file
+
+TL;DR: Product descriptions now need `"quotes"` around them.
+
+This version comes with breaking changes to the `revbank.products` syntax, to
+expand the capabilities of the file in a more future-proof way. Bitlair
+(Hackerspace Amersfoort) has requested a way to add metadata to products for
+automation, which together with recent other additions to the format, made
+clear a more structured approach was needed.
+
+The line format for the products file is now like the input format of the
+command line interface. This means that if product descriptions contain spaces,
+as they typically do, quotes are needed around them. You can pick between
+`"double"` and `'single'` quotes. Any backslashes and quotes within the same
+kind of quotes need escaping by adding a `\` in front, e.g. `\"` and `\\`.
+
+```
+# Old format:
+product_id    0.42    Can't think of a good description +addon1 +addon2
+
+# New format, recommended style:
+product_id    0.42    "Can't think of a good description" +addon1 +addon2
+
+# Automatically generated? You may wish to quote all fields:
+"product_id" "0.42" "Can't think of a good description" "+addon1" "+addon2"
+
+# Escaping also works:
+product_id 0.42 Can\'t\ think\ of\ a\ good\ description +addon1 +addon2
+```
+
+To convert your `revbank.products` file to the recommended style automatically,
+you could use:
+
+```sh
+# The following is one command. It was obviously not optimized for readability :)
+
+perl -i.backupv6 -ple'unless (/^\s*#/ or /^\s*$/) {
+  my ($pre, $desc) = /(^\s*\S+\s+\S+\s*)(.*)/; $pre .= " " if $pre !~ /\s$/;
+  my @a; unshift @a, $1 while $desc =~ s/\s\+(\S+)$//;
+  $desc =~ s/([\"\\])/\\$1/g; $_ = "$pre\"$desc\"";
+  for my $a (@a) { $_ .= " +$a" }
+}' revbank.products
+```
+
+Note that this will leave commented lines unchanged! If those contain disabled
+products, you'll have to add the quotes yourself.
+
+## New feature: hashtags in `revbank.products`
+
+After the description field, you can add hashtag fields. These begin with `#`
+and may take the form of a lone `#hashtag`, or they may be used as a
+`#key=value` pair. The hashtags can be read by plugins. Out of the box, they
+currently do nothing.
+
+```
+8711327538481  0.80  "Ola Liuk"   #ah=wi162664 #q=8
+8712100340666  0.45  "Ola Raket"  #ah=wi209562 #q=12
+5000112659184,5000112658873  0.95  "Coca-Cola Cola Zero Sugar (33 cl)" #sligro +sb
+
+# equivalent:
+"8711327538481" "0.80" "Ola Liuk" "#ah=wi162664" "#q=8"
+```
+
+See https://github.com/bitlair/revbank-inflatinator/ for a possible use of adding metadata.
+
 # (2023-12-26) RevBank 5.0.0
 
 This version comes with breaking changes to the command line syntax, to shield
@@ -231,19 +307,25 @@ list from within RevBank, add `edit` to `revbank.plugins`.
 
 ## Check your `revbank.products`
 
-There's new syntax for `revbank.products`: addons. Check that your lines don't
-have `+foo` at the end, where `foo` can be anything.
+> Added 2024-01-20 v6.0.0: if you're upgrading to v6.0.0 from a version before
+> v3.6, instead of following these instructions, you can just add quotes to the
+> descriptions (when using the perl oneliner from the v6.0.0 upgrade
+> instructions, check if any `+something` that got placed outside of the quotes
+> should have been within the quotes.)
+
+~~There's new syntax for `revbank.products`: addons. Check that your lines don't
+have `+foo` at the end, where `foo` can be anything.~~
 
-Also check that you don't have any product ids that start with `+`; those can
-no longer be entered as this syntax now has special semantics.
+~~Also check that you don't have any product ids that start with `+`; those can
+no longer be entered as this syntax now has special semantics.~~
 
-So these don't work as before:
+~~So these don't work as before:~~
 
     example_id      1.00  Example product +something
     +something      1.00  Product id that starts with plus
     example,+alias  1.00  Alias that starts with plus
 
-These will keep working as they were:
+~~These will keep working as they were:~~
 
     example_id1     1.00  Example product+something
     example_id2     1.00  Example product + something

plugins/products

@@ -13,16 +13,61 @@ sub read_products() {
     %products = ();
     $mtime = -M $filename;
 
-    my $line = 0;
+    my $linenr = 0;
+    my $warnings = 0;
 
-    for (slurp $filename) {
-        $line++;
+    for my $line (slurp $filename) {
+        $linenr++;
 
-        s/^\s+|\s+$//g;  # trim
-        next if /^#/;
-        next if not length;
+        next if $line =~ m[
+            ^\s*\#    # comment line
+            |
+            ^\s*$     # empty line, or only whitespace
+        ]x;
+
+        my @split = RevBank::Prompt::split_input($line);
+
+        if (grep /\0SEPARATOR/, @split) {
+            warn "Invalid character in $filename line $linenr.\n";
+            next;
+        }
+        if (grep /\0/, @split) {
+            warn "Invalid value in $filename line $linenr.\n";
+            next;
+        }
+
+        my ($ids, $p, $desc, @extra) = @split;
+
+        my @addon_ids;
+        my %tags;
+
+        my $compat = 0;
+        if (@split == 1 and ref $split[0]) {
+            $compat = 1;
+        } else {
+            for (@extra) {
+                if (/^\+(.*)/) {
+                    push @addon_ids, $1;
+                } elsif (/^\#(\w+)(=(.*))/) {
+                    $tags{$1} = $2 ? $3 : 1;
+                } else {
+                    $compat = 1;
+                    last;
+                }
+            }
+        }
+
+        if ($compat) {
+            $warnings++;
+            warn "$filename line $linenr: can't parse as new format; assuming old format.\n" if $warnings < 4;
+            warn "Too many warnings; suppressing the rest. See UPGRADING.md for instructions.\n" if $warnings == 4;
+
+            ($ids, $p, $desc) = split " ", $line, 3;
+
+            @addon_ids = ();
+            unshift @addon_ids, $1 while $desc =~ s/\s+ \+ (\S+)$//x;
+        }
 
-        my ($ids, $p, $desc) = split " ", $_, 3;
         my @ids = split /,/, $ids;
 
         $p ||= "invalid";
@@ -35,29 +80,26 @@ sub read_products() {
 
         if ($percent) {
             if (grep !/^\+/, @ids) {
-                warn "Percentage invalid for non-addon at $filename line $line.\n";
+                warn "Percentage invalid for non-addon at $filename line $linenr.\n";
                 next;
             }
             $price = 0 + $price;
         } else {
             $price = eval { parse_amount($price) };
             if (not defined $price) {
-                warn "Invalid price for '$ids[0]' at $filename line $line.\n";
+                warn "Invalid price for '$ids[0]' at $filename line $linenr.\n";
                 next;
             }
         }
-
-        my @addon_ids;
-        unshift @addon_ids, $1 while $desc =~ s/\s+ \+ (\S+)$//x;
-
         $products{$_} = {
             id          => $ids[0],
             price       => $sign * $price,
             percent     => $percent,
             description => $desc,
             contra      => $contra || $default_contra,
             _addon_ids  => \@addon_ids,
-            line        => $line,
+            line        => $linenr,
+            tags        => \%tags,
         } for @ids;
     }
 

plugins/products.pod

@@ -4,10 +4,14 @@ products - RevBank plugin for selling products
 
 =head1 SYNOPISIS
 
-	8710447032756              0.80          Festini Peer
-	4029764001807,clubmate     1.40          Club-Mate +half +pf
-	pf                         0.15@+pfand   Pfand NRW-Flasche
-	+half                      -50%          50% discount \o/
+	# Comments are lines that begin with a # character.
+	# Empty lines are ignored.
+
+	8710447032756          0.80        "Festini Peer"
+	4029764001807,clubmate 1.40        "Club-Mate" +half +pf
+	pf                     0.15@+pfand "Pfand NRW-Flasche"
+	+half                  -50%        "50% discount \\o/"
+	123                    0.42        "Hashtag example" #tag #tag2=42
 
 =head1 DESCRIPTION
 
@@ -63,9 +67,15 @@ C<revbank.products>. User accounts are liability accounts.)
 
 =head2 Description
 
-The description may contain whitespace.
+The description, like other columns, may contain whitespace, but to use
+whitespace, either the entire field "needs quotes" around it, or the whitespace
+can be escaped with backslashes.
+
+It is suggested to always use quotes around the description.
+
+=head2 Additional fields
 
-=head2 Addons
+=head3 Addons
 
 Addons are products that are added as part of the main product. They are
 specified after the description, with a C<+> sign that has whitespace before
@@ -76,9 +86,9 @@ the product id C<foo> is used instead. The difference is that a product id
 C<+foo> can only be used as an addon for another product, while C<foo> can be
 used either as an addon or a manually entered as a standalone product.
 
-	example_id          2.20     Example product +first +second
-	+first              1.20     First thing
-	second              0.80     Second thing
+	example_id          2.20     "Example product" +first +second
+	+first              1.20     "First thing"
+	second              0.80     "Second thing"
 
 In this example, the final price of the example product will be 4.20. It is not
 possible to buy the first thing separate, but it is possible to buy the second
@@ -97,20 +107,49 @@ listed as a component named "Product".
 A product can have multiple addons. Addon products themselves can also have
 further addons, but circular recursion is not supported.
 
-=head3 Percentage addons
+=head4 Percentage addons
 
 As a special case, an addon's price can be a percentage. In this case, the
 price is calculated from the sum of the the product components I<up to that
 point> that have I<the same contra account> as the percentage addon.
 
 So, given the following example,
 
-	example_id          0.90          Example product +some_fee +discount
-	+some_fee           0.15@+fees    Some fee; might be a bottle deposit
-	+discount           -50%          Special offer discount!
+	example_id          0.90          "Example product" +some_fee +discount
+	+some_fee           0.15@+fees    "Some fee; might be a bottle deposit"
+	+discount           -50%          "Special offer discount!"
 
 only 0.45 is discounted, because the 0.15 has a different contra account. While
 complicated, this is probably what you want in most cases. There is currently
 no way to apply a discount to the product with all of its addons.
 
 A percentage addon must have a product_id that begins with C<+>.
+
+=head3 Tags
+
+Additional metadata can be given in additional fields that begin with C<#> and
+the name of the tag, optionally followed by C<=> and a value to turn it into a
+key/value pair. If no value is specified, a value of C<1> is used.
+
+The name of a hashtag must contain only C<A-Z a-z 0-9 _> characters. There must
+not be whitespace after the C<#> or around the C<=>.
+
+Like all the fields, the field can be quoted to contain whitespace. Note,
+however, that the quotes must be placed around the entire field, not just the
+value part.
+
+	ht1         0.42           "Just one hashtag" #tag
+	ht2         0.42           "Two hashtags!" #tag #key=value
+	ht3         0.42           "Surprising syntax" "#x=spaces in value"
+
+Tags can be accessed by custom plugins, but are currently ignored by upstream
+RevBank and its plugins.
+
+=head3 Other additional fields
+
+When any field is added after the description, that does not begin with C<+> or
+C<#>, RevBank currently assumes it's the old syntax (which is not described in
+the current version of this document!), and parses it using the old semantics
+while showing a warning.
+
+This compatibility feature will be removed from a future version of RevBank.

plugins/statiegeld.pod

@@ -6,11 +6,11 @@ statiegeld - RevBank plugin for return deposits
 
 revbank.products:
 
-	clubmate     1.40                   Club-Mate bottle +sb
-	cola         0.90                   Cola can +sc
-	+sb          0.15@+statiegeld       Bottle deposit
-	+sc          0.25@+statiegeld       Can deposit
-	matecrate    1.50@+statiegeld       Mate crate (empty)
+	clubmate     1.40                   "Club-Mate bottle" +sb
+	cola         0.90                   "Cola can" +sc
+	+sb          0.15@+statiegeld       "Bottle deposit"
+	+sc          0.25@+statiegeld       "Can deposit"
+	matecrate    1.50@+statiegeld       "Mate crate (empty)"
 
 =head1 DESCRIPTION
 

plugins/vat.pod

@@ -13,8 +13,8 @@ C<revbank.vat>
 
 C<revbank.products>
 
-	123123123   1.00      Example product that gets the default contra
-	42424242    1.00@+sales/products/hoogbtw  Example with high VAT rate
+	123123123   1.00      "Example product that gets the default contra"
+	42424242    1.00@+sales/products/hoogbtw  "Example with high VAT rate"
 
 =head1 DESCRIPTION
 

revbank

@@ -18,7 +18,7 @@ use RevBank::Messages;
 use RevBank::Cart;
 use RevBank::Prompt;
 
-our $VERSION = "5.1.3";
+our $VERSION = "6.0.0";
 our %HELP1 = (
     "abort" => "Abort the current transaction",
 );