The Story

Migration from RHEL 6 to RHEL 7

Impact of the changes

• Some numbers started to appear in double quotes;
• Some were no longer quoted.

Inconsistencies

Handling of magic variables changes with $]

encode_json([$$, "" . $$])

Inconsistencies (2)

Handling of “unicode numbers” changes with $]

$x = 12; utf8::decode($x); encode_json([$x])

Inconsistencies (3)

Cpanel::JSON::XS as a 1:1 replacement?

$x = '19a'; $x += 0; encode_json([$x])

Inconsistencies (4)

Not only is JSON::PP slow…

use Data::Dumper; Dumper(decode_json('[1e4]'))

Inconsistencies (5)

Another inconsistency of JSON::PP

$x = 1844674407370955161; encode_json([$x, $x / 10])

Inconsistencies (6)

JSON::XS and tied variables

#! /usr/bin/perl
use warnings;
use strict;

{   package MyIncrementer;
    use Tie::Scalar;
    use parent -norequire => 'Tie::StdScalar';
    sub TIESCALAR { my ($class, $val) = @_; bless \$val, $class }
    sub FETCH { my $s = shift; $$s++ }
}

use JSON::XS;
my $json = 'JSON::XS'->new->allow_nonref;

tie my $x, 'MyIncrementer', 'Xa';
print $json->encode($x) for 1 .. 4;

Inconsistencies (7)

JSON::XS calls FETCH twice

Inconsistencies (8)

Handling of “unicode numbers” changes across versions

$j = 'JSON::XS'->new->allow_nonref;
$x = 12;
print $j->decode($x), $j->encode($x);

Don’t use the system Perl

• Good advice.
• We would have probably discovered the problems earlier.
• When upgrading our Perl and/or JSON::XS, the problems would have still appeared.

Hotfix

#! /usr/bin/perl
use warnings;
use strict;

use JSON::XS;

my $integer = "12";
my $string  = 42;
my $float   = "122e-1";

print encode_json([
    int $integer,
    "" . $string,
    0 + $float
]);

Output:

[12,"42",12.2]

Hotfix (2)

• Complex nested structures hard to track
• Just inspecting a value might change its format
• Confusing for non-Perl teams
• Redundant: formal API description

Cpanel::JSON::XS::Type

• Constants for atomic types:
  JSON_TYPE_INT, JSON_TYPE_FLOAT, JSON_TYPE_STRING, JSON_TYPE_BOOL, (+ JSON_TYPE_NULL)
• Complex structures:
  [], {}
• Functions for complex structures:
  json_type_arrayof(), json_type_hashof()
• + a special function for “alternatives”:
  json_type_anyof()

null (undef)

• Atomic types:
  JSON_TYPE_INT_OR_NULL, …
• Alternatives:
  json_type_null_or_anyof()

Limitations

• json_type_anyof()
  max one scalar, one array, and one hash

Solution

• Parse the formal API definitons into type structures
• Pass the type structures to Cpanel::JSON::XS

my $type = parse_api_definition($symbol);
$json->encode($data, $type);

Examples

#! /usr/bin/perl
use warnings;
use strict;

use Cpanel::JSON::XS;
use Cpanel::JSON::XS::Type;

my $type = {count      => JSON_TYPE_INT,
            average    => JSON_TYPE_FLOAT,
            name       => JSON_TYPE_STRING,
            is_enabled => JSON_TYPE_BOOL,
            orders     => json_type_arrayof(JSON_TYPE_INT)};

print 'Cpanel::JSON::XS'->new->pretty->canonical
    ->encode({count      => '12',
              average    => '11.2',
              name       => 100 / 3,
              is_enabled => 1,
              orders     => [ 1 .. 10 ]
    }, $type);

Examples (2)

{
   "average" : 11.2,
   "count" : 12,
   "is_enabled" : true,
   "name" : "33.3333333333333",
   "orders" : [
      1,
      2,
      3,
      4,
      5,
      6,
      7,
      8,
      9,
      10
   ]
}

Recursive structures

#! /usr/bin/perl
use warnings;
use strict;

use Cpanel::JSON::XS; use Cpanel::JSON::XS::Type;

my $node = {value => JSON_TYPE_STRING};
$node->{children} = json_type_arrayof($node);


print 'Cpanel::JSON::XS'->new->pretty->encode({
    value => 'root',
    children => [{ value => 'child1', children => [
        {value => 'grandchild1'}
    ] }],
}, $node);

Recursive structures

#! /usr/bin/perl
use warnings;
use strict;

use Cpanel::JSON::XS; use Cpanel::JSON::XS::Type;

my $node = {value => JSON_TYPE_STRING};
$node->{children} = json_type_arrayof($node);
# Creates reference cycle: -----------^^^^^

print 'Cpanel::JSON::XS'->new->pretty->encode({
    value => 'root',
    children => [{ value => 'child1', children => [
        {value => 'grandchild1'}
    ] }],
}, $node);

Recursive structures (2)

#! /usr/bin/perl
use warnings;
use strict;

use Cpanel::JSON::XS; use Cpanel::JSON::XS::Type;

my $node = {value => JSON_TYPE_STRING};
$node->{children} = json_type_arrayof(
    json_type_weaken($node)
);
print 'Cpanel::JSON::XS'->new->pretty->encode({
    value => 'root',
    children => [{ value => 'child1', children => [
        {value => 'grandchild1'}
    ] }],
}, $node);

Recursive structures (3)

json_type_weaken() already merged, but not yet released to CPAN

{
   "children" : [
      {
         "children" : [
            {
               "value" : "grandchild1"
            }
         ],
         "value" : "child1"
      }
   ],
   "value" : "root"
}

Type decoder

#! /usr/bin/perl
use warnings;
use strict;

use Cpanel::JSON::XS; use Cpanel::JSON::XS::Type;

my $struct = 'Cpanel::JSON::XS'->new
    ->decode('[null,1,1.1,"1",[0],true]', my $type);

/* types */
#define JSON_TYPE_SCALAR      0x0000
#define JSON_TYPE_BOOL        0x0001
#define JSON_TYPE_INT         0x0002
#define JSON_TYPE_FLOAT       0x0003
#define JSON_TYPE_STRING      0x0004
/* flags */
#define JSON_TYPE_CAN_BE_NULL 0x0100
/* null type */
#define JSON_TYPE_NULL        JSON_TYPE_CAN_BE_NULL

$struct = [
            undef,
            1,
            '1.1',
            '1',
            [
              0
            ],
            1
          ];
$type = [
          256,
          2,
          3,
          4,
          [
            2
          ],
          1
        ];

Similar story

Did the JSON module change?

It’s not just about JSON

DBI

SQL is typed

• Why does DBI implicitly change integers to strings

Types in Perl

The bitwise feature

• The only operators that depend on types of its arguments: ~ ^ & |

Thanks

• Pali
• Reini Urban
• GoodData