GoodData

Types in JSON

Cpanel::JSON::XS::Type

E. Choroba

GoodData

PerlMonks, CPAN, StackOverflow

choroba@matfyz.cz

The Story

Migration from RHEL 6 to RHEL 7

Vendor Perl and libraries

What From To
Perl 5.10.1 5.16.3
JSON::XS 2.27 3.01

What has changed

  • Perl
    • Internal flags
  • JSON::XS
    • Type conversion heuristics

Impact of the changes

Perl doesn’t care.

Inconsistencies

Handling of magic variables changes with $]

encode_json([$$, "" . $$])
Perl Version Library Result
5.10.1 JSON::XS ["213","213"]
5.16.3 JSON::XS [213,"213"]
5.16.3 JSON::PP [213,"213"]
5.18.2 Cpanel::JSON::XS [213,"213"]

Inconsistencies (2)

Handling of “unicode numbers” changes with $]

$x = 12; utf8::decode($x); encode_json([$x])
Perl Version Library Result
5.10.1 JSON::XS [12]
5.16.3 JSON::XS ["12"]
5.16.3 JSON::PP ["12"]
5.18.2 Cpanel::JSON::XS ["12"]

Inconsistencies (3)

Cpanel::JSON::XS as a 1:1 replacement?

$x = '19a'; $x += 0; encode_json([$x])
Perl Version Library Result
5.10.1 JSON::XS [19]
5.16.3 JSON::XS [19]
5.16.3 JSON::PP [19]
5.18.2 Cpanel::JSON::XS [19.0]

Inconsistencies (4)

Not only is JSON::PP slow…

use Data::Dumper; Dumper(decode_json('[1e4]'))
Perl Version Library Result
5.10.1 JSON::XS ['10000']
5.16.3 JSON::XS ['10000']
5.16.3 JSON::PP [10000]
5.18.2 Cpanel::JSON::XS ['10000']

Inconsistencies (5)

Another inconsistency of JSON::PP

$x = 1844674407370955161; encode_json([$x, $x / 10])
Perl Version Library Result
5.10.1 JSON::XS [1.84467440737096e+18,1.84467440737096e+17]
5.16.3 JSON::XS [1.84467440737096e+18,1.84467440737096e+17]
5.16.3 JSON::PP [1844674407370955161,1.84467440737096e+17]
5.18.2 Cpanel::JSON::XS [1.84467440737096e+18,1.84467440737096e+17]

Inconsistencies (6)

JSON::XS and tied variables

#! /usr/bin/perl
use warnings;
use strict;

{   package MyIncrementer;
    use Tie::Scalar;
    use parent -norequire => 'Tie::StdScalar';
    sub TIESCALAR { my ($class, $val) = @_; bless \$val, $class }
    sub FETCH { my $s = shift; $$s++ }
}

use JSON::XS;
my $json = 'JSON::XS'->new->allow_nonref;

tie my $x, 'MyIncrementer', 'Xa';
print $json->encode($x) for 1 .. 4;

Inconsistencies (7)

JSON::XS calls FETCH twice

Perl Version Library Result
5.10.1 JSON::XS "Xb""Xd""Xf""Xh"
5.16.3 JSON::XS "Xb""Xd""Xf""Xh"
5.16.3 JSON::PP "Xa""Xb""Xc""Xd"
5.18.2 Cpanel::JSON::XS "Xa""Xb""Xc""Xd"

Inconsistencies (8)

Handling of “unicode numbers” changes across versions

$j = 'JSON::XS'->new->allow_nonref;
$x = 12;
print $j->decode($x), $j->encode($x);
Perl Version Library Result
5.10.1 JSON::XS !! 2.27 12"12"
5.10.1 JSON::XS !! 3.01 1212
5.16.3 JSON::PP 1212
5.18.2 Cpanel::JSON::XS 1212

Don’t use the system Perl

Angry woman
Angry woman

Hotfix

#! /usr/bin/perl
use warnings;
use strict;

use JSON::XS;

my $integer = "12";
my $string  = 42;
my $float   = "122e-1";

print encode_json([
    int $integer,
    "" . $string,
    0 + $float
]);

Output:

[12,"42",12.2]

Hotfix (2)

Cpanel::JSON::XS::Type

null (undef)

Limitations

Solution

my $type = parse_api_definition($symbol);
$json->encode($data, $type);

Examples

#! /usr/bin/perl
use warnings;
use strict;

use Cpanel::JSON::XS;
use Cpanel::JSON::XS::Type;

my $type = {count      => JSON_TYPE_INT,
            average    => JSON_TYPE_FLOAT,
            name       => JSON_TYPE_STRING,
            is_enabled => JSON_TYPE_BOOL,
            orders     => json_type_arrayof(JSON_TYPE_INT)};

print 'Cpanel::JSON::XS'->new->pretty->canonical
    ->encode({count      => '12',
              average    => '11.2',
              name       => 100 / 3,
              is_enabled => 1,
              orders     => [ 1 .. 10 ]
    }, $type);

Examples (2)

{
   "average" : 11.2,
   "count" : 12,
   "is_enabled" : true,
   "name" : "33.3333333333333",
   "orders" : [
      1,
      2,
      3,
      4,
      5,
      6,
      7,
      8,
      9,
      10
   ]
}

Recursive structures

#! /usr/bin/perl
use warnings;
use strict;

use Cpanel::JSON::XS; use Cpanel::JSON::XS::Type;

my $node = {value => JSON_TYPE_STRING};
$node->{children} = json_type_arrayof($node);


print 'Cpanel::JSON::XS'->new->pretty->encode({
    value => 'root',
    children => [{ value => 'child1', children => [
        {value => 'grandchild1'}
    ] }],
}, $node);

Recursive structures

#! /usr/bin/perl
use warnings;
use strict;

use Cpanel::JSON::XS; use Cpanel::JSON::XS::Type;

my $node = {value => JSON_TYPE_STRING};
$node->{children} = json_type_arrayof($node);
# Creates reference cycle: -----------^^^^^

print 'Cpanel::JSON::XS'->new->pretty->encode({
    value => 'root',
    children => [{ value => 'child1', children => [
        {value => 'grandchild1'}
    ] }],
}, $node);

Recursive structures (2)

#! /usr/bin/perl
use warnings;
use strict;

use Cpanel::JSON::XS; use Cpanel::JSON::XS::Type;

my $node = {value => JSON_TYPE_STRING};
$node->{children} = json_type_arrayof(
    json_type_weaken($node)
);
print 'Cpanel::JSON::XS'->new->pretty->encode({
    value => 'root',
    children => [{ value => 'child1', children => [
        {value => 'grandchild1'}
    ] }],
}, $node);

Recursive structures (3)

json_type_weaken() already merged, but not yet released to CPAN

{
   "children" : [
      {
         "children" : [
            {
               "value" : "grandchild1"
            }
         ],
         "value" : "child1"
      }
   ],
   "value" : "root"
}

Type decoder

Coming soon!

Pull request not yet merged

#! /usr/bin/perl
use warnings;
use strict;

use Cpanel::JSON::XS; use Cpanel::JSON::XS::Type;

my $struct = 'Cpanel::JSON::XS'->new
    ->decode('[null,1,1.1,"1",[0],true]', my $type);

/* types */
#define JSON_TYPE_SCALAR      0x0000
#define JSON_TYPE_BOOL        0x0001
#define JSON_TYPE_INT         0x0002
#define JSON_TYPE_FLOAT       0x0003
#define JSON_TYPE_STRING      0x0004
/* flags */
#define JSON_TYPE_CAN_BE_NULL 0x0100
/* null type */
#define JSON_TYPE_NULL        JSON_TYPE_CAN_BE_NULL

$struct = [
            undef,
            1,
            '1.1',
            '1',
            [
              0
            ],
            1
          ];
$type = [
          256,
          2,
          3,
          4,
          [
            2
          ],
          1
        ];

Similar story

Did the JSON module change?

It’s not just about JSON

DBI

SQL is typed

Types in Perl

The bitwise feature

Thanks

Thank you.

Questions?

http://e-choroba.eu/18-yapc/