Internet-Draft SID Extension March 2024
Toutain, et al. Expires 27 September 2024 [Page]
Workgroup:
t2t Research Group
Internet-Draft:
draft-toutain-t2t-sid-extension-00
Published:
Intended Status:
Standards Track
Expires:
Authors:
L. Toutain
Institut MINES TELECOM; IMT Atlantique
M. Gudi
Institut MINES TELECOM; IMT Atlantique
J. A. Fernandez
Institut MINES TELECOM; IMT Atlantique

SID Extension to efficiently manipulate YANG Data Models

Abstract

As the Internet of Things (IoT) systems are becoming more pervasive with their rapid adoption, they are also becoming more complex in their architecture. Hence, a tool is required to generate prototype code based on the YANG models for constrained devices [RFC7228] to improve interoperability and increase the reusability of software components. A novel approach is introduced in this document to generate software prototypes (also called stubs) in the C language for the CORECONF protocol [I-D.ietf-core-comi] using YANG Schema Item iDentifiers (YANG SID [I-D.ietf-core-sid]). These stubs greatly reduce the complexity of navigating the CORECONF structure by abstracting the corresponding YANG SIDs. This document elaborates on the procedure to generate YANG SIDs for a given YANG model of a system, which then generates C stubs using the tools developed by the authors with the help of an example (sensor.yang file).

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on 27 September 2024.

▲

Table of Contents

1. Introduction

The YANG modeling language is very popular for node configuration and retrieving the the state of a device.
XML and JSON are also two popular formats to serialize data in conformance with the data model. Recently, a new serialization format has been published, allowing a more compact representation of the information. This new format is based on CBOR and the YANG identifier; instead of being represented by a unique ASCII string, values are mapped to an unique integer.

The mapping between strings and integer is kept in a .sid file usually generated by tools such as pyang.

This document presents some extensions to the sid files that allow to ease the conversion between CBOR and JSON, as well as manipulating YANG Data Models in a constraint environment.

2. YANG Data model

YANG is a modeling language to structure information and check its conformity. Each element of a YANG Data Model is identified through an unique identifier. YANG is based on a hierarchical approach. During the serialization phase, data is represented either in XML or JSON. In this document we will use the JSON representation. [RFC7951] indicates how to form a JSON structure conforming to a YANG Data Model:

The YANG Data Model, in Figure 1, is used to illustrate how data will be serialized. It defines a container representing a physical object able to perform several measurements. The physical object is battery powered, has a status LED able to change color, and a list of attached sensors returning an integer value.

<CODE BEGINS> file "sensor.yang"

module sensor {
  yang-version 1.1;
  namespace "http://sensorexample.in/sensor-example";
  prefix sen1;

  identity battery-indicator-base-type {
    description
      "a base identity for battery level indicator";
  }

  identity high-level {
    base battery-indicator-base-type;
  }

  identity med-level {
    base battery-indicator-base-type;
  }

  identity low-level {
    base battery-indicator-base-type;
  }

  typedef battery-level {
    type identityref {
      base battery-indicator-base-type;
    }
  }

  container sensorObject {
    leaf statusLED {
      type enumeration {
        enum green  {value 0;}
        enum yellow {value 1;}
        enum red {value 2;}
      }
    }
    leaf battery {
        type battery-level;
    }
    list sensorReadings {
      key "index";
      leaf index {
        type uint8;
      }
      leaf sensorValue {
        type uint32;
      }

    }
  }
}

<CODE ENDS>
Figure 1: Example of a YANG Data Model for sensors

The YANG tree regarding this Data Model is given in Figure 2. The tree displays the module hierarchy. For the module "sensor", a container "sensorObject" contains two leaves ("statusLED", "battery") and a list ("sensorReadings").

$ pyang -f tree sensor.yang
module: sensor
  +--rw sensorObject
     +--rw statusLED?        enumeration
     +--rw battery?          battery-level
     +--rw sensorReadings* [index]
        +--rw index          uint8
        +--rw sensorValue?   uint32
Figure 2: YANG tree associated to the DM.

3. JSON serialization

An example of data, serialized with JSON, and conforming to the YANG Data Model, is given in Figure 3. Embedded JSON Objects allow to represent the hierarchy. The key of the outer JSON Object is composed of the module name and the container name. The embedded JSON Object associated to this contains the leaves as keys. The YANG list is represented by an JSON Array.

{
  "sensor:sensorObject": {
    "statusLED": "green",
    "battery": "sensor:med-level",
    "sensorReadings": [
      {
        "index": 0,
        "sensorValue": 42
      },
      {
        "index": 1,
        "sensorValue": 22
      }
    ]
  }
}
Figure 3: JSON structure conform to the YANG DM.

4. CBOR Serialization

JSON notation is verbose for constrained networks and objects. To optimize the size of the representation, [RFC9254] defines a CBOR serialization, also used in CORECONF. YANG ASCII identifiers are replaced by unique number. In JSON, the uniqueness is guaranteed by the "namespace" URI, as shown in Figure 1. By construction, the rest of the identifiers are unique.

In CORECONF, the uniqueness is guaranteed through the use of positive integers called SID, which replace the ASCII identifiers. [I-D.ietf-core-sid] defines the allocation process. Module developers may ask for a SID range from their authority. For example, for an IETF module, the IANA will allocate a SID range.

The Figure 4 shows an example of this conversion. The range is arbitrarily fixed between 60000 and 60099. Note that the module, the identities, and the leaves have an assigned SID.

$ pyang --sid-generate-file=60000:100 --sid-list sensor.yang

SID        Assigned to
​---------  --------------------------------------------------
60000      module sensor
60001      identity battery-indicator-base-type
60002      identity high-level
60003      identity low-level
60004      identity med-level
60005      data /sensor:sensorObject
60006      data /sensor:sensorObject/battery
60007      data /sensor:sensorObject/sensorReadings
60008      data /sensor:sensorObject/sensorReadings/index
60009      data /sensor:sensorObject/sensorReadings/sensorValue
60010      data /sensor:sensorObject/statusLED

File sensor@unknown.sid created
Number of SIDs available : 100
Number of SIDs used : 11
Figure 4: JSON structure conform to the YANG DM.

To perform the allocation, the pyang utility generates a .sid file in the JSON format, resulting in a YANG Data Model specified in [I-D.ietf-core-sid]. The Figure 5 gives an excerpt.

{
  "assignment-range": [
    {
      "entry-point": 60000,
      "size": 100
    }
  ],
  "module-name": "sensor",
  "module-revision": "unknown",
  "item": [
    {
      "namespace": "module",
      "identifier": "sensor",
      "status": "unstable",
      "sid": 60000
    },
    {
      "namespace": "identity",
      "identifier": "battery-indicator-base-type",
      "status": "unstable",
      "sid": 60001
    },
    ...
Figure 5: JSON structure conform to the YANG DM.

The serialization in CBOR of the JSON example in Figure 3 is given in Figure 6. Compared to the compacted representation of the JSON structure (152 Bytes), the CORECONF structure is 24 bytes long. Although the size is different, the structure remains the same. It is composed of embedded CBOR Maps (equivalent of JSON Object). The first key is a SID (60005 corresponds to the container). Embedded CBOR Maps use a delta notation to encode the keys. The key 5 corresponds to SID 60005+5, thus to the leaf "statusLED". Key 2 in the second CBOR Map corresponds to the YANG list "sensorReadings", and the elements of the list are stored in a CBOR Array.

Note that in this example, the enum for "statusLED" (60005+5) is an integer and the identityref for "battery" (60005+1) is also an integer pointing to the SID "med-level" (60004).

b'a119ea65a305000119ea640282a2010002182aa201010216'

Diagnostic notation:
{60005:
  {5: 0,
   1: 60004,
   2: [
     {1: 0,
      2: 42},
     {1: 1,
      2: 22}
     ]
  }
}
Figure 6: JSON structure conform to the YANG DM.

5. Conversion between JSON and CBOR

Even if the conversion between CBOR and JSON formats might look obvious, including data compatible with a YANG Data Model is not so trivial. The reason is that JSON uses ASCII identifiers for readability and CBOR prefers integers for compactness. The Table 1 summarizes some YANG types when coded in JSON ([RFC7951]) or CBOR ([RFC8949]).

Table 1: YANG basic types in JSON and CBOR.
YANG JSON CBOR
int8, int16, int32, uint8, uint16, uint32 number +int, -int
int64, uint64 string +int, -int
decimal64 string CBOR Tag 4
binary base64 byte array
bits string array
boolean boolean boolean
identityref string +int
enumeration string +int

The conversion from CBOR to JSON is not direct, since the YANG type needs to be considered. For instance, a integer could be converted into a number, an enum, or an identityref. Note that for union, the conversion is simple since some CBOR Tags may be used to indicate how the integer is converted, but outside of the union, for a single value, there is no such clue.

On the other direction, a JSON string may correspond to a 64-bit long number, an enumeration, or an identityref.

To perform the conversion, the YANG type is needed, but popular tools such as yanglint or pyang are not currently manipulating CBOR representation. Furthermore, these tools cannot run on constrained devices. To overcome these problems, we propose to extend the .sid file with more information. The modified pyang code (see https://github.com/ltn22/pyang/tree/sid-extension) adds useful information to the .sid file when the "--sid-extension" argument is provided.

This extension contains a "type" key in the JSON Object describing the mapping between the SID and the identifier if the node is a leaf (see Figure 7).

...
    {
      "namespace": "data",
      "identifier": "/sensor:sensorObject",
      "status": "unstable",
      "sid": 60005
    },
    {
      "namespace": "data",
      "identifier": "/sensor:sensorObject/battery",
      "status": "unstable",
      "sid": 60006,
      "type": "identityref"
    },
    {
      "namespace": "data",
      "identifier": "/sensor:sensorObject/sensorReadings",
      "status": "unstable",
      "sid": 60007
    },
    {
      "namespace": "data",
      "identifier": "/sensor:sensorObject/sensorReadings/index",
      "status": "unstable",
      "sid": 60008,
      "type": "uint8"
    },
    {
      "namespace": "data",
      "identifier": "/sensor:sensorObject/sensorReadings/sensorValue",
      "status": "unstable",
      "sid": 60009,
      "type": "uint32"
    },
    {
      "namespace": "data",
      "identifier": "/sensor:sensorObject/statusLED",
      "status": "unstable",
      "sid": 60010,
      "type": {
        "0": "green",
        "1": "yellow",
        "2": "red"
      }
    }
...
Figure 7: SID "type

The "type" key added to leaf nodes contains several information:

We developed the pycoreconf Python module to facilitate the conversion between JSON and CBOR (https://github.com/ltn22/pycoreconf). The Figure 8 gives an example of a Python script using this module. It takes as input the .sid file and a JSON structure. It transforms it into a CBOR structure, and back to JSON representation.

# pycoreconf sample: "basic"
# This script demonstrates the basic usage of pycoreconf
#  using a simple YANG datamodel.

import pycoreconf
import binascii
import cbor2 as cbor

# Create the model object
ccm = pycoreconf.CORECONFModel("sensor@unknown.sid")

# Read JSON configuration file
config_file = "output.json"

with open(config_file, "r") as f:
    json_data = f.read()
print("Input JSON config data =\n", json_data, sep='')

# Convert configuration to CORECONF/CBOR
cbor_data = ccm.toCORECONF(config_file) # can also take json_data
print("Encoded CBOR data (CORECONF payload) =", binascii.hexlify(cbor_data))
print (cbor.loads(cbor_data))

# Decode CBOR data back to JSON configuration data
decoded_json = ccm.toJSON(cbor_data)
print("Decoded config data =", decoded_json)
Figure 8: pycoreconf module.

The resulting JSON structure of Figure 3 is given in Figure 9. It shows that the JSON format can be converted to CBOR and vice versa, just by using the extended .sid file.

Input JSON config data =
{
  "sensor:sensorObject": {
    "statusLED": "green",
    "battery": "sensor:med-level",
    "sensorReadings": [
      {
        "index": 0,
        "sensorValue": 42
      },
      {
        "index": 1,
        "sensorValue": 22
      }
    ]
  }
}

Encoded CBOR data (CORECONF payload) = b'a119ea65a305000119ea640282a2010002182aa201010216'
{60005: {5: 0, 1: 60004, 2: [{1: 0, 2: 42}, {1: 1, 2: 22}]}}
Decoded config data = {"sensor:sensorObject": {"statusLED": "green", "battery": "med-level", "sensorReadings": [{"index": 0, "sensorValue": 42}, {"index": 1, "sensorValue": 22}]}}
Figure 9: pycoreconf module.

7. Linking to real values

In some YANG Data Models, as the one provided in Figure 1, some leaf nodes might encompass information that extends beyond the model's boundaries. For example, the "statusLED" leaf may require an action to be triggered when a value is written onto it, or the managed device might need to interact with a physical sensor when the "sensorValue" leaf is queried. On the other hand, certain leaf nodes, such as "index", do not require any interaction as their values are directly stored in the database.

We wrote a script taking the .sid file obtained from pyang with the "--sid-extension" argument to produce template C code for these functions. We try keep SID transparent and use the YANG identifiers as much as possible, which are easier to manipulate by a programmer.

The tool aims to generate aliases for identifiers that also include details about their associated SIDs. This eliminates the need for developers to recall or verify SIDs during software development.

$ python generateStubs.py sensor@unknown.sid "sensor_prototypes"

//Headers
//-----------
#include <stdlib.h>
#include <stdint.h>
#include <stdbool.h>
#include <string.h>
#include <cbor.h>

#define  SID_BATTERY             60006
#define  SID_INDEX               60008
#define  SID_SENSORVALUE         60009
#define  SID_STATUSLED           60010


#define read_sensorObject        read_60005
#define read_battery             read_60006
#define read_sensorReadings      read_60007
#define read_sensorValue         read_60009
#define read_statusLED           read_60010

char* keyMapping = "\xa1\x19\xeag\x81\x19\xeah";
enum StatusledEnum {green = 0, yellow = 1, red = 2};

void  read_sensorObject(void);
char *  read_battery(void);
void  read_sensorReadings(uint8_t index);
uint32_t  read_sensorValue(uint8_t index);
enum StatusledEnum read_statusLED(void);

//Code File
//-----------
#include <stdlib.h>
#include <stdint.h>
#include <stdbool.h>
#include <string.h>
#include "sensor_prototypes.h"

/*
    This is an autogenerated function associated to
    SID: 60005
    Module: data
    Identifier: /sensor:sensorObject
    function params:
    Stable: false
*/
void  read_sensorObject(void){
    // Initialize the leaf if it has a return type with a default value;
    // Do something with the leaf
}

/*
    This is an autogenerated function associated to
    SID: 60006
    Module: data
    Identifier: /sensor:sensorObject/battery
    function params:
    Stable: false
*/
char *  read_battery(void){
    // Initialize the leaf if it has a return type with a default value;
    char * batteryInstance  = NULL;

    // Do something with the leaf
    // Return the leaf
    return batteryInstance;
}

/*
    This is an autogenerated function associated to
    SID: 60007
    Module: data
    Identifier: /sensor:sensorObject/sensorReadings
    function params: /sensor:sensorObject/sensorReadings/index
    Stable: false
*/
void  read_sensorReadings(uint8_t index){
    // Initialize the leaf if it has a return type with a default value;
    // Do something with the leaf
}

/*
    This is an autogenerated function associated to
    SID: 60009
    Module: data
    Identifier: /sensor:sensorObject/sensorReadings/sensorValue
    function params: /sensor:sensorObject/sensorReadings/index
    Stable: false
*/
uint32_t  read_sensorValue(uint8_t index){
    // Initialize the leaf if it has a return type with a default value;
    uint32_t sensorValueInstance  = 0;

    // Do something with the leaf
    // Return the leaf
    return sensorValueInstance;
}

/*
    This is an autogenerated function associated to
    SID: 60010
    Module: data
    Identifier: /sensor:sensorObject/statusLED
    function params:
    Stable: false
*/
enum StatusledEnum read_statusLED(void){
    // Initialize the leaf if it has a return type with a default value;
    enum StatusledEnum statusLEDInstance;

    // Do something with the leaf
    // Return the leaf
    return statusLEDInstance;
}

8. Understanding Stubs

Let us take a quick look at the generated code snippet to understand how they help abstract SIDs from the developers:

The first part of the sensor_prototypes.h contains pre-processor directives which basically store the identifiers with their corresponding SID values for quick access. The second part contains function prototypes which can be implemented later. Note that there will not be any function prototype generated to read SID keys (in this case index, with SID 60008).

// Aliases of the leaves mapped to their corresponding SID
#define  SID_BATTERY             60006
#define  SID_INDEX               60008
#define  SID_SENSORVALUE         60009
#define  SID_STATUSLED           60010

// Aliases of function prototypes linked to their function name containing SID value for quick and easy access
#define read_sensorObject        read_60005
#define read_battery             read_60006
#define read_sensorReadings      read_60007
#define read_sensorValue         read_60009
#define read_statusLED           read_60010

// The relation between certain identifiers and their sid keys is stored in keyMapping as a CBOR map
char* keyMapping = "\xa1\x19\xeag\x81\x19\xeah";
enum StatusledEnum {green = 0, yellow = 1, red = 2};

// Function prototypes of the getters which can be implemented by the developer later in its corresponding C file
// The return types of the functions are mapped according to the type of the leaf. The getters for non-leaves are mapped as void
void  read_sensorObject(void);
char *  read_battery(void);
void  read_sensorReadings(uint8_t index);
uint32_t  read_sensorValue(uint8_t index);
enum StatusledEnum read_statusLED(void);

Now let us understand what is generated in sid_prototypes.c by looking at the auto-generated function prototype read_sensorValue. The program auto-infers that the leaf sensorValue can only be reached through a valid SID key (that is, index), so a function trying to read sensorValue from its CORECONF instance will require a parameter index. Also, since sensorValue has a well-defined type in C, the function will try to return that uint32_t.

The program also maps the enum type from SID to a corresponding enum type in C, as shown for the leaf statusLED. The value of these enums is as specified in the corresponding YANG model. If unspecified, the values are auto-assigned.

For rest of the non-leaf functions and leaf nodes with type identityref, the program infers it to be a char * (since the types are non-standard).

uint32_t  read_sensorValue(uint8_t index){
    // Initialize the leaf if it has a return type with a default value;
    uint32_t sensorValueInstance  = 0;

    // Do something with the leaf
    // Return the leaf
    return sensorValueInstance;
}

9. Normative References

[I-D.ietf-core-comi]
Veillette, M., Van der Stok, P., Pelov, A., Bierman, A., and C. Bormann, "CoAP Management Interface (CORECONF)", Work in Progress, Internet-Draft, draft-ietf-core-comi-17, , <https://datatracker.ietf.org/doc/html/draft-ietf-core-comi-17>.
[I-D.ietf-core-sid]
Veillette, M., Pelov, A., Petrov, I., Bormann, C., and M. Richardson, "YANG Schema Item iDentifier (YANG SID)", Work in Progress, Internet-Draft, draft-ietf-core-sid-24, , <https://datatracker.ietf.org/doc/html/draft-ietf-core-sid-24>.
[RFC2119]
Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, , <https://www.rfc-editor.org/rfc/rfc2119>.
[RFC7228]
Bormann, C., Ersue, M., and A. Keranen, "Terminology for Constrained-Node Networks", RFC 7228, DOI 10.17487/RFC7228, , <https://www.rfc-editor.org/rfc/rfc7228>.
[RFC7951]
Lhotka, L., "JSON Encoding of Data Modeled with YANG", RFC 7951, DOI 10.17487/RFC7951, , <https://www.rfc-editor.org/rfc/rfc7951>.
[RFC8949]
Bormann, C. and P. Hoffman, "Concise Binary Object Representation (CBOR)", STD 94, RFC 8949, DOI 10.17487/RFC8949, , <https://www.rfc-editor.org/rfc/rfc8949>.
[RFC9254]
Veillette, M., Ed., Petrov, I., Ed., Pelov, A., Bormann, C., and M. Richardson, "Encoding of Data Modeled with YANG in the Concise Binary Object Representation (CBOR)", RFC 9254, DOI 10.17487/RFC9254, , <https://www.rfc-editor.org/rfc/rfc9254>.

Authors' Addresses

Laurent Toutain
Institut MINES TELECOM; IMT Atlantique
2 rue de la Chataigneraie
CS 17607
35576 Cesson-Sevigne Cedex
France
Manoj Gudi
Institut MINES TELECOM; IMT Atlantique
2 rue de la Chataigneraie
CS 17607
35576 Cesson-Sevigne Cedex
France
Javier A. Fernandez
Institut MINES TELECOM; IMT Atlantique
2 rue de la Chataigneraie
CS 17607
35576 Cesson-Sevigne Cedex
France