Verifiable Distributed Aggregation Functions

1.1. Change Log

(*) Indicates a change that breaks wire compatibility with the previous draft.¶

08:¶

• Poplar1: Bind the report nonce to the authenticator vector programmed into the IDPF. (*)¶

• IdpfPoplar: Modify extend() by stealing each control bit from its corresponding seed. This improves performance by reducing the number of AES calls per level from 3 to 2. The cost is a slight reduction in the concrete privacy bound. (*)¶

• Prio3: Add support for generating and verifying mutliple proofs per measurement. This enables a trade-off between communication cost and runtime: if more proofs are used, then a smaller field can be used without impacting robustness. (*)¶

• Replace SHAKE128 with TurboSHAKE128. (*)¶

07:¶

• Rename PRG to XOF ("eXtendable Output Function"). Accordingly, rename PrgSha3 to XofShake128 and PrgFixedKeyAes128 to XofFixedKeyAes128. "PRG" is a misnomer since we don't actually treat this object as a pseudorandom generator in existing security analysis.¶

• Replace cSHAKE128 with SHAKE128, re-implementing domain separation for the customization string using a simpler scheme. This change addresses the reality that implementations of cSHAKE128 are less common. (*)¶

• Define a new VDAF, called Prio3SumVec, that generalizes Prio3Sum to a vector of summands.¶

• Prio3Histogram: Update the codepoint and use the parallel sum optimization introduced by Prio3SumVec to reduce the proof size. (*)¶

• Daf, Vdaf: Rename interface methods to match verbiage in the draft.¶

• Daf: Align with Vdaf by adding a nonce to shard() and prep().¶

• Vdaf: Have prep_init() compute the first prep share. This change is intended to simplify the interface by making the input to prep_next() not optional.¶

• Prio3: Split sharding into two auxiliary functions, one for sharding with joint randomness and another without. This change is intended to improve readability.¶

• Fix bugs in the ping-pong interface discovered after implementing it.¶

06:¶

• Vdaf: Define a wrapper interface for preparation that is suitable for the "ping-pong" topology in which two Aggregators exchange messages over a request/response protocol, like HTTP, and take turns executing the computation until input from the peer is required.¶

• Prio3Histogram: Generalize the measurement type so that the histogram can be used more easily with discrete domains. (*)¶

• Daf, Vdaf: Change the aggregation parameter validation algorithm to take the set of previous parameters rather than a list. (The order of the parameters is irrelevant.)¶

• Daf, Vdaf, Idpf: Add parameter RAND_SIZE that specifies the number of random bytes consumed by the randomized algorithm (shard() for Daf and Vdaf and gen() for Idpf).¶

05:¶

• IdpfPoplar: Replace PrgSha3 with PrgFixedKeyAes128, a fixed-key mode for AES-128 based on a construction from [GKWWY20]. This change is intended to improve performance of IDPF evaluation. Note that the new PRG is not suitable for all applications. (*)¶

• Idpf: Add a binder string to the key-generation and evaluation algorithms. This is used to plumb the nonce generated by the Client to the PRG.¶

• Plumb random coins through the interface of randomized algorithms. Specifically, add a random input to (V)DAF sharding algorithm and IDPF key-generation algorithm and require implementations to specify the length of the random input. Accordingly, update Prio3, Poplar1, and IdpfPoplar to match the new interface. This change is intended to improve coverage of test vectors.¶

• Use little-endian byte-order for field element encoding. (*)¶

• Poplar1: Move the last step of sketch evaluation from prep_next() to prep_shares_to_prep().¶

04:¶

• Align security considerations with the security analysis of [DPRS23].¶

• Vdaf: Pass the nonce to the sharding algorithm.¶

• Vdaf: Rather than allow the application to choose the nonce length, have each implementation of the Vdaf interface specify the expected nonce length. (*)¶

• Prg: Split "info string" into two components: the "customization string", intended for domain separation; and the "binder string", used to bind the output to ephemeral values, like the nonce, associated with execution of a (V)DAF.¶

• Replace PrgAes128 with PrgSha3, an implementation of the Prg interface based on SHA-3, and use the new scheme as the default. Accordingly, replace Prio3Aes128Count with Prio3Count, Poplar1Aes128 with Poplar1, and so on. SHA-3 is a safer choice for instantiating a random oracle, which is used in the analysis of Prio3 of [DPRS23]. (*)¶

• Prio3, Poplar1: Ensure each invocation of the Prg uses a distinct customization string, as suggested by [DPRS23]. This is intended to make domain separation clearer, thereby simplifying security analysis. (*)¶

• Prio3: Replace "joint randomness hints" sent in each input share with "joint randomness parts" sent in the public share. This reduces communication overhead when the number of shares exceeds two. (*)¶

• Prio3: Bind nonce to joint randomness parts. This is intended to address birthday attacks on robustness pointed out by [DPRS23]. (*)¶

• Poplar1: Use different Prg invocations for producing the correlated randomness for inner and leaf nodes of the IDPF tree. This is intended to simplify implementations. (*)¶

• Poplar1: Don't bind the candidate prefixes to the verifier randomness. This is intended to improve performance, while not impacting security. According to the analysis of [DPRS23], it is necessary to restrict Poplar1 usage such that no report is aggregated more than once at a given level of the IDPF tree; otherwise, attacks on privacy may be possible. In light of this restriction, there is no added benefit of binding to the prefixes themselves. (*)¶

• Poplar1: During preparation, assert that all candidate prefixes are unique and appear in order. Uniqueness is required to avoid erroneously rejecting a valid report; the ordering constraint ensures the uniqueness check can be performed efficiently. (*)¶

• Poplar1: Increase the maximum candidate prefix count in the encoding of the aggregation parameter. (*)¶

• Poplar1: Bind the nonce to the correlated randomness derivation. This is intended to provide defense-in-depth by ensuring the Aggregators reject the report if the nonce does not match what the Client used for sharding. (*)¶

• Poplar1: Clarify that the aggregation parameter encoding is OPTIONAL. Accordingly, update implementation considerations around cross-aggregation state.¶

• IdpfPoplar: Add implementation considerations around branching on the values of control bits.¶

• IdpfPoplar: When decoding the the control bits in the public share, assert that the trailing bits of the final byte are all zero. (*)¶

03:¶

• Define codepoints for (V)DAFs and use them for domain separation in Prio3 and Poplar1. (*)¶

• Prio3: Align joint randomness computation with revised paper [BBCGGI19]. This change mitigates an attack on robustness. (*)¶

• Prio3: Remove an intermediate PRG evaluation from query randomness generation. (*)¶

• Add additional guidance for choosing FFT-friendly fields.¶

02:¶

• Complete the initial specification of Poplar1.¶

• Extend (V)DAF syntax to include a "public share" output by the Client and distributed to all of the Aggregators. This is to accommodate "extractable" IDPFs as required for Poplar1. (See [BBCGGI21], Section 4.3 for details.)¶

• Extend (V)DAF syntax to allow the unsharding step to take into account the number of measurements aggregated.¶

• Extend FLP syntax by adding a method for decoding the aggregate result from a vector of field elements. The new method takes into account the number of measurements.¶

• Prio3: Align aggregate result computation with updated FLP syntax.¶

• Prg: Add a method for statefully generating a vector of field elements.¶

• Field: Require that field elements are fully reduced before decoding. (*)¶

• Define new field Field255.¶

01:¶

• Require that VDAFs specify serialization of aggregate shares.¶

• Define Distributed Aggregation Functions (DAFs).¶

• Prio3: Move proof verifier check from prep_next() to prep_shares_to_prep(). (*)¶

• Remove public parameter and replace verification parameter with a "verification key" and "Aggregator ID".¶

4.1. Sharding

In order to protect the privacy of its measurements, a DAF Client shards its measurements into a sequence of input shares. The shard method is used for this purpose.¶

• Daf.shard(measurement: Measurement, nonce: bytes[Daf.NONCE_SIZE], rand: bytes[Daf.RAND_SIZE]) -> tuple[PublicShare, list[InputShare]] is the randomized sharding algorithm run by each Client. The input rand consists of the random bytes consumed by the algorithm. This value MUST be generated using a cryptographically secure pseudorandom number generator (CSPRNG). It consumes the measurement and produces a "public share", distributed to each of the Aggregators, and a corresponding sequence of input shares, one for each Aggregator. The length of the output vector MUST be SHARES.¶

    Client
    ======

    measurement
      |
      V
    +----------------------------------------------+
    | shard                                        |
    +----------------------------------------------+
      |              |              |     |
      |              |         ...  |    public_share
      |              |              |     |
      |    +---------|-----+--------|-----+
      |    |         |     |        |     |
      V    |         V     |        V     |
     input_share_0  input_share_1  input_share_[SHARES-1]
      |    |         |     |   ...  |     |
      V    V         V     V        V     V
    Aggregator 0   Aggregator 1    Aggregator SHARES-1

4.2. Preparation

Once an Aggregator has received the public share and one of the input shares, the next step is to prepare the input share for aggregation. This is accomplished using the following algorithm:¶

• Daf.prep(agg_id: Unsigned, agg_param: AggParam, nonce: bytes[NONCE_SIZE], public_share: PublicShare, input_share: InputShare) -> OutShare is the deterministic preparation algorithm. It takes as input the public share and one of the input shares generated by a Client, the Aggregator's unique identifier, the aggregation parameter selected by the Collector, and a nonce and returns an output share.¶

The protocol in which the DAF is used MUST ensure that the Aggregator's identifier is equal to the integer in range [0, SHARES) that matches the index of input_share in the sequence of input shares output by the Client.¶

4.3. Validity of Aggregation Parameters

Concrete DAFs implementations MAY impose certain restrictions for input shares and aggregation parameters. Protocols using a DAF MUST ensure that for each input share and aggregation parameter agg_param, Daf.prep is only called if Daf.is_valid(agg_param, previous_agg_params) returns True, where previous_agg_params contains all aggregation parameters that have previously been used with the same input share.¶

DAFs MUST implement the following function:¶

• Daf.is_valid(agg_param: AggParam, previous_agg_params: set[AggParam]) -> Bool: Checks if the agg_param is compatible with all elements of previous_agg_params.¶

4.4. Aggregation

Once an Aggregator holds output shares for a batch of measurements (where batches are defined by the application), it combines them into a share of the desired aggregate result:¶

• Daf.aggregate(agg_param: AggParam, out_shares: list[OutShare]) -> AggShare is the deterministic aggregation algorithm. It is run by each Aggregator a set of recovered output shares.¶

    Aggregator 0    Aggregator 1        Aggregator SHARES-1
    ============    ============        ===================

    out_share_0_0   out_share_1_0       out_share_[SHARES-1]_0
    out_share_0_1   out_share_1_1       out_share_[SHARES-1]_1
    out_share_0_2   out_share_1_2       out_share_[SHARES-1]_2
         ...             ...                     ...
    out_share_0_B   out_share_1_B       out_share_[SHARES-1]_B
      |               |                   |
      V               V                   V
    +-----------+   +-----------+       +-----------+
    | aggregate |   | aggregate |   ... | aggregate |
    +-----------+   +-----------+       +-----------+
      |               |                   |
      V               V                   V
    agg_share_0     agg_share_1         agg_share_[SHARES-1]

For simplicity, we have written this algorithm in a "one-shot" form, where all output shares for a batch are provided at the same time. Many DAFs may also support a "streaming" form, where shares are processed one at a time.¶

Implementation note: For most natural DAFs (and VDAFs) it is not necessary for an Aggregator to store all output shares individually before aggregating. Typically it is possible to merge output shares into aggregate shares as they arrive, merge these into other aggregate shares, and so on. In particular, this is the case when the output shares are vectors over some finite field and aggregating them involves merely adding up the vectors element-wise. Such is the case for Prio3 Section 7 and Poplar1 Section 8.¶

4.5. Unsharding

After the Aggregators have aggregated a sufficient number of output shares, each sends its aggregate share to the Collector, who runs the following algorithm to recover the following output:¶

• Daf.unshard(agg_param: AggParam, agg_shares: list[AggShare], num_measurements: Unsigned) -> AggResult is run by the Collector in order to compute the aggregate result from the Aggregators' shares. The length of agg_shares MUST be SHARES. num_measurements is the number of measurements that contributed to each of the aggregate shares. This algorithm is deterministic.¶

    Aggregator 0    Aggregator 1        Aggregator SHARES-1
    ============    ============        ===================

    agg_share_0     agg_share_1         agg_share_[SHARES-1]
      |               |                   |
      V               V                   V
    +-----------------------------------------------+
    | unshard                                       |
    +-----------------------------------------------+
      |
      V
    agg_result

    Collector
    =========

• QUESTION Maybe the aggregation algorithms should be randomized in order to allow the Aggregators (or the Collector) to add noise for differential privacy. (See the security considerations of [DAP].) Or is this out-of-scope of this document? See https://github.com/ietf-wg-ppm/ppm-specification/issues/19.¶

4.6. Execution of a DAF

Securely executing a DAF involves emulating the following procedure.¶

def run_daf(Daf,
            agg_param: Daf.AggParam,
            measurements: list[Daf.Measurement],
            nonces: list[bytes[Daf.NONCE_SIZE]]):
    out_shares = [[] for j in range(Daf.SHARES)]
    for (measurement, nonce) in zip(measurements, nonces):
        # Each Client shards its measurement into input shares and
        # distributes them among the Aggregators.
        rand = gen_rand(Daf.RAND_SIZE)
        (public_share, input_shares) = \
            Daf.shard(measurement, nonce, rand)

        # Each Aggregator prepares its input share for aggregation.
        for j in range(Daf.SHARES):
            out_shares[j].append(
                Daf.prep(j, agg_param, nonce,
                         public_share, input_shares[j]))

    # Each Aggregator aggregates its output shares into an aggregate
    # share and sends it to the Collector.
    agg_shares = []
    for j in range(Daf.SHARES):
        agg_share_j = Daf.aggregate(agg_param,
                                    out_shares[j])
        agg_shares.append(agg_share_j)

    # Collector unshards the aggregate result.
    num_measurements = len(measurements)
    agg_result = Daf.unshard(agg_param, agg_shares,
                             num_measurements)
    return agg_result

The inputs to this procedure are the same as the aggregation function computed by the DAF: An aggregation parameter and a sequence of measurements. The procedure prescribes how a DAF is executed in a "benign" environment in which there is no adversary and the messages are passed among the protocol participants over secure point-to-point channels. In reality, these channels need to be instantiated by some "wrapper protocol", such as [DAP], that realizes these channels using suitable cryptographic mechanisms. Moreover, some fraction of the Aggregators (or Clients) may be malicious and diverge from their prescribed behaviors. Section 9 describes the execution of the DAF in various adversarial environments and what properties the wrapper protocol needs to provide in each.¶

5.1. Sharding

Sharding transforms a measurement into input shares as it does in DAFs (cf. Section 4.1); in addition, it takes a nonce as input and produces a public share:¶

• Vdaf.shard(measurement: Measurement, nonce: bytes[Vdaf.NONCE_SIZE], rand: bytes[Vdaf.RAND_SIZE]) -> tuple[PublicShare, list[InputShare]] is the randomized sharding algorithm run by each Client. Input rand consists of the random bytes consumed by the algorithm. It consumes the measurement and the nonce and produces a public share, distributed to each of Aggregators, and the corresponding sequence of input shares, one for each Aggregator. Depending on the VDAF, the input shares may encode additional information used to verify the recovered output shares (e.g., the "proof shares" in Prio3 Section 7). The length of the output vector MUST be SHARES.¶

In order to ensure privacy of the measurement, the Client MUST generate the random bytes and nonce using a CSPRNG. (See Section 9 for details.)¶

5.2. Preparation

To recover and verify output shares, the Aggregators interact with one another over ROUNDS rounds. Prior to each round, each Aggregator constructs an outbound message. Next, the sequence of outbound messages is combined into a single message, called a "preparation message", or "prep message" for short. (Each of the outbound messages are called "preparation-message shares", or "prep shares" for short.) Finally, the preparation message is distributed to the Aggregators to begin the next round.¶

An Aggregator begins the first round with its input share and it begins each subsequent round with the previous prep message. Its output in the last round is its output share and its output in each of the preceding rounds is a prep share.¶

This process involves a value called the "aggregation parameter" used to map the input shares to output shares. The Aggregators need to agree on this parameter before they can begin preparing the measurement shares for aggregation.¶

    Aggregator 0   Aggregator 1        Aggregator SHARES-1
    ============   ============        ===================

    input_share_0  input_share_1       input_share_[SHARES-1]
      |              |              ...  |
      V              V                   V
    +-----------+  +-----------+       +-----------+
    | prep_init |  | prep_init |       | prep_init |
    +-----------+  +------------+      +-----------+
      |              |              ...  |
      V              V                   V
    +----------------------------------------------+   \
    | prep_shares_to_prep                          |   |
    +----------------------------------------------+   |
      |              |              ...  |             |
      V              V                   V             | x ROUNDS
    +-----------+  +-----------+       +-----------+   |
    | prep_next |  | prep_next |       | prep_next |   |
    +-----------+  +-----------+       +-----------+   |
      |              |              ...  |             |
      V              V                   V             /
     ...            ...                 ...
      |              |              ...  |
      V              V                   V
    out_share_0    out_share_1         out_share_[SHARES-1]

To facilitate the preparation process, a concrete VDAF implements the following methods:¶

• Vdaf.prep_init(verify_key: bytes[Vdaf.VERIFY_KEY_SIZE], agg_id: Unsigned, agg_param: AggParam, nonce: bytes[Vdaf.NONCE_SIZE], public_share: PublicShare, input_share: InputShare) -> tuple[PrepState, PrepShare] is the deterministic preparation-state initialization algorithm run by each Aggregator to begin processing its input share into an output share. Its inputs are the shared verification key (verify_key), the Aggregator's unique identifier (agg_id), the aggregation parameter (agg_param), the nonce provided by the environment (nonce, see Figure 7), the public share (public_share), and one of the input shares generated by the Client (input_share). Its output is the Aggregator's initial preparation state and initial prep share.¶

  It is up to the high level protocol in which the VDAF is used to arrange for the distribution of the verification key prior to generating and processing reports. (See Section 9 for details.)¶

  Protocols using the VDAF MUST ensure that the Aggregator's identifier is equal to the integer in range [0, SHARES) that matches the index of input_share in the sequence of input shares output by the Client.¶

  Protocols MUST ensure that public share consumed by each of the Aggregators is identical. This is security critical for VDAFs such as Poplar1.¶

• Vdaf.prep_next(prep_state: PrepState, prep_msg: PrepMessage) -> Union[tuple[PrepState, PrepShare], OutShare] is the deterministic preparation-state update algorithm run by each Aggregator. It updates the Aggregator's preparation state (prep_state) and returns either its next preparation state and its message share for the current round or, if this is the last round, its output share. An exception is raised if a valid output share could not be recovered. The input of this algorithm is the inbound preparation message.¶

• Vdaf.prep_shares_to_prep(agg_param: AggParam, prep_shares: list[PrepShare]) -> PrepMessage is the deterministic preparation-message pre-processing algorithm. It combines the prep shares generated by the Aggregators in the previous round into the prep message consumed by each in the next round.¶

In effect, each Aggregator moves through a linear state machine with ROUNDS states. The Aggregator enters the first state on using the initialization algorithm, and the update algorithm advances the Aggregator to the next state. Thus, in addition to defining the number of rounds (ROUNDS), a VDAF instance defines the state of the Aggregator after each round.¶

• TODO Consider how to bake this "linear state machine" condition into the syntax. Given that Python 3 is used as our pseudocode, it's easier to specify the preparation state using a class.¶

The preparation-state update accomplishes two tasks: recovery of output shares from the input shares and ensuring that the recovered output shares are valid. The abstraction boundary is drawn so that an Aggregator only recovers an output share if it is deemed valid (at least, based on the Aggregator's view of the protocol). Another way to draw this boundary would be to have the Aggregators recover output shares first, then verify that they are valid. However, this would allow the possibility of misusing the API by, say, aggregating an invalid output share. Moreover, in protocols like Prio+ [AGJOP21] based on oblivious transfer, it is necessary for the Aggregators to interact in order to recover aggregatable output shares at all.¶

Note that it is possible for a VDAF to specify ROUNDS == 0, in which case each Aggregator runs the preparation-state update algorithm once and immediately recovers its output share without interacting with the other Aggregators. However, most, if not all, constructions will require some amount of interaction in order to ensure validity of the output shares (while also maintaining privacy).¶

• OPEN ISSUE accommodating 0-round VDAFs may require syntax changes if, for example, public keys are required. On the other hand, we could consider defining this class of schemes as a different primitive. See issue#77.¶

5.3. Validity of Aggregation Parameters

Similar to DAFs (see Section 4.3), VDAFs MAY impose restrictions for input shares and aggregation parameters. Protocols using a VDAF MUST ensure that for each input share and aggregation parameter agg_param, the preparation phase (including Vdaf.prep_init, Vdaf.prep_next, and Vdaf.prep_shares_to_prep; see Section 5.2) is only called if Vdaf.is_valid(agg_param, previous_agg_params) returns True, where previous_agg_params contains all aggregation parameters that have previously been used with the same input share.¶

VDAFs MUST implement the following function:¶

• Vdaf.is_valid(agg_param: AggParam, previous_agg_params: set[AggParam]) -> Bool: Checks if the agg_param is compatible with all elements of previous_agg_params.¶

5.4. Aggregation

VDAF Aggregation is identical to DAF Aggregation (cf. Section 4.4):¶

• Vdaf.aggregate(agg_param: AggParam, out_shares: list[OutShare]) -> AggShare is the deterministic aggregation algorithm. It is run by each Aggregator over the output shares it has computed for a batch of measurements.¶

The data flow for this stage is illustrated in Figure 3. Here again, we have the aggregation algorithm in a "one-shot" form, where all shares for a batch are provided at the same time. VDAFs typically also support a "streaming" form, where shares are processed one at a time.¶

5.5. Unsharding

VDAF Unsharding is identical to DAF Unsharding (cf. Section 4.5):¶

• Vdaf.unshard(agg_param: AggParam, agg_shares: list[AggShare], num_measurements: Unsigned) -> AggResult is run by the Collector in order to compute the aggregate result from the Aggregators' shares. The length of agg_shares MUST be SHARES. num_measurements is the number of measurements that contributed to each of the aggregate shares. This algorithm is deterministic.¶

The data flow for this stage is illustrated in Figure 4.¶

5.6. Execution of a VDAF

Secure execution of a VDAF involves simulating the following procedure.¶

def run_vdaf(Vdaf,
             verify_key: bytes[Vdaf.VERIFY_KEY_SIZE],
             agg_param: Vdaf.AggParam,
             nonces: list[bytes[Vdaf.NONCE_SIZE]],
             measurements: list[Vdaf.Measurement]):
    out_shares = []
    for (nonce, measurement) in zip(nonces, measurements):
        # Each Client shards its measurement into input shares.
        rand = gen_rand(Vdaf.RAND_SIZE)
        (public_share, input_shares) = \
            Vdaf.shard(measurement, nonce, rand)

        # Each Aggregator initializes its preparation state.
        prep_states = []
        outbound = []
        for j in range(Vdaf.SHARES):
            (state, share) = Vdaf.prep_init(verify_key, j,
                                            agg_param,
                                            nonce,
                                            public_share,
                                            input_shares[j])
            prep_states.append(state)
            outbound.append(share)

        # Aggregators recover their output shares.
        for i in range(Vdaf.ROUNDS-1):
            prep_msg = Vdaf.prep_shares_to_prep(agg_param,
                                                outbound)
            outbound = []
            for j in range(Vdaf.SHARES):
                out = Vdaf.prep_next(prep_states[j], prep_msg)
                (prep_states[j], out) = out
                outbound.append(out)

        # The final outputs of the prepare phase are the output shares.
        prep_msg = Vdaf.prep_shares_to_prep(agg_param,
                                            outbound)
        outbound = []
        for j in range(Vdaf.SHARES):
            out_share = Vdaf.prep_next(prep_states[j], prep_msg)
            outbound.append(out_share)
        out_shares.append(outbound)

    # Each Aggregator aggregates its output shares into an
    # aggregate share. In a distributed VDAF computation, the
    # aggregate shares are sent over the network.
    agg_shares = []
    for j in range(Vdaf.SHARES):
        out_shares_j = [out[j] for out in out_shares]
        agg_share_j = Vdaf.aggregate(agg_param, out_shares_j)
        agg_shares.append(agg_share_j)

    # Collector unshards the aggregate.
    num_measurements = len(measurements)
    agg_result = Vdaf.unshard(agg_param, agg_shares,
                              num_measurements)
    return agg_result

The inputs to this algorithm are the aggregation parameter, a list of measurements, and a nonce for each measurement. This document does not specify how the nonces are chosen, but security requires that the nonces be unique. See Section 9 for details. As explained in Section 4.6, the secure execution of a VDAF requires the application to instantiate secure channels between each of the protocol participants.¶

5.7. Communication Patterns for Preparation

In each round of preparation, each Aggregator writes a prep share to some broadcast channel, which is then processed into the prep message using the public prep_shares_to_prep() algorithm and broadcast to the Aggregators to start the next round. In this section we describe some approaches to realizing this broadcast channel functionality in protocols that use VDAFs.¶

The state machine of each Aggregator is shown in Figure 8.¶

               +----------------+
               |                |
               v                |
Start ------> Continued(prep_state) --> Finished(out_share)
 |                |
 |                |
 +--> Rejected <--+

State transitions are made when the state is acted upon by the host's local inputs and/or messages sent by the peers. The initial state is Start. The terminal states are Rejected, which indicates that the report cannot be processed any further, and Finished(out_share), which indicates that the Aggregator has recovered an output share out_share.¶

class State:
    pass

class Start(State):
    pass

class Continued(State):
    def __init__(self, prep_state):
        self.prep_state = prep_state

class Finished(State):
    def __init__(self, output_share):
        self.output_share = output_share

class Rejected(State):
    def __init__(self):
        pass

Note that there is no representation of the Start state as it is never instantiated in the ping-pong topology.¶

For convenience, the methods described in this section are defined in terms of opaque byte strings. A compatible Vdaf MUST specify methods for encoding public shares, input shares, prep shares, prep messages, and aggregation parameters. Minimally:¶

• Vdaf.decode_public_share(encoded: bytes) -> Vdaf.PublicShare decodes a public share.¶

• Vdaf.decode_input_share(agg_id: Unsigned, encoded: bytes) -> Vdaf.InputShare decodes an input share, using the aggregator ID as optional context.¶

• Vdaf.encode_prep_share(prep_share: Vdaf.PrepShare) -> bytes encodes a prep share.¶

• Vdaf.decode_prep_share(prep_state: Vdaf.PrepState, encoded: bytes) -> Vdaf.PrepShare decodes a prep share, using the prep state as optional context.¶

• Vdaf.encode_prep_msg(prep_msg: Vdaf.PrepMessage) -> bytes encodes a prep message.¶

• Vdaf.decode_prep_msg(prep_state: Vdaf.PrepState, encoded: bytes) -> Vdaf.PrepMessage decodes a prep message, using the prep state as optional decoding context.¶

• Vdaf.decode_agg_param(encoded: bytes) -> Vdaf.AggParam decodes an aggregation parameter.¶

• Vdaf.encode_agg_param(agg_param: Vdaf.AggParam) -> bytes encodes an aggregation parameter.¶

Implementations of Prio3 and Poplar1 MUST use the encoding scheme specified in Section 7.2.7 and Section 8.2.6 respectively.¶

5.8. Ping-Pong Topology (Only Two Aggregators)

For VDAFs with precisely two Aggregators (i.e., Vdaf.SHARES == 2), the following "ping pong" communication pattern can be used. It is compatible with any request/response transport protocol, such as HTTP.¶

Let us call the initiating party the "Leader" and the responding party the "Helper". The high-level idea is that the Leader and Helper will take turns running the computation locally until input from their peer is required:¶

• For a 1-round VDAF (e.g., Prio3 (Section 7)), the Leader sends its prep share to the Helper, who computes the prep message locally, computes its output share, then sends the prep message to the Leader. Preparation requires just one round trip between the Leader and the Helper.¶

• For a 2-round VDAF (e.g., Poplar1 (Section 8)), the Leader sends its first-round prep share to the Helper, who replies with the first-round prep message and its second-round prep share. In the next request, the Leader computes its second-round prep share locally, computes its output share, and sends the second-round prep message to the Helper. Finally, the Helper computes its own output share.¶

• In general, each request includes the Leader's prep share for the previous round and/or the prep message for the current round; correspondingly, each response consists of the prep message for the current round and the Helper's prep share for the next round.¶

The Aggregators proceed in this ping-ponging fashion until a step of the computation fails (indicating the report is invalid and should be rejected) or preparation is completed. All told there there are ceil((Vdaf.ROUNDS+1)/2) requests sent.¶

Each message in the ping-pong protocol is structured as follows (expressed in TLS syntax as defined in Section 3 of [RFC8446]):¶

enum {
  initialize(0),
  continue(1),
  finish(2),
  (255)
} MessageType;

struct {
  MessageType type;
  select (Message.type) {
    case initialize:
      opaque prep_share<0..2^32-1>;
    case continue:
      opaque prep_msg<0..2^32-1>;
      opaque prep_share<0..2^32-1>;
    case finish:
      opaque prep_msg<0..2^32-1>;
  };
} Message;

These messages are used to transition between the states described in Section 5.7. They are encoded and decoded to or from byte buffers as described Section 3 of [RFC8446]) using the following routines:¶

• encode_ping_pong_message(message: Message) -> bytes encodes a Message into an opaque byte buffer.¶

• decode_pong_pong_message(encoded: bytes) -> Message decodes an opaque byte buffer into a Message, raising an error if the bytes are not a valid encoding.¶

The Leader's initial transition is computed with the following procedure:¶

def ping_pong_leader_init(
            Vdaf,
            vdaf_verify_key: bytes[Vdaf.VERIFY_KEY_SIZE],
            agg_param: bytes,
            nonce: bytes[Vdaf.NONCE_SIZE],
            public_share: bytes,
            input_share: bytes,
        ) -> tuple[State, bytes]:
    try:
        (prep_state, prep_share) = Vdaf.prep_init(
            vdaf_verify_key,
            0,
            Vdaf.decode_agg_param(agg_param),
            nonce,
            Vdaf.decode_public_share(public_share),
            Vdaf.decode_input_share(0, input_share),
        )
        outbound = Message.initialize(
            Vdaf.encode_prep_share(prep_share))
        return (Continued(prep_state), encode_ping_pong_message(outbound))
    except:
        return (Rejected(), None)

The output is the State to which the Leader has transitioned and an encoded Message. If the Leader's state is Rejected, then processing halts. Otherwise, if the state is Continued, then processing continues.¶

The Leader sends the outbound message to the Helper. The Helper's initial transition is computed using the following procedure:¶

def ping_pong_helper_init(
            Vdaf,
            vdaf_verify_key: bytes[Vdaf.VERIFY_KEY_SIZE],
            agg_param: bytes,
            nonce: bytes[Vdaf.NONCE_SIZE],
            public_share: bytes,
            input_share: bytes,
            inbound_encoded: bytes,
        ) -> tuple[State, bytes]:
    try:
        (prep_state, prep_share) = Vdaf.prep_init(
            vdaf_verify_key,
            1,
            Vdaf.decode_agg_param(agg_param),
            nonce,
            Vdaf.decode_public_share(public_share),
            Vdaf.decode_input_share(1, input_share),
        )

        inbound = decode_ping_pong_message(inbound_encoded)

        if inbound.type != 0: # initialize
            return (Rejected(), None)

        prep_shares = [
            Vdaf.decode_prep_share(prep_state, inbound.prep_share),
            prep_share,
        ]
        return Vdaf.ping_pong_transition(
            agg_param,
            prep_shares,
            prep_state,
        )
    except:
        return (Rejected(), None)

Procedure ping_pong_transition() takes in the prep shares, combines them into the prep message, and computes the next prep state of the caller:¶

def ping_pong_transition(
            Vdaf,
            agg_param: Vdaf.AggParam,
            prep_shares: list[Vdaf.PrepShare],
            prep_state: Vdaf.PrepState,
         ) -> (State, bytes):
    prep_msg = Vdaf.prep_shares_to_prep(agg_param,
                                        prep_shares)
    out = Vdaf.prep_next(prep_state, prep_msg)
    if type(out) == Vdaf.OutShare:
        outbound = Message.finish(Vdaf.encode_prep_msg(prep_msg))
        return (Finished(out), encode_ping_pong_message(outbound))
    (prep_state, prep_share) = out
    outbound = Message.continue(
        Vdaf.encode_prep_msg(prep_msg),
        Vdaf.encode_prep_share(prep_share),
    )
    return (Continued(prep_state), encode_ping_pong_message(outbound))

The output is the State to which the Helper has transitioned and an encoded Message. If the Helper's state is Finished or Rejected, then processing halts. Otherwise, if the state is Continued, then processing continues.¶

Next, the Helper sends the outbound message to the Leader. The Leader computes its next state transition using the function ping_pong_leader_continued:¶

def ping_pong_leader_continued(
            Vdaf,
            agg_param: bytes,
            state: State,
            inbound_encoded: bytes,
        ) -> (State, Optional[bytes]):
    return Vdaf.ping_pong_continued(
        True,
        agg_param,
        state,
        inbound_encoded,
    )

def ping_pong_continued(
            Vdaf,
            is_leader: bool,
            agg_param: bytes,
            state: State,
            inbound_encoded: bytes,
        ) -> (State, Optional[bytes]):
    try:
        inbound = decode_ping_pong_message(inbound_encoded)

        if inbound.type == 0: # initialize
            return (Rejected(), None)

        if !isinstance(state, Continued):
            return (Rejected(), None)

        prep_msg = Vdaf.decode_prep_msg(state.prep_state, inbound.prep_msg)
        out = Vdaf.prep_next(state.prep_state, prep_msg)
        if type(out) == tuple[Vdaf.PrepState, Vdaf.PrepShare] \
                and inbound.type == 1:
            # continue
            (prep_state, prep_share) = out
            prep_shares = [
                Vdaf.decode_prep_share(prep_state, inbound.prep_share),
                prep_share,
            ]
            if is_leader:
                prep_shares.reverse()
            return Vdaf.ping_pong_transition(
                Vdaf.decode_agg_param(agg_param),
                prep_shares,
                prep_state,
            )
        elif type(out) == Vdaf.OutShare and inbound.type == 2:
            # finish
            return (Finished(out), None)
        else:
            return (Rejected(), None)

    except:
        return (Rejected(), None)

If the Leader's state is Finished or Rejected, then processing halts. Otherwise, the Leader sends the outbound message to the Helper. The Helper computes its next state transition using the function ping_pong_helper_continued:¶

def ping_pong_helper_continued(
            Vdaf,
            agg_param: bytes,
            state: State,
            inbound_encoded: bytes,
        ) -> (State, Optional[bytes]):
    return Vdaf.ping_pong_continued(
        False,
        agg_param,
        state,
        inbound_encoded,
    )

They continue in this way until processing halts. Note that, depending on the number of rounds of preparation that are required, there may be one more message to send before the peer can also finish processing (i.e., outbound != None).¶

5.9. Star Topology (Any Number of Aggregators)

The ping-pong topology of the previous section is only suitable for VDAFs involving exactly two Aggregators. If more Aggregators are required, the star topology described in this section can be used instead.¶

• TODO Describe the Leader-emulated broadcast channel architecture that was originally envisioned for DAP. (As of DAP-05 we are going with the ping pong architecture described in the previous section.)¶

6.1. Finite Fields

Both Prio3 and Poplar1 use finite fields of prime order. Finite field elements are represented by a class Field with the following associated parameters:¶

• MODULUS: Unsigned is the prime modulus that defines the field.¶

• ENCODED_SIZE: Unsigned is the number of bytes used to encode a field element as a byte string.¶

A concrete Field also implements the following class methods:¶

• Field.zeros(length: Unsigned) -> output: Vec[Field] returns a vector of zeros. The length of output MUST be length.¶

• Field.rand_vec(length: Unsigned) -> output: Vec[Field] returns a vector of random field elements. The length of output MUST be length.¶

A field element is an instance of a concrete Field. The concrete class defines the usual arithmetic operations on field elements. In addition, it defines the following instance method for converting a field element to an unsigned integer:¶

• elem.as_unsigned() -> Unsigned returns the integer representation of field element elem.¶

Likewise, each concrete Field implements a constructor for converting an unsigned integer into a field element:¶

• Field(integer: Unsigned) returns integer represented as a field element. The value of integer MUST be less than Field.MODULUS.¶

Each concrete Field has two derived class methods, one for encoding a vector of field elements as a byte string and another for decoding a vector of field elements.¶

def encode_vec(Field, data: Vec[Field]) -> Bytes:
    encoded = Bytes()
    for x in data:
        encoded += to_le_bytes(x.as_unsigned(), Field.ENCODED_SIZE)
    return encoded

def decode_vec(Field, encoded: Bytes) -> Vec[Field]:
    L = Field.ENCODED_SIZE
    if len(encoded) % L != 0:
        raise ERR_DECODE

    vec = []
    for i in range(0, len(encoded), L):
        encoded_x = encoded[i:i+L]
        x = from_le_bytes(encoded_x)
        if x >= Field.MODULUS:
            raise ERR_DECODE # Integer is larger than modulus
        vec.append(Field(x))
    return vec

Finally, Field implements the following methods for representing a value as a sequence of field elements, each of which represents a bit of the input.¶

def encode_into_bit_vector(Field,
                           val: Unsigned,
                           bits: Unsigned) -> Vec[Field]:
    """
    Encode the bit representation of `val` with at most `bits` number
    of bits, as a vector of field elements.
    """
    if val >= 2 ** bits:
        # Sanity check we are able to represent `val` with `bits`
        # number of bits.
        raise ValueError("Number of bits is not enough to represent "
                         "the input integer.")
    encoded = []
    for l in range(bits):
        encoded.append(Field((val >> l) & 1))
    return encoded

def decode_from_bit_vector(Field, vec: Vec[Field]) -> Field:
    """
    Decode the field element from the bit representation, expressed
    as a vector of field elements `vec`.
    """
    bits = len(vec)
    if Field.MODULUS >> bits == 0:
        raise ValueError("Number of bits is too large to be "
                         "represented by field modulus.")
    decoded = Field(0)
    for (l, bit) in enumerate(vec):
        decoded += Field(1 << l) * bit
    return decoded

def vec_sub(left: Vec[Field], right: Vec[Field]):
    """
    Subtract the right operand from the left and return the result.
    """
    return list(map(lambda x: x[0] - x[1], zip(left, right)))

def vec_add(left: Vec[Field], right: Vec[Field]):
    """Add the right operand to the left and return the result."""
    return list(map(lambda x: x[0] + x[1], zip(left, right)))

6.2. Extendable Output Functions

VDAFs in this specification use extendable output functions (XOFs) to extract short, fixed-length strings we call "seeds" from long input strings and expand seeds into long output strings. We specify a single interface that is suitable for both purposes.¶

XOFs are defined by a class Xof with the following associated parameter and methods:¶

• SEED_SIZE: Unsigned is the size (in bytes) of a seed.¶

• Xof(seed: Bytes[Xof.SEED_SIZE], dst: Bytes, binder: Bytes) constructs an instance of Xof from the given seed, domain separation tag, and binder string. (See below for definitions of these.) The seed MUST be of length SEED_SIZE and MUST be generated securely (i.e., it is either the output of gen_rand or a previous invocation of the XOF).¶

• xof.next(length: Unsigned) returns the next length bytes of output of xof.¶

Each Xof has two derived methods. The first is used to derive a fresh seed from an existing one. The second is used to compute a sequence of field elements.¶

def derive_seed(Xof,
                seed: Bytes[Xof.SEED_SIZE],
                dst: Bytes,
                binder: Bytes):
    """Derive a new seed."""
    xof = Xof(seed, dst, binder)
    return xof.next(Xof.SEED_SIZE)

def next_vec(self, Field, length: Unsigned):
    """Output the next `length` elements of `Field`."""
    m = next_power_of_2(Field.MODULUS) - 1
    vec = []
    while len(vec) < length:
        x = from_le_bytes(self.next(Field.ENCODED_SIZE))
        x &= m
        if x < Field.MODULUS:
            vec.append(Field(x))
    return vec

def expand_into_vec(Xof,
                    Field,
                    seed: Bytes[Xof.SEED_SIZE],
                    dst: Bytes,
                    binder: Bytes,
                    length: Unsigned):
    """
    Expand the input `seed` into vector of `length` field elements.
    """
    xof = Xof(seed, dst, binder)
    return xof.next_vec(Field, length)

class XofTurboShake128(Xof):
    """XOF wrapper for TurboSHAKE128."""

    # Associated parameters
    SEED_SIZE = 16

    def __init__(self, seed, dst, binder):
        self.l = 0
        self.m = to_le_bytes(len(dst), 1) + dst + seed + binder

    def next(self, length: Unsigned) -> Bytes:
        self.l += length

        # Function `TurboSHAKE128(M, D, L)` is as defined in
        # Section 2.2 of [TurboSHAKE].
        #
        # Implementation note: Rather than re-generate the output
        # stream each time `next()` is invoked, most implementations
        # of TurboSHAKE128 will expose an "absorb-then-squeeze" API that
        # allows stateful handling of the stream.
        stream = TurboSHAKE128(self.m, 1, self.l)
        return stream[-length:]

class XofFixedKeyAes128(Xof):
    """
    XOF based on a circular collision-resistant hash function from
    fixed-key AES.
    """

    # Associated parameters
    SEED_SIZE = 16

    def __init__(self, seed, dst, binder):
        self.length_consumed = 0

        # Use TurboSHAKE128 to derive a key from the binder string and
        # domain separation tag. Note that the AES key does not need
        # to be kept secret from any party. However, when used with
        # IdpfPoplar, we require the binder to be a random nonce.
        #
        # Implementation note: This step can be cached across XOF
        # evaluations with many different seeds.
        dst_length = to_le_bytes(len(dst), 1)
        self.fixed_key = TurboSHAKE128(dst_length + dst + binder, 2, 16)
        self.seed = seed

    def next(self, length: Unsigned) -> Bytes:
        offset = self.length_consumed % 16
        new_length = self.length_consumed + length
        block_range = range(
            int(self.length_consumed / 16),
            int(new_length / 16) + 1)
        self.length_consumed = new_length

        hashed_blocks = [
            self.hash_block(xor(self.seed, to_le_bytes(i, 16))) \
                         for i in block_range
        ]
        return concat(hashed_blocks)[offset:offset+length]

    def hash_block(self, block):
        """
        The multi-instance tweakable circular correlation-robust hash
        function of [GKWWY20] (Section 4.2). The tweak here is the key
        that stays constant for all XOF evaluations of the same Client,
        but differs between Clients.

        Function `AES128(key, block)` is the AES-128 blockcipher.
        """
        lo, hi = block[:8], block[8:]
        sigma_block = concat([hi, xor(hi, lo)])
        return xor(AES128(self.fixed_key, sigma_block), sigma_block)

def format_dst(algo_class: Unsigned,
               algo: Unsigned,
               usage: Unsigned) -> Bytes:
    """Format XOF domain separation tag for use within a (V)DAF."""
    return concat([
        to_be_bytes(VERSION, 1),
        to_be_bytes(algo_class, 1),
        to_be_bytes(algo, 4),
        to_be_bytes(usage, 2),
    ])

7.1. Fully Linear Proof (FLP) Systems

Conceptually, an FLP is a two-party protocol executed by a prover and a verifier. In actual use, however, the prover's computation is carried out by the Client, and the verifier's computation is distributed among the Aggregators. The Client generates a "proof" of its measurement's validity and distributes shares of the proof to the Aggregators. Each Aggregator then performs some computation on its measurement share and proof share locally and sends the result to the other Aggregators. Combining the exchanged messages allows each Aggregator to decide if it holds a share of a valid measurement. (See Section 7.2 for details.)¶

As usual, we will describe the interface implemented by a concrete FLP in terms of an abstract base class Flp that specifies the set of methods and parameters a concrete FLP must provide.¶

The parameters provided by a concrete FLP are listed in Table 4.¶

An FLP specifies the following algorithms for generating and verifying proofs of validity (encoding is described below in Section 7.1.1):¶

• Flp.prove(meas: Vec[Field], prove_rand: Vec[Field], joint_rand: Vec[Field]) -> Vec[Field] is the deterministic proof-generation algorithm run by the prover. Its inputs are the encoded measurement, the "prover randomness" prove_rand, and the "joint randomness" joint_rand. The prover randomness is used only by the prover, but the joint randomness is shared by both the prover and verifier.¶

• Flp.query(meas: Vec[Field], proof: Vec[Field], query_rand: Vec[Field], joint_rand: Vec[Field], num_shares: Unsigned) -> Vec[Field] is the query-generation algorithm run by the verifier. This is used to "query" the measurement and proof. The result of the query (i.e., the output of this function) is called the "verifier message". In addition to the measurement and proof, this algorithm takes as input the query randomness query_rand and the joint randomness joint_rand. The former is used only by the verifier. num_shares specifies how many shares were generated.¶

• Flp.decide(verifier: Vec[Field]) -> Bool is the deterministic decision algorithm run by the verifier. It takes as input the verifier message and outputs a boolean indicating if the measurement from which it was generated is valid.¶

Our application requires that the FLP is "fully linear" in the sense defined in [BBCGGI19]. As a practical matter, what this property implies is that, when run on a share of the measurement and proof, the query-generation algorithm outputs a share of the verifier message. Furthermore, the privacy property of the FLP system ensures that the verifier message reveals nothing about the measurement other than whether it is valid. Therefore, to decide if a measurement is valid, the Aggregators will run the query-generation algorithm locally, exchange verifier shares, combine them to recover the verifier message, and run the decision algorithm.¶

The query-generation algorithm includes a parameter num_shares that specifies the number of shares that were generated. If these data are not secret shared, then num_shares == 1. This parameter is useful for certain FLP constructions. For example, the FLP in Section 7.3 is defined in terms of an arithmetic circuit; when the circuit contains constants, it is sometimes necessary to normalize those constants to ensure that the circuit's output, when run on a valid measurement, is the same regardless of the number of shares.¶

An FLP is executed by the prover and verifier as follows:¶

def run_flp(flp, meas: Vec[Flp.Field], num_shares: Unsigned):
    joint_rand = flp.Field.rand_vec(flp.JOINT_RAND_LEN)
    prove_rand = flp.Field.rand_vec(flp.PROVE_RAND_LEN)
    query_rand = flp.Field.rand_vec(flp.QUERY_RAND_LEN)

    # Prover generates the proof.
    proof = flp.prove(meas, prove_rand, joint_rand)

    # Shard the measurement and the proof.
    meas_shares = additive_secret_share(meas, num_shares, flp.Field)
    proof_shares = additive_secret_share(proof, num_shares, flp.Field)

    # Verifier queries the meas shares and proof shares.
    verifier_shares = [
        flp.query(
            meas_share,
            proof_share,
            query_rand,
            joint_rand,
            num_shares,
        )
        for meas_share, proof_share in zip(meas_shares, proof_shares)
    ]

    # Combine the verifier shares into the verifier.
    verifier = flp.Field.zeros(len(verifier_shares[0]))
    for verifier_share in verifier_shares:
        verifier = vec_add(verifier, verifier_share)

    # Verifier decides if the measurement is valid.
    return flp.decide(verifier)

The proof system is constructed so that, if meas is valid, then run_flp(Flp, meas, 1) always returns True. On the other hand, if meas is invalid, then as long as joint_rand and query_rand are generated uniform randomly, the output is False with overwhelming probability.¶

We remark that [BBCGGI19] defines a much larger class of fully linear proof systems than we consider here. In particular, what is called an "FLP" here is called a 1.5-round, public-coin, interactive oracle proof system in their paper.¶

7.2. Construction

This section specifies Prio3, an implementation of the Vdaf interface (Section 5). It has two generic parameters: an Flp (Section 7.1) and a Xof (Section 6.2). It also has an associated constant, PROOFS, with a value within the range of [1, 256), denoting the number of FLPs generated by the Client (Section 7.1.2). The value of PROOFS is 1 unless explicitly specified.¶

The associated constants and types required by the Vdaf interface are defined in Table 5. The methods required for sharding, preparation, aggregation, and unsharding are described in the remaining subsections. These methods refer to constants enumerated in Table 6.¶

def shard(Prio3, measurement, nonce, rand):
    l = Prio3.Xof.SEED_SIZE
    seeds = [rand[i:i+l] for i in range(0, Prio3.RAND_SIZE, l)]

    meas = Prio3.Flp.encode(measurement)
    if Prio3.Flp.JOINT_RAND_LEN > 0:
        return Prio3.shard_with_joint_rand(meas, nonce, seeds)
    else:
        return Prio3.shard_without_joint_rand(meas, seeds)

def shard_without_joint_rand(Prio3, meas, seeds):
  k_helper_seeds, seeds = front((Prio3.SHARES-1) * 2, seeds)
  k_helper_meas_shares = [
      k_helper_seeds[i]
      for i in range(0, (Prio3.SHARES-1) * 2, 2)
  ]
  k_helper_proofs_shares = [
      k_helper_seeds[i]
      for i in range(1, (Prio3.SHARES-1) * 2, 2)
  ]
  (k_prove,), seeds = front(1, seeds)

  # Shard the encoded measurement into shares.
  leader_meas_share = meas
  for j in range(Prio3.SHARES-1):
      leader_meas_share = vec_sub(
          leader_meas_share,
          Prio3.helper_meas_share(j+1, k_helper_meas_shares[j]),
      )

  # Generate and shard each proof into shares.
  prove_rands = Prio3.prove_rands(k_prove)
  leader_proofs_share = []
  for _ in range(Prio3.PROOFS):
      prove_rand, prove_rands = front(
          Prio3.Flp.PROVE_RAND_LEN, prove_rands)
      leader_proofs_share += Prio3.Flp.prove(meas, prove_rand, [])
  for j in range(Prio3.SHARES-1):
      leader_proofs_share = vec_sub(
          leader_proofs_share,
          Prio3.helper_proofs_share(j+1, k_helper_proofs_shares[j]),
      )

  # Each Aggregator's input share contains its measurement share
  # and share of proof(s).
  input_shares = []
  input_shares.append((
      leader_meas_share,
      leader_proofs_share,
      None,
  ))
  for j in range(Prio3.SHARES-1):
      input_shares.append((
          k_helper_meas_shares[j],
          k_helper_proofs_shares[j],
          None,
      ))
  return (None, input_shares)

def shard_with_joint_rand(Prio3, meas, nonce, seeds):
  k_helper_seeds, seeds = front((Prio3.SHARES-1) * 3, seeds)
  k_helper_meas_shares = [
      k_helper_seeds[i]
      for i in range(0, (Prio3.SHARES-1) * 3, 3)
  ]
  k_helper_proofs_shares = [
      k_helper_seeds[i]
      for i in range(1, (Prio3.SHARES-1) * 3, 3)
  ]
  k_helper_blinds = [
      k_helper_seeds[i]
      for i in range(2, (Prio3.SHARES-1) * 3, 3)
  ]
  (k_leader_blind,), seeds = front(1, seeds)
  (k_prove,), seeds = front(1, seeds)

  # Shard the encoded measurement into shares and compute the
  # joint randomness parts.
  leader_meas_share = meas
  k_joint_rand_parts = []
  for j in range(Prio3.SHARES-1):
      helper_meas_share = Prio3.helper_meas_share(
          j+1, k_helper_meas_shares[j])
      leader_meas_share = vec_sub(leader_meas_share,
                                  helper_meas_share)
      k_joint_rand_parts.append(Prio3.joint_rand_part(
          j+1, k_helper_blinds[j], helper_meas_share, nonce))
  k_joint_rand_parts.insert(0, Prio3.joint_rand_part(
      0, k_leader_blind, leader_meas_share, nonce))

  # Generate and shard each proof into shares.
  prove_rands = Prio3.prove_rands(k_prove)
  joint_rands = Prio3.joint_rands(
      Prio3.joint_rand_seed(k_joint_rand_parts))
  leader_proofs_share = []
  for _ in range(Prio3.PROOFS):
      prove_rand, prove_rands = front(
          Prio3.Flp.PROVE_RAND_LEN, prove_rands)
      joint_rand, joint_rands = front(
          Prio3.Flp.JOINT_RAND_LEN, joint_rands)
      leader_proofs_share += Prio3.Flp.prove(meas,
                                             prove_rand, joint_rand)
  for j in range(Prio3.SHARES-1):
      leader_proofs_share = vec_sub(
          leader_proofs_share,
          Prio3.helper_proofs_share(j+1, k_helper_proofs_shares[j]),
      )

  # Each Aggregator's input share contains its measurement share,
  # share of proof(s), and blind. The public share contains the
  # Aggregators' joint randomness parts.
  input_shares = []
  input_shares.append((
      leader_meas_share,
      leader_proofs_share,
      k_leader_blind,
  ))
  for j in range(Prio3.SHARES-1):
      input_shares.append((
          k_helper_meas_shares[j],
          k_helper_proofs_shares[j],
          k_helper_blinds[j],
      ))
  return (k_joint_rand_parts, input_shares)

def prep_init(Prio3, verify_key, agg_id, _agg_param,
            nonce, public_share, input_share):
  k_joint_rand_parts = public_share
  (meas_share, proofs_share, k_blind) = \
      Prio3.expand_input_share(agg_id, input_share)
  out_share = Prio3.Flp.truncate(meas_share)

  # Compute the joint randomness.
  joint_rand = []
  k_corrected_joint_rand, k_joint_rand_part = None, None
  if Prio3.Flp.JOINT_RAND_LEN > 0:
      k_joint_rand_part = Prio3.joint_rand_part(
          agg_id, k_blind, meas_share, nonce)
      k_joint_rand_parts[agg_id] = k_joint_rand_part
      k_corrected_joint_rand = Prio3.joint_rand_seed(
          k_joint_rand_parts)
      joint_rands = Prio3.joint_rands(k_corrected_joint_rand)

  # Query the measurement and proof share.
  query_rands = Prio3.query_rands(verify_key, nonce)
  verifiers_share = []
  for _ in range(Prio3.PROOFS):
      proof_share, proofs_share = front(
          Prio3.Flp.PROOF_LEN, proofs_share)
      query_rand, query_rands = front(
          Prio3.Flp.QUERY_RAND_LEN, query_rands)
      if Prio3.Flp.JOINT_RAND_LEN > 0:
          joint_rand, joint_rands = front(
              Prio3.Flp.JOINT_RAND_LEN, joint_rands)
      verifiers_share += Prio3.Flp.query(meas_share,
                                          proof_share,
                                          query_rand,
                                          joint_rand,
                                          Prio3.SHARES)

  prep_state = (out_share, k_corrected_joint_rand)
  prep_share = (verifiers_share, k_joint_rand_part)
  return (prep_state, prep_share)

def prep_next(Prio3, prep, prep_msg):
  k_joint_rand = prep_msg
  (out_share, k_corrected_joint_rand) = prep

  # If joint randomness was used, check that the value computed by the
  # Aggregators matches the value indicated by the Client.
  if k_joint_rand != k_corrected_joint_rand:
      raise ERR_VERIFY  # joint randomness check failed

  return out_share

def prep_shares_to_prep(Prio3, _agg_param, prep_shares):
  # Unshard the verifier shares into the verifier message.
  verifiers = Prio3.Flp.Field.zeros(
      Prio3.Flp.VERIFIER_LEN * Prio3.PROOFS)
  k_joint_rand_parts = []
  for (verifiers_share, k_joint_rand_part) in prep_shares:
      verifiers = vec_add(verifiers, verifiers_share)
      if Prio3.Flp.JOINT_RAND_LEN > 0:
          k_joint_rand_parts.append(k_joint_rand_part)

  # Verify that each proof is well-formed and the input is valid
  for _ in range(Prio3.PROOFS):
      verifier, verifiers = front(Prio3.Flp.VERIFIER_LEN, verifiers)
      if not Prio3.Flp.decide(verifier):
          raise ERR_VERIFY  # proof verifier check failed

  # Combine the joint randomness parts computed by the
  # Aggregators into the true joint randomness seed. This is
  # used in the last step.
  k_joint_rand = None
  if Prio3.Flp.JOINT_RAND_LEN > 0:
      k_joint_rand = Prio3.joint_rand_seed(k_joint_rand_parts)
  return k_joint_rand

def is_valid(agg_param, previous_agg_params):
    return len(previous_agg_params) == 0

def aggregate(Prio3, _agg_param, out_shares):
    agg_share = Prio3.Flp.Field.zeros(Prio3.Flp.OUTPUT_LEN)
    for out_share in out_shares:
        agg_share = vec_add(agg_share, out_share)
    return agg_share

def unshard(Prio3, _agg_param,
            agg_shares, num_measurements):
    agg = Prio3.Flp.Field.zeros(Prio3.Flp.OUTPUT_LEN)
    for agg_share in agg_shares:
        agg = vec_add(agg, agg_share)
    return Prio3.Flp.decode(agg, num_measurements)

def helper_meas_share(Prio3, agg_id, k_share):
    return Prio3.Xof.expand_into_vec(
        Prio3.Flp.Field,
        k_share,
        Prio3.domain_separation_tag(USAGE_MEAS_SHARE),
        byte(agg_id),
        Prio3.Flp.MEAS_LEN,
    )

def helper_proofs_share(Prio3, agg_id, k_share):
  return Prio3.Xof.expand_into_vec(
      Prio3.Flp.Field,
      k_share,
      Prio3.domain_separation_tag(USAGE_PROOF_SHARE),
      byte(Prio3.PROOFS) + byte(agg_id),
      Prio3.Flp.PROOF_LEN * Prio3.PROOFS,
  )

def expand_input_share(Prio3, agg_id, input_share):
  (meas_share, proofs_share, k_blind) = input_share
  if agg_id > 0:
      meas_share = Prio3.helper_meas_share(agg_id, meas_share)
      proofs_share = Prio3.helper_proofs_share(agg_id, proofs_share)
  return (meas_share, proofs_share, k_blind)

def prove_rands(Prio3, k_prove):
  return Prio3.Xof.expand_into_vec(
      Prio3.Flp.Field,
      k_prove,
      Prio3.domain_separation_tag(USAGE_PROVE_RANDOMNESS),
      byte(Prio3.PROOFS),
      Prio3.Flp.PROVE_RAND_LEN * Prio3.PROOFS,
  )

def query_rands(Prio3, verify_key, nonce):
  return Prio3.Xof.expand_into_vec(
      Prio3.Flp.Field,
      verify_key,
      Prio3.domain_separation_tag(USAGE_QUERY_RANDOMNESS),
      byte(Prio3.PROOFS) + nonce,
      Prio3.Flp.QUERY_RAND_LEN * Prio3.PROOFS,
  )

def joint_rand_part(Prio3, agg_id, k_blind, meas_share, nonce):
    return Prio3.Xof.derive_seed(
        k_blind,
        Prio3.domain_separation_tag(USAGE_JOINT_RAND_PART),
        byte(agg_id) + nonce + Prio3.Flp.Field.encode_vec(meas_share),
    )

def joint_rand_seed(Prio3, k_joint_rand_parts):
    """Derive the joint randomness seed from its parts."""
    return Prio3.Xof.derive_seed(
        zeros(Prio3.Xof.SEED_SIZE),
        Prio3.domain_separation_tag(USAGE_JOINT_RAND_SEED),
        concat(k_joint_rand_parts),
    )

def joint_rands(Prio3, k_joint_rand_seed):
  """Derive the joint randomness from its seed."""
  return Prio3.Xof.expand_into_vec(
      Prio3.Flp.Field,
      k_joint_rand_seed,
      Prio3.domain_separation_tag(USAGE_JOINT_RANDOMNESS),
      byte(Prio3.PROOFS),
      Prio3.Flp.JOINT_RAND_LEN * Prio3.PROOFS,
  )

opaque Prio3Seed[S];

opaque Prio3Field[F];

struct {
    Prio3Seed k_joint_rand_parts[S * Prio3.SHARES];
} Prio3PublicShareWithJointRand;

struct {
    Prio3Field meas_share[F * Prio3.Flp.MEAS_LEN];
    Prio3Field proofs_share[F * Prio3.Flp.PROOF_LEN * Prio3.PROOFS];
} Prio3LeaderShare;

struct {
    Prio3Seed k_meas_share;
    Prio3Seed k_proofs_share;
} Prio3HelperShare;

struct {
    Prio3LeaderShare inner;
    Prio3Seed k_blind;
} Prio3LeaderShareWithJointRand;

struct {
    Prio3HelperShare inner;
    Prio3Seed k_blind;
} Prio3HelperShareWithJointRand;

struct {
    Prio3Field verifiers_share[F * Prio3.Flp.VERIFIER_LEN * Prio3.PROOFS];
} Prio3PrepShare;

struct {
    Prio3Field verifiers_share[F * Prio3.Flp.VERIFIER_LEN * Prio3.PROOFS];
    Prio3Seed k_joint_rand_part;
} Prio3PrepShareWithJointRand;

struct {
    Prio3Seed k_joint_rand;
} Prio3PrepMessageWithJointRand;

struct {
    Prio3Field agg_share[F * Prio3.Flp.OUTPUT_LEN];
} Prio3AggShare;

7.3. A General-Purpose FLP

This section describes an FLP based on the construction from in [BBCGGI19], Section 4.2. We begin in Section 7.3.1 with an overview of their proof system and the extensions to their proof system made here. The construction is specified in Section 7.3.3.¶

• OPEN ISSUE We're not yet sure if specifying this general-purpose FLP is desirable. It might be preferable to specify specialized FLPs for each data type that we want to standardize, for two reasons. First, clear and concise specifications are likely easier to write for specialized FLPs rather than the general one. Second, we may end up tailoring each FLP to the measurement type in a way that improves performance, but breaks compatibility with the general-purpose FLP.¶

  In any case, we can't make this decision until we know which data types to standardize, so for now, we'll stick with the general-purpose construction. The reference implementation can be found at https://github.com/cfrg/draft-irtf-cfrg-vdaf/tree/main/poc.¶

• OPEN ISSUE Chris Wood points out that the this section reads more like a paper than a standard. Eventually we'll want to work this into something that is readily consumable by the CFRG.¶

C(x_shares[0] + ... + x_shares[SHARES-1]) =
    C(x_shares[0]) + ... + C(x_shares[SHARES-1])

C(x) = (x[0] - 1) * x[0] = x[0]^2 - x[0]

Mul(left, right) = left * right

C(x[0]) = Mul(x[0], x[0]) - x[0]

C(meas, r) = r * Range2(meas[0]) + ... + r^N * Range2(meas[N-1])

C(meas, r) = r * Range2(meas[0]) + ... + r^L * Range2(meas[L-1]) + \
            r^(L+1) * Range3(meas[L]) + ... + r^N * Range3(meas[N-1])

def prove_rand_len(self):
    """Length of the prover randomness."""
    return sum(g.ARITY for g in Valid.GADGETS)

def query_rand_len(self):
    """Length of the query randomness."""
    return len(Valid.GADGETS)

def proof_len(self):
    """Length of the proof."""
    length = 0
    for (g, g_calls) in zip(self.GADGETS, self.GADGET_CALLS):
        P = next_power_of_2(1 + g_calls)
        length += g.ARITY + g.DEGREE * (P - 1) + 1
    return length

def verifier_len(self):
    """Length of the verifier message."""
    length = 1
    for g in self.GADGETS:
        length += g.ARITY + 1
    return length

7.4. Instantiations

This section specifies instantiations of Prio3 for various measurement types. Each uses FlpGeneric as the FLP (Section 7.3) and is determined by a validity circuit (Section 7.3.2) and a XOF (Section 6.2). Test vectors for each can be found in Appendix "Test Vectors".¶

• NOTE Reference implementations of each of these VDAFs can be found in https://github.com/cfrg/draft-irtf-cfrg-vdaf/blob/main/poc/vdaf_prio3.sage.¶

def eval(self, Field, inp):
    self.check_gadget_eval(inp)
    return inp[0] * inp[1]

def eval(self, meas, joint_rand, _num_shares):
    return self.GADGETS[0].eval(self.Field, [meas[0], meas[0]]) \
        - meas[0]

def encode(self, measurement):
    if 0 > measurement or measurement >= 2 ** self.MEAS_LEN:
        raise ERR_INPUT

    return self.Field.encode_into_bit_vector(measurement,
                                             self.MEAS_LEN)

def truncate(self, meas):
    return [self.Field.decode_from_bit_vector(meas)]

def decode(self, output, _num_measurements):
    return output[0].as_unsigned()

def eval(self, Field, inp):
    self.check_gadget_eval(inp)
    return inp[0] * inp[0] - inp[0]

def eval(self, meas, joint_rand, _num_shares):
    self.check_valid_eval(meas, joint_rand)
    out = self.Field(0)
    r = joint_rand[0]
    for b in meas:
        out += r * self.GADGETS[0].eval(self.Field, [b])
        r *= joint_rand[0]
    return out

def encode(self, measurement: Vec[Unsigned]):
    if len(measurement) != self.length:
        raise ERR_INPUT

    encoded = []
    for val in measurement:
        if 0 > val or val >= 2 ** self.bits:
            raise ERR_INPUT

        encoded += self.Field.encode_into_bit_vector(val, self.bits)
    return encoded

def truncate(self, meas):
    truncated = []
    for i in range(self.length):
        truncated.append(self.Field.decode_from_bit_vector(
            meas[i * self.bits: (i + 1) * self.bits]
        ))
    return truncated

def decode(self, output, _num_measurements):
    return [x.as_unsigned() for x in output]

def eval(self, Field, inp):
    self.check_gadget_eval(inp)
    out = Field(0)
    for i in range(self.count):
        start_index = i * self.subcircuit.ARITY
        end_index = (i + 1) * self.subcircuit.ARITY
        out += self.subcircuit.eval(Field, inp[start_index:end_index])
    return out

def eval(self, meas, joint_rand, num_shares):
    self.check_valid_eval(meas, joint_rand)

    out = Field128(0)
    r = joint_rand[0]
    r_power = r
    shares_inv = self.Field(num_shares).inv()

    for i in range(self.GADGET_CALLS[0]):
        inputs = [None] * (2 * self.chunk_length)
        for j in range(self.chunk_length):
            index = i * self.chunk_length + j
            if index < len(meas):
                meas_elem = meas[index]
            else:
                meas_elem = self.Field(0)

            inputs[j * 2] = r_power * meas_elem
            inputs[j * 2 + 1] = meas_elem - shares_inv

            r_power *= r

        out += self.GADGETS[0].eval(self.Field, inputs)

    return out

def encode(self, measurement):
    encoded = [self.Field(0)] * self.length
    encoded[measurement] = self.Field(1)
    return encoded

def truncate(self, meas):
    return meas

def decode(self, output, _num_measurements):
    return [bucket_count.as_unsigned() for bucket_count in output]

def eval(self, meas, joint_rand, num_shares):
    self.check_valid_eval(meas, joint_rand)

    # Check that each bucket is one or zero.
    range_check = self.Field(0)
    r = joint_rand[0]
    r_power = r
    shares_inv = self.Field(num_shares).inv()
    for i in range(self.GADGET_CALLS[0]):
        inputs = [None] * (2 * self.chunk_length)
        for j in range(self.chunk_length):
            index = i * self.chunk_length + j
            if index < len(meas):
                meas_elem = meas[index]
            else:
                meas_elem = self.Field(0)

            inputs[j * 2] = r_power * meas_elem
            inputs[j * 2 + 1] = meas_elem - shares_inv

            r_power *= r

        range_check += r * self.GADGETS[0].eval(self.Field, inputs)

    # Check that the buckets sum to 1.
    sum_check = -shares_inv
    for b in meas:
        sum_check += b

    out = joint_rand[1] * range_check + \
        joint_rand[1] ** 2 * sum_check
    return out

8.1. Incremental Distributed Point Functions (IDPFs)

An IDPF is defined over a domain of size 2^BITS, where BITS is constant defined by the IDPF. Indexes into the IDPF tree are encoded as integers in range [0, 2^BITS). The Client specifies an index alpha and a vector of values beta, one for each "level" L in range [0, BITS). The key generation algorithm generates one IDPF "key" for each Aggregator. When evaluated at level L and index 0 <= prefix < 2^L, each IDPF key returns an additive share of beta[L] if prefix is the L-bit prefix of alpha and shares of zero otherwise.¶

An index x is defined to be a prefix of another index y as follows. Let LSB(x, N) denote the least significant N bits of positive integer x. By definition, a positive integer 0 <= x < 2^L is said to be the length-L prefix of positive integer 0 <= y < 2^BITS if LSB(x, L) is equal to the most significant L bits of LSB(y, BITS), For example, 6 (110 in binary) is the length-3 prefix of 25 (11001), but 7 (111) is not.¶

Each of the programmed points beta is a vector of elements of some finite field. We distinguish two types of fields: One for inner nodes (denoted FieldInner), and one for leaf nodes (FieldLeaf). (Our instantiation of Poplar1 (Section 8.4) will use a much larger field for leaf nodes than for inner nodes. This is to ensure the IDPF is "extractable" as defined in [BBCGGI21], Definition 1.)¶

A concrete IDPF defines the types and constants enumerated in Table 13. In the remainder we write Output as shorthand for the type Union[list[list[FieldInner]], list[list[FieldLeaf]]]. (This type denotes either a vector of inner node field elements or leaf node field elements.) The scheme is comprised of the following algorithms:¶

• Idpf.gen(alpha: Unsigned, beta_inner: list[list[FieldInner]], beta_leaf: list[FieldLeaf], binder: bytes, rand: bytes[Idpf.RAND_SIZE]) -> tuple[bytes, list[bytes]] is the randomized IDPF-key generation algorithm. (Input rand consists of the random bytes it consumes.) Its inputs are the index alpha the values beta, and a binder string. The value of alpha MUST be in range [0, 2^BITS). The output is a public part that is sent to all Aggregators and a vector of private IDPF keys, one for each aggregator. The binder string is used to derive the key in the underlying XofFixedKeyAes128 XOF that is used for expanding seeds at each level. It MUST be chosen uniformly at random by the Client (see Section 9.2).¶

  • TODO(issue #255) Decide whether to treat the public share as an opaque byte string or to replace it with an explicit type.¶

• Idpf.eval(agg_id: Unsigned, public_share: bytes, key: bytes, level: Unsigned, prefixes: tuple[Unsigned, ...], binder: Bytes) -> Output is the deterministic, stateless IDPF-key evaluation algorithm run by each Aggregator. Its inputs are the Aggregator's unique identifier, the public share distributed to all of the Aggregators, the Aggregator's IDPF key, the "level" at which to evaluate the IDPF, the sequence of candidate prefixes, and a binder string. It returns the share of the value corresponding to each candidate prefix.¶

  The output type (i.e., Output) depends on the value of level: If level < Idpf.BITS-1, the output is the value for an inner node, which has type list[list[Idpf.FieldInner]]; otherwise, if level == Idpf.BITS-1, then the output is the value for a leaf node, which has type list[list[Idpf.FieldLeaf]].¶

  The value of level MUST be in range [0, BITS). The indexes in prefixes MUST all be distinct and in range [0, 2^level).¶

  Applications MUST ensure that the Aggregator's identifier is equal to the integer in range [0, SHARES) that matches the index of key in the sequence of IDPF keys output by the Client.¶

In addition, the following method is derived for each concrete Idpf:¶

def current_field(Idpf, level):
    return Idpf.FieldInner if level < Idpf.BITS-1 \
                else Idpf.FieldLeaf

Finally, an implementation note. The interface for IDPFs specified here is stateless, in the sense that there is no state carried between IDPF evaluations. This is to align the IDPF syntax with the VDAF abstraction boundary, which does not include shared state across across VDAF evaluations. In practice, of course, it will often be beneficial to expose a stateful API for IDPFs and carry the state across evaluations. See Section 8.3 for details.¶

8.2. Construction

This section specifies Poplar1, an implementation of the Vdaf interface (Section 5). It is defined in terms of any Idpf (Section 8.1) for which Idpf.SHARES == 2 and Idpf.VALUE_LEN == 2 and an implementation of Xof (Section 6.2). The associated constants and types required by the Vdaf interface are defined in Table 14. The methods required for sharding, preparation, aggregation, and unsharding are described in the remaining subsections. These methods make use of constants defined in Table 15.¶

    A = -2*a + k
    B = a^2 + b - k*a + c

def shard(Poplar1, measurement, nonce, rand):
    l = Poplar1.Xof.SEED_SIZE

    # Split the random input into random input for IDPF key
    # generation, correlated randomness, and sharding.
    if len(rand) != Poplar1.RAND_SIZE:
        raise ERR_INPUT # unexpected length for random input
    idpf_rand, rand = front(Poplar1.Idpf.RAND_SIZE, rand)
    seeds = [rand[i:i+l] for i in range(0,3*l,l)]
    corr_seed, seeds = front(2, seeds)
    (k_shard,), seeds = front(1, seeds)

    xof = Poplar1.Xof(
        k_shard,
        Poplar1.domain_separation_tag(USAGE_SHARD_RAND),
        nonce,
    )

    # Construct the IDPF values for each level of the IDPF tree.
    # Each "data" value is 1; in addition, the Client generates
    # a random "authenticator" value used by the Aggregators to
    # compute the sketch during preparation. This sketch is used
    # to verify the one-hotness of their output shares.
    beta_inner = [
        [Poplar1.Idpf.FieldInner(1), k]
        for k in xof.next_vec(Poplar1.Idpf.FieldInner,
                              Poplar1.Idpf.BITS - 1)
    ]
    beta_leaf = [Poplar1.Idpf.FieldLeaf(1)] + \
        xof.next_vec(Poplar1.Idpf.FieldLeaf, 1)

    # Generate the IDPF keys.
    (public_share, keys) = Poplar1.Idpf.gen(measurement,
                                            beta_inner,
                                            beta_leaf,
                                            nonce,
                                            idpf_rand)

    # Generate correlated randomness used by the Aggregators to
    # compute a sketch over their output shares. XOF seeds are
    # used to encode shares of the `(a, b, c)` triples.
    # (See [BBCGGI21, Appendix C.4].)
    corr_offsets = vec_add(
        Poplar1.Xof.expand_into_vec(
            Poplar1.Idpf.FieldInner,
            corr_seed[0],
            Poplar1.domain_separation_tag(USAGE_CORR_INNER),
            byte(0) + nonce,
            3 * (Poplar1.Idpf.BITS-1),
        ),
        Poplar1.Xof.expand_into_vec(
            Poplar1.Idpf.FieldInner,
            corr_seed[1],
            Poplar1.domain_separation_tag(USAGE_CORR_INNER),
            byte(1) + nonce,
            3 * (Poplar1.Idpf.BITS-1),
        ),
    )
    corr_offsets += vec_add(
        Poplar1.Xof.expand_into_vec(
            Poplar1.Idpf.FieldLeaf,
            corr_seed[0],
            Poplar1.domain_separation_tag(USAGE_CORR_LEAF),
            byte(0) + nonce,
            3,
        ),
        Poplar1.Xof.expand_into_vec(
            Poplar1.Idpf.FieldLeaf,
            corr_seed[1],
            Poplar1.domain_separation_tag(USAGE_CORR_LEAF),
            byte(1) + nonce,
            3,
        ),
    )

    # For each level of the IDPF tree, shares of the `(A, B)`
    # pairs are computed from the corresponding `(a, b, c)`
    # triple and authenticator value `k`.
    corr_inner = [[], []]
    for level in range(Poplar1.Idpf.BITS):
        Field = Poplar1.Idpf.current_field(level)
        k = beta_inner[level][1] if level < Poplar1.Idpf.BITS - 1 \
            else beta_leaf[1]
        (a, b, c), corr_offsets = corr_offsets[:3], corr_offsets[3:]
        A = -Field(2) * a + k
        B = a ** 2 + b - a * k + c
        corr1 = xof.next_vec(Field, 2)
        corr0 = vec_sub([A, B], corr1)
        if level < Poplar1.Idpf.BITS - 1:
            corr_inner[0] += corr0
            corr_inner[1] += corr1
        else:
            corr_leaf = [corr0, corr1]

    # Each input share consists of the Aggregator's IDPF key
    # and a share of the correlated randomness.
    input_shares = list(zip(keys, corr_seed, corr_inner, corr_leaf))
    return (public_share, input_shares)

def prep_init(Poplar1, verify_key, agg_id, agg_param,
              nonce, public_share, input_share):
    (level, prefixes) = agg_param
    (key, corr_seed, corr_inner, corr_leaf) = input_share
    Field = Poplar1.Idpf.current_field(level)

    # Ensure that candidate prefixes are all unique and appear in
    # lexicographic order.
    for i in range(1,len(prefixes)):
        if prefixes[i-1] >= prefixes[i]:
            raise ERR_INPUT # out-of-order prefix

    # Evaluate the IDPF key at the given set of prefixes.
    value = Poplar1.Idpf.eval(
        agg_id, public_share, key, level, prefixes, nonce)

    # Get shares of the correlated randomness for computing the
    # Aggregator's share of the sketch for the given level of the IDPF
    # tree.
    if level < Poplar1.Idpf.BITS - 1:
        corr_xof = Poplar1.Xof(
            corr_seed,
            Poplar1.domain_separation_tag(USAGE_CORR_INNER),
            byte(agg_id) + nonce,
        )
        # Fast-forward the XOF state to the current level.
        corr_xof.next_vec(Field, 3 * level)
    else:
        corr_xof = Poplar1.Xof(
            corr_seed,
            Poplar1.domain_separation_tag(USAGE_CORR_LEAF),
            byte(agg_id) + nonce,
        )
    (a_share, b_share, c_share) = corr_xof.next_vec(Field, 3)
    (A_share, B_share) = corr_inner[2*level:2*(level+1)] \
        if level < Poplar1.Idpf.BITS - 1 else corr_leaf

    # Compute the Aggregator's first round of the sketch. These are
    # called the "masked input values" [BBCGGI21, Appendix C.4].
    verify_rand_xof = Poplar1.Xof(
        verify_key,
        Poplar1.domain_separation_tag(USAGE_VERIFY_RAND),
        nonce + to_be_bytes(level, 2),
    )
    verify_rand = verify_rand_xof.next_vec(Field, len(prefixes))
    sketch_share = [a_share, b_share, c_share]
    out_share = []
    for (i, r) in enumerate(verify_rand):
        [data_share, auth_share] = value[i]
        sketch_share[0] += data_share * r
        sketch_share[1] += data_share * r ** 2
        sketch_share[2] += auth_share * r
        out_share.append(data_share)

    prep_mem = [A_share, B_share, Field(agg_id)] + out_share
    return ((b'sketch round 1', level, prep_mem),
            sketch_share)

def prep_next(Poplar1, prep_state, prep_msg):
    prev_sketch = prep_msg
    (step, level, prep_mem) = prep_state
    Field = Poplar1.Idpf.current_field(level)

    if step == b'sketch round 1':
        if prev_sketch == None:
            prev_sketch = Field.zeros(3)
        elif len(prev_sketch) != 3:
            raise ERR_INPUT  # prep message malformed
        (A_share, B_share, agg_id), prep_mem = \
            prep_mem[:3], prep_mem[3:]
        sketch_share = [
            agg_id * (prev_sketch[0] ** 2
                      - prev_sketch[1]
                      - prev_sketch[2])
            + A_share * prev_sketch[0]
            + B_share
        ]
        return ((b'sketch round 2', level, prep_mem),
                sketch_share)

    elif step == b'sketch round 2':
        if prev_sketch == None:
            return prep_mem  # Output shares
        else:
            raise ERR_INPUT  # prep message malformed

    raise ERR_INPUT  # unexpected input

def prep_shares_to_prep(Poplar1, agg_param, prep_shares):
    if len(prep_shares) != 2:
        raise ERR_INPUT  # unexpected number of prep shares
    (level, _) = agg_param
    Field = Poplar1.Idpf.current_field(level)
    sketch = vec_add(prep_shares[0], prep_shares[1])
    if len(sketch) == 3:
        return sketch
    elif len(sketch) == 1:
        if sketch == Field.zeros(1):
            # In order to reduce communication overhead, let `None`
            # denote a successful sketch verification.
            return None
        else:
            raise ERR_VERIFY  # sketch verification failed
    else:
        raise ERR_INPUT  # unexpected input length

def is_valid(agg_param, previous_agg_params):
    (level, _) = agg_param
    return all(
        level != other_level
        for (other_level, _) in previous_agg_params
    )

def aggregate(Poplar1, agg_param, out_shares):
    (level, prefixes) = agg_param
    Field = Poplar1.Idpf.current_field(level)
    agg_share = Field.zeros(len(prefixes))
    for out_share in out_shares:
        agg_share = vec_add(agg_share, out_share)
    return agg_share

def unshard(Poplar1, agg_param,
            agg_shares, _num_measurements):
    (level, prefixes) = agg_param
    Field = Poplar1.Idpf.current_field(level)
    agg = Field.zeros(len(prefixes))
    for agg_share in agg_shares:
        agg = vec_add(agg, agg_share)
    return list(map(lambda x: x.as_unsigned(), agg))

opaque Poplar1Seed[S];

opaque Poplar1FieldInner[Fi];

opaque Poplar1FieldLeaf[Fl];

struct {
    opaque idpf_key[Poplar1.Idpf.KEY_SIZE];
    Poplar1Seed corr_seed;
    Poplar1FieldInner corr_inner[Fi * 2 * (Poplar1.Idpf.BITS - 1)];
    Poplar1FieldLeaf corr_leaf[Fl * 2];
} Poplar1InputShare;

struct {
    Poplar1FieldInner sketch_share[Fi * 3];
} Poplar1PrepShareRoundOneInner;

struct {
    Poplar1FieldLeaf sketch_share[Fl * 3];
} Poplar1PrepShareRoundOneLeaf;

For the second round and inner field:

struct {
    Poplar1FieldInner sketch_share;
} Poplar1PrepShareRoundTwoInner;

struct {
    Poplar1FieldLeaf sketch_share;
} Poplar1PrepShareRoundTwoLeaf;

struct {
    Poplar1FieldInner[Fi * 3];
} Poplar1PrepMessageRoundOneInner;

struct {
    Poplar1FieldLeaf sketch[Fl * 3];
} Poplar1PrepMessageRoundOneLeaf;

struct {
    Poplar1FieldInner agg_share[Fi * prefix_count];
} Poplar1AggShareInner;

struct {
    Poplar1FieldLeaf agg_share[Fl * prefix_count];
} Poplar1AggShareLeaf;

def encode_agg_param(Poplar1, (level, prefixes)):
    if level > 2 ** 16 - 1:
        raise ERR_INPUT # level too deep
    if len(prefixes) > 2 ** 32 - 1:
        raise ERR_INPUT # too many prefixes
    encoded = Bytes()
    encoded += to_be_bytes(level, 2)
    encoded += to_be_bytes(len(prefixes), 4)
    packed = 0
    for (i, prefix) in enumerate(prefixes):
        packed |= prefix << ((level+1) * i)
    l = ((level+1) * len(prefixes) + 7) // 8
    encoded += to_be_bytes(packed, l)
    return encoded

def decode_agg_param(Poplar1, encoded):
    encoded_level, encoded = encoded[:2], encoded[2:]
    level = from_be_bytes(encoded_level)
    encoded_prefix_count, encoded = encoded[:4], encoded[4:]
    prefix_count = from_be_bytes(encoded_prefix_count)
    l = ((level+1) * prefix_count + 7) // 8
    encoded_packed, encoded = encoded[:l], encoded[l:]
    packed = from_be_bytes(encoded_packed)
    prefixes = []
    m = 2 ** (level+1) - 1
    for i in range(prefix_count):
        prefixes.append(packed >> ((level+1) * i) & m)
    if len(encoded) != 0:
        raise ERR_INPUT
    return (level, tuple(prefixes))

8.3. The IDPF scheme of [BBCGGI21]

In this section we specify a concrete IDPF, called IdpfPoplar, suitable for instantiating Poplar1. The scheme gets its name from the name of the protocol of [BBCGGI21].¶

• TODO We should consider giving IdpfPoplar a more distinctive name.¶

The constant and type definitions required by the Idpf interface are given in Table 16.¶

IdpfPoplar requires a XOF for deriving the output shares, as well as a variety of other artifacts used internally. For performance reasons, we instantiate this object using XofFixedKeyAes128 (Section 6.2.2). See Section 9.4 for justification of this choice.¶

def gen(IdpfPoplar, alpha, beta_inner, beta_leaf, binder, rand):
    if alpha >= 2 ** IdpfPoplar.BITS:
        raise ERR_INPUT # alpha too long
    if len(beta_inner) != IdpfPoplar.BITS - 1:
        raise ERR_INPUT # beta_inner vector is the wrong size
    if len(rand) != IdpfPoplar.RAND_SIZE:
        raise ERR_INPUT # unexpected length for random input

    init_seed = [
        rand[:XofFixedKeyAes128.SEED_SIZE],
        rand[XofFixedKeyAes128.SEED_SIZE:],
    ]

    seed = init_seed.copy()
    ctrl = [Field2(0), Field2(1)]
    correction_words = []
    for level in range(IdpfPoplar.BITS):
        Field = IdpfPoplar.current_field(level)
        keep = (alpha >> (IdpfPoplar.BITS - level - 1)) & 1
        lose = 1 - keep
        bit = Field2(keep)

        (s0, t0) = IdpfPoplar.extend(seed[0], binder)
        (s1, t1) = IdpfPoplar.extend(seed[1], binder)
        seed_cw = xor(s0[lose], s1[lose])
        ctrl_cw = (
            t0[0] + t1[0] + bit + Field2(1),
            t0[1] + t1[1] + bit,
        )

        x0 = xor(s0[keep], ctrl[0].conditional_select(seed_cw))
        x1 = xor(s1[keep], ctrl[1].conditional_select(seed_cw))
        (seed[0], w0) = IdpfPoplar.convert(level, x0, binder)
        (seed[1], w1) = IdpfPoplar.convert(level, x1, binder)
        ctrl[0] = t0[keep] + ctrl[0] * ctrl_cw[keep]
        ctrl[1] = t1[keep] + ctrl[1] * ctrl_cw[keep]

        b = beta_inner[level] if level < IdpfPoplar.BITS-1 \
                else beta_leaf
        if len(b) != IdpfPoplar.VALUE_LEN:
            raise ERR_INPUT # beta too long or too short

        w_cw = vec_add(vec_sub(b, w0), w1)
        # Implementation note: Here we negate the correction word if
        # the control bit `ctrl[1]` is set. We avoid branching on the
        # value in order to reduce leakage via timing side channels.
        mask = Field(1) - Field(2) * Field(ctrl[1].as_unsigned())
        for i in range(len(w_cw)):
            w_cw[i] *= mask

        correction_words.append((seed_cw, ctrl_cw, w_cw))

    public_share = IdpfPoplar.encode_public_share(correction_words)
    return (public_share, init_seed)

def eval(IdpfPoplar, agg_id, public_share, init_seed,
         level, prefixes, binder):
    if agg_id >= IdpfPoplar.SHARES:
        raise ERR_INPUT # invalid aggregator ID
    if level >= IdpfPoplar.BITS:
        raise ERR_INPUT # level too deep
    if len(set(prefixes)) != len(prefixes):
        raise ERR_INPUT # candidate prefixes are non-unique

    correction_words = IdpfPoplar.decode_public_share(public_share)
    out_share = []
    for prefix in prefixes:
        if prefix >= 2 ** (level+1):
            raise ERR_INPUT # prefix too long

        # The Aggregator's output share is the value of a node of
        # the IDPF tree at the given `level`. The node's value is
        # computed by traversing the path defined by the candidate
        # `prefix`. Each node in the tree is represented by a seed
        # (`seed`) and a set of control bits (`ctrl`).
        seed = init_seed
        ctrl = Field2(agg_id)
        for current_level in range(level+1):
            bit = (prefix >> (level - current_level)) & 1

            # Implementation note: Typically the current round of
            # candidate prefixes would have been derived from
            # aggregate results computed during previous rounds. For
            # example, when using `IdpfPoplar` to compute heavy
            # hitters, a string whose hit count exceeded the given
            # threshold in the last round would be the prefix of each
            # `prefix` in the current round. (See [BBCGGI21,
            # Section 5.1].) In this case, part of the path would
            # have already been traversed.
            #
            # Re-computing nodes along previously traversed paths is
            # wasteful. Implementations can eliminate this added
            # complexity by caching nodes (i.e., `(seed, ctrl)`
            # pairs) output by previous calls to `eval_next()`.
            (seed, ctrl, y) = IdpfPoplar.eval_next(
                seed,
                ctrl,
                correction_words[current_level],
                current_level,
                bit,
                binder,
            )
        out_share.append(y if agg_id == 0 else vec_neg(y))
    return out_share

def eval_next(IdpfPoplar, prev_seed, prev_ctrl,
              correction_word, level, bit, binder):
    """
    Compute the next node in the IDPF tree along the path determined by
    a candidate prefix. The next node is determined by `bit`, the bit of
    the prefix corresponding to the next level of the tree.

    TODO Consider implementing some version of the optimization
    discussed at the end of [BBCGGI21, Appendix C.2]. This could on
    average reduce the number of AES calls by a constant factor.
    """

    Field = IdpfPoplar.current_field(level)
    (seed_cw, ctrl_cw, w_cw) = correction_word
    (s, t) = IdpfPoplar.extend(prev_seed, binder)
    s[0] = xor(s[0], prev_ctrl.conditional_select(seed_cw))
    s[1] = xor(s[1], prev_ctrl.conditional_select(seed_cw))
    t[0] += ctrl_cw[0] * prev_ctrl
    t[1] += ctrl_cw[1] * prev_ctrl

    next_ctrl = t[bit]
    (next_seed, y) = IdpfPoplar.convert(level, s[bit], binder)
    # Implementation note: Here we add the correction word to the
    # output if `next_ctrl` is set. We avoid branching on the value of
    # the control bit in order to reduce side channel leakage.
    mask = Field(next_ctrl.as_unsigned())
    for i in range(len(y)):
        y[i] += w_cw[i] * mask

    return (next_seed, next_ctrl, y)

def extend(IdpfPoplar, seed, binder):
    xof = XofFixedKeyAes128(seed, format_dst(1, 0, 0), binder)
    s = [
        bytearray(xof.next(XofFixedKeyAes128.SEED_SIZE)),
        bytearray(xof.next(XofFixedKeyAes128.SEED_SIZE)),
    ]
    # Use the least significant bits as the control bit correction,
    # and then zero it out. This gives effectively 127 bits of
    # security, but reduces the number of AES calls needed by 1/3.
    t = [Field2(s[0][0] & 1), Field2(s[1][0] & 1)]
    s[0][0] &= 0xFE
    s[1][0] &= 0xFE
    return (s, t)

def convert(IdpfPoplar, level, seed, binder):
    xof = XofFixedKeyAes128(seed, format_dst(1, 0, 1), binder)
    next_seed = xof.next(XofFixedKeyAes128.SEED_SIZE)
    Field = IdpfPoplar.current_field(level)
    w = xof.next_vec(Field, IdpfPoplar.VALUE_LEN)
    return (next_seed, w)

def encode_public_share(IdpfPoplar, correction_words):
    encoded = Bytes()
    control_bits = list(itertools.chain.from_iterable(
        cw[1] for cw in correction_words
    ))
    encoded += pack_bits(control_bits)
    for (level, (seed_cw, _, w_cw)) \
        in enumerate(correction_words):
        Field = IdpfPoplar.current_field(level)
        encoded += seed_cw
        encoded += Field.encode_vec(w_cw)
    return encoded

def decode_public_share(IdpfPoplar, encoded):
    l = (2*IdpfPoplar.BITS + 7) // 8
    encoded_ctrl, encoded = encoded[:l], encoded[l:]
    control_bits = unpack_bits(encoded_ctrl, 2 * IdpfPoplar.BITS)
    correction_words = []
    for level in range(IdpfPoplar.BITS):
        Field = IdpfPoplar.current_field(level)
        ctrl_cw = (
            control_bits[level * 2],
            control_bits[level * 2 + 1],
        )
        l = XofFixedKeyAes128.SEED_SIZE
        seed_cw, encoded = encoded[:l], encoded[l:]
        l = Field.ENCODED_SIZE * IdpfPoplar.VALUE_LEN
        encoded_w_cw, encoded = encoded[:l], encoded[l:]
        w_cw = Field.decode_vec(encoded_w_cw)
        correction_words.append((seed_cw, ctrl_cw, w_cw))
    if len(encoded) != 0:
        raise ERR_DECODE
    return correction_words

8.4. Instantiation

By default, Poplar1 is instantiated with IdpfPoplar (VALUE_LEN == 2) and XofTurboShake128 (Section 6.2.1). This VDAF is suitable for any positive value of BITS. Test vectors can be found in Appendix "Test Vectors".¶

9.1. Requirements for the Verification Key

The Aggregators are responsible for exchanging the verification key in advance of executing the VDAF. Any procedure is acceptable as long as the following conditions are met:¶

1. To ensure robustness of the computation, the Aggregators MUST NOT reveal the verification key to the Clients. Otherwise, a malicious Client might be able to exploit knowledge of this key to craft an invalid report that would be accepted by the Aggregators.¶

2. To ensure privacy of the measurements, the Aggregators MUST commit to the verification key prior to processing reports generated by Clients. Otherwise, a malicious Aggregator may be able to craft a verification key that, for a given report, causes an honest Aggregator to leak information about the measurement during preparation.¶

Meeting these conditions is required in order to leverage security analysis in the framework of [DPRS23]. Their definition of robustness allows the attacker, playing the role of a cohort of malicious Clients, to submit arbitrary reports to the Aggregators and eavesdrop on their communications as they process them. Security in this model is achievable as long as the verification key is kept secret from the attacker.¶

The privacy definition of [DPRS23] considers an active attacker that controls the network and a subset of Aggregators; in addition, the attacker is allowed to choose the verification key used by each honest Aggregator over the course of the experiment. Security is achievable in this model as long as the key is picked at the start of the experiment, prior to any reports being generated. (The model also requires nonces to be generated at random; see Section 9.2 below.)¶

Meeting these requirements is relatively straightforward. For example, the Aggregators may designate one of their peers to generate the verification key and distribute it to the others. To assure Clients of key commitment, the Clients and (honest) Aggregators could bind reports to a shared context string derived from the key. For instance, the "task ID" of DAP [DAP] could be set to the hash of the verification key; then as long as honest Aggregators only consume reports for the task indicated by the Client, forging a new key after the fact would reduce to finding collisions in the underlying hash function. (Keeping the key secret from the Clients would require the hash function to be one-way.) However, since rotating the key implies rotating the task ID, this scheme would not allow key rotation over the lifetime of a task.¶

9.2. Requirements for the Nonce

The sharding and preparation steps of VDAF execution depend on a nonce associated with the Client's report. To ensure privacy of the underlying measurement, the Client MUST generate this nonce using a CSPRNG. This is required in order to leverage security analysis for the privacy definition of [DPRS23], which assumes the nonce is chosen at random prior to generating the report.¶

Other security considerations may require the nonce to be non-repeating. For example, to achieve differential privacy it is necessary to avoid "over exposing" a measurement by including it too many times in a single batch or across multiple batches. It is RECOMMENDED that the nonce generated by the Client be used by the Aggregators for replay protection.¶

9.3. Requirements for the Aggregation Parameters

As described in Section 4.3 and Section 5.3 respectively, DAFs and VDAFs may impose restrictions on the re-use of input shares. This is to ensure that correlated randomness provided by the Client through the input share is not used more than once, which might compromise confidentiality of the Client's measurements.¶

Protocols that make use of VDAFs therefore MUST call Vdaf.is_valid on the set of all aggregation parameters used for a Client's input share, and only proceed with the preparation and aggregation phases if that function call returns True.¶

9.4. Requirements for XOFs

As described in Section 6.2, our constructions rely on eXtendable Output Functions (XOFs). In the security analyses of our protocols, these are usually modeled as random oracles. XofTurboShake128 is designed to be indifferentiable from a random oracle [MRH04], making it a suitable choice for most situations.¶

The one exception is the Idpf implementation IdpfPoplar Section 8.3. Here, a random oracle is not needed to prove privacy, since the analysis of [BBCGGI21], Proposition 1, only requires a Pseudorandom Generator (PRG). As observed in [GKWY20], a PRG can be instantiated from a correlation-robust hash function H. Informally, correlation robustness requires that for a random r, H(xor(r, x)) is computationally indistinguishable from a random function of x. A PRG can therefore be constructed as¶

PRG(r) = H(xor(r, 1)) || H(xor(r, 2)) || ...`

since each individual hash function evaluation is indistinguishable from a random function.¶

Our construction at Section 6.2.2 implements a correlation-robust hash function using fixed-key AES. For security, it assumes that AES with a fixed key can be modeled as a random permutation [GKWY20]. Additionally, we use a different AES key for every client, which in the ideal cipher model leads to better concrete security [GKWWY20].¶

We note that for robustness, the analysis of [BBCGGI21] still assumes a random oracle to make the Idpf extractable. While XofFixedKeyAes128 has been shown to be differentiable from a random oracle [GKWWY20], there are no known attacks exploiting this difference. We also stress that even if the Idpf is not extractable, Poplar1 guarantees that every client can contribute to at most one prefix among the ones being evaluated by the helpers.¶

9.5. Choosing the Number of Proofs to Use for Prio3

• TODO Add guidance for choosing PROOFS (Section 7.1.2) for Prio3. In particular when we go for a smaller field for a given circuit. See this for details.¶

Prio3Count

• TODO Copy the machine readable vectors from the source repository (https://github.com/cfrg/draft-irtf-cfrg-vdaf/tree/main/poc/test_vec) and format them for humans.¶

Prio3Sum

• TODO Copy the machine readable vectors from the source repository (https://github.com/cfrg/draft-irtf-cfrg-vdaf/tree/main/poc/test_vec) and format them for humans.¶

Prio3SumVec

• TODO Copy the machine readable vectors from the source repository (https://github.com/cfrg/draft-irtf-cfrg-vdaf/tree/main/poc/test_vec) and format them for humans.¶

Prio3Histogram

• TODO Copy the machine readable vectors from the source repository (https://github.com/cfrg/draft-irtf-cfrg-vdaf/tree/main/poc/test_vec) and format them for humans.¶

Poplar1

• TODO Copy the machine readable vectors from the source repository (https://github.com/cfrg/draft-irtf-cfrg-vdaf/tree/main/poc/test_vec) and format them for humans.¶