DSNP Specification

Welcome to the Decentralized Social Networking Protocol (DSNP) specification. Here you can find the detailed specification documentation for DSNP, official DSNP system specifications, and associated specifications.

Goals & Purpose

Free communication among users on the Internet faces a variety of problems in the modern day. These challenges include censorship by state and corporate actors, the amplification of misinformation through viral content, and an ever-shrinking collection of near monopolies with absolute power over social interaction in the twenty-first century. Through the DSNP, we hope to mitigate and ideally solve these challenges in the way social interaction operates online.

How to Read This Specification

There are three core specifications currently.

NameVersionDescription
DSNP1.2.0The system-agnostic DSNP specification
Activity Content1.2.0A specification for DSNP-referenced content (subset of W3C Activity Streams)
DSNP Over Frequency-DSNP Over Frequency specification

Each specification is divided into several modules. Each module describes a mostly self-contained aspect of the specification.

Versioning

DSNP specification versions follow Semantic Versioning 2.0 for releases.

Contributions

Development occurs on GitHub. All interactions must follow the Code of Conduct and Contribution Guidelines.

Learning More

In addition to this document, more resources regarding the project can be found at DSNP.org, including the blog, forum, code repository, and SDK.

DSNP Specification

Version 1.2.0

DSNP (Decentralized Social Networking Protocol) is a social networking protocol designed to run on a blockchain. It specifies a set of social primitives along with requirements for interoperability. Go to Systems for specifics on how DSNP is expressed on a specific blockchain.

Overview

What is a DSNP System?

A DSNP system is a state machine which generates an ongoing, publicly observable and verifiable stream of state change records in response to public input. This specification defines a set of operations (the DSNP Operations) that a compliant DSNP system MUST make available. For each DSNP Operation, the DSNP specification defines the data to be provided as input, the data that will be generated as output, and the state change records (the DSNP State Change Records) that will be observable as a result of the Operation, depending on the input and prior state of the system.

The nature of blockchain technology means that all DSNP Operations are potentially asynchronous; that is, DSNP State Change Records need not be created immediately upon invoking a DSNP Operation, and may appear in the stream of state changes at any subsequent time. A DSNP Operation can therefore only be confirmed to have completed successfully by observing the stream of newly-generated DSNP Records.

All DSNP Records MUST be publicly visible. On a blockchain, for example, all nodes must share a common set of Records, and a DSNP system specification MUST define how external systems can subscribe to or otherwise view all state changes. In practical terms, this means that all users of a given DSNP system exist conceptually within a single shared social graph, and a given user MUST have the ability at a protocol level to see and participate in all public discourse. Note, however, that this is not an application-level requirement; an application that interacts with DSNP might choose to filter what content is shown.

Finally, to ensure decentralization, a DSNP system MUST avoid having any single point of failure (many nodes) and MUST avoid having any single entity that can override its consensus mechanisms (many operators).

DSNP System Compliance

Compliant DSNP system specifications MUST document how each of the required DSNP Operations (for a given version of the DSNP specification) can be invoked, including, for example, what wire protocols should be used for discovery, authentication and operation execution, and how inputs and outputs will be serialized.

A compliant specification MUST specify a mapping from its system-specific state change data (for example, the events emitted by a blockchain) to the DSNP State Change Records that data represents.

Releases

DSNP system specifications MUST specify the version(s) of the DSNP specification that may be used.

VersionDescriptionRelease DateChangelog
1.2.0DIP-210, DIP-216, DIP-220, DIP-225, DIP-226, DIP-227, DIP-2282023-04-11Changelog
1.1.0DIP-148, DIP-149, DIP-150, DIP-165, DIP-1802022-05-06Changelog
1.0.0Initial Release2021-09-09Changelog

Identity

This specification is intended to cover the concept of identity and how identity is represented within the protocol.

Identifiers

DSNP Identifiers form the basis for pseudo-anonymous streams of content. While some users may choose to link or expose their real-world identity, DSNP implementations MUST NOT require such data exposure for account creation. The social graph is formed using this Identifier so that a user's connections maintain integrity regardless of changes in any user's client choices or access changes.

Pseudo Anonymity

  • An Identifier MUST default to being disconnected from a real-world identity.
  • An Identifier MUST be unique to the implementation.

Ownership

A user's ownership of their identity is expressed via ownership and control of their pseudo-anonymous Identifier(s). Control entails the power to invoke DSNP Operations including publishing Announcements that create, update and delete (Tombstone) content associated with the Identifier, delegating these powers to others, managing keys associated with the Identifier, and retiring the Identifier.

Key Management

An initial control key must be created in order to acquire an Identifier. Each distinct Identifier MUST have distinct control keys; that is, the same key MUST NOT be linked to multiple Identifiers. Optionally, an implementation MAY allow the user to add and remove additional keys. An implementation MUST NOT allow the user to remove the only or last remaining control key.

Retirement

A user may choose to retire their Identifier at any time. Once an Identifier is retired, an implementation MAY remove all state data associated with that Identifier, provided that an indication that the Identifier is retired remains, so it may not be reused in the future. This means that all data previously sent from the Identifier, the keys associated with the Identifier, and the delegations (see next section) associated with the Identifier may be removed.

After an Identifier is retired, any existing or future Announcements from the Identifier should be treated as if they have been tombstoned (for Announcement Types that support tombstoning). A retired Identifier MUST NOT be allowed to act as a principal for any additional DSNP Operations.

Delegation

  • An owner MUST be able to delegate permission to announce on their behalf to other parties.
  • A user MUST be able to revoke delegated permissions.
  • Announcements from a delegate MUST be able to be verify which delegate made the specific Announcement.
  • Delegation revocation MUST NOT be retroactive.

Non-normative

Retroactive Revocation of Delegation

There are times when one might desire retroactive revocation of delegation, if for example, a key were found to have been compromised at an earlier time. However, retroactive revocation is much more difficult from a caching and performance perspective. Instead reverting any undesirable Announcements via Tombstones allows a user to choose which specific events need reverting and then notify the network of that change.

To this end, any Batch Publications can always be validated from the perspective of the time they were published, and do not require re-validation at future read times. The validity of a Batch is thus immutable.

  1. Collect all the DSNP Ids used in a Batch.
  2. Validate that each DSNP Id had delegated to the publisher of the Batch at the time of publishing.
  3. Validate that any failures from step 2 were from DSNP Ids that revoked delegation within the acceptance window of prior blocks.
  4. Record the batch as valid or invalid.

Identifiers

The DSNP Identifiers form the basis for pseudo-anonymous streams of content. Graph connections are formed through the DSNP User Id.

DSNP User Id

  • 64-bit Unsigned Integer
  • MUST be serialized as decimal
  • MUST be unique per implementation

DSNP Content Hash

  • Variable length byte array (fixed length for a given hashing algorithm)
  • MUST be a valid multihash encoding of the hash output for the bytes of the content, generated with a Supported Hashing Algorithm

Supported Hashing Algorithms

AlgorithmMultihash NameLeading bytes (as varint)ReferenceDSNP Version Added
SHA-256sha2-2560x1220RFC 62341.2.0
BLAKE2bblake2b-2560xa0e40220RFC 76931.2.0

DSNP Protocol Scheme

  • MUST always be the string dsnp://

DSNP User URI

The DSNP User URI consists of two parts: the scheme and the user id. It is used to identify a user via a URI.

Example

dsnp://1311768467294899700
partvalue
Schemedsnp://
User Id1311768467294899700

DSNP Content URI

The DSNP Content URI consists of three parts: the scheme, the user id, and the content hash. It is used to uniquely identify an Announcement from a given user with content.

Any Announcement Types with a fromId and contentHash have a DSNP Content URI.

Example

dsnp://78187493520/0x1234567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef
partvalue
Schemedsnp://
User Id78187493520
Content Hash0x1234567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef

Graph

This specification describes the social network graph and how it is represented in the protocol.

In this context a social graph means a graph that represents social relationships between entities.

Nodes

Users are represented in the DSNP graph by nodes uniquely identified by their DSNP User Ids. Relationships between users are the graph edges.

Edges

Graph edges that originate from a user are stored as DSNP User Data associated with a given user. DSNP user data structures capture three distinct sets of relationships, as described below.

Public Follows

Each public follow records a unidirectional relationship where one user (the controller of the graph) is interested in following the public speech of another user. When follows are public, they are visible to any other user or system, including the user being followed.

Private Follows

A private follow records an interest in following the public speech of another user, but the data is stored in encrypted form. The relationship is not visible to the followed user or any other user that does not have the decryption key.

Private Connections

Private connections represent one half of a bidirectional relationship between two DSNP Users. Private connections are stored on both sides of the bidirectional relationship between two DSNP Users. Each user can then update the status of the relationship without loss of privacy. Each DSNP user maintains and controls two sets of data:

  1. An encrypted data set that records which users are recognized as connections.
  2. A public data set that is used to signal connection status via Pseudonymous Relationship Identifiers.

Pseudonymous Relationship Identifiers

Private connections are complemented by publishing private connection Pseudonymous Relationship Identifiers (PRIds), which provide a means of making the connection relationship visible to the other user while maintaining privacy with respect to any third parties. Applications are encouraged to verify connection relationships by checking to see that they are reciprocated, with the appropriate identifier present in both users' data.

For each pair of users Alice and Bob, and a specified PRId context, a pair of PRIds can be generated from Alice and Bob's shared secret, with one representing a simplex channel from Alice to Bob, and the other representing the corresponding simplex channel from Bob to Alice.

PRIds have the following properties:

  • If Alice has generated a PRId from Alice to Bob, only Bob can generate the same PRId. This allows Bob to check if he is listed in Alice's published connections.
  • Bob can prove to a third party that a given PRId in Alice's published connections is his, without allowing the third party to discover Alice or Bob's relationships with any other user.

A detailed discussion of the cryptography for PRIds can be found in the PRId object definition.

User Data

DSNP stores user-related data of well defined formats associated with a user's DSNP Identifier. DSNP User Data is designed for data with the following characteristics:

  • The data is intended to be accessible across the network with the strongest possible durability guarantees
  • The data may change frequently.
  • The data does not need to be widely announced and instead can be read just-in-time for a particular user.

Data may be public or stored in encrypted form. The Avro schema specification is used to define the binary representation of relevant data types.

User Data Types

DSNP implementations MUST support the following User Data Types:

System NameVersionEncryption AlgorithmCompression CodecAvro Object Type
publicFollows1.2NONEDEFLATEGraphEdge
privateFollows1.2curve25519xsalsa20poly1305DEFLATEGraphEdge
privateConnections1.2curve25519xsalsa20poly1305DEFLATEGraphEdge
privateConnectionPRIds1.2NONENONEPRId

Data for each data type is initially formatted as a stream of Avro objects that should conform to the schema specified. Avro file- and block-level information (including in-stream schema) is omitted. The Avro stream is then compressed and/or encrypted as specified.

curve25519xsalsa20poly1305 (that is, X25519 key exchange, XSalsa20 encryption, and Poly1305 message authentication) is the default authenticated encryption algorithm used in the NaCl ("Salt") library, and its successor libsodium. In the specification of cryptographic operations below, relevant methods from these libraries are noted. While these specific libraries are not required for DSNP compatibility, they are highly recommended.

Data Chunks

Because blockchain systems often have specific limits to the amount of data that can be included in a given transaction, operations on user data deal with the data in discrete chunks. As implementation strategies may vary, implementations MUST define their own maximum chunk size in bytes to be used in the operations described below.

Entity Tags

To allow for application-layer caching and prevent race conditions between different applications attempting to update the same data, an implementation MUST provide a non-empty etag (entity tag) value with each chunk fetched, and update this for each chunk replaced.

An etag is any ASCII-encoded string with a minimum length of one byte and a maximum length of 255 bytes.

An implementation MUST fail the Replace User Data Operation if the etag values supplied do not match the current etag values for all chunks of the specified data type.

Replace User Data Operation

The Replace User Data Operation takes the following parameters:

  • A DSNP User Id
    • Implementations MUST ensure that the principal invoking this Operation is this user, or a transparent chain of delegation from the user to the principal exists.
  • A Key Identifier for the keyAgreement key pair used to encrypt any private data in the operation. (If only unencrypted user data types are included, the key identifier is optional.)
  • A map containing the set of data types to update as the keys, and tuples consisting of (1) the schema version used to encode the data type, and (2) a list where each element includes a data chunk and its associated entity tag, as the values.

If the Operation is successful, any previous data associated with the user for each data type included in the input MUST be removed and replaced by the new data.

Data chunks should be generated for each included data type using the following sequence of operations:

  1. Serialize the data records, in Avro binary format, according to the versioned schema.
  2. Divide the data records into one or more chunks, with each chunk containing a maximum byte count specified by the implementation.
  • Note that the combined chunks constitute a different, and smaller, binary than a complete Avro file, as they exclude the header information, schema, and any block-level metadata. Each chunk should start and end with a complete Avro object; in other words, chunk boundaries MUST NOT occur mid-record. Applications SHOULD try to include as many records as possible within the byte length limit, but because compression and encryption may alter the byte size versus the raw Avro binary records, they may wish to use a heuristic approach that sometimes produces non-optimal chunks.
  1. For each chunk generated, the application should then:
    1. If the data type requires compression, apply the compression codec noted.
    2. If the data type requires encryption,
      1. Retrieve the user's active (most recently announced) keyAgreement public key, Upublic. The keyId in the announcement should match the key identifier provided for this Operation. If no key exists, one should be created and published as an Announcement before invoking the Operation.
      2. Create a sealed box (a payload encrypted with a symmetric key derived from an ephemeral key pair, and accompanied by the ephemeral public key), as in the libsodium function crypto_box_seal, using Upublic.
      3. Include the previous etag value for the chunk. If the chunk is new, etag should be set to null. If any chunks are to be deleted, they should be included in the input identified with the existing etag and a null value for the data.

If the Operation is invoked successfully the implementation MUST synchronously return a new set of etag values for each data type replaced, corresponding to the updated state of the data (with new chunks added and deleted chunks removed). Applications should not interpret this response as an indication that the operation was completed and a state change record emitted, as this typically occurs asynchronously. However, this strategy allows applications to make the optimistic assumption that, in due course, the DSNP system will reflect the intended changes, without needing to wait for asynchronous confirmation.

If, on the other hand, an invocation of Replace User Data is rejected due to entity tag discrepancies, this indicates that the relevant data on the network has changed since the entity tags were acquired by the application, and the application should fetch the most current version with the Get User Data Operation, reapply any intended changes, and retry the operation.

The Replace User Data Operation MUST generate a User Data Replaced Record containing the DSNP User Id and the set of updated User Data Types (but not the data itself). If the implementation detects that no change has occurred, it SHOULD omit the relevant unchanged data types from the state change record.

Examples

The following section is non-normative. The JSON schema and encoding used is provided for illustration only and implementations are free to define their own encoding.

The following example illustrates the input to the Replace User Data Operation corresponding to the following scenario (utilizing social graph data types):

  • The user's previously stored data consists of one chunk for public follows, two chunks for private connections, and one chunk for private connection declarations.
  • The user adds several follows to their public list, causing it to exceed the maximum capacity for a single chunk and require a new chunk.
  • The user adds a relationship to their private connections, but it still fits in two chunks. The private connection's PRId is added to the private connection PRId list, but it remains a single chunk.
{
  "publicFollows": {
    "version": "1.2",
    "chunks": [
      {
        "etag": string                             // unchanged chunk
      },
      {
        "data": base64(compress(chunk2)),
        "etag": null,                              // new chunk
      }
    ]
  },
  "privateFollows": {
    "version": "1.2",
    "chunks": [
      {
        "etag": string                             // unchanged chunk
      },
      {
        "data": base64(encrypt(compress(chunk2))), // updated chunk
        "etag": string
      }
    ]
  },
  "privateConnectionPRIds": {
    "version": "1.2",
    "chunks": [
      {
        "data": base64(compress(chunk1)),
        "etag": string
      }
    ]
  }
}

Deletion of records may cause situations where the number of chunks decreases. To ensure that the deleted chunk was up to date, the deleted chunk should still be included in the array with the relevant entity tag value, using an explicit null value for the data field to indicate deletion, as in the following snippet:

  "publicFollows": {
    "version": "1.2",
    "chunks": [
      {
        "etag": string                     // unchanged chunk
      },
      {
        "data": null,                      // deleted chunk
        "etag": string,
      }
    ]
  }

Get User Data Operation

The Get User Data Operation takes the following parameters:

  • The DSNP User Id of the user who controls the data
    • Note: While writing user data is reserved for the user and any delegates, anyone on the network can read any user's data (though it may be encrypted).
  • The User Data Types (by system name) that should be retrieved.

The operation returns a mapping of User Data Type to data chunks, with each data chunk annotated with an entity tag and (optionally) a key identifier. (Note that this is the same general structure as the input data for Replace User Data, for each requested data type. If no chunks for a requested data type exist, an implementation MAY omit that data type from the response.

To transform the data from the output to Avro binary records, a consumer should apply the following algorithm to each data type included:

  1. Determine the relevant encryption algorithm, compression codec, and object schema from the User Data Type and version noted.
  2. For each chunk,
    1. If encryption is indicated, decrypt the chunk data using the user's secret key (identified using the key identifier) as in the libsodium function crypto_box_seal_open.
    2. If compression is required, uncompress the chunk data using the specified codec.
    3. Deserialize the uncompressed data to logical records according to the Avro object schema.
    4. Retain the chunk's etag value if needed for any updates.

Examples

The following section is non-normative. The JSON schema and encoding used is provided for illustration only and implementations are free to define their own encoding.

The following example illustrates the output of a Get User Data Operation invocation requesting data for publicFollows, privateConnections, and privateConnectionPRIds:

{
  "publicFollows": {
    "version": "1.2",
    "chunks": [
      {
        "data": base64_string,
        "etag": string
      },
      {
        "data": base64_string,
        "etag": string
      }
    ]
  },
  "privateConnections": {
    "version": "1.2",
    "chunks": [
      {
        "data": base64_string,
        "etag": string,
        "keyId": integer
      },
      {
        "data": base64_string,
        "etag": string,
        "keyId": integer
      }
    ]
  },
  "privateConnectionPRIds": {
    "version": "1.2",
    "chunks": [
      {
        "data": base64_string,
        "etag": string
      }
    ]
  }
}

Graph Edge

Relationships between users are represented using the GraphEdge object.

Serialization

GraphEdge object serialization MUST conform to the following Avro schema:

{
    "namespace": "org.dsnp",
    "name": "GraphEdge",
    "type": "record",
    "doc": "A relationship to another DSNP user",
    "fields": [
        {
            "name": "userId",
            "type": "long",
            "doc": "The other user's DSNP User Id"
        },
        {
            "name": "since",
            "type": "long",
            "doc": "Timestamp in Unix epoch seconds when this relationship was originally established"
        }
    ]
}

Generation

userId

To allow for optimal compression, User Ids are stored using the long type in Avro schema, which is a 64-bit signed integer. Care should be taken to ensure that User Id values greater than or equal to 263, where used by an implementation, are converted correctly between signed and unsigned representations.

since

PRId

A Pseudonymous Relationship Identifier is represented by the PRId object type.

Serialization

PRId object serialization MUST conform to the following Avro schema:

{
    "namespace": "org.dsnp",
    "name": "PRId",
    "type": "fixed",
    "size": 8,
    "doc": "Pseudonymous Relationship Identifier"
}

Generation

PRIds are generated cryptographically to represent a relationship from one user to another within a specified context, in a privacy-preserving manner.

Contexts

The following context values are currently defined for PRIds. All other values are reserved for future use.

Context IdDescriptionContext string
0ConnectionPRIdCtx0

Algorithm

In the following section, the Alice to Bob identifier for context C is called PRIdA→B,C, and the corresponding Bob to Alice identifier is called PRIdB→A,C.

A PRId is derived from Alice and Bob's keyAgreement key pairs, using a key exchange protocol as follows. To illustrate the cryptographic operations required, the relevant functions from libsodium are noted. Sodium is a stable, fast, free, and cross-platform cryptography library, and supports all encryption algorithms used in the DSNP specification out of the box.

Definitions:

  • IdA = DSNP User Id of A (little-endian)
  • IdB = DSNP User Id of B (little-endian)

Algorithm:

  1. Both Alice and Bob generate an asymmetric key pair for use with X25519 ECIES. Each publishes a Public Key Announcement with their generated public key with a keyType value of keyAgreement.
LibsodiumAlgorithm
crypto_box_keypair(
  &a_public,
  &a_secret);
crypto_box_keypair(
  &b_public, 
  &b_secret);
  
(Apublic, Asecret) ← KGF()
(Bpublic, Bsecret) ← KGF()
  1. When Alice wants to interact with Bob, she looks up Bob's public key and performs an X25519 Elliptic-curve Diffie-Hellman key exchange operation using her secret key and Bob's public key, generating a root shared secret.
LibsodiumAlgorithm
crypto_box_beforenm(
  &root_shared_secret,
  b_public,
  a_secret);
RootSharedSecretABECDH(Bpublic, Asecret)
  1. Alice derives a context-specific subkey CtxSharedSecretBob from the shared secret RootSharedSecret as the master key, Bob's DSNP User Id as the 64-bit key identifier, and the ASCII encoding of the PRId Context string ("PRIdCtx0" for connections).
LibsodiumAlgorithm
crypto_kdf_derive_from_key(
  ctx_shared_secret,
  32,
  b_user_id,
  "PRIdCtx0",
  root_shared_secret);
CtxSharedSecretA→B ←
  Blake2b256(
    key = RootSharedSecretAB,
    message = {},
    salt = IdB || {0},
    personal = "PRIdCtx0" || {0})
  1. Alice uses Bob's DSNP User Id to form an 8-byte little-endian message. Alice encrypts this message using XSalsa20 with the PRId key CtxSharedSecretA→B and a nonce of her own User Id (little-endian) followed by 16 zero bytes.
LibsodiumAlgorithm
char nonce[24] = {0};
int i;
for (i = 0; i < 8; i++) {
  nonce[i] = (user_id_a >> (i*8))
    & 0xff;
}
crypto_secretbox_detached( &prid, &mac_unused, user_id_b, 8, nonce, ctx_shared_secret);
  • Alice's act of publishing provides authentication, so the MAC is unused.
PRIdA→B,C ←
  XSalsa20(
    message = IdB,
    key = CtxSharedSecretA→B,
    nonce = Padded24BytesLE(IdA)
  )
  1. Alice adds the generated PRId to the relevant list of PRIds and publishes an updated copy via the Replace User Data Operation.

Similarly, Bob can calculate the same root shared secret RootSharedSecret using Alicepublic and Bobsecret and derive the same PRIdA→B,C in order to check if it is in Alice's published PRIds. Bob can also derive the PRId subkey for Alice's DSNP User Id and encrypt Alice's User Id, using his own as the nonce, to generate the Bob-to-Alice PRId (PRIdB→A,C), and then publish it to his own list, if desired.

If Alice or Bob wants to prove to a third party that their PRIds are in each other's PRId list, they can provide the third party with their own subkey CtxSharedSecretA→B or CtxSharedSecretB→A. The third party can repeat the encryption step using Alice and Bob's User Ids, and check that the output is present in the published set of PRIds. The root shared secret RootSharedSecret (used as a master key in this algorithm) should not be divulged.

Test Vector

For the following inputs:

InputValue
Asecret0xc9432ed5c0c5c24e8a4ff190619893918b4d1265a67d123895023fa7324b43e0
Apublic0x0fea2cafabdc83752be36fa5349640da2c828add0a290df13cd2d8173eb2496f
Bsecret0xdc106e1371293ee9536956e1253f43f8941d4a5c4e40f15968d24b75512b6920
Bpublic0xd0d4eb21db1df63369c147e63b2573816dd4b3fe513e95bf87f7ed1835407e62
IdA42
IdB478
ContextPRIdCtx0

An implementation of the PRId generation algorithm should produce the following outputs:

OutputValue
PRIdA→B0xace4d2995b1a829c
CtxSharedSecretA→B0x37cb1a870f0c1dce06f5116faf145ac2cf7a2f7d30136be4eea70c324932e6d2
PRIdB→A0x1a53b02a26503600
CtxSharedSecretB→A0x32c45c49fcfe12f9db60e74fa66416c5a05832c298814d82032a6783a4b1fca0

Batch Publications

A Batch Publication is an Apache Parquet file with a collection of Announcements.

Implementation Requirements

Discoverable

Implementations MUST have publicly discoverable Batch Publications.

Validity

Implementations MUST be able to validate Parquet file contents. Validity MUST be immutable.

Historical

Implementations MUST retain proof of existence of a Batch Publication.

Transparent Chain of Delegation

All Announcements in a Batch file MUST be able to be proven to be from or have a chain of delegation to the publisher of the Batch.

File Requirements

Batch files are stored and transferred in Apache Parquet format.

  • Batch files MUST match the spec for a single Announcement Type.
  • Batch files MUST have Bloom filters set in accordance with the Announcement Type Spec.
  • Batch files MUST have NO MORE THAN 128*1024 rows.

Bloom Filter

Calculation for filter bits is different and is nearly a factor of 10 lower than for a standard Bloom filter: 128*1024 rows with a 0.001 false-positive rate results in around 29,000 bits for a Split Block Bloom filter.

Bloom filters are ONLY added to some fields. See also Announcement Types.

Columns with Bloom Filters

ColumnParquet Type
contentHashBYTE_ARRAY
emojiBYTE_ARRAY
fromIdBYTE_ARRAY
inReplyToBYTE_ARRAY
objectIdBYTE_ARRAY

Non-Normative

Design Requirements

Batch files need to be quickly and easily searchable. Minimal storage size and fast, simple querying are preferred over guarantees of no false positives or advanced data manipulation and column relationships. The files are parseable by client applications, web views, or browsers running pure JavaScript without needing to convert the format.

Applications need to know if a given Batch file has any information of interest without downloading the file first.

Why Parquet?

  1. Parquet is a column-oriented format. Since DSNP Batch Message data will have a very small column-to-row ratio compared to a typical web application database, it makes sense to prefer a column-oriented format.
  2. Parquet format has been field tested under extreme network conditions. It has broad support in cloud storage solutions, with libraries in multiple languages.
  3. Bloom filters are already supported in the Parquet specification, which allows for fast and accurate searching (with caveats for proper configuration).
  4. Amazon S3 support: We anticipate that some Batch Announcers (and possibly Archivists) will store Batch files on Amazon S3. Amazon Athena also supports storage in Parquet, and its API supports SQL-like queries.
  5. Parquet also allows references to the same column across files, which could enable multi-file querying in the future.
  6. Parquet supports compression formats such as Brötli, which is already a browser standard and offers a demonstrated improvement in compression speed and file size over older formats.
  7. Parquet files can be transferred directly to clients, which can parse the files in the app or browser. No conversion to a serialization format is necessary. This eliminates an entire class of bugs and makes both fetching and querying faster.
  8. Parquet uses schemas, which additionally reduce file size.

Rejected Alternatives

  1. Cassandra, RocksDB, CouchDB, MongoDB, and HBASE were rejected since DSNP data needs neither a database for storage nor the overhead of one. Each of these was designed for use cases ranging from somewhat different to drastically different from the DSNP network.
  2. JSON, BSON, and SQLite, while sometimes used for storage, are intended for serialization. They are schemaless, which results in redundant information and therefore a larger size than formats with schemas. They also don't support Bloom filters; thus, indexing would be required, or new batches would need to be downloaded entirely. The exception is SQLite, which does support more advanced queries. However, SQLite was designed for in-memory storage.

Batch Validity and Order

Batch validity is immutable and is usually based (in part) on the validation of the delegation of authors listed inside the batch to the publisher. Due to the nature of distributed systems, it is possible that a race condition occurs such that a user's delegation revocation presents before a Batch that contains a message from that user via the revoked delegate. While those individual messages should be considered invalid, a window of time for historical testing is suggested before considering the entire batch invalid. This is analogous to the idea of a confirmation time, but only applies to the past rather than the future.

Announcements Overview

Announcements are content or references to content that communicate new user activity to the rest of the network. Announcements are associated with an Identifier that can be validated as the creator of the Announcement. Depending on the implementation, Announcements may be published directly to the network, included in Batch Publication Files, or some combination of those two.

Announcement Types

Each Announcement has an enumerated type for use when separating out a stream of Announcements.

ValueNameDescriptionDSNP Content URITombstone Allowed
0Tombstonean invalidation of previously announced contentnono
1Graph Changeasocial graph changesnono
2Broadcasta public postYESYES
3Replya public response to a BroadcastYESYES
4Reactiona public visual reply to a Broadcastnono
5Profilea profileYESno
6Updatean update to contentYESno
7Public Keya public key for secure communicationnono

a Since DSNP version 1.2, social graph changes use User Data operations as described in the Graph section.

Announcement Validation

There is no guarantee that, at time of creation, a given Announcement will be from the fromId claimed in the Announcement. The reader MUST perform a validation of the Announcement at read time to ensure authenticity. Implementations MUST provide a way to validate that the identifier associated with a given Announcement is authentic.

External Content URLs and Hashes

Where Announcements refer to external documents (such as Activity Content documents), these are referenced by both a URL and a DSNP Content Hash. The content hash MUST be generated by applying a Supported Hashing Algorithm to the full, unaltered contents retrieved from the URL. When readers retrieve content referenced in an Announcement, they can validate the authenticity of the content by regenerating the hash output and comparing it with the content hash recorded in the Announcement.

Duplicate Handling

Due to the nature of asynchronous communication, duplicate Announcements may occur. In the case of duplicates, the first Announcement should be considered the ONLY valid Announcement. Additional duplicate Announcements MUST be rejected or ignored.

Ordering Announcements

  1. Order Batch Publications by implementation order.
  2. Order Announcements in a Batch Publication File by row appearance order.

Reverting an Announcement

Announcements may not be deleted, but may be marked invalid by using a Tombstone Announcement, or updated by using an Update Announcement. For example, if a user creates a Reaction Announcement, they may remove that reaction by creating a Tombstone Announcement.

Non-Normative

Duplicate Announcements

Due to the distributed nature of DSNP, duplicate Announcements are possible from time to time. These should be discarded and ignored.

Replay Attacks

Implementations typically restrict replay attacks by testing that the chain transaction sender is authorized (often via delegation) to publish an Announcement.

Announcement Ordering and Activity Content Published Timestamp

Activity Content has a published field that contains a user-generated timestamp. User-generated timestamps cannot be validated, but may be used to indicate ordering other than the network order for Announcements (which are not time dependent.)

Announcement Reference Ordering

Some Announcements contain references to other Announcements via the inReplyTo field. Due to the distributed nature of DSNP, the canonical order can have an Announcement that refers to another Announcement appearing later in the network order. For display purposes, these messages should be considered to have occurred after the reference.

DSNP v1.0 Announcement Signatures

In DSNP v1.0, Announcements had individual signatures, producing Batch Publications that were generic and disconnected from the user. Announcements could be submitted to the chain via anyone—not just delegates or users.

In DSNP v1.1, Announcement signatures were removed in favor of the implementation being responsible for the connection between the on-chain signature and the user. Implementations require that the transaction that produces a Batch be performed by the user or delegate directly. This creates batches that are delegate specific, but allows for faster testing of the validity of individual Announcements in a Batch.

For more information see DIP-145.

Tombstone Announcement

A Tombstone Announcement is a way to note that a previously announced content is invalid and the related Announcement should be considered reverted. It is NOT possible to revert a tombstone.

Fields

FieldDescriptionData TypeSerializationParquet TypeBloom Filter
announcementTypeAnnouncement Type Enum (0)enumdecimalINT32no
fromIdId of the user creating the Announcement and the Tombstoned Announcement64-bit unsigned integerdecimalUINT_64YES
targetAnnouncementTypetarget tombstoned Announcement typeenumdecimalINT32no
targetContentHashtarget contentHash of the original Announcement to tombstonevariable length byte arrayhexadecimalBYTE_ARRAYYES

Field Requirements

announcementType

  • MUST be fixed to 0

fromId

  • MUST be a DSNP User Id
  • MUST have authorized the creation of the Announcement, either directly or via a transparent chain of delegation

targetAnnouncementType

  • MUST be the Announcement Type of the target Announcement
  • MUST ONLY be a Tombstone allowed Announcement Type

Tombstone Allowed Announcement Types

ValueName
2Broadcast
3Reply

targetContentHash

  • MUST match a contentHash of previous Announcement with the same fromId as the Tombstone Announcement

Broadcast Announcement

A Broadcast Announcement is a way to send a public message to everyone.

Fields

FieldDescriptionData TypeSerializationParquet TypeBloom Filter
announcementTypeAnnouncement Type Enum (2)enumdecimalINT32no
contentHashmultihash-encoded hash of content stored at URLvariable length byte arrayhexadecimalBYTE_ARRAYYES
fromIdid of the user creating the Announcement64-bit unsigned integerdecimalUINT_64YES
urlcontent URLUTF-8UTF-8UTF8no

Field Requirements

announcementType

  • MUST be fixed to 2

contentHash

fromId

  • MUST be a DSNP User Id
  • MUST have authorized the creation of the Announcement, either directly or via a transparent chain of delegation

url

  • MUST NOT refer to localhost or any reserved IP addresses as defined in RFC6890
  • Resource MUST be one of the supported Activity Content Types
  • MUST use one of the supported URL Schemes

Supported URL Schemes

SchemeDescriptionReferenceDSNP Version Added
HTTPSHypertext Transfer Protocol SecureRFC28181.0

Reply Announcement

A Reply Announcement is the same as a Broadcast Announcement, but includes an inReplyTo field for noting it as a reply to a given DSNP Content URI.

Fields

FieldDescriptionData TypeSerializationParquet TypeBloom Filter
announcementTypeAnnouncement Type Enum (3)enumdecimalINT32no
contentHashmultihash-encoded hash of content stored at URLvariable length byte arrayhexadecimalBYTE_ARRAYYES
fromIdid of the user creating the Announcement64-bit unsigned integerdecimalUINT_64YES
inReplyToTarget DSNP Content URIUTF-8UTF-8UTF8YES
urlcontent URLUTF-8UTF-8UTF8no

Field Requirements

announcementType

  • MUST be fixed to 3

contentHash

fromId

  • MUST be a DSNP User Id
  • MUST have authorized the creation of the Announcement, either directly or via a transparent chain of delegation

inReplyTo

url

  • MUST NOT refer to localhost or any reserved IP addresses as defined in RFC6890
  • Resource MUST be one of the supported Activity Content Types
  • MUST use one of the supported URL Schemes

Supported URL Schemes

SchemeDescriptionReferenceDSNP Version Added
HTTPSHypertext Transfer Protocol SecureRFC28181.0

Reaction Announcement

A Reaction Announcement is for publishing emoji reactions to anything with a DSNP Content URI.

Fields

FieldDescriptionData TypeSerializationParquet TypeBloom Filter
announcementTypeAnnouncement Type Enum (4)enumdecimalINT32no
emojithe encoded reactionUTF-8UTF-8UTF8YES
applyhow to apply the reaction8-bit unsigned integerdecimalUINT_8no
fromIdid of the user creating the relationship64-bit unsigned integerdecimalUINT_64YES
inReplyToTarget DSNP Content URIUTF-8UTF-8UTF8YES

Field Requirements

announcementType

  • MUST be fixed to 4

emoji

  • Emoji fields must not be empty
  • Emoji fields must consist only of Unicode points from U+2000 to U+2BFF, from U+E000 to U+FFFF, or from U+1F000 to U+10FFFF

Examples

All of the following should be considered valid emojis:

"😀", "🤌🏼", "👩🏻‍🎤", "🧑🏿‍🏫", "🏳️‍🌈", "🏳️‍⚧️", "⚛︎", "🃑", "♻︎"

None of the following should be considered valid:

"F", ":custom-emoji:", "<custom-emoji>", "ᚱ", "ᘐ", "״"

apply

  • MUST be an UINT_8
  • Indicates whether the emoji should be applied and if so, at what "strength".

Potential uses:

  • a single reaction
  • ratings
  • a range of responses, e.g. "strongly disagree" --> "strongly agree" = 1 --> 5 stars.
  • recommendation engines

Apply Enums

ValueNameDescription
0retractRemove the referenced emoji
napplyApply the referenced emoji N times

fromId

  • MUST be a DSNP User Id
  • MUST have authorized the creation of the Announcement, either directly or via a transparent chain of delegation

inReplyTo

Non-Normative

Likes

Generic "likes" should default to the "❤️" or Unicode U+FE0F as the emoji in the reaction.

Profile Announcement

A Profile Announcement is a constrained version of a Broadcast Announcement. The reference content MUST be of profile type.

Fields

FieldDescriptionData TypeSerializationParquet TypeBloom Filter
announcementTypeAnnouncement Type Enum (5)enumdecimalINT32no
contentHashmultihash-encoded hash of content stored at URLvariable length byte arrayhexadecimalBYTE_ARRAYYES
fromIdid of the user creating the Announcement64-bit unsigned integerdecimalUINT_64YES
urlprofile content URLUTF-8UTF-8UTF8no

Field Requirements

announcementType

  • MUST be fixed to 5

contentHash

fromId

  • MUST be a DSNP User Id
  • MUST have authorized the creation of the Announcement, either directly or via a transparent chain of delegation

url

  • MUST NOT refer to localhost or any reserved IP addresses as defined in RFC6890
  • Resource MUST be a valid Profile Activity Content Type
  • MUST use one of the supported URL Schemes

Supported URL Schemes

SchemeDescriptionReferenceDSNP Version Added
HTTPSHypertext Transfer Protocol SecureRFC28181.0

Non-Normative

Most Recent Profile

When displaying a DSNP user's profile, the most recent profile should be considered the complete and correct version. Previous Profile Announcements from the same fromId may be disregarded.

Update Announcement

An Update Announcement is a way to note intent to update previously announced content. If the original Broadcast/Reply is Tombstoned, subsequent Updates should be ignored.

Fields

FieldDescriptionData TypeSerializationParquet TypeBloom Filter
announcementTypeAnnouncement Type Enum (6)enumdecimalINT32no
fromIdid of the user creating the announcement64 bit unsigned integerdecimalUINT_64YES
contentHashmultihash-encoded hash of content stored at URLvariable length byte arrayhexadecimalBYTE_ARRAYYES
urlupdated content URLUTF-8UTF-8UTF8no
targetAnnouncementTypetarget updated Announcement typeenumdecimalINT32no
targetContentHashmultihash-encoded hash of target contentvariable length byte arrayhexadecimalBYTE_ARRAYYES

Field Requirements

announcementType

  • MUST be fixed to 6

fromId

contentHash

url

  • MUST NOT refer to localhost or any reserved IP addresses as defined in RFC6890
  • Resource MUST be one of the supported Activity Content Types
  • MUST use one of the supported URL Schemes

targetAnnouncementType

  • MUST be the Announcement Type of the target Announcement
  • MUST ONLY be an Update allowed Announcement Type

Update Allowed Announcement Types

ValueName
2Broadcast
3Reply

targetContentHash

  • MUST be the contentHash of an allowed Announcement type with the same fromId as the Update Announcement

Public Key Announcement

A Public Key Announcement is a way to note a new cryptographic key that can be used in DSNP to secure and verify the authenticity of communications.

The most recently published key (if one exists) for a given key type should be treated as the active key of that key type.

Fields

FieldDescriptionData TypeSerializationParquet TypeBloom Filter
announcementTypeAnnouncement Type Enum (7)enumdecimalINT32no
fromIdid of the user creating the Announcement64 bit unsigned integerdecimalUINT_64YES
keyTypeKey Type EnumenumdecimalINT32YES
keyIduser-assigned identifier64 bit unsigned integerdecimalUINT_64no
publicKeypublic key in multikey formatvariable length byte arrayUTF-8BYTE_ARRAYno

Field Requirements

announcementType

  • MUST be fixed to 7

fromId

keyType

  • MUST be an allowed Key Type value

Allowed Key Types

ValueNameAllowed Algorithms (multicodec)Purpose
1keyAgreementx25519-pubA Curve25519 public key that can be used in key exchange protocols to generate a shared secret

keyId

  • A user-assigned 64-bit identifier for the key.

The user may assign a new keyId each time they announce a new key of a given keyType. A keyId value is useful when invoking certain DSNP Operations in order to indicate which key was used to encrypt data. It may also provide a hint to the user if they ever need to regenerate their private key (for example, many key derivation functions enable the use of a subkey identifier to deterministically create a subkey from a root key).

publicKey

  • MUST be a public key of an allowed algorithm for keyType, encoded in multikey format

The multikey encoding of public keys is described in the draft did:key Method specification. The byte encoding consists of a multicodec key identifier (as a varint) followed by the public key's binary data in the codec's described format.

If serializing the multicodec value as a string, base58btc encoding is recommended. For example, the string z6LStiZsmxiK4odS4Sb6JmdRFuJ6e1SYP157gtiCyJKfrYha decodes as a Base58 string using the x25519-pub multicodec value with a 32-byte raw key of 0xfd3384e132ad02a56c78f45547ee40038dc79002b90d29ed90e08eee762ae715.

Migrated Announcements

Graph Change Announcement

Since DSNP version 1.2, social graph changes use User Data operations as described in the Graph section.

A Graph Change Announcement is for publishing relationship state changes for a user.

Fields

FieldDescriptionData TypeSerializationParquet TypeBloom Filter
announcementTypeAnnouncement Type Enum (1)enumdecimalINT32no
changeTypeType of relationship changeenumdecimalINT32no
fromIdId of the user creating the relationship64-bit unsigned integerdecimalUINT_64YES
objectIdId of the target of the relationship64-bit unsigned integerdecimalUINT_64YES

Field Requirements

announcementType

  • MUST be fixed to 1

changeType

  • MUST be one of the Change Type Enum

Change Type Enum

Different change types have different meanings.

ValueNameDescription
0UnfollowRemove a Follow relationship
1FollowCreate a Follow relationship

fromId

  • MUST be a DSNP User Id
  • MUST have authorized the creation of the Announcement, either directly or via a transparent chain of delegation

objectId

Non-Normative

Graph Retrieval, Ordering and Reading

Each Graph Change event represents a state transition for the graph. The state of the graph at any time is given by taking the state of the graph at a previous time and applying all Graph Change events not previously applied in the order specified by Announcement Ordering.

Once those Graph Change events are retrieved, they can be ordered to reflect the current graph state (i.e. Charlie has followed Bob, then he unfollowed him, and then followed him again. The graph state reflects that Charlie is following Bob.)

To retrieve the graph, do the following:

  1. Retrieve the events with announcementType matching the enum for Graph Change.
  2. Filter the events to a particular DSNP User Id to retrieve information about the respective graph.
  3. Order the retrieved data by Announcement Ordering.

Operations

DSNP implementations perform well-defined DSNP Operations and generate DSNP State Change Records.

Control Keys and Proofs

Each invocation of a DSNP Operation MUST have verifiable approval of the acting principal(s) via a Control Key Ownership Proof. The precise data type and representation of both the Control Key and the Control Key Ownership Proof MUST be defined by each DSNP implementation. For example, an implementation might use the public key of an asymmetric key pair as a control key, and provide a proof for each operation by producing a cryptographic signature of the user's DSNP Identifier and some nonce value.

Where operations are listed as using control keys or ownership proofs as input parameters, this indicates that these keys or proofs should be provided in addition to those needed for invocation authentication.

Transaction Identifiers

Each invocation of a DSNP Operation should be associated with a Transaction Identifier. Transaction Identifiers are used to associate Operation invocations with asynchronously emitted State Change Records. It MUST be possible to associate a DSNP State Change Record with a Transaction Identifier from a particular DSNP Operation invocation. Transaction Identifiers MUST be unique within an implementation. Transaction Identifiers MUST be serializable as a string.

Failure Handling

Compliant implementations may respond to error conditions either synchronously, as a response to the invocation request, or asynchronously, by emitting a Failure Record.

List of Operations

OperationOptional?Principal(s)InputsState Change Record or Output
Create IdentifiernoNoneControl Key, Control Key Ownership ProofIdentifier Creation Record
Retire IdentifiernoUserNoneIdentifier Retirement Record
Define DelegationnoUser AND DelegateUser's Identifier, Delegate's Identifier, Set of Allowed Announcement Types, Set of Allowed User Data TypesDelegation Definition Record
Revoke DelegationnoUser OR DelegateUser's Identifier, Delegate's IdentifierDelegation Revocation Record
Add Control KeyYESUserKey, Key Ownership ProofControl Key Addition Record
Remove Control KeyYESUserKeyControl Key Removal Record
Publish Announcementno*User OR DelegateAnnouncementAnnouncement Published Record
Publish Batchno*User OR DelegateAnnouncement Type, Batch Publication URL, Batch Publication Content HashBatch Published Record
Get User DatanoAnyUser's Identifier, Set of Requested User Data TypesMap of User Data Types to Data Chunks with optional Key Identifiers
Replace User DatanoUser OR DelegateUser's Identifier, Key Identifier, Map of User Data Types to Data ChunksUser Data Replaced Record

* For each Announcement Type, an implementation may support one or both of these operations. Implementations MUST document which of the operations is available for each Announcement Type.

State Change Records

State Change Records constitute the observable output of a DSNP system. Implementations MUST specify how applications can translate implementation-specific output into State Change Records.

Each Record is associated with a Transaction Identifier, which allows Operations to be asynchronously associated with their results.

Records consists of one or more fields.

Record Type Field Field Data Type
Identifier Creation Record Control Key Implementation dependent
Control Key Ownership Proof Implementation dependent
Identifier Retirement Record User's Identifier DSNP User Id
Delegation Definition Record User Id DSNP User Id
Delegate Id DSNP User Id
Allowed Announcement Types List of enum values
Allowed User Data Types List of enum values
Delegation Revocation Record User Id DSNP User Id
Delegate Id DSNP User Id
Control Key Addition Record User Id DSNP User Id
Key Implementation dependent
Key Ownership Proof Implementation dependent
Control Key Removal Record User Id DSNP User Id
Key Implementation dependent
Announcement Published Record Announcement One of the types described in Announcements
Batch Published Record From Id DSNP User Id
Announcement Type Enum value
URL String
Content Hash DSNP Content Hash
Failure Record Message String
User Data Replaced Record User Id DSNP User Id
User Data Types Set of User Data Types

Serializations

Serialization is how the value should be stringified for signing and for transfer between systems. Most serializations use outside standards, but some requiring additional clarifications are provided here.

decimal

Used to represent integers. Strings are used to avoid issues with different implementations of numbers.

  • MUST use 0-9 representation
  • MUST NOT have spaces or separators
  • MUST be a string
InvalidWhyValid
0x123Must be decimal"291"
291Must be a string"291"
291nBigInt(291) serialization appends an n"291"

hexadecimal

Used to represent bytes.

  • MUST use 0-9 and a-f representation
  • MUST be lowercase
  • MUST be prefixed with a 0x
  • MUST NOT have spaces or separators
  • MUST have two characters per byte in addition to the 0x characters
BytesInvalidValid
20x1230x0123
2123h0x0123
20x0ABC0x0abc
80xabc0x0000000000000abc
320x3e34c4325f4461b9355027b314f3eb56d31af549f7da7bd9ef1ce951651e0x00003e34c4325f4461b9355027b314f3eb56d31af549f7da7bd9ef1ce951651e

DSNP Systems

DSNP Over Frequency

Language and Framework

Frequency is a Polkadot Parachain written in Rust, using the Substrate framework.

DSNP Over Frequency Schemas

Official schemas may be found in GitHub.

NameSchema Id MainnetSchema Id Rococo
Tombstone11
Broadcast22
Reply33
Reaction44
Profile65
Update56
Public Key718
Public Follows813
Private Follows914
Private Connections1015
Last Update DateFrequency ReleaseDSNP Version
2023-07-171.5.2+1.2.0

Frequency Identity

Identity

  • Name: Message Source Account or MSA
  • Docs: MSA Pallet
  • Representation: The following data that constitute a Message Source Account are stored in the MSA pallet:
    • The DSNP User Id associated with this MSA
    • Delegation relationships to Providers
    • Schema permissions granted to Providers
    • MSA user state (Profile, Graph, etc...)
    • Control keys

DSNP User Identifier

  • Name: Message Source Account Identifier, or MSA Id.
  • Data Type: uint64
  • Docs: MessageSourceId
  • Mapping: The MSA Id is able to be used directly as the DSNP User Id
  • Description: At least one public key MUST be associated with an MSA Id for it to be considered active.

Control Keys

  • Name: Referred to as: public_key, provider_key, or delegator_key
  • Data Type: AccountId, Schnorrkel/Ristretto X25519 ("sr25519") derived cryptographic public key
  • Docs: AccountId
  • Description: See Cryptography on Polkadot and Polkadot Protocol Specification. A public key CANNOT be associated with more than one MSA at a time.

Delegation

  • Name: delegation
  • Docs: Delegation
  • Representation: The following data storage relates necessary information for retrieving and validating delegations:
    • Provider registry
    • Delegations
    • Schema permissions granted to Providers

User

  • Name: Delegator
  • Representation: MSA Id

Delegate

  • Name: Provider
  • Representation: MSA Id
  • Description: A Provider MUST already have an MSA Id (via msa::create()) and be approved as a Provider (via msa::propose_to_be_provider()).

Announcement Publishing

On Frequency, Announcements are mapped to Schemas which in turn publish Frequency Messages or Stateful Storage changes. Frequency Messages are either individual Announcements from a particular user, or a Batch Publication with a multitude of possible users. Frequency Stateful Storage is either direct Announcements from a particular user or User Data changes.

Announcement Type EnumAnnouncementTypeSchema Id MainnetSchema Id RococoFrequency Model TypeFrequency Payload Location
0TombstoneBatched11ParquetIPFS
2BroadcastBatched22ParquetIPFS
3ReplyBatched33ParquetIPFS
4ReactionBatched44ParquetIPFS
5ProfileBatched65ParquetIPFS
6UpdateBatched56ParquetIPFS
7Public KeyStatefulTBD18AvroBinaryItemized

Source code for each schema is located in the LibertyDSNP/schemas repository.

Batch Publications

Frequency uses DSNP Batch Publications for some types of Announcements. Parquet files are stored on IPFS, but are discovered through Frequency Messages.

DSNP Batch Publications MUST be validated upon fetching to ensure data and permission integrity.

Announcement Validation

DSNP Announcements are validated differently depending on the type of Announcement. Non-batched Announcements are on chain, are validated at write time, and do not need to be re-validated at read time. Batched Announcements are off chain and MUST be validated at read time (See: Validation).

Ordering Announcements

Frequency Messages are well ordered:

  1. Frequency: Block number ascending
  2. Frequency: Block index ascending (unique)
  3. DSNP Standard: Order Announcements in a Batch Publication File by row appearance order.

Human Order

Due to the asynchronous nature of networks and batching, it is possible that the canonical ordering of Announcements is wrong from a human viewpoint. With dependent Announcements, where one Announcement refers to another Announcement, the order may be inferred differently than the canonical ordering. It is left to user interfaces to handle these situations.

Retrieval

Frequency nodes provide an RPC interface messages.getBySchemaId() with paginated responses that differ based on the Schema.

Frequency nodes can provide a websocket interface that will emit an event for each block that has one or more messages of a given schema in that block. The messages::MessagesStored event can be used to know when to call the RPC interface to retrieve the messages.

See the Frequency Documentation for more details on Message retrieval.

Batch Publication Validation

The Frequency Message for a Batch Publication has several important fields for validation:

FieldDescription
provider_msa_idMSA Id of the provider sending the message
cidThe Content IDentifier v1 for IPFS content
payload_lengthExpected length of the content from IPFS
block_numberBlock number that the message was recorded on the chain.

File Validation

  1. Retrieve the file from the IPFS network using the cid.
  2. Verify the file hash by comparing it to the hash included in the cid. (Required for non-trusted IPFS nodes.)
  3. Verify that the byte length of the retrieved file matches the payload_length.

Publication Announcements Validation

  1. Collect the unique set of fromId values.
  2. Use the Custom RPC msa.checkDelegations() with the fromId values as the delegator_msa_ids and the provider_msa_id at the block_number.
  3. The fromId values that msa.checkDelegations() verifies as having a delegation at block_number are valid Announcements.
  4. Set the schema_id parameter to the Schema Id used in the Frequency Message

Announcement Duplicates

Duplicate Announcements MUST be rejected or ignored.

Operations

Method of execution

DSNP Operations are executed on Frequency via on-chain transactions, also known as extrinsic calls. An extrinsic is a type of function defined in a Substrate Pallet. A Pallet is a Substrate runtime module and also a Cargo crate.

MSA Pallet

Responsible for DSNP Identity Operations and Delegation management.

Schema Pallet

Responsible for managing the data structures for DSNP Announcements and User Data.

Messages Pallet

Responsible for most DSNP Announcement Operations (see Publishing).

Stateful Storage Pallet

Responsible for DSNP User Data (see User Data) and select Announcement Operations (see Publishing).

Principals

Every Frequency transaction for DSNP is accompanied by an Schnorrkel/Ristretto X25519 ("sr25519") derived cryptographic signature and associated public key. When the transaction occurs, the signature is validated and the MSA Id is retrieved.

DSNP TermFrequency Term
UserUser / Delegator
DelegateProvider

Failure Handling

Frequency has a variety of errors that fall into these classes:

  • Timeout: A transaction with an unknown status after a set time.
  • Node Rejection: Public nodes may reject transactions from unknown parties.
  • Node Validation Error: A node will reject malformed or invalid transactions when possible.
  • Execution Error: A transaction that was included in a block but failed upon execution.

Transactions are not automatically resubmitted. Check with the Frequency Documentation for more information regarding errors.

List of Operations

Write operations are via Transactions (also called Extrinsics): pallet::extrinsic()

State Change Records

State Change Records constitute the observable output of a DSNP system. Frequency uses a combination of on chain data to store and Events to notify for Records.

Frequency Transaction Identifier

Frequency uses the hash of the Operation transaction to then request a node report the transaction status and any Events related to that hash. (Client libraries usually have this built in.)

Frequency Records

Frequency produces three types of data that map or point to DSNP Records.

Events

Events are generated on each block and may contain a pointer to the Record data instead of the entire Record.

Events are referenced by pallet::EventName.

Messages

Frequency Messages store Announcements or Batch Publication Records. It uses the Messages Pallet.

Messages are retrieved via state queries (pallet.stateQuery) or RPC calls (pallet.rpcCall()).

State

Frequency State stores data associated with an Identity. It can also be used to look up the state of prior Records related to an Identity.

State data is retrieved via state queries (pallet.stateQuery) or RPC calls (pallet.rpcCall()).

Record Mappings

Frequency User Data

On Frequency, User Data and select Announcements are mapped to Schemas which use Stateful Storage for storage and retrieval of the data.

User Data Sets

User Data SetSchema Id MainnetSchema Id RococoFrequency Model TypeFrequency Payload LocationSettings
Public Follows813AvroBinaryPaginatedNone
Private Follows914AvroBinaryPaginatedNone
Private Connections1015AvroBinaryPaginatedNone

Pseudonymous Relationship Identifiers (PRIds) are stored along side Private Connections in the same Stateful Storage page.

Source code for each schema is located in the LibertyDSNP/schemas repository.

Announcements

AnnouncementSchema Id MainnetSchema Id RococoFrequency Model TypeFrequency Payload LocationSettings
Public Key77 (v1.3.0+)AvroBinaryItemizedAppend Only, Signature Required

Read Operation Mapping

Stateful data is retrieved via state queries (pallet.stateQuery) or RPC calls (pallet.rpcCall()).

Model TypeDSNP PropertiesQuery/Call
ItemizedEntity Tagstateful_storage.getItemizedStorage()
PaginatedEntity Tag, Chunkedstateful_storage.getPaginatedStorage()

Write Operation Mapping

Write operations are via Transactions (also called Extrinsics): pallet::extrinsic()

Entity Tags

Frequency requires the hash of current state for any Stateful Storage change.

Activity Content Specification

Version 1.2.0

Content references shared via the DSNP consist of URLs pointing to documents containing Activity Streams JSON objects. For the purposes of the DSNP, restrictions are placed on the Activity Streams 2.0 specification.

JSON-LD and Activity Streams

All DSNP Activity Content is compatible with the Activity Streams 2.0 specification. While there are some DSNP extensions, they are guaranteed to use non-colliding terms. Therefore, the JSON-LD @context field is set to https://www.w3.org/ns/activitystreams according to Activity Streams 2.0 §2.1.

Core Activity Content Types

DSNP uses only the following content types at the root level:

NameDescriptionDSNP Announcements
Notestandard user contentBroadcast, Reply, Update
Profileuser profile contentProfile

Associated Types

NameDescriptionSpecification
Locationadd a location to contentActivity Vocabulary
Tagadd a tag to contentActivity Vocabulary
Attachmentssupported attachment typesActivity Vocabulary
Hashcontent validation hashDSNP Extension

Supported URL Schema

URLs in DSNP-compatible Activity Content MUST use one of the following URL schemes:

SchemeDescriptionReferenceDSNP Version Added
HTTPSHypertext Transfer Protocol SecureRFC28181.0
HTTPHypertext Transfer ProtocolRFC26161.0

Libraries

Releases

VersionDescriptionRelease DateChangelog
1.2.0DIP-2102023-04-11Changelog
1.1.0DIP-1582022-05-05Changelog
1.0.0Initial Release2021-09-09Changelog

Non-Normative

Additional Fields

Implementers may choose to support more of the Activity Streams standard as long as it does not conflict with this specification, but should note that other implementations may not recognize those additions. Implementers who extend their support for Activity Streams objects beyond the subset defined here do so at their own risk.

Activity Stream Type: Note

Activity Vocabulary: Note

PropertyBase SpecRequiredDescriptionRestrictions
@contextActivity Streams 2.0YESJSON-LD @contextMUST be set to https://www.w3.org/ns/activitystreams
typeActivity Vocabulary 2.0YESIdentifies the type of the objectMUST be set to Note
contentActivity Vocabulary 2.0YESText content of the note
mediaTypeActivity Vocabulary 2.0YESMIME type for the content fieldMUST be set to a supported MIME type
publishedActivity Vocabulary 2.0YESThe time of publishingMUST be ISO8601
nameActivity Vocabulary 2.0noThe display name for the note
attachmentActivity Vocabulary 2.0noArray of attached links or mediaMUST be one of the Supported Attachments
tagActivity Vocabulary 2.0noArray of tags/mentionsMUST follow Tag Type
locationActivity Vocabulary 2.0noFor locationMUST follow Location Type

Supported Content MIME Types

FormatMIME TypeSpecification(s)
Plaintext/plain

Examples

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Note",
  "content": "Hello world!",
  "mediaType": "text/plain",
  "published": "1970-01-01T00:00:00+00:00"
}
{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Note",
  "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.",
  "mediaType": "text/plain",
  "attachment": [
    {
      "type": "Link",
      "href": "https://en.wikipedia.org/wiki/Citation_needed"
    }
  ],
  "published": "1970-01-01T00:00:00+00:00"
}

Activity Stream Type: Profile

Profiles are used to provide additional user information display.

Activity Vocabulary: Profile

PropertyBase SpecRequiredDescriptionRestrictions
@contextActivity Streams 2.0YESJSON-LD @contextMUST be set to https://www.w3.org/ns/activitystreams
typeActivity Vocabulary 2.0YESIdentifies the type of the objectMUST be set to Profile
nameActivity Vocabulary 2.0noThe display name for the profile
iconActivity Vocabulary 2.0noAn array of avatars of the profileMUST follow Image Link Type
summaryActivity Vocabulary 2.0noUsed as a plain text biography of the profile
publishedActivity Vocabulary 2.0noThe time of publishingMUST be ISO8601
locationActivity Vocabulary 2.0noFor locationMUST follow Location Type
tagActivity Vocabulary 2.0noFor tags or mentionsMUST follow Tag Type

Examples

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Profile",
  "name": "John Doe",
  "summary": "John Doe is actually a small kitten. See pfp.",
  "icon": [
    {
      "type": "Link",
      "href": "https://placekitten.com/256/256",
      "mediaType": "image/jpeg",
      "width": "256",
      "height": "256",
      "hash": [
        "QmVmUqGYtHcVgpTFR64bHNcLGGFEeWxmUP6pV2C2RbWpKT"
      ]
    },
    {
      "type": "Link",
      "href": "https://placekitten.com/64/64",
      "mediaType": "image/jpeg",
      "width": "64",
      "height": "64",
      "hash": [
        "QmcAh1rov5GcddekCffGeRnaSyiji6ATmfGWpxXYJHgJZx"
      ]
    }
  ],
  "published": "1970-01-01T00:00:00+00:00"
}

Associated Type: Hash

NOT part of the Activity Streams 2.0 Vocabulary.

Activity objects linking to external content such as audio, image or video files must include a "hash" field for users to validate linked content. The value of this "hash" field must be an array of strings, each representing a hash output using a specific algorithm. AT LEAST ONE hash in the array MUST be one of the supported algorithms, although others may also be used.

Hashes MUST be encoded using the multihash specification, and serialized as a multibase string.

Supported Algorithms

AlgorithmMultihash NameLeading bytes (as varint)ReferenceDSNP Version Added
SHA-256sha2-2560x1220RFC 62341.2.0
BLAKE2bblake2b-2560xa0e40220RFC 76931.2.0

Example

This example gives SHA-256 and BLAKE2b hashes for the PDF version of the DSNP whitepaper.

{
  "hash": [
    "QmQNHNfHnbgJJ6nK4UPx2VtTUCafAKCbqZJ6ZRYUGjoeFj",
    "2DrjgbGgSsXRhTiBWckoVwBFC6H4qiBWWNumSsRwdUt82YnTdN"
  ]
}

Associated Type: Location

Activity Vocabulary: location

PropertyBase SpecRequiredDescriptionRestrictions
typeActivity Vocabulary 2.0YESIdentifies the type of the objectMUST be set to Place
nameActivity Vocabulary 2.0YESThe display name for the location
accuracyActivity Vocabulary 2.0noThe accuracy of the coordinates as a percentage. (e.g. "94.0" means "94.0% accurate")
altitudeActivity Vocabulary 2.0noThe altitude of the location
latitudeActivity Vocabulary 2.0noThe latitude of the location
longitudeActivity Vocabulary 2.0noThe longitude of the location
radiusActivity Vocabulary 2.0noThe area around the given point that comprises the location
unitsActivity Vocabulary 2.0noThe units for radius and altitude (defaults to meters)MUST be one of these: cm, feet, inches, km, m, miles

Warning

Location data may pose a privacy danger to users. Users should be warned before publishing location data.

Example

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Note",
  "content": "I'm in NYC!",
  "mediaType": "text/plain",
  "location": {
    "type": "Place",
    "name": "New York City, NY",
    "latitude": "40.73",
    "longitude": "-73.93",
    "accuracy": "94.0"
  },
  "published": "1970-01-01T00:00:00+00:00"
}

Associated Type: Tag

Activity Vocabulary: tag

Hashtag

PropertyBase SpecRequiredDescriptionRestrictions
nameActivity Vocabulary 2.0YESThe text of the tag

Example

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Note",
  "content": "I love the #dsnp spec!",
  "mediaType": "text/plain",
  "tag": [
    {
      "name": "#dsnp"
    }
  ],
  "published": "1970-01-01T00:00:00+00:00"
}

Mention

PropertyBase SpecRequiredDescriptionRestrictions
nameActivity Vocabulary 2.0noThe text of the tag
typeActivity Vocabulary 2.0YESIdentifies the tag as type MentionMUST be Mention
idActivity Vocabulary 2.0YESLink to the user mentionedMUST be a DSNP User URI

Example

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Note",
  "content": "@sally should check out the #dsnp spec!",
  "mediaType": "text/plain",
  "tag": [
    {
      "name": "#dsnp"
    },
    {
      "name": "@sally",
      "type": "Mention",
      "id": "dsnp://12345678"
    }
  ],
  "published": "1970-01-01T00:00:00+00:00"
}

Associated Attachments

Audio

Activity Vocabulary: Audio

PropertyBase SpecRequiredDescriptionRestrictions
typeActivity Vocabulary 2.0YESIdentifies the type of the objectMUST be set to Audio
urlActivity Vocabulary 2.0YESAn array of links for given audio content in different formatsMUST be an Audio Link AND MUST have at least one supported audio MIME type
nameActivity Vocabulary 2.0noThe display name for the audio file
durationActivity Vocabulary 2.0noApproximate duration of the audio
PropertyBase SpecRequiredDescriptionRestrictions
typeActivity Vocabulary 2.0YESIdentifies the type of the objectMUST be set to Link
hrefActivity Vocabulary 2.0YESThe URL for the given audio contentMUST be a Supported URL Schema
mediaTypeActivity Vocabulary 2.0YESMIME type of href contentMUST follow
hashDSNP 1.0YESArray of hashes for linked content validationMUST include at least one supported hash

Supported Audio MIME Types

FormatMIME TypeSpecification(s)
MP3audio/mpegRFC3003
OGGaudio/oggRFC5334
WebMaudio/webmWebM standard

Example

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Note",
  "attachments": [
    {
      "type": "Audio",
      "name": "The Scream",
      "url": [
        {
          "type": "Link",
          "href": "https://upload.wikimedia.org/wikipedia/commons/d/d9/Wilhelm_Scream.ogg",
          "mediaType": "audio/ogg",
          "hash": [
		    "QmQrGdv6Ky5sJhaVdw27y4aod5pdfihDkBTxiBkRaSGJJ7"
          ]
        }
      ],
      "duration": "1S"
    }
  ],
  "published": "1970-01-01T00:00:00+00:00"
}

Image

Activity Vocabulary: Image

PropertyBase SpecRequiredDescriptionRestrictions
typeActivity Vocabulary 2.0YESIdentifies the type of the objectMUST be set to Image
urlActivity Vocabulary 2.0YESAn array of links for given image content in different formatsMUST be an Image Link AND MUST have at least one supported image MIME type
nameActivity Vocabulary 2.0noThe display name or alt text for the image
PropertyBase SpecRequiredDescriptionRestrictions
typeActivity Vocabulary 2.0YESIdentifies the type of the objectMUST be set to Link
hrefActivity Vocabulary 2.0YESThe URL for the given imageMUST be a Supported URL Schema
mediaTypeActivity Vocabulary 2.0YESMIME type of href content
hashDSNP 1.0YESArray of hashes for linked content validationMUST include at least one supported hash
heightActivity Vocabulary 2.0noA hint as to the rendering height in device-independent pixels
widthActivity Vocabulary 2.0noA hint as to the rendering width in device-independent pixels

Supported Image MIME Types

FormatMIME TypeSpecification(s)
JPEGimage/jpegRFC2045
PNGimage/pngW3C PNG Standard
SVGimage/svg+xmlW3C SVG standard
WebPimage/webpWebP standard
GIFimage/gifRFC2045

Example

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Note",
  "attachments": [
    {
      "type": "Image",
      "name": "One of the founders of DSNP",
      "url": [
        {
          "type": "Link",
          "href": "https://upload.wikimedia.org/wikipedia/commons/a/ae/Mccourt.jpg",
          "width": 350,
          "height": 228,
          "mediaType": "image/jpg",
          "hash": [
            "2Drjgb5yoVWTpubcWmDBLJqkxrFkZamekzJoYLSWwM2ezpFkab"
          ]
        }
      ]
    }
  ],
  "published": "1970-01-01T00:00:00+00:00"
}

Video

Activity Vocabulary: Video

PropertyBase SpecRequiredDescriptionRestrictions
typeActivity Vocabulary 2.0YESIdentifies the type of the objectMUST be set to Video
urlActivity Vocabulary 2.0YESAn array of links for given video content in different formatsMUST be a Video Link AND MUST have at least one supported video MIME type
nameActivity Vocabulary 2.0noThe display name for the video
durationActivity Vocabulary 2.0noApproximate duration of the video
PropertyBase SpecRequiredDescriptionRestrictions
typeActivity Vocabulary 2.0YESIdentifies the type of the objectMUST be set to Link
hrefActivity Vocabulary 2.0YESThe URL for the given contentMUST be a Supported URL Schema
mediaTypeActivity Vocabulary 2.0YESMIME type of href content
hashDSNP 1.0YESArray of hashes for linked content validationMUST include at least one supported hash
heightActivity Vocabulary 2.0noA hint as to the rendering height in device-independent pixels
widthActivity Vocabulary 2.0noA hint as to the rendering width in device-independent pixels

Supported Video MIME Types

FormatMIME TypeSpecification(s)
MPEGvideo/mpegRFC2045
OGGvideo/oggRFC5334
WebMvideo/webmWebM standard
H265video/H265RFC7798
MP4video/mp4RFC4337

Example

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Note",
  "attachments": [
    {
      "type": "Video",
      "name": "One of the founders of DSNP",
      "duration": "PT10M32S",
      "url": [
        {
          "type": "Link",
          "href": "https://upload.wikimedia.org/wikipedia/commons/c/c0/Big_Buck_Bunny_4K.webm",
          "width": 4000,
          "height": 2250,
          "mediaType": "video/webm",
          "hash": [
            "2Drjgb4a8eC4XheBKCBcbAcaVdEWcKjMbCSZ2L2c9CQs4x98jf"
          ]
        }
      ]
    }
  ],
  "published": "1970-01-01T00:00:00+00:00"
}

Activity Vocabulary: Link

PropertyBase SpecRequiredDescriptionRestrictions
typeActivity Vocabulary 2.0YESIdentifies the type of the objectMUST be set to Link
hrefActivity Vocabulary 2.0YESThe URL for the given linkMUST be a Supported URL Schema
nameActivity Vocabulary 2.0noThe display name for the link

Examples

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Note",
  "content": "Check out the DSNP Website!",
  "mediaType": "text/plain",
  "attachment": [
    {
      "type": "Link",
      "name": "DSNP Website",
      "href": "https://dsnp.org"
    }
  ],
  "published": "1970-01-01T00:00:00+00:00"
}

Glossary

  • Activity Content - User-generated social networking content in a JSON format defined by the Activity Streams 2.0 specification with DSNP extensions. Some DSNP Announcements contain the URL and hash for an Activity Content document.
  • Announcement - Content, or references to content, that communicate user activity to the network.
  • Application - A computer program that helps a User interact with a DSNP system
  • Batch Publication - A collection of Announcements bundled together into a single file using the Parquet format
  • Consumer - An application or user who reads content from a DSNP system
  • DSNP System - A consensus system (and its surrounding services) that enables DSNP operations and generates DSNP state change records
  • Hash - A string of bytes generated from a hash function, a cryptographic function whose output is effectively unique for any given input without any information from the input being accessible in the output
  • User - A human using a DSNP application. See article for an exception