Network Status Documents¶
Parsing for Tor network status documents. This supports both the v2 and v3 dir-spec. Documents can be obtained from a few sources…
The ‘cached-consensus’ file in Tor’s data directory.
Archived descriptors provided by CollecTor.
Directory authorities and mirrors via their DirPort.
… and contain the following sections…
document header
list of
stem.descriptor.router_status_entry.RouterStatusEntry
document footer
For a great graphical overview see Jordan Wright’s chart describing the anatomy of the consensus.
Of these, the router status entry section can be quite large (on the order of
hundreds of kilobytes). As such we provide a couple of methods for reading
network status documents through parse_file()
.
For more information see DocumentHandler()
…
from stem.descriptor import parse_file, DocumentHandler
with open('.tor/cached-consensus', 'rb') as consensus_file:
# Processes the routers as we read them in. The routers refer to a document
# with an unset 'routers' attribute.
for router in parse_file(consensus_file, 'network-status-consensus-3 1.0', document_handler = DocumentHandler.ENTRIES):
print router.nickname
Module Overview:
NetworkStatusDocument - Network status document
|- NetworkStatusDocumentV2 - Version 2 network status document
|- NetworkStatusDocumentV3 - Version 3 network status document
+- BridgeNetworkStatusDocument - Version 3 network status document for bridges
KeyCertificate - Certificate used to authenticate an authority
DocumentSignature - Signature of a document by a directory authority
DetachedSignature - Stand alone signature used when making the consensus
DirectoryAuthority - Directory authority as defined in a v3 network status document
-
class
stem.descriptor.networkstatus.
PackageVersion
[source]¶ Bases:
stem.descriptor.networkstatus.PackageVersion
Latest recommended version of a package that’s available.
- Variables
name (str) – name of the package
version (str) – latest recommended version
url (str) – package’s url
digests (dict) – mapping of digest types to their value
Bases:
stem.descriptor.networkstatus.SharedRandomnessCommitment
Directory authority’s commitment for generating the next shared random value.
- Variables
version (int) – shared randomness protocol version
algorithm (str) – hash algorithm used to make the commitment
identity (str) – authority’s sha1 identity fingerprint
commit (str) – base64 encoded commitment hash to the shared random value
reveal (str) – base64 encoded commitment to the shared random value, None of not provided
-
class
stem.descriptor.networkstatus.
DocumentDigest
[source]¶ Bases:
stem.descriptor.networkstatus.DocumentDigest
Digest of a consensus document.
New in version 1.8.0.
- Variables
flavor (str) – consensus type this digest is for (for example, ‘microdesc’)
algorithm (str) – hash algorithm used to make the digest
digest (str) – digest value of the consensus
-
class
stem.descriptor.networkstatus.
NetworkStatusDocument
(contents, lazy_load=False)[source]¶ Bases:
stem.descriptor.Descriptor
Common parent for network status documents.
-
digest
(hash_type='SHA1', encoding='HEX')[source]¶ Digest of this descriptor’s content. These are referenced by…
DetachedSignature
Referer:
DetachedSignature
consensus_digest attributeFormat: SHA1/HEX
New in version 1.8.0.
- Parameters
hash_type (stem.descriptor.DigestHash) – digest hashing algorithm
encoding (stem.descriptor.DigestEncoding) – digest encoding
- Returns
hashlib.HASH or str based on our encoding argument
-
-
class
stem.descriptor.networkstatus.
NetworkStatusDocumentV2
(raw_content, validate=False)[source]¶ Bases:
stem.descriptor.networkstatus.NetworkStatusDocument
Version 2 network status document. These have been deprecated and are no longer generated by Tor.
- Variables
routers (dict) – fingerprints to
RouterStatusEntryV2
contained in the documentversion (int) – * document version
hostname (str) – * hostname of the authority
address (str) – * authority’s IP address
dir_port (int) – * authority’s DirPort
fingerprint (str) – * authority’s fingerprint
contact (str) – * authority’s contact information
signing_key (str) – * authority’s public signing key
client_versions (list) – list of recommended client tor version strings
server_versions (list) – list of recommended server tor version strings
published (datetime) – * time when the document was published
options (list) – * list of things that this authority decides
signing_authority (str) – * name of the authority signing the document
signature (str) – * authority’s signature for the document
* attribute is either required when we’re parsed with validation or has a default value, others are left as None if undefined
-
TYPE_ANNOTATION_NAME
= 'network-status-2'¶
-
classmethod
content
(attr=None, exclude=(), sign=False)[source]¶ Creates descriptor content with the given attributes. Mandatory fields are filled with dummy information unless data is supplied. This doesn’t yet create a valid signature.
New in version 1.6.0.
- Parameters
attr (dict) – keyword/value mappings to be included in the descriptor
exclude (list) – mandatory keywords to exclude from the descriptor, this results in an invalid descriptor
sign (bool) – includes cryptographic signatures and digests if True
- Returns
str with the content of a descriptor
- Raises
ImportError if cryptography is unavailable and sign is True
NotImplementedError if not implemented for this descriptor type
-
class
stem.descriptor.networkstatus.
NetworkStatusDocumentV3
(raw_content, validate=False, default_params=True)[source]¶ Bases:
stem.descriptor.networkstatus.NetworkStatusDocument
Version 3 network status document. This could be either a vote or consensus.
- Variables
routers (dict) – fingerprint to
RouterStatusEntryV3
mapping for relays contained in the documentversion (int) – * document version
version_flavor (str) – * flavor associated with the document (such as ‘ns’ or ‘microdesc’)
is_consensus (bool) – * True if the document is a consensus
is_vote (bool) – * True if the document is a vote
is_microdescriptor (bool) – * True if this is a microdescriptor flavored document, False otherwise
valid_after (datetime) – * time when the consensus became valid
fresh_until (datetime) – * time when the next consensus should be produced
valid_until (datetime) – * time when this consensus becomes obsolete
vote_delay (int) – * number of seconds allowed for collecting votes from all authorities
dist_delay (int) – * number of seconds allowed for collecting signatures from all authorities
client_versions (list) – list of recommended client tor versions
server_versions (list) – list of recommended server tor versions
packages (list) – * list of
PackageVersion
entriesknown_flags (list) – * list of
Flag
for the router’s flagsparams (dict) – * dict of parameter(str) => value(int) mappings
directory_authorities (list) – * list of
DirectoryAuthority
objects that have generated this documentsignatures (list) – *
DocumentSignature
of the authorities that have signed the document
Consensus Attributes:
- Variables
consensus_method (int) – method version used to generate this consensus
bandwidth_weights (dict) – dict of weight(str) => value(int) mappings
shared_randomness_current_reveal_count (int) – number of commitments used to generate the current shared random value
shared_randomness_current_value (str) – base64 encoded current shared random value
shared_randomness_previous_reveal_count (int) – number of commitments used to generate the last shared random value
shared_randomness_previous_value (str) – base64 encoded last shared random value
Vote Attributes:
- Variables
consensus_methods (list) – list of ints for the supported method versions
published (datetime) – time when the document was published
flag_thresholds (dict) – * mapping of internal performance thresholds used while making the vote, values are ints or floats
recommended_client_protocols (dict) – recommended protocols for clients
recommended_relay_protocols (dict) – recommended protocols for relays
required_client_protocols (dict) – required protocols for clients
required_relay_protocols (dict) – required protocols for relays
bandwidth_file_headers (dict) – headers from the bandwidth authority that generated this vote
bandwidth_file_digest (dict) – hashes of the bandwidth authority file used to generate this vote, this is a mapping of hash functions to their resulting digest value
* attribute is either required when we’re parsed with validation or has a default value, others are left as None if undefined
Changed in version 1.4.0: Added the packages attribute.
Changed in version 1.5.0: Added the is_shared_randomness_participate, shared_randomness_commitments, shared_randomness_previous_reveal_count, shared_randomness_previous_value, shared_randomness_current_reveal_count, and shared_randomness_current_value attributes.
Changed in version 1.6.0: Added the recommended_client_protocols, recommended_relay_protocols, required_client_protocols, and required_relay_protocols attributes.
Changed in version 1.6.0: The is_shared_randomness_participate and shared_randomness_commitments were misdocumented in the tor spec and as such never set. They’re now an attribute of votes in the directory_authorities.
Changed in version 1.7.0: The shared_randomness_current_reveal_count and shared_randomness_previous_reveal_count attributes were undocumented and not provided properly if retrieved before their shred_randomness_*_value counterpart.
Changed in version 1.7.0: Added the bandwidth_file_headers attributbute.
Changed in version 1.8.0: Added the bandwidth_file_digest attributbute.
-
classmethod
content
(attr=None, exclude=(), sign=False, authorities=None, routers=None)[source]¶ Creates descriptor content with the given attributes. Mandatory fields are filled with dummy information unless data is supplied. This doesn’t yet create a valid signature.
New in version 1.6.0.
- Parameters
attr (dict) – keyword/value mappings to be included in the descriptor
exclude (list) – mandatory keywords to exclude from the descriptor, this results in an invalid descriptor
sign (bool) – includes cryptographic signatures and digests if True
- Returns
str with the content of a descriptor
- Raises
ImportError if cryptography is unavailable and sign is True
NotImplementedError if not implemented for this descriptor type
-
classmethod
create
(attr=None, exclude=(), validate=True, sign=False, authorities=None, routers=None)[source]¶ Creates a descriptor with the given attributes. Mandatory fields are filled with dummy information unless data is supplied. This doesn’t yet create a valid signature.
New in version 1.6.0.
- Parameters
attr (dict) – keyword/value mappings to be included in the descriptor
exclude (list) – mandatory keywords to exclude from the descriptor, this results in an invalid descriptor
validate (bool) – checks the validity of the descriptor’s content if True, skips these checks otherwise
sign (bool) – includes cryptographic signatures and digests if True
- Returns
Descriptor
subclass- Raises
ValueError if the contents is malformed and validate is True
ImportError if cryptography is unavailable and sign is True
NotImplementedError if not implemented for this descriptor type
-
type_annotation
()[source]¶ Provides the Tor metrics annotation of this descriptor type. For example, “@type server-descriptor 1.0” for server descriptors.
Please note that the version number component is specific to CollecTor, and for the moment hardcode as 1.0. This may change in the future.
New in version 1.8.0.
- Returns
TypeAnnotation
with our type information
-
is_valid
()[source]¶ Checks if the current time is between this document’s valid_after and valid_until timestamps. To be valid means the information within this document reflects the current network state.
New in version 1.8.0.
- Returns
True if this consensus is presently valid and False otherwise
-
is_fresh
()[source]¶ Checks if the current time is between this document’s valid_after and fresh_until timestamps. To be fresh means this should be the latest consensus.
New in version 1.8.0.
- Returns
True if this consensus is presently fresh and False otherwise
-
validate_signatures
(key_certs)[source]¶ Validates we’re properly signed by the signing certificates.
New in version 1.6.0.
- Parameters
key_certs (list) –
KeyCertificates
to validate the consensus against- Raises
ValueError if an insufficient number of valid signatures are present.
-
get_unrecognized_lines
()[source]¶ Provides a list of lines that were either ignored or had data that we did not know how to process. This is most common due to new descriptor fields that this library does not yet know how to process. Patches welcome!
- Returns
list of lines of unrecognized content
-
meets_consensus_method
(method)[source]¶ Checks if we meet the given consensus-method. This works for both votes and consensuses, checking our ‘consensus-method’ and ‘consensus-methods’ entries.
- Parameters
method (int) – consensus-method to check for
- Returns
True if we meet the given consensus-method, and False otherwise
-
class
stem.descriptor.networkstatus.
DirectoryAuthority
(raw_content, validate=False, is_vote=False)[source]¶ Bases:
stem.descriptor.Descriptor
Directory authority information obtained from a v3 network status document.
Authorities can optionally use a legacy format. These are no longer found in practice, but have the following differences…
The authority’s nickname ends with ‘-legacy’.
There’s no contact or vote_digest attribute.
- Variables
nickname (str) – * authority’s nickname
v3ident (str) – * identity key fingerprint used to sign votes and consensus
hostname (str) – * hostname of the authority
address (str) – * authority’s IP address
dir_port (int) – * authority’s DirPort
or_port (int) – * authority’s ORPort
is_legacy (bool) – * if the authority’s using the legacy format
contact (str) – contact information, this is included if is_legacy is False
Consensus Attributes:
- Variables
vote_digest (str) – digest of the authority that contributed to the consensus, this is included if is_legacy is False
Vote Attributes:
- Variables
legacy_dir_key (str) – fingerprint of and obsolete identity key
key_certificate (stem.descriptor.networkstatus.KeyCertificate) – * authority’s key certificate
is_shared_randomness_participate (bool) – * True if this authority participates in establishing a shared random value, False otherwise
shared_randomness_commitments (list) – * list of
SharedRandomnessCommitment
entriesshared_randomness_previous_reveal_count (int) – number of commitments used to generate the last shared random value
shared_randomness_previous_value (str) – base64 encoded last shared random value
shared_randomness_current_reveal_count (int) – number of commitments used to generate the current shared random value
shared_randomness_current_value (str) – base64 encoded current shared random value
* mandatory attribute
Changed in version 1.4.0: Renamed our ‘fingerprint’ attribute to ‘v3ident’ (prior attribute exists for backward compatability, but is deprecated).
Changed in version 1.6.0: Added the is_shared_randomness_participate, shared_randomness_commitments, shared_randomness_previous_reveal_count, shared_randomness_previous_value, shared_randomness_current_reveal_count, and shared_randomness_current_value attributes.
-
classmethod
content
(attr=None, exclude=(), sign=False, is_vote=False)[source]¶ Creates descriptor content with the given attributes. Mandatory fields are filled with dummy information unless data is supplied. This doesn’t yet create a valid signature.
New in version 1.6.0.
- Parameters
attr (dict) – keyword/value mappings to be included in the descriptor
exclude (list) – mandatory keywords to exclude from the descriptor, this results in an invalid descriptor
sign (bool) – includes cryptographic signatures and digests if True
- Returns
str with the content of a descriptor
- Raises
ImportError if cryptography is unavailable and sign is True
NotImplementedError if not implemented for this descriptor type
-
classmethod
create
(attr=None, exclude=(), validate=True, sign=False, is_vote=False)[source]¶ Creates a descriptor with the given attributes. Mandatory fields are filled with dummy information unless data is supplied. This doesn’t yet create a valid signature.
New in version 1.6.0.
- Parameters
attr (dict) – keyword/value mappings to be included in the descriptor
exclude (list) – mandatory keywords to exclude from the descriptor, this results in an invalid descriptor
validate (bool) – checks the validity of the descriptor’s content if True, skips these checks otherwise
sign (bool) – includes cryptographic signatures and digests if True
- Returns
Descriptor
subclass- Raises
ValueError if the contents is malformed and validate is True
ImportError if cryptography is unavailable and sign is True
NotImplementedError if not implemented for this descriptor type
-
class
stem.descriptor.networkstatus.
KeyCertificate
(raw_content, validate=False)[source]¶ Bases:
stem.descriptor.Descriptor
Directory key certificate for a v3 network status document.
- Variables
version (int) – * version of the key certificate
address (str) – authority’s IP address
dir_port (int) – authority’s DirPort
fingerprint (str) – * authority’s fingerprint
identity_key (str) – * long term authority identity key
published (datetime) – * time when this key was generated
expires (datetime) – * time after which this key becomes invalid
signing_key (str) – * directory server’s public signing key
crosscert (str) – signature made using certificate’s signing key
certification (str) – * signature of this key certificate signed with the identity key
* mandatory attribute
-
TYPE_ANNOTATION_NAME
= 'dir-key-certificate-3'¶
-
classmethod
content
(attr=None, exclude=(), sign=False)[source]¶ Creates descriptor content with the given attributes. Mandatory fields are filled with dummy information unless data is supplied. This doesn’t yet create a valid signature.
New in version 1.6.0.
- Parameters
attr (dict) – keyword/value mappings to be included in the descriptor
exclude (list) – mandatory keywords to exclude from the descriptor, this results in an invalid descriptor
sign (bool) – includes cryptographic signatures and digests if True
- Returns
str with the content of a descriptor
- Raises
ImportError if cryptography is unavailable and sign is True
NotImplementedError if not implemented for this descriptor type
-
class
stem.descriptor.networkstatus.
DocumentSignature
(method, identity, key_digest, signature, flavor=None, validate=False)[source]¶ Bases:
object
Directory signature of a v3 network status document.
- Variables
method (str) – algorithm used to make the signature
identity (str) – fingerprint of the authority that made the signature
key_digest (str) – digest of the signing key
signature (str) – document signature
flavor (str) – consensus type this signature is for (such as ‘microdesc’), None if for the standard consensus
- Parameters
validate (bool) – checks validity if True
- Raises
ValueError if a validity check fails
-
class
stem.descriptor.networkstatus.
DetachedSignature
(raw_content, validate=False)[source]¶ Bases:
stem.descriptor.Descriptor
Stand alone signature of the consensus. These are exchanged between directory authorities when determining the next hour’s consensus.
Detached signatures are defined in section 3.10 of the dir-spec, and only available to be downloaded for five minutes between minute 55 until the end of the hour.
New in version 1.8.0.
- Variables
consensus_digest (str) – * digest of the consensus being signed
valid_after (datetime) – * time when the consensus became valid
fresh_until (datetime) – * time when the next consensus should be produced
valid_until (datetime) – * time when this consensus becomes obsolete
additional_digests (list) – *
DocumentDigest
for additional consensus flavorsadditional_signatures (list) – *
DocumentSignature
for additional consensus flavorssignatures (list) – *
DocumentSignature
of the authorities that have signed the document
* mandatory attribute
-
TYPE_ANNOTATION_NAME
= 'detached-signature-3'¶
-
classmethod
content
(attr=None, exclude=(), sign=False)[source]¶ Creates descriptor content with the given attributes. Mandatory fields are filled with dummy information unless data is supplied. This doesn’t yet create a valid signature.
New in version 1.6.0.
- Parameters
attr (dict) – keyword/value mappings to be included in the descriptor
exclude (list) – mandatory keywords to exclude from the descriptor, this results in an invalid descriptor
sign (bool) – includes cryptographic signatures and digests if True
- Returns
str with the content of a descriptor
- Raises
ImportError if cryptography is unavailable and sign is True
NotImplementedError if not implemented for this descriptor type
-
class
stem.descriptor.networkstatus.
BridgeNetworkStatusDocument
(raw_content, validate=False)[source]¶ Bases:
stem.descriptor.networkstatus.NetworkStatusDocument
Network status document containing bridges. This is only available through the metrics site.
- Variables
routers (dict) – fingerprint to
RouterStatusEntryV3
mapping for relays contained in the documentpublished (datetime) – time when the document was published
-
TYPE_ANNOTATION_NAME
= 'bridge-network-status'¶