Skip to content

libraries.md

Latest commit

 

History

History
329 lines (254 loc) · 11.9 KB

libraries.md

File metadata and controls

329 lines (254 loc) · 11.9 KB

Lantern - Libraries

Extensions to, or code closely associated with, third-party libraries relied on by this application.

Markdown

assets_tracking_service.lib.markdown

Markdown plain text plugin

A plugin based on https://github.com/kostyachum/python-markdown-plain-text is used to strip Markdown formatting from text for use in HTML titles for example.

Markdown automatic links plugin

A plugin based on https://github.com/daGrevis/mdx_linkify is used to convert inline URLs and email addresses in Markdown text into HTML links.

Markdown list formatting plugin

A plugin based on https://gitlab.com/ayblaq/prependnewline/ is used to automatically add additional line breaks to correctly paragraphs from lists in Markdown and ensure proper formatting.

BAS Metadata Library

Important

These are rough/working notes that will be written up properly when this module is extracted.

lantern.lib.metadata_library

Includes classes for Records.

These redesigned and refactored classes will replace core parts of the Metadata Library project.

Records

Records are a partial representation of the ISO 19115 information model implemented as a base data class (lantern.lib.metadata_library.models.record.Record). They generically describe resources (maps [products], datasets, collections, etc.).

Important

The Records model does not support all properties supported by the BAS ISO 19115 JSON Schema. See the Record Limitations section for more information.

Note

When encoded as XML records are interoperable with applications that support ISO 19139 encoded records.

Warning

When encoded as JSON, records are only interoperable with applications that support the BAS ISO 19115 JSON Schema.

The Record data class provides:

  • access to typed record properties
  • record configuration validation
  • filtering list based properties such as contacts, aggregations, etc.
  • loading and dumping record configurations (including from/to JSON documents)

Sub-properties are implemented as additional data classes (e.g. an Identification class). Code list properties are implemented using Enum classes.

Record validation

The Record data class includes a validate() method which will:

Records will be validated automatically when needed. Invalid records will raise a lantern.lib.metadata_library.models.record.RecordInvalidError exception.

Record limitations

Supported common elements (references not normative or exhaustive):

  • *.citation.title
  • *.citation.dates
  • *.citation.edition
  • *.citation.identifiers
  • *.citation.contacts (except contact.position)
  • *.citation.series (with local workaround for series.page until v5 schema)
  • *.constraints (limited restriction code list options)
  • *.maintenance
  • *.online resource (partial)

Supported elements (references not normative or exhaustive):

  • $schema
  • file_identifier
  • file_revision (non-ISO 19115 property, see RecordRevision)
  • hierarchy_level
  • metadata.character_set (hard-coded to 'utf8')
  • metadata.language (hard-coded to 'eng')
  • metadata.contacts (see *.citation.contacts)
  • metadata.constraints
  • metadata.maintenance
  • metadata.date_stamp
  • metadata.metadata_standard
  • reference_system_info
  • identification.title (via *.citation.title)
  • identification.dates (via *.citation.dates)
  • identification.edition (via *.citation.edition)
  • identification.identifiers (via *.citation.identifiers)
  • identification.contacts (except *.citation.contacts)
  • identification.abstract
  • identification.purpose
  • identification.other_citation_details
  • identification.supplemental_information
  • identification.aggregations
  • identification.extents (temporal and bounding box extents only)
  • identification.graphic_overviews
  • identification.spatial_resolution
  • identification.maintenance
  • identification.character_set (hard-coded to 'utf8')
  • identification.language (hard-coded to 'eng')
  • (identification.)data_quality.domain_consistency
  • (identification.)data_quality.lineage.statement
  • distribution.distributor
  • distribution.format (format and href only)
  • distribution.transfer_option

Unsupported elements (not normative or exhaustive):

  • *.contact.position
  • *.online_resource.protocol
  • (identification.)data_quality.lineage.process_step
  • (identification.)data_quality.lineage.sources
  • distribution.format (except name and URL)
  • identification.credit
  • identification.extent.geographic.identifier
  • identification.extent.vertical
  • identification.keywords
  • identification.resource_formats
  • identification.spatial_representation_type
  • identification.status
  • identification.topics

Record authoring

Records can be authored using any tool or system that can produce a valid record configuration. These may be created directly as JSON documents, or constructed as Record data class instances and then dumped to JSON.

Tip

For manual editing, consider an editor that supports JSON schemas for inline validation and enum auto-completion.

Within Python applications or scripts, consider using Record data classes for typed record properties, validation and easy serialisation to JSON.

Note

There is no formal guidance on what to include in record configurations. However, a starting point may be the Examples Records defined for the MAGIC Discovery ISO 19115 Profile.

Tip

See the Guide for how titles, summaries, abstracts and lineage statements can be formatted.

Record presets

If authoring Records using data classes, a set of presets in the lantern.lib.metadata_library.models.record.presets package are available to create common config subsets and improve consistency across records.

For example:

  • lantern.lib.metadata_library.models.record.presets.extents.make_bbox_extent:
    • simplifies creating a bounding box extent from a set of coordinates
  • lantern.lib.metadata_library.models.record.presets.constraints.OGL_V3:
    • provides a constant for the Open Government Licence

Tip

A larger scale present (lantern.lib.metadata_library.models.record.presets.base.RecordMagic) exists for creating typical MAGIC records, valid against the MAGIC Discovery (v2) and Administration (v1) profiles.

Record utilities

A set of utility functions in the lantern.lib.metadata_library.models.record.utils package are available to perform common or complex tasks.

Record key value data

To support properties that cannot be represented natively in the ISO 19115 information model, key value data can be encoded in a JSON string within the identifification.supplemental_information element of a Record.

Warning

The use of key values is non-standard and exclusive. If used, other content MUST NOT be included in the element.

Keys in this data are not controlled and must be accessed defensively.

Tip

The lantern.lib.metadata_library.models.record.utils.kv.get_kv and set_kv Utility Functions MAY be used to access and update key value data.

Record administrative metadata

The lantern.lib.metadata_library.models.record.utils.admin.get_admin and set_admin utility functions extend the Metadata Library methods to access and update MAGIC Administration metadata to work with Records.

Adding new Record properties

Warning

This section is Work in Progress (WIP) and may not be complete/accurate.

To add support for a new ISO element within Records:

  1. create a new data class for the new element in the relevant top module (i.e. identification.py)
  2. define enums for code lists if needed
  3. define a cattrs (un)structure hook if needed
  4. include the new class as a property in the relevant top-level class (i.e. Identification)
  5. register the cattrs (un)structure hook in the top-level class hooks if needed
  6. add tests for the new class testing all permutations, and cattrs hook if needed
  7. amend tests for top-level class (i.e. TestIdentification) variant:
    1. add variant for minimal instance of the new class if optional
    2. amend all variants with a minimal instance of the new class if required
    3. amend asserts to check new class as required
    4. amend tests for top-level cattrs hooks if changed
  8. if new class part of minimal record, update fx_record_config_minimal fixture
  9. amend tests for root-level class (i.e. TestRecord):
    1. amend tests for root-level cattrs hooks if top-level hooks changed (as an integration check)
    2. amend variants in test_loop as needed (include all possible options in complete variant)
  10. amend list of unsupported properties in /docs/data-model.md#record-limitations as needed

ArcGIS

assets_tracking_service.lib.arcgis

ArcGIS API for Python

The Esri ArcGIS API for Python package is not used in this project due to the number of dependencies it relies on (see Esri/arcgis-python-api#1692).

Instead, direct HTTP requests are made the relevant operations in the ArcGIS REST API.

ArcGIS vendored classes

A vendored subset of data classes and enums for representing content items are maintained in this package.

Warning

These classes have been modified from their original versions in some cases to remove unneeded functionality and to adjust types to comply with linting rules in this project.

ArcGIS item JSON properties

Supported item JSON properties:

  • id
  • owner
  • org_id
  • url (service endpoint)
  • access (sharing level)
  • title
  • type [enum]
  • snippet
  • description
  • accessInformation (attribution)
  • licenseInfo

Unsupported item JSON properties:

  • created
  • isOrgItem
  • modified
  • guid
  • name
  • typeKeywords
  • tags
  • thumbnail
  • documentation
  • extent
  • categories
  • spatialReference
  • classification
  • culture
  • properties
  • advancedSettings
  • proxyFilter
  • size
  • subInfo
  • appCategories
  • industries
  • languages
  • largeThumbnail
  • banner
  • screenshots
  • listed
  • ownerFolder
  • protected
  • commentsEnabled
  • numComments
  • numRatings
  • avgRating
  • cnumViews
  • itemControl
  • scoreCompleteness
  • groupDesignations
  • apiToken1ExpirationDate
  • apiToken2ExpirationDate
  • lastViewed

ArcGIS limitations

Warning

This section is Work in Progress (WIP) and may not be complete/accurate.

Note

Overall, this package is limited to functionality required by the esri-record and esri-item dev tasks only.

Partially supported features (not-exhaustive):

  • the metadataEditable property
    • not exposed as a property via these classes
    • used only in the _update_agol_item() method of the esri-item dev task (force set)

Known unsupported features (not-exhaustive):
⚡