NO JIRA. Misc fixes and refactoring (#23)

# Description of changes
This PR contains a lot of fixes, refactorings and a couple of new features:

## Fixes
* The documentations is made up-to-date with the actual funtionality and now is more similar in format to the other microservices in that it lists the exposed and consumed interfaces.
* In particular the record is no set straight on the version info JSON files, which are required.
* Validation of object import directories is now more strict:
   * version dirs must all have a corresonding version info JSON file which must be well-formed.
   * versions must be consecutive

## Refactorings
* To avoid confusion between the object-version-properties extension and the `vN.json` version info JSON files, names and docs have been updated to make the use more consistent.
* The object import directory validation is broken up into smaller methods for readability.


## New features
* It is now possible to archive an old (closed) layer. This feature is for now only useful for recovery scenario where some manual steps outside the service are likely to be necessary. The advantage of having at least the archiving be done through the service, is that the consistency check is automatically triggered and the dmftar command is executed with the exact same parameters and environment as when a new top layer is created.


## Dropped feature
* It was possible before to allow timestamps instead of version strings such as `v1` as version directory names. This feature was not used and has been dropped to reduce code complexity.

Author: janvanmansum

docs/dev.md

@@ -7,6 +7,11 @@ Local testing
 -------------
 Local testing uses the same [set-up]{:target=_blank} as other DANS microservices.
 
-[set-up]: https://dans-knaw.github.io/dans-module-archetype/common-practices/#debugging
+### Creating Object Import Directories
 
+If you want to create object import directories for testing, you can use the helper script
+`create-object-import-dir.py` from [dans-dev-scripts]{:target=_blank}.
+
+[set-up]: {{ local_testing_setup }}
+[dans-dev-scripts]: {{ dans_dev_scripts_url }}
 

docs/img/overview.graphml

@@ -5,65 +5,103 @@ Manages a DANS Data Vault Storage Root
 
 Purpose
 -------
-A DANS Data Vault Storage Root is an OCFL storage root that is used to store a collection of long term preservation objects.
+A DANS Data Vault Storage Root is an OCFL storage root used to store a collection of long-term preservation objects.
 
 Interfaces
 ----------
+This service has the following interfaces:
 
-### Batches and Object Import Directories
+![](img/overview.png){width="70%"}
 
-Objects versions to be stored must be placed under the inbox in a batch directory. The layout of the batch directory is as follows:
+### Provided interfaces
+
+#### Inbox
+
+* _Protocol type_: Shared filesystem
+* _Internal or external_: **internal**
+* _Purpose_: to receive [Object Import Directories](#object-import-directories)
+
+#### Command API
+
+* _Protocol type_: HTTP
+* _Internal or external_: **internal**
+* _Purpose_: to manage the service including starting imports
+
+#### Admin console
+
+* _Protocol type_: HTTP
+* _Internal or external_: **internal**
+* _Purpose_: application monitoring and management
+
+### Consumed interfaces
+
+#### DMFTAR (optional)
+
+* _Protocol type_: Local command invocation
+* _Internal or external_: **external**
+* _Purpose_: to create DMFTAR archives in the [SURF Data Archive]{:target=_blank}. This interface is optional because it is only used if the DMFTAR
+  archive provider has been configured. (The other archive providers use Java code to create the archives and do not require an external interface.)
+
+### Object Import Directories
+
+Objects versions to be imported must be placed under the inbox in a batch directory. The layout of the batch directory is as follows:
 
 ```plaintext
 batch-dir
  ├── urn:nbn:nl:ui:13-26febff0-4fd4-4ee7-8a96-b0703b96f812
  │   ├── v1
  │   │   └── <content files>
+ │   ├── v1.json
  │   ├── v2
  │   │   └── <content files>
  │   ├── v2.json
- │   └── v3
- │       └── <content files>
+ │   ├── v3
+ │   │   └── <content files>
+ │   └── v3.json
  ├── urn:nbn:nl:ui:13-2ced2354-3a9d-44b1-a594-107b3af99789
- │   └── v3
- │       └── <content files>
+ │   ├── v3
+ │   │   └── <content files>
+ │   └── v3.json
  └── urn:nbn:nl:ui:13-b7c0742f-a9b2-4c11-bffe-615dbe24c8a0
-      └── v1
-          └── <content files> 
+      ├── v1
+      │   └── <content files>
+      └── v1.json
 ```
 
 * `batch-dir` - The batch directory is the directory where the batch of objects to be imported is placed.
-* `urn:nbn:nl:ui:13-26febff0-4fd4-4ee7-8a96-b0703b96f812` - The directory name is the identifier of the object. The pattern that an
-  identifier must match can be configured in the configuration file.
-* `v1`, `v2`, `v3` - The version directories contain the content of the object version. The version directories must be named `v1`, `v2`, `v3`, etc.
-  When updating an existing object, the first version directory must be named after the next version to be created in the OCFL object.
-* The service can also be configured to accept timestamps as version directories. In that case, the version directories are expected to be numbers,
-  representing the timestamp of the version in milliseconds since the epoch. This timestamp is only used for ordering the versions in the OCFL object, so
-  any number can be used as long as it is unique for the object. This option is mainly used for testing purposes.
-* A version directory must be accompanied by a JSON file named `vN.json`, where `N` is the version number (e.g. `v2.json` for
-  version 2). This file is required for every version. It must have the following structure:
-  
-  ```json
-  {
-    "version-info": {
-      "user": {
-        "name": "John Doe",
-        "email": "john.doe@mail.com"
-      },
-      "message": "Commit message"
+* `urn:nbn:nl:ui:13-26febff0-4fd4-4ee7-8a96-b0703b96f812` - The directory name is the identifier of the object in the OCFL Storage Root. The pattern that an
+  identifier must match can be [configured]{:target=_blank}.
+* `v1`, `v2`, `v3` - The version directories contain the content of the object versions. The version directories must be named `v1`, `v2`, `v3`, etc.
+  The first version directory must be named after the next version to be created in the OCFL object.
+* A version directory must be accompanied by a version info JSON file named `vN.json`, where `N` is the version number (e.g., `v2.json` for
+  version 2). This version info JSON file is required for every version. It must have a structure as in the example below.
+
+[configured]: {{ config_file_url }}
+
+##### Example version info JSON file
+
+```json      
+{
+  "version-info": {
+    "user": {
+      "name": "John Doe",
+      "email": "john.doe@mail.com"
     },
-    "object-version-properties": {
-      "dataset-version": "1.2",
-      "packaging-format": "DANS RDA BagPack/1.0.0"
-    }
+    "message": "Commit message"
+  },
+  "object-version-properties": {
+    "dataset-version": "1.2",
+    "packaging-format": "DANS RDA BagPack/1.0.0"
   }
-  ```
-  
-  Requirements and notes:
-  - The `version-info` object is mandatory and must include `user.name`, `user.email`, and `message`.
-  - `version-info.user.email` may be specified with or without the `mailto:` prefix; the service will normalize it to `mailto:`.
-  - The `object-version-properties` object is optional and may contain any custom properties to be stored for the object version. These are written to the
-    Object Version Properties extension.
+}
+```
+
+Requirements and notes:
+
+- The `version-info` object is mandatory and must include `user.name`, `user.email`, and `message`.
+- `version-info.user.email` may be specified with or without the `mailto:` prefix; the service will normalize it to `mailto:`.
+- The `object-version-properties` object is optional and may contain any custom properties to be stored for the object version. These are written to the
+  [Object Version Properties]{:target=_blank} extension.
 
 [Object Version Properties]: {{ object_version_properties_ext }}
 

docs/img/overview.png

@@ -35,6 +35,9 @@ nav:
 
 extra:
   object_version_properties_ext: https://dans-knaw.github.io/dans-ocfl-extensions/object-version-properties/object-version-properties/
+  config_file_url: https://github.com/DANS-KNAW/dd-data-vault/blob/master/src/main/assembly/dist/cfg/config.yml
+  dans_dev_scripts_url: https://github.com/DANS-KNAW/dans-dev-scripts
+  local_testing_setup: https://dans-knaw.github.io/dans-datastation-architecture/dev-common-practices/#debugging
 
 plugins:
   - markdownextradata

docs/index.md

@@ -26,7 +26,7 @@
     </parent>
 
     <artifactId>dd-data-vault</artifactId>
-    <version>3.4.1-SNAPSHOT</version>
+    <version>4.0.0-SNAPSHOT</version>
 
     <name>DD Data Vault</name>
     <url>https://github.com/DANS-KNAW/dd-data-vault</url>
@@ -37,6 +37,9 @@
         <!-- TODO: move to dd-parent -->
         <commons-validator.version>1.7</commons-validator.version>
         <dans-ocfl-extensions.version>1.0.0</dans-ocfl-extensions.version>
+        <dans-ocfl-java-extensions-lib.version>1.1.0</dans-ocfl-java-extensions-lib.version>
+        <dans-layer-store-lib.version>1.1.0</dans-layer-store-lib.version>
+        <dd-data-vault-api.version>1.0.0</dd-data-vault-api.version>
         <main-class>nl.knaw.dans.datavault.DdDataVaultApplication</main-class>
     </properties>
 

mkdocs.yml

@@ -159,4 +159,4 @@ logging:
       # Used in combination with journald, which already adds the timestamp
       logFormat: "%-5p [%t] %c{0}: %m%n%dwREx"
   loggers:
-    'org.hibernate.engine.internal.StatisticalLoggingSessionEventListener': 'OFF'
+    'org.hibernate.engine.internal.StatisticalLoggingSessionEventListener': 'OFF'

pom.xml

@@ -54,9 +54,6 @@ public enum Status {
     @Column
     private boolean singleObject;
 
-    @Column
-    private boolean acceptTimestampVersionDirectories;
-
     @Column(nullable = false)
     private OffsetDateTime created;