Merge pull request #351 from poseidon-framework/Schema_300_dev

Version 3.0.0 changes

Author: nevrome

CHANGELOG.md

@@ -1,3 +1,25 @@
+- V 1.7.0.0:
+    - Changes to .janno columns according to Poseidon v3.0.0:
+        - Replaced column `Source_Tissue`  with column `Source_Material`.
+        - New column `Individual_ID`.
+        - New column `Species`.
+        - New column `Alternative_IDs_Context` linked to `Alternative_IDs`.
+        - New column `Custodian_Institution`.
+        - New columns `Cultural_Era` + `Cultural_Era_URL` and `Archaeological_Culture` + `Archaeological_Culture_URL`.
+        - New column `Chromosomal_Anomalies`.
+        - Made column `Collection_ID` a list column.
+        - Soft-retired the option `ReferenceGenome` in the column `Capture_Type`.
+        - Added rescaling feature for the columns `Endogenous` and `Damage` for packages below Poseidon v3.0.0..
+        - Made column `Damage` a list column.
+        - Added the option `WISC2013` to the column `Capture_Type`.
+        - Changed the handling of `_Note` columns. Previously they were explicitly specified and part of the `JannoRow` record type. Now they are just treated as arbitrary additional columns that get algorithmically sorted in when writing .janno files (e.g. in `forge`). See `makeHeaderWithAdditionalColumns`.
+    - Changes to .ssf columns according to Poseidon v3.0.0:
+        - New column `submitted_md5`.
+    - Changes to trident to accomodate the schema changes:
+        - Added warnings for characters outside of the recommended range in `Poseidon_ID`s and `Group_Names`.
+        - Introduced smart .janno field construction based on the relevant Poseidon version. Smart here means that different checks and even minor data transformations are applied depending on the input version. The written output always adheres to the latest Poseidon version a given trident version supports. trident does not perform a comprehensive "upgrade" of old data, though, which would e.g. entail replacing .janno columns.
+        - Added command line arguments to set the expected Poseidon version for individual input files when no `POSEIDON.yml` file is available in `validate` and `jannocoalesce`: `--pvJanno`, `--pvSSF`, `--pvSource`, and `--pvTarget`.
+        - Made `-o,--outFile` mandatory in `jannocoalesce`, even when a `-t,--targetFile` is overwritten, to avoid any confusion which version is written (always the latest trident supports!).
 - V 1.6.9.1:
     - Fixed [a bug](https://github.com/poseidon-framework/poseidon-hs/issues/365) related to forge-names of packages with hyphens and numbers.
 - V 1.6.9.0:

CHANGELOGRELEASE.md

@@ -1,3 +1,42 @@
+### V 1.7.0.0
+
+This is a major release to add compatibility with Poseidon v3.0.0. It includes features to accomodate the new schema, and various other changes added since the last release V 1.6.7.3.
+
+#### `.janno`-related changes for Poseidon v3.0.0
+
+For the new schema release we modified the data structures that internally store `.janno`, `.ssf` and even `POSEIDON.yml` files. Please consult the [schema changelog](https://www.poseidon-adna.org/#/changelog) for the list of affected columns.
+
+To keep it possible to read older Poseidon packages we introduced "smart" `.janno` (and `.ssf`) field constructors based on the relevant Poseidon version. Smart here means that different checks and even minor data transformations are applied depending on the input version. This renders all valid input minimally compatible with Poseidon v3.0.0. trident does not perform a comprehensive "upgrade" of old data, though. That would also entail replacing outdated `.janno` columns like `Source_Tissue`. The most intrusive change that is actually implemented is a rescaling of the columns `Endogenous` and `Damage`, which are not stored as percentages any more in Poseidon v3.0.0. The output of trident, e.g. of `forge`, thus always adheres to the latest supported Poseidon version, but may carry along additional columns as free-text.
+
+Another possibly surprising change in this context concerns the handling of `_Note` columns in the `.janno` file. Poseidon v3.0.0 does not explcitly define individual `_Note` columns any more, so trident equally does not validate them. It instead treats them as unspecified, free-text columns. It does sort them, though, so that `_Note` columns are at least positioned sensibly when trident writes `.janno` files.
+
+#### Minor interface changes that emerged as a result
+
+Beyond these changes in the handling of `.janno` files, V 1.7.0.0 also comes with some minor changes in the trident CLI interface:
+
+1. The fact that we introduced smart, version-aware constructors when reading `.janno` and `.ssf` files has the consequence that the schema version must be known upon reading. We therefore added command line arguments for `validate` and `jannocoalesce` to set the expected Poseidon version when no `POSEIDON.yml` file is available: `--pvJanno`, `--pvSSF`, `--pvSource`, and `--pvTarget`. By default the latest supported schema version is assumed.
+2. As explained above trident now can read different Poseidon version more explicitly, but it can always only write data following the latest schema. To avoid any confusion we made `-o,--outFile` mandatory in `jannocoalesce`, even when a `-t,--targetFile` is overwritten. Otherwise `--pvTarget` may be confused for a way to set the output version number.
+3. Poseidon v3.0.0 recommends that `Poseidon_ID`s and `Group_Names` only include the ASCII characters "A-Za-z0-9_-.". trident now prints a warning if it encounters any characters outside of this recommended range in these fields.
+
+#### New features for archive maintenance
+
+trident is not only a CLI tool for personal data management, but also includes essential tooling for the maintenance and distribution of the public Poseidon archives. https://server.poseidon-adna.org is run by trident. In this context trident V 1.7.0.0 sports two new features:
+
+1. The `--archiveConfigFile`, so the archive specification YAML file of the server, can now include a `retiredPackagesFile` field to specify retired packages. Retired packages are by default ignored in the `/packages`, `/groups`, `/bibliography` and `/individuals` endpoints,of the web API, as well as ignored in the archive HTML page of the explorer. However, the `/zip_file` API endpoint still serves retired packages, so that they can be downloaded. The retired packages are still available in the per-package explorer HTML page. This feature allows us to retire outdated packages, e.g. in the [community-archive](https://github.com/poseidon-framework/community-archive/blob/master/archive.retired).
+2. `validate` now includes a mechanism to check for the presence and completeness of usually optional `.janno` and `.ssf` columns with `-j,--mandatoryJannoColumn` and `-s,--mandatorySSFColumn`. This feature will allow us to gradually make more fields mandatory in the public archives, beyond the three that are already required by the schema (`Poseidon_ID`, `Genetic_Sex`, `Group_Names`).
+
+#### Fixed a subtle bug in the forge language
+
+A user reported [an issue](https://github.com/poseidon-framework/poseidon-hs/issues/365) in the selection language parsing of `forge`, where package names with multiple hyphens and numbers caused the parsing to fail:
+
+```
+option --forgeString: Error when parsing the forge selection (either -f or --forgeFile):
+unexpected "-"
+expecting digit
+```
+
+We identified and fixed this bug.
+
 ### V 1.6.7.3
 
 This is a minor release with few changes in the behaviour of trident. It mainly includes internal alterations that allow for better error reporting. On the user side there are three notable changes:

poseidon-hs.cabal

@@ -1,5 +1,5 @@
 name:                poseidon-hs
-version:             1.6.9.1
+version:             1.7.0.0
 synopsis:            A package with tools for working with Poseidon genotype data
 description:         The tools in this package read and analyse Poseidon-formatted genotype databases, a modular system for storing genotype data from thousands of individuals.
 license:             MIT

src-executables/Main-trident.hs

@@ -273,7 +273,7 @@ serveOptParser = ServeOptions <$> parseArchiveConfig
 jannocoalesceOptParser :: OP.Parser JannoCoalesceOptions
 jannocoalesceOptParser = JannoCoalesceOptions <$> parseJannocoalSourceSpec
                                               <*> parseJannocoalTargetFile
-                                              <*> parseJannocoalOutSpec
+                                              <*> parseJannocoalOutFile
                                               <*> parseJannocoalJannoColumns
                                               <*> parseJannocoalOverride
                                               <*> parseJannocoalSourceKey

src/Poseidon/CLI/Forge.hs

@@ -4,6 +4,7 @@ module Poseidon.CLI.Forge where
 
 import           Poseidon.BibFile            (BibEntry (..), BibTeX,
                                               writeBibTeXFile)
+import           Poseidon.ColumnTypesJanno   (PoseidonID (..))
 import           Poseidon.ColumnTypesUtils   (ListColumn (..),
                                               getMaybeListColumn)
 import           Poseidon.EntityTypes        (EntityInput,
@@ -51,7 +52,7 @@ import           Control.Exception           (catch, throwIO)
 import           Control.Monad               (filterM, forM, forM_, unless,
                                               when)
 import           Data.List                   (intercalate, nub)
-import           Data.Maybe                  (mapMaybe)
+import           Data.Maybe                  (catMaybes, mapMaybe)
 import           Data.Time                   (getCurrentTime)
 import qualified Data.Vector                 as V
 import qualified Data.Vector.Unboxed         as VU
@@ -186,6 +187,7 @@ runForge (
             maybeSnpFile of
                 Nothing -> snpSetMergeList snpSetList intersect_
                 Just _  -> SNPSetOther
+    (newRefName, newRefUrl) <- fillMissingReferenceAssemblyInfo relevantPackages
     -- compile genotype data structure
     let gz = if outZip then "gz" else ""
     genotypeFileData <- case outFormat of
@@ -198,7 +200,7 @@ runForge (
                                    (outName <.> "bim" <.> gz)  Nothing
                                    (outName <.> "fam")  Nothing
             GenotypeOutFormatVCF        -> return $ GenotypeVCF (outName <.> "vcf" <.> gz) Nothing
-    let genotypeData = GenotypeDataSpec genotypeFileData (Just newSNPSet)
+    let genotypeData = GenotypeDataSpec genotypeFileData (Just newSNPSet) newRefName newRefUrl
 
     -- assemble and write result depending on outMode --
     logInfo "Creating new package entity"
@@ -329,9 +331,9 @@ sumNonMissingSNPs accumulator (_, geno) = do
 filterSeqSourceRows :: JannoRows -> SeqSourceRows -> SeqSourceRows
 filterSeqSourceRows (JannoRows jRows) (SeqSourceRows sRows) =
     let desiredPoseidonIDs = map jPoseidonID jRows
-    in SeqSourceRows $ filter (hasAPoseidonID desiredPoseidonIDs) sRows
+    in  SeqSourceRows $ filter (hasAPoseidonID desiredPoseidonIDs) sRows
     where
-        hasAPoseidonID :: [String] -> SeqSourceRow -> Bool
+        hasAPoseidonID :: [PoseidonID] -> SeqSourceRow -> Bool
         hasAPoseidonID jIDs seqSourceRow =
             let sIDs = getMaybeListColumn $ sPoseidonID seqSourceRow
             in any (`elem` jIDs) sIDs
@@ -351,3 +353,23 @@ fillMissingSnpSets packages = forM packages $ \pac -> do
             logWarning $ "Warning for package " ++ show pac_ ++ ": field \"snpSet\" \
                 \is not set. I will interpret this as \"snpSet: Other\""
             return SNPSetOther
+
+fillMissingReferenceAssemblyInfo :: [PoseidonPackage] -> PoseidonIO (Maybe String, Maybe String)
+fillMissingReferenceAssemblyInfo packages = do
+    let refNames = map (genotypeRefAssemblyName . posPacGenotypeData) packages
+        refUrls  = map (genotypeRefAssemblyName  . posPacGenotypeData) packages
+        uniqueRefNames = nub $ catMaybes refNames
+        uniqueRefUrls  = nub $ catMaybes refUrls
+    when (length uniqueRefNames > 1) $
+        logWarning $ "different reference genome assembly names given: " ++ show uniqueRefNames ++
+            ". I will pick the first for the forge output file"
+    when (length uniqueRefUrls > 1) $
+        logWarning $ "different reference genome assembly URLs given: " ++ show uniqueRefUrls ++
+            ". I will pick the first for the forge output file"
+    let finalRefName = case uniqueRefNames of
+            []    -> Nothing
+            (x:_) -> Just x
+        finalRefUrl = case uniqueRefUrls of
+            []    -> Nothing
+            (x:_) -> Just x
+    return (finalRefName, finalRefUrl)

src/Poseidon/CLI/Genoconvert.hs

@@ -157,7 +157,7 @@ convertGenoTo outFormat onlyGeno outPath removeOld outPlinkPopMode outZip pac =
                 GenotypeOutFormatPlink      -> return $
                     GenotypePlink      (outFilesRel !! 0) Nothing (outFilesRel !! 1) Nothing (outFilesRel !! 2) Nothing
                 GenotypeOutFormatVCF        -> return $ GenotypeVCF (outFilesRel !! 0) Nothing
-        let newGenotypeData = GenotypeDataSpec gFileSpec (genotypeSnpSet . posPacGenotypeData $ pac)
+        let newGenotypeData = GenotypeDataSpec gFileSpec (genotypeSnpSet . posPacGenotypeData $ pac) Nothing Nothing
             newPac = pac { posPacGenotypeData = newGenotypeData }
         logInfo $ "Adjusting POSEIDON.yml for " ++ show (posPacNameAndVersion pac)
         liftIO $ writePoseidonPackage newPac

src/Poseidon/CLI/Jannocoalesce.hs

@@ -3,31 +3,33 @@
 
 module Poseidon.CLI.Jannocoalesce where
 
-import           Poseidon.Janno         (JannoRow (..), JannoRows (..),
-                                         parseJannoRowFromNamedRecord,
-                                         readJannoFile, writeJannoFile)
-import           Poseidon.Package       (PackageReadOptions (..),
-                                         defaultPackageReadOptions,
-                                         getJointJanno,
-                                         readPoseidonPackageCollection)
-import           Poseidon.Utils         (PoseidonException (..), PoseidonIO,
-                                         logDebug, logInfo, logWarning)
+import           Poseidon.Janno           (JannoRow (..), JannoRows (..),
+                                           parseJannoRowFromNamedRecord,
+                                           readJannoFile, writeJannoFile)
+import           Poseidon.Package         (PackageReadOptions (..),
+                                           defaultPackageReadOptions,
+                                           getJointJanno,
+                                           readPoseidonPackageCollection)
+import           Poseidon.PoseidonVersion (VersionedFile (..),
+                                           latestPoseidonVersion)
+import           Poseidon.Utils           (PoseidonException (..), PoseidonIO,
+                                           logDebug, logInfo, logWarning)
 
-import           Control.Monad          (filterM, forM_, when)
-import           Control.Monad.Catch    (MonadThrow, throwM)
-import           Control.Monad.IO.Class (liftIO)
-import qualified Data.ByteString.Char8  as BSC
-import qualified Data.Csv               as Csv
-import qualified Data.HashMap.Strict    as HM
-import qualified Data.IORef             as R
-import           Data.List              ((\\))
-import           Data.Text              (pack, replace, unpack)
-import           System.Directory       (createDirectoryIfMissing)
-import           System.FilePath        (takeDirectory)
-import           Text.Regex.TDFA        ((=~))
+import           Control.Monad            (filterM, forM_, when)
+import           Control.Monad.Catch      (MonadThrow, throwM)
+import           Control.Monad.IO.Class   (liftIO)
+import qualified Data.ByteString.Char8    as BSC
+import qualified Data.Csv                 as Csv
+import qualified Data.HashMap.Strict      as HM
+import qualified Data.IORef               as R
+import           Data.List                ((\\))
+import           Data.Text                (pack, replace, unpack)
+import           System.Directory         (createDirectoryIfMissing)
+import           System.FilePath          (takeDirectory)
+import           Text.Regex.TDFA          ((=~))
 
 -- the source can be a single janno file, or a set of base directories as usual.
-data JannoSourceSpec = JannoSourceSingle FilePath | JannoSourceBaseDirs [FilePath]
+data JannoSourceSpec = JannoSourceSingle VersionedFile | JannoSourceBaseDirs [FilePath]
 
 data CoalesceJannoColumnSpec =
       AllJannoColumns
@@ -36,8 +38,8 @@ data CoalesceJannoColumnSpec =
 
 data JannoCoalesceOptions = JannoCoalesceOptions
     { _jannocoalesceSource           :: JannoSourceSpec
-    , _jannocoalesceTarget           :: FilePath
-    , _jannocoalesceOutSpec          :: Maybe FilePath -- Nothing means "in place"
+    , _jannocoalesceTarget           :: VersionedFile
+    , _jannocoalesceOutFile          :: FilePath
     , _jannocoalesceJannoColumns     :: CoalesceJannoColumnSpec
     , _jannocoalesceOverwriteColumns :: Bool
     , _jannocoalesceSourceKey        :: String -- by default set to "Poseidon_ID"
@@ -46,9 +48,9 @@ data JannoCoalesceOptions = JannoCoalesceOptions
     }
 
 runJannocoalesce :: JannoCoalesceOptions -> PoseidonIO ()
-runJannocoalesce (JannoCoalesceOptions sourceSpec target outSpec fields overwrite sKey tKey maybeStrip) = do
+runJannocoalesce (JannoCoalesceOptions sourceSpec (VersionedFile targetPV targetPath) outPath fields overwrite sKey tKey maybeStrip) = do
     JannoRows sourceRows <- case sourceSpec of
-        JannoSourceSingle sourceFile -> readJannoFile [] sourceFile
+        JannoSourceSingle (VersionedFile sourcePV sourcePath) -> readJannoFile sourcePV [] sourcePath
         JannoSourceBaseDirs sourceDirs -> do
             let pacReadOpts = defaultPackageReadOptions {
                       _readOptIgnoreChecksums      = True
@@ -57,11 +59,10 @@ runJannocoalesce (JannoCoalesceOptions sourceSpec target outSpec fields overwrit
                     , _readOptOnlyLatest           = True
                 }
             getJointJanno <$> readPoseidonPackageCollection pacReadOpts sourceDirs
-    JannoRows targetRows <- readJannoFile [] target
+    JannoRows targetRows <- readJannoFile targetPV [] targetPath
 
     newJanno <- makeNewJannoRows sourceRows targetRows fields overwrite sKey tKey maybeStrip
 
-    let outPath = maybe target id outSpec
     logInfo $ "Writing to file (directory will be created if missing): " ++ outPath
     liftIO $ do
         createDirectoryIfMissing True (takeDirectory outPath)
@@ -123,7 +124,7 @@ mergeRow cp targetRow sourceRow fields overwrite sKey tKey = do
         -- fill in the target row with dummy values for desired fields that might not be present yet
         targetComplete    = HM.union targetRowRecord (HM.fromList $ map (, BSC.empty) sourceKeysDesired)
         newRowRecord      = HM.mapWithKey fillFromSource targetComplete
-        parseResult       = Csv.runParser . parseJannoRowFromNamedRecord [] $ newRowRecord
+        parseResult       = Csv.runParser . parseJannoRowFromNamedRecord latestPoseidonVersion [] $ newRowRecord
     logInfo $ "matched target " ++ BSC.unpack (targetComplete  HM.! BSC.pack tKey) ++
               " with source "   ++ BSC.unpack (sourceRowRecord HM.! BSC.pack sKey)
     case parseResult of