cpu & gpu versions for most taggers & pie model update

Author: PrinsINT

.github/workflows/dev-to-docker.yml

@@ -20,6 +20,6 @@ jobs:
           password: ${{ secrets.DOCKERHUB_TOKEN }}
 
       - name: Build and push 
-        run: ./buildandpushall.sh
+        run: ./push-and-build.sh
         env:
           VERSION: dev

.github/workflows/prod-to-docker.yml

@@ -21,6 +21,6 @@ jobs:
           password: ${{ secrets.DOCKERHUB_TOKEN }}
 
       - name: Build and push 
-        run: ./buildandpushall.sh
+        run: ./push-and-build.sh
         env:
-          VERSION: latest
+          VERSION: $GITHUB_REF_NAME

TaggerAPITest.py

@@ -1,4 +1,4 @@
-FROM python:3.13.2-slim-bookworm
+FROM python:3.12.9-slim-bookworm
 
 # Copy python source and requirements.
 COPY . ./

base/Dockerfile

@@ -1,5 +1,5 @@
 """
-Initialize the tagger if needed and process input files by calling the specific tagger implementation 
+Initialize the tagger if needed and process input files by calling the specific tagger implementation
 and ensuring the output is written to the expected file.
 """
 

base/process.py

@@ -1,12 +1,12 @@
 """
 Acts as a daemon that checks on an interval if there are any pending tasks (i.e. documents) to be processed.
-It then processes them one by one (i.e. running the tagger process), keeps track of their statusses, 
+It then processes them one by one (i.e. running the tagger process), keeps track of their statusses,
 and sends the results to the callback server. The server then responds with KEEP or DELETE,
 which determines if the resulting tagged output file is kept or deleted.
 Input files are deleted automatically after processing, or moved to the error folder if processing fails.
 
 We use a multiprocessing pool to process files, because we want to kill the process if needed.
-Additonally the taggers need to be initialized only once (and in the same thread), 
+Additonally the taggers need to be initialized only once (and in the same thread),
 so that can be done at pool initialization.
 """
 

base/tagger_worker.py

@@ -5,7 +5,7 @@
 File are processed automatically once uploaded, with the result being sent back to the callback server.
 Input files are deleted automatically after being processed.
 
-Deleting files also stops the tagger if that file was being processed. 
+Deleting files also stops the tagger if that file was being processed.
 (Thus, deleting all input files is equivalent to stopping the tagger.)
 """
 

base/webservice.py

@@ -0,0 +1,41 @@
+# load the .env file
+source .env
+
+# Set the default label
+: ${VERSION:=latest}
+
+echo "Will build taggers with version <$VERSION>. Set VERSION to override this."
+
+# Base image
+docker build -t instituutnederlandsetaal/galahad-taggers-base:$VERSION base
+
+# PIE
+# base
+docker build --build-arg VERSION=$VERSION --build-arg CPU_GPU=cpu -t instituutnederlandsetaal/galahad-taggers-pie-cpu-base:$VERSION pie/base
+docker build --build-arg VERSION=$VERSION --build-arg CPU_GPU=gpu -t instituutnederlandsetaal/galahad-taggers-pie-gpu-base:$VERSION pie/base
+# all
+docker build --build-arg VERSION=$VERSION --build-arg CPU_GPU=cpu -t instituutnederlandsetaal/galahad-taggers-pie-cpu-tdn-all:$VERSION pie/TDN-ALL
+docker build --build-arg VERSION=$VERSION --build-arg CPU_GPU=gpu -t instituutnederlandsetaal/galahad-taggers-pie-gpu-tdn-all:$VERSION pie/TDN-ALL
+# tdn-1200-1600
+docker build --build-arg VERSION=$VERSION --build-arg CPU_GPU=cpu -t instituutnederlandsetaal/galahad-taggers-pie-cpu-tdn-1200-1600:$VERSION pie/TDN-1200-1600
+docker build --build-arg VERSION=$VERSION --build-arg CPU_GPU=gpu -t instituutnederlandsetaal/galahad-taggers-pie-gpu-tdn-1200-1600:$VERSION pie/TDN-1200-1600
+# tdn-1600-1900
+docker build --build-arg VERSION=$VERSION --build-arg CPU_GPU=cpu -t instituutnederlandsetaal/galahad-taggers-pie-cpu-tdn-1600-1900:$VERSION pie/TDN-1600-1900
+docker build --build-arg VERSION=$VERSION --build-arg CPU_GPU=gpu -t instituutnederlandsetaal/galahad-taggers-pie-gpu-tdn-1600-1900:$VERSION pie/TDN-1600-1900
+
+# UD-parsers
+# flair
+docker build --build-arg VERSION=$VERSION --build-arg CPU_GPU=cpu -t instituutnederlandsetaal/galahad-taggers-flair-cpu:$VERSION flair
+docker build --build-arg VERSION=$VERSION --build-arg CPU_GPU=gpu -t instituutnederlandsetaal/galahad-taggers-flair-gpu:$VERSION flair
+# # spacy
+#docker build --build-arg VERSION=$VERSION --build-arg SPACY_MODEL=nl_core_news_lg --build-arg CPU_GPU=cpu -t instituutnederlandsetaal/galahad-taggers-spacy-cpu:$VERSION spacy
+# docker build --build-arg VERSION=$VERSION --build-arg SPACY_MODEL=nl_core_news_lg --build-arg CPU_GPU=gpu -t instituutnederlandsetaal/galahad-taggers-spacy-gpu:$VERSION spacy
+# stanza
+docker build --build-arg VERSION=$VERSION --build-arg CPU_GPU=cpu -t instituutnederlandsetaal/galahad-taggers-stanza-cpu:$VERSION stanza
+docker build --build-arg VERSION=$VERSION --build-arg CPU_GPU=gpu -t instituutnederlandsetaal/galahad-taggers-stanza-gpu:$VERSION stanza
+# udpipe
+# docker build --build-arg VERSION=$VERSION -t instituutnederlandsetaal/galahad-taggers-udpipe:$VERSION udpipe
+# docker build --build-arg VERSION=$VERSION -t instituutnederlandsetaal/galahad-taggers-stanza:$VERSION stanza
+
+# Huggingface
+# TODO