exported tei xml dataset questions

[jameshowison]

We're parsing the tei xml file and ran into a minor issue. @Kermit2 any thoughts?

The rs tag on line 9804 seems to have both an id and an xml:id while the others don't?

<rs id="10.1257%2Faer.20150592-software-0" resp="#curator" type="software" xml:id="10.1257%2Faer.20150592-software-simple-0">Google Scholar</rs>

[kermitt2]

Sorry clearly an error! Probably something I didn't clean correctly.
The attribute @id should have been deleted here.
I don't see another occurrence of an @id attribute, but if there are more, they should be removed too.

[jameshowison]

Ok, best to just do this manually in the xml file? (which we've added to the softcite-dataset repo, btw).

…

On Fri, Apr 10, 2020 at 2:08 PM Patrice Lopez ***@***.***> wrote: Sorry clearly an error! Probably something I didn't clean correctly. The attribute @id should have been deleted here. I don't see another occurrence of an @id attribute, but if there are more, they should be removed too. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#657 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAWOUVRCC35J3X3UHUOD53RL5VCVANCNFSM4MFUQ7MA> .

[caifand]

If the source file changes I can link it back to here.

[kermitt2]

I've updated in the original repo softcite/software-mentions@eb683b6

[caifand]

Spotted some rs tags without attribute publisher but creator. Another edit is needed :) @kermitt2

id,publisher,creator
a2008-39-NAT_BIOTECHNOL-software-9,NA,Molecular Devices
a2008-39-NAT_BIOTECHNOL-software-5,NA,Chang Bioscience
a2008-39-NAT_BIOTECHNOL-software-0,NA,National Institutes of Health
a2008-39-NAT_BIOTECHNOL-software-7,NA,Chang Bioscience
a2008-39-NAT_BIOTECHNOL-software-8,NA,Chang Bioscience
a2007-48-UNDERSEA_HYPERBAR_M-software-0,NA,Microsoft Corp 

[caifand]

Sorry, there is still another one:
at line 13505:

<rs resp="#curator" type="software" xml:id="PMC3481138-software-1">Pictar</rs> (<rs corresp="#PMC3481138-software-0" resp="#curator" type="url">pictar.mdc-berlin.de</rs>))

Here the corresp attribute of the url tag does not match the xml:id of the linked software tag.

Those are a bit trivial, thanks for fixing them! @kermitt2

[kermitt2]

Good catch @caifand !
This is corrected with softcite/software-mentions@ebf5766

[caifand]

Some mismatched fields in rs tags:
xml:id,software,version,publisher,url,description
PMC4475901-software-2,MASCOT,NA,Matrix Science Ltd,NA,publisher rs tag is associated with wrong xml:id
PMC4938145-software-0,Nanoscope,6.13R1,Digital Instruments,NA,publisher rs tag is associated with wrong xml:id
PMC3123123-software-0,R,2.12.0,NA,NA,version rs tag is associated with wrong xml:id
PMC4589266-software-2,TBSS,1.2,http://fsl.fmrib.ox.ac.uk/ fsl/fslwiki/TBSS/UserGuide,url rs tag type="version"
PMC5018359-software-1,Stata,12,StataCorp,NA,repeated xml:id in the same para

@kermitt2 Thanks!

[kermitt2]

Thank you, good catch !

I have updated the reviewed TEI export -> softcite/software-mentions@bdbb49f

Sorry for this slow correction, long and lot's of internet downtime in the last days.

[caifand]

@kermitt2 Thanks! These are indeed small pieces of things. I am happy to fix those if it won't break the file.

[kermitt2]

sure I've invited you on the ourresearch/software-mentions so that you can make corrections directly on the file, thank you!

[caifand]

Hi @kermitt2 , I found that in all_clean_post_processed_with_no_mention.tei.xml, the TEI element marking up article titles has two types: </titleStatement> & </titleStmt>. It seems to me that all the articles with mentions have </titleStatement> while all articles with zero mentions have </titleStmt>.

In all_clean_post_processed.tei.xml, there are all </titleStatement>, other than the top block including the whole corpus metadata (i.e., there is just </titleStmt> in the very first </fileDesc> block.)

@kermitt2 Should we seek to uniformize all of them? If it is just a legacy thing and does not involve changes in the pipeline, I can help with the manual edits.

[kermitt2]

Thank you @caifand, good catch !
It should be <titleStmt> everywhere indeed. I corrected it at some point in the program that generates the XML, but it was still incorrect in the file I edited for the manual corrections, so still propagating to the final version.
This is fixed with commit softcite/software-mentions@da7f761

[caifand]

I've found annotated mentions from early-stage training articles in csv files. Though training articles are marked in softcite_articles.csv, there are still some not in softcite_articles.csv because their RDF annotations have different schemas from our established coding scheme. But their detailed annotations went into other csv files. I suspect those different schemas were used in early training when our coding scheme hadn't been stabilized yet?  @jameshowison Just checking with you to see if I am correct.

I've removed all the annotations from training articles from all the csv files now. So, @kermitt2 , sorry that I have to ask for another corpus run :)

(I did this check since I've found there are still some training article annotations in TEI file)

[kermitt2]

I've regenerated the tei corpus with the updated csv:
https://github.com/ourresearch/software-mentions/blob/master/resources/dataset/software/corpus/softcite_corpus.tei.xml

These articles have disappeared:

• a2008-39-NAT_BIOTECHNOL
• a2010-05-BMC_MOL_BIOL
• a2007-48-UNDERSEA_HYPERBAR_M
• a2001-40-MOL_ECOL
• a2008-02-WATERBIRDS