Bug 36147 - Load Austlang dataset from source to make available as authorised values
Summary: Load Austlang dataset from source to make available as authorised values
Status: RESOLVED MOVED
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Aleisha Amohia
QA Contact: Testopia
URL: https://gitlab.com/aleishaamohia/koha...
Keywords:
Depends on:
Blocks:
 
Reported: 2024-02-22 04:16 UTC by Aleisha Amohia
Modified: 2024-02-26 22:27 UTC (History)
4 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 36147: [WIP] Load Austlang dataset from source to make available as authorised values (5.11 KB, patch)
2024-02-22 04:22 UTC, Aleisha Amohia
Details | Diff | Splinter Review
Bug 36147: Script to load Austlang dataset from source to make available as authorised values (9.81 KB, patch)
2024-02-23 00:36 UTC, Aleisha Amohia
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Aleisha Amohia 2024-02-22 04:16:46 UTC
Austlang provides information about Indigenous Australian languages which has been assembled from referenced sources.

The dataset provided at https://collection.aiatsis.gov.au/datasets/austlang/001 includes the language names, each with a unique alpha-numeric code which functions as a stable identifier, alternative/variant names and spellings and the approximate location of each language variety.

This is a live dataset and is updated regularly.

It would be great to have a cronjob that regularly pulled this dataset to keep an authorised value category of AUSTLANG updated with Indigenous Australian languages
Comment 1 Aleisha Amohia 2024-02-22 04:22:37 UTC
Created attachment 162330 [details] [review]
Bug 36147: [WIP] Load Austlang dataset from source to make available as authorised values
Comment 2 Aleisha Amohia 2024-02-23 00:36:39 UTC
Created attachment 162354 [details] [review]
Bug 36147: Script to load Austlang dataset from source to make available as authorised values

This enhancement adds a new cronjob that can be scheduled to regularly load Indigenous Australian languages (Austlang) data from an external source, process it, then create or update authorised values of this data.

This enhancement also adds an AUSTLANG authorised value category which can be used for the Austlang data.

To test:

1) Check the documentation for the script is clear and accurate
  perldoc misc/cronjobs/download_austlang_for_authvalues.pl
2) Run the script with the verbose flag to see an error. You should see the error "No AUSTLANG authorised value category found."
  ./misc/cronjobs/download_austlang_for_authvalues.pl -v
3) Install the database update and confirm in Koha Administration -> Authorized values that the AUSTLANG category exists, with no values in it
4) Run the script with a fake URL to see the error "Error accessing data."
  ./misc/cronjobs/download_austlang_for_authvalues.pl -v --url 'bad-url'
5) Run the script normally. When you refresh the AUSTLANG authorised values page, there should be languages loaded as authorised values.
  ./misc/cronjobs/download_austlang_for_authvalues.pl -v [--category AUSTLANG]
6) Run the script again to confirm it skips matches by default, i.e. a message is printed "Authorised value match found, not updating: Y80"
  ./misc/cronjobs/download_austlang_for_authvalues.pl -v [--category AUSTLANG]
7) Run the script with the --update flag so it updates matches. Confirm a count of updated values is printed at the end.
  ./misc/cronjobs/download_austlang_for_authvalues.pl -v --update [--category AUSTLANG]

Sponsored-by: Education Services Australia SCIS
Comment 3 Martin Renvoize 2024-02-23 13:45:39 UTC
This should really be a plugin in my opinion. I don't see why it should be another script added to Core.

There is a cron hook that could be used perfectly for this I believe.
Comment 4 David Cook 2024-02-25 22:15:41 UTC
(In reply to Martin Renvoize from comment #3)
> This should really be a plugin in my opinion. I don't see why it should be
> another script added to Core.
> 
> There is a cron hook that could be used perfectly for this I believe.

+1
Comment 5 Martin Renvoize 2024-02-26 10:02:26 UTC
I will add that I'm really looking forward to having a more public way of discovering plugins.. that's one of my big goals at Hackfest.. so this isn't a 'No, don't do this'.. it's "lets do this is a more modular/manageable way" :)
Comment 6 Aleisha Amohia 2024-02-26 10:11:46 UTC
Totally hear you both, and agree this is better as a plugin :)

Working on that over at https://gitlab.com/aleishaamohia/koha-plugin-austlang
Comment 7 David Cook 2024-02-26 22:20:19 UTC
(In reply to Aleisha Amohia from comment #6)
> Totally hear you both, and agree this is better as a plugin :)
> 
> Working on that over at https://gitlab.com/aleishaamohia/koha-plugin-austlang

Sounds good :)

I meant to mention earlier that I have a library who use MarcEdit to convert the Austlang CSV into MARC Authority records, if I recall correctly.

I think though that they've created them with their own authority type, which I think might have some negative consequences with indexing. I recommended they just add them as Topic Terms for indexing purposes but I think they stuck with the custom type.
Comment 8 Aleisha Amohia 2024-02-26 22:27:34 UTC
(In reply to David Cook from comment #7)
> (In reply to Aleisha Amohia from comment #6)
> > Totally hear you both, and agree this is better as a plugin :)
> > 
> > Working on that over at https://gitlab.com/aleishaamohia/koha-plugin-austlang
> 
> Sounds good :)
> 
> I meant to mention earlier that I have a library who use MarcEdit to convert
> the Austlang CSV into MARC Authority records, if I recall correctly.
> 
> I think though that they've created them with their own authority type,
> which I think might have some negative consequences with indexing. I
> recommended they just add them as Topic Terms for indexing purposes but I
> think they stuck with the custom type.

Ah yes that sounds interesting.

This feature was so they could link Indigenous Australian languages specifically in the 041$a, hence the use of authorised values.