Dev notes - Hydra camp

Owned by Anna Headley (Unlicensed)
Last updated: Mar 18, 2015
10 min read

Questions and practicalities

practicalities

  • Environment var management?

    • envyable?

  • How to curate, manage, maintain a set of test files / data?

    • use rspec

  • Does Blacklight provide REST APIs?

    • You can get search results but maybe not facets?

  • What db are most people using? postgres? How to move from sqlite to postgres?

    • use what you know or is supported by your IT department

    • (sqlite is not for production)

  • What decisions do I need to make before I launch into production? what decisions can I change later on?

    • what id do you want to use?

    • object model in fedora - don't even worry about it for the first 50-100 thousand items

    • don't store anything exclusively in solr


Questions

  • What are the encoding defaults of metadata and databases in Hydra?

  • what data modeling needs to be done in addition to what fedora / hydra already provide (or will provide)? default URIs? Should these be changed? If so, to what?

  • DEFER (may not be relevant): what are the graph manipulation tools in fedora? how are these accessed from hydra? Or is it just taken care of when API requests are made?

  • Re: Ordering, what is a proxy? Also has a meaning outside the context of ordering.


Answers

  • is XML completely deprecated in fedora 4?

    • Doesn't sound like it. Get more info. Mark recommends examining your use cases – legacy XML from fedora 3? maybe model it that way for fed4. Ultimately he says it doesn't matter what format you store the data in, i.e. same from a user standpoint. (fair enough)

    • Reasons to use XML

      • You have very hierarchical data in nice, clean XML that someone is attached to

    • Reasons to use RDF

      • It's the current direction of the community

      • code is a little bit simpler

      • more specific / self-describing / natively self-authoritizing

      • You have data-sharing use cases

  • how / when is bagit used with fedora? hydra?

    • bagit is for storing collections of files / metadata in remote locations. we have no reason to worry about this at the moment.

  • MIRA: open source?

  • General Linked Data conversation

    • marmotta and fedora both implement LDP (linked data platform) which is a brand new w3c specification

    • Marmotta is currently being used in conjunction with fedora by people who want to cache external linked data sets for local use. (e.g., only the relevant portions of LCSH, and only on an as-needed basis)

    • Linked Data Fragments is super new and it's not clear yet how useful it will be

    • LDP: http://www.w3.org/TR/ldp/

    • Fedora3 used "data streams" Fedora4 uses the concept of "attachments" (see LDP). Fedora uses an RDF triplestore (fedora3 had to do a lot of serializing and deserializing and parsing). Fedora still has binary datastreams (used for attachments, non-RDF stuff).

    • You can theoretically use SPARQL directly into fedora's triplestore. but this would be a bad idea slash only for very expert people.

  • How does hydra/fedora handle preservation? E.g. checksums.

    • Fedora4 doesn't yet carry forward all the technical support for this that Fedora3 had. That's why there is current work around an audit service.

    • archivematica people believe that there are holes in hydra preservation. archivematica

    • Can you use fixity checks built in to your storage platform? If so, don't do it in fedora / hydra.

    • https://github.com/psu-stewardship/scholarsphere/wiki/Fixity-in-Sufia-with-Fedora-4 - confirm that this doc is still relevant and ask to talk to these people.

    • There is a Hydra preservation interest group (note for Michelle)

    • Other things to be aware of: provenance and history of edits - planned auditing service will take this on. Currently / formerly everything was recorded as a single user.

    • Consider thinking about fedora storage formats. What happens if you lose fedora?

  • What does it look like to upgrade?

    • Have sufficient test coverage

    • Make sure all tests are passing

    • Increase versions of dependencies in your Gemfile

    • Run tests and observe failures.

    • Start turning tests green




Uncategoriezed notes

  • Reindex everything into Solr from Fedora!
    • ActiveFedora::Base.reindex_everything
  • opaquenamespaces: a community registry / namespace for RDF properties. Probably best practice to try to put locally-required properties there or somewhere similar. This was started by Karen and Tom at U Oregon.
  • Page numbers: can use this as literal sequence and put marked pagenumbers in page label property.
  • Book.where(title:"hat") # note this returns an array, not a single book object (the way find does)
  • re: IDs. consider using the default id as an internal identifier and another field as a local id for human use.
  • one challenge of hosting on amazon is moving large files around. sufia uses ffmpeg and converts them to playable proxies. anything that can be handled by imagemagick, openoffice / libreoffice (creates a thumbnail from the first page of the doc)
  • Currently recommending different heads for different types of collections. use a shared gem for the data models. Then create a separate admin head for managing them all in one place.
  • sufia has a new wiki page on metadata modeling https://github.com/projecthydra/sufia/wiki/Customizing-Metadata
  • DCE also recommends keeping number of servers as low as possible until metrics indicate you should make changes (and can measure those changes on desired vectors)
  • Dspace, libguides, digital library publishing (drupal, node apps), archive-it – NYU harvests into hydra (ichabod) using Internet Archive API. metadata of record is in another system, but can be supplemented in the admin backend. Also have batch-loaded enrichment data.
  • fedora 4 implements w3c standard for access controls
  • REST vs. CRUD
    • GET, POST, PUT/PATCH, DELETE (ActiveFedora, HTTP)
    • Read, Ceate, Update, Delete - ActiveRecord (RDMS)
  • ORM - object relational mapper - the code that interfaces with the actual database
  • LDP is a way to use REST to talk about objects with containment relationships
  • Gemfile.lock shows the expansion of the gemfile
  • If developing on core, replace hydra gem with the gemspec contents of same if you want to mess with changing versions of those dependencies. (or if you want to manage / change these versions manually)
  • look at other peoples' .gitignore files for rails and sufia projects
  • What should our ID be? sufia uses NOIDs. note: fedora has a concept of different minters so this may be a factor here as well. NOID translates to fedora as pair paths, but fedora doesn't actually store it that way. so why did sufia do it this way?
  • characterization object contains xml. you could take each of those values and store them as properties.
  • different 2nd-level facets: see dl.tufts.edu
  • one-off pages / static pages
    • curationexperts/alexandra-v2 see welcome/about see also index, contact form. note empty controllers and views that go with them.

 

Deployment

  • travis-ci.org/curationexperts/alexandria-v2/builds
  • https://travis-ci.org/projecthydra/sufia

  • capistrano: deployment manager. redundant with a ci workflow?

  • bambu - stanford's environment management solution.

  • PRODUCTION setup
    • tomcat, solr, fedora replace hydra-jetty / jetty wrapper
    • postgres as opposed to sqlite
    • WATCH RAILS VERSIONS between multithreading and databases
  • staging
    • may be as much like production as possible. may have less CPU power, less memory, smaller HD. May be an exact clone. Also take into account how much time / effort this may require.
  • ditch testunit and install rspec. spec directory:
    • spec
      • fixtures
        • pbcore
          • artesia
            • joyce_chen
              • image_1.xml
          • mars (filemaker database)
            • audio_1.xml
            • image_1.xml
  • look at fixtures vs. factories
    • fixtures have to be maintained.
    • factories behave the way you tell them to behave; sometimes you need to put in real data.
  • sandy metz railsconf 2013 presentation video
  • github.com/afred/openvault - look at factories here.
  • github.com/projecthydra-labs/hydradam
  • github.com/WGBH/pbucore
    • huge xslt stylesheet to convert XML into RDF-XML, which they will then use to load into fedora4.
  • Amazon Ops products: elastic beanstalk, opsworks

 

Testing

  • rspec only? selinium with rspec?
  • or capybara with cucumber?

Example fedora instances:

  • scholarsphere
  • dl.tufts.edu - tufts digital library - put a hydra head on top of existing fedora repo. awesome transcription / TEI w/ embedded timecode / audio player
  • levysheetmusic - changes / customizations to interface
  • hullhistorycentre.org.uk - hull city archives - example of EADs. (nice search box page!)
  • hydra.hull.ac.uk - has a backend with workflow stuff. would likely be happy to give a short demo. (also note interesting icons)
  • alexandria digital research
  • spotlight (stanford) - library.stanford.edu/projects/spotlight - for exhibit building. - note: blacklight gallery gem gives you different views of results lists.
  • another gem: date slider
  • digital.case.edu (built on worthwhile, rdf-driven) - open seadragon + iiif-compliant server for amazing image viewing. view metadata / different formats.
  • dl.tufts.edu - MIRA (management of institutional repository assets. more workflow-type, controlled deposit. 
  • http://demo.curationexperts.com/
  • WGBH - digitize on-demand. Metadata is published and there's a button.
  • HydraDAM (replaced Artesia at WGBH)
  • single EAD site (blacklight-only) http://bassiveratti.stanford.edu/en/catalog

 

Syllabus

  

March 9th-13th, 2015

Yale University Library

New Haven, CT

Course Goals  

The goal of Hydra Camp is to introduce new developers to the skills and tools they will need to successfully build Hydra based digital repository solutions.  There’s a lot of ground to cover and you won’t walk out at the end of the week a complete expert, but we hope we’ll have provided you enough of a scaffolding to jump-start your own work and keep learning like the rest of us.  We hope that the topics covered at Hydra Camp provide enough breadcrumbs that you’ll have a good idea where to start looking once you get home and start digging into problems on your own!

Supplies

  • Laptop & Power Supply

  • Headphones/Earbuds

  • Water bottle/Travel Mug


Location

Prerequisites  


  • Local Development Environment Requirements

    • If you’re using a Mac, install/update XCode & homebrew.  

    • Install Git (on Mac, we recommend using homebrew to install git)

    • Install RVM + Ruby 2.1.5 using RVM `rvm install ruby-2.1.5`

(if you follow the installfest exactly, 2.0.0 also works fine)

    • Install Java 7 runtime (if you already have 6 installed, that also works)

    • Install Rails 4.1.9: `gem install rails 4.1.9`

    • Install a Text Editor of your choice.   KomodoEdit is a popular free option.  Many people use TextMate (not free). VIM is hardcore but some of us do it.

    • Create a (free) Github Account if you don't already have one: https://github.com/signup/free

    • Create Github SSH Keys and set them up for your development machine

  • NOTE: We’ll have help available in class if you run into any troubles getting your system set up.  We’ll have the VirtualBox image configured with Ubuntu and the necessary tools to complete class exercises or can help install all necessary software locally.


Syllabus

https://docs.google.com/document/d/10YeaUkYV-akfLQhqVd7Zffy_UxosdDUgzu6RxuIsxBA/edit#

Day 1: Monday, March 9th

  • MORNING - start 9:00am

  • DINNER  - on your own


Day 2: Tuesday, March 10th

See links to examples under Resources

  • LUNCH - on your own

    • [Optional] Start modelling your own metadata

    • [Optional] Create some additional content (books) to search

    • [Optional] http://vim-adventures.com 

  • DINNER  - group dinner @ TBA (?)


Day 3: Wednesday, March 11th

https://github.com/curationexperts/hydradam/wiki/Production-Installation%3A-Overview


Day 4: Thursday, March 12th


Day 5 [BONUS]: Friday, March 13th

  • Blacklight workshop







 


RESOURCES





  • Sample code for exercises

https://gist.github.com/afred/660e8f43026ad08a992d




 def show_my_stuff(solr_parameters, user_parameters)

   solr_parameters[:fq] ||= []

   solr_parameters[:fq] << "has_model_ssim:(\"#{Book.to_class_uri}\" OR \"#{Manuscript.to_class_uri}\")"

 end

 

 

 

Blacklight slides (content only)

 

 

Customization Examples

 

Documentation & Getting Help

Setting up the Dev Environment

 

  • Get the VM running

    • Install VirtualBox & Vagrant

    • Copy tutorial directory

    • Open a terminal window & cd to tutorial

      • vagrant up

      • vagrant ssh


Customizing catalog_controller.rb

  • Open the file in your editor

  • Read through the comments, try changes

  • Change how fields display on index views
    (i.e. search results)

  • Turn off author search, turn on year

  • What else can you do via catalog controller configuration?

Internationalization

Advanced Customization

 


Customization best practices

 

Other examples