fcrepo-message-consumer

Owned by Anna Headley (Unlicensed)
Apr 07, 2015
3 min read

 jcoyne

cam156: did you set up fcrepo-message-consumer to use the FileSerializer on your prod install?

11:18

cam156

jcoyne: I have no idea… awed hector^

11:19

cam156

awead^^

11:19

awead

jcoyne: I haven't heard of that, so I'm guessing no

11:19

hector

cam156 jcoyne: I have no idea either.

11:20

jcoyne

I don't think it's totally obvious that that's the preservation strategy for Fedora 4.

11:20

jcoyne

Needs better instructions

11:20

jcoyne

escowles: Is that about right?

11:21

escowles

jcoyne: yes, that's about right -- things are moving towards using camel for that kind of thing now though, but i don't think it's gotten as far as the message-consumer yet

11:27

jonibar has left IRC (Quit: Leaving.)

11:28

awoods has joined (~awoods@c-67-165-245-76.hsd1.co.comcast.net)

11:29

jonibar has joined (~Adium@pool-71-162-33-59.altnpa.east.verizon.net)

11:29

mejackreed has left IRC (Quit: Leaving.)

11:30

jonibar has left IRC (Client Quit)

11:43

pgwillia has joined (~chatzilla@129.128.46.157)

11:49

mejackreed has joined (~mejackree@50.141.67.0)

11:52

jonibar has joined (~Adium@h34.184.101.208.static.ip.windstream.net)

11:53

umgrosscol has joined (~gcol@grosscol.umdl.umich.edu)

11:55

mejackreed has left IRC (Ping timeout: 250 seconds)

12:04

hortongn has left IRC (Ping timeout: 255 seconds)

12:08

mjgiarlo

+1, not totally obviou.

12:08

mjgiarlo

s

12:10

cam156

jcoyne: escowles: Is this something that goes in a config file?

12:10

cam156

fcrepo-message-consumer?

12:10

jcoyne

cam156: it's a totally separate java app

12:11

jcoyne

cam156: https://github.com/fcrepo4/fcrepo-message-consumer#running-the-indexer

12:11

cam156

jcoyne:  That should be running on the servers?

12:11

escowles

cam156: jcoyne: yep, it's a separate app that listens for messages coming from fcrepo4 -- it can be on a separate server or the same one

12:12

escowles

cam156: there are modules for indexing updated records in solr, syncing them to a triplestore, or saving them to disk

12:13

escowles

the advantage (over nightly backups) is you get a message every time a record is updated, so you can backup multiple versions throughout the day, and capture changes that would have been overwritten

12:15

cam156

escowles: Is the indexer a necessary item?  This is the first I am hearing of it

12:15

escowles

cam156: no, it's not necessary -- you only need it if you want to do something every time records change

12:16

escowles

cam156: in theory a lot of the resque jobs could be done in the fcrepo-message-consumer or fcrepo-camel

12:16

cam156

escowles: I think we are leaning on the old hydra way of index after save at the moment

12:17

awead

cam156 escowles: yeah, we'd want to investigate that as a possible replacement for reque, but not now... :)

12:17

mjgiarlo

what's the current recommendation for folks who care about having filesystems with both binaries and serialized metadata written to disk?

12:18

mjgiarlo

fcr4 already does binaries, it seems, but it'd be great to have "whole objects" on disk at some point.

12:18

awead

mjgiarlo: are you talking about things at the modeshape level?

12:19

mjgiarlo

I'm not sure about levels.

12:20

mjgiarlo

when I have fcr4 running, binaries I upload get spooled off to disk. properties do not, apparently. this is a thing we had in fcr3 and that a bunch of us would like to see in some form.

12:20

escowles

mjgiarlo: awead: the metadata is written to disk in a binary format (modeshape using infinispan store) - if you want to write all the metadata to disk in a human-readable form, then either use message-consumer/camel to do it in real time, or write a tool to crawl the rest api and download everything

12:20

mjgiarlo

ok, so it's the message-consumer again. hence what jcoyne was talking about re: preservation strategy.

12:20

escowles

mjgiarlo: yep, he's referring to this from the tech working group: https://wiki.duraspace.org/display/FF/Outcomes+-+Preservation+Worthiness

12:21

awead

escowles: so where does F4 store metadata, the triples? To a different disk?

12:21

mjgiarlo

ok, good to know.  thx

12:21

escowles

awead: f4 has one store for binaries and a separate store for properties

12:21

escowles

binaries are written to disk as-is, named after their sha1 checksums

12:22

awead

escowles: huh. I thought modeshape kept metadata in nodes together with the files.

12:22

escowles

awead: that's what the API looks like (binaries are just another property in the JCR/Modeshape API)

12:23

escowles

awead: but when they're written to disk, they use two separate stores

12:23

mjgiarlo

escowles++

12:23

awead

escowles++ # thanks