...

Solr

The “search engine” that gives us a “google-like” search, powers the search functionality.

Need to verify we have as much customization of schema.xml (and solrconfig.xml?) as we need.

Current AWS

• Has 8000 docs (maxdocs 10,000). Basically scales with number of Works we have.

• Listed as 70MB size

• Trying to figure out how many solr queries we do per day or hour, since some solr heroku add-ons are metered requests… I’m not sure we’re logging the right stuff and am not sure I’m interpreting the logs right, but I am going to say somewhere under 10K requests per day.

Heroku Estimate

There is one plugin for Solr available on heroku marketplace, from third-party vendor “WebSolr”. The “Standard Small” plan seems to be sufficient for us.

• 1 million document limit => plenty

• 40K Daily requests => should be enough, although not an order of magnitude extra, about right

• Storage capacity 1GB – we are only using 70MB currently (including transcript full text indexing, no problem.

• “5 concurrent connection” limit could be a problem.

• $59/month, next step up is a big step to $189 (150K requests/day, 10 concurrent connections) or $299/month (250K requests/day, 15 concurrent requests)

Considerations

• I am concerned that the “concurrent request limit” of “5” in “Standard small” might be a problem, considering updates/reindexes as well as searches are concurrent requests. I don’t entirely understand how this is metered, and what happens if it’s exceeded

  • WebSolr seems to suggest custom logic to queue/throttle solr queries, which would be something we’d have to develop additionally in our app and could be a pain. https://docs.websolr.com/article/178-http-429-too-many-requests

• Doing lots of bulk operations might make lots of requests per day, since our app is set up to do a solr update on every ‘save’.

• Larger plans can get expensive quickly. “Standard Medium” with 150K daily requests and 10 concurrent requests (which still seems small to me) is $189/month

• If we ended up with anything running on raw AWS instead of heroku, Solr would probably be the first thing. It’s what I’m most worried about being able to run affordably on heroku.

• ElasticSearch is a competitor to Solr; it has many more heroku plugin offerings from multiple vendors at differnet price points, which are not typically metered by concurrent connections (the most troublesome meter in WebSolr plans). But getting our app to run on ElasticSearch instead of Solr would take significant development (we are not as familiar with it; it is unclear to what extent Blacklight supports it, or we’d need to develop new stuff on top of or instead of Blacklight; we are definitely using some Blacklight plugins like date_range_limit that are Solr-only).