Responding to an error report because you are on-call, and need some ideas for how to get started with some quick actions? We got you.
| Table of Contents |
|---|
To execute any of these on staging instead of production replace -a scihist-digicoll-production with -a scihist-digicoll-staging.
Check on status of heroku dynos
...
There are ways to set up command-line access to papertrail logs too. https://github.com/papertrail/papertrail-heroku-plugin https://github.com/papertrail/papertrail-cli
Even without papertrail you can use heroku cli
heroku logs -a scihist-digicoll-productionto look at running trail of current heroku logs, but the display isn’t nearly as nice as papertrail, and includes more noise that papertrail filters out.
...
You can enable it again with enable instead of disable. You can see if it’s enabled with heroku features -a scihist-digicoll-production
Is heroku itself having problems? Or are other platforms we use?
...
Look at release history with heroku CLI:
heroku releases -a scihist-digicoll-production
Failed releases will be in red. With the id from the left-most column, you can look at specific log output (mainly of our custom release phase) for the failed or successful release, eg:
heroku releases:output v323 -a scihist-digicoll-production
You can also see some limited release status info in the Web GUI on the Activity tab.
...
Login to searchstax
Use shared credentials stored in our credential spot on the P:\ drive
Click on the instance you want to restart (
scihist_digicoll(production), orscihist-digicoll-staging)At bottom of page there is a single node listed (our plan only has one node), you can click “stop solr”, and then “Start solr”
...
No worries, just login to http://hirefire.io (we each have our own login), and you can click the “enable” toggle on or off next to each autoscale worker, right on the initial dashboard. (We may only have one worker).
Note: If you turn off auto-scaling when workers are scaled up, they will probably stay scaled up! Look at the minimum scale value (2, as I write this), you may want to scale down to that manually after turning off auto-scaling:
| Code Block |
|---|
# how many workers are there?
$ heroku ps worker -a scihist-digicoll-production
# set em back to two
$ heroku ps:scale worker=2 -a scihist-digicoll-production |
Put entire app into maintenance mode
...
In heroku CLI , run heroku maintenance:on -a scihist-digicoll-production and heroku maintenance:off -a scihist-digicoll-production
(Note: Right now, this is just a generic heroku maintenance message. It is possible to customize/brand this page, we may get to that eventually. https://github.com/sciencehistory/scihist_digicoll/issues/1201 See more on our custom maintenance page configuration at Heroku custom maintenance page )
Disable staff logins
We can effectively make the app “read-only” but still available to the public by disabling staff logins. So we don’t have a public facing outage, but if we’re dealing with some kind of data corruption issue we’re trying to diagnose, we might want to ‘freeze’ staff out.
In heroku config vars on section of the heroku dashboard settings tab, just set LOGINS_DISABLED to true.
...
Restore postgres database from backups
See separate page.