Journal of heroku investigations. Most recent entries on top? See also Heroku Consideration

Wed Oct 14

For RAM comparison, on our current EC2 production, after being up for some time, passenger reports this memory use:

------ Passenger processes ------
PID    VMSize     Private   Name
---------------------------------
18108  299.2 MB   2.0 MB    Passenger watchdog
18114  1082.6 MB  5.3 MB    Passenger core
18139  30.4 MB    0.4 MB    /usr/local/lib/ruby/gems/2.6.0/gems/passenger-5.3.7/buildout/support-binaries/PassengerAgent temp-dir-toucher /tmp/passenger-standalone.11jhb2e --cleanup --daemonize --pid-file /tmp/passenger-standalone.11jhb2e/temp_dir_toucher.pid --log-file /opt/scihist_digicoll/shared/passenger.log --user digcol --nginx-pid 18123
18187  958.4 MB   340.2 MB  Passenger AppPreloader: /opt/scihist_digicoll/current (forking...)
18206  873.4 MB   281.1 MB  Passenger AppPreloader: /opt/scihist_digicoll/current (forking...)
18225  738.5 MB   197.3 MB  Passenger AppPreloader: /opt/scihist_digicoll/current (forking...)
18244  736.6 MB   160.8 MB  Passenger AppPreloader: /opt/scihist_digicoll/current (forking...)
18261  736.7 MB   158.1 MB  Passenger AppPreloader: /opt/scihist_digicoll/current (forking...)
18278  736.8 MB   169.9 MB  Passenger AppPreloader: /opt/scihist_digicoll/current (forking...)
18295  736.9 MB   163.0 MB  Passenger AppPreloader: /opt/scihist_digicoll/current (forking...)
18312  737.0 MB   169.8 MB  Passenger AppPreloader: /opt/scihist_digicoll/current (forking...)
18329  737.1 MB   163.2 MB  Passenger AppPreloader: /opt/scihist_digicoll/current (forking...)
18346  737.2 MB   162.4 MB  Passenger AppPreloader: /opt/scihist_digicoll/current (forking...)

So actually it’s true that the Private RSS was getting up to 340MB, although after more use. One difference is that on heroku it seems to balloon up memory quicker. But I may have under-estimated our RAM use – although it still isn’t the 400-500MB+ that we’re seeing in heroku.

An app with the work show page almost entirely disabled is at sample#memory_total=277.77MB sample#memory_rss=269.82MB 

We might be able to get under 300 by making the work/show page avoid loading all children at once with an “infinite scroll” technique. This would also take care of our slowest pages. Pages we are trying that are NOT large-membered-work-show seem to currently on ‘standard’ rather than ‘hobby’ resources be loading at similar times to current EC2, we think? Fixity report 3s on heroku compared to 3.5s on EC2, so actually faster on heroku?

If we limit to only 50 children on a page, ramelli loads from heroku in about 2.6s (yeah, still slow), and takes RAM: sample#memory_total=447.76MB sample#memory_rss=399.51MB gah why is this still so much!! – I guess the way we did it we still loaded all children into memory but just didn’t display them, let’s chagne that…. after a few loads, still up to sample#memory_total=466.50MB sample#memory_rss=398.28MB  gahhhh.

If we limit to the 5 child work, we get a more reasonable sample#memory_total=321.20MB sample#memory_rss=252.28MB… to compare, let’s slice ramelli to actually 5 children… it’s still taking more than 2 seconds to return (what’s it doing?), but is sample#memory_total=395.11MB sample#memory_rss=326.82MB  , ok i guess?

without actual member display code, and limited to 5…. sample#memory_total=391.79MB sample#memory_rss=323.62MB  … about the same… aha, it’s partially our viewer_images_info taking up all the memory, that one still has full list. (but doesn’t explain why the page load time is so slow) Just curl… no, still slow still same memory.

A moment to look at speed again

Yes, even with standard-2x and standard pg, ramelli is taking 4-6s on heroku, compared to 2-2.5s on our current EC2. 😞 Smaller 115-item work goes from 0.5-0.6s on EC2 to ~0.9-1.2S on heroku, what.

RAM how many threads can we get away with

RAILS_MAX_THREADSour puma config pays attention to that heroku config env, making it easy to switch.

One worker 5 threads on a standard-1x (512MB) dyno – we exceeded memory capacity repeatedly requesting ramelli. 😞

three threads – yep, still exceeded quota.

two threads? seems to be okay, but pushing it! We wouldn’t want our app to expand it’s wasteland any further. sample#memory_total=497.73MB sample#memory_rss=473.06MB

going back to one thread for a consistent baseline for exploring how changes effect memory.

Monday Oct 12/Tuesday Oct 13

...