Why was the website so slow for so long?
The cause of the slowdown was a change to the ZFS dataset. In conjunction with the database server, very little caching was being done. This combination resulted in increased disk I/O as the system churned through the database. Details in the blog post which outlines the various things which changed. Many graphs.