notbugAs an Amazon Associate I earn from qualifying purchases.
Want a good read? Try FreeBSD Mastery: Jails (IT Mastery Book 15)

Why was the website so slow for so long?

The cause of the slowdown was a change to the ZFS dataset. In conjunction with the database server, very little caching was being done. This combination resulted in increased disk I/O as the system churned through the database. Details in the blog post which outlines the various things which changed. Many graphs.

How big is it

It was a few days ago that I was thinking about search engines crawling through this website. I began to wonder just how many web pages there are here. To calculate this total, it's not just a simple matter of counting files on disk. Most of the web pages are created from entries in the database. One recent evening, I started to design a formula to find out how many web pages there are. Roughly. This will not be 100% accurate, but it will be close.

Pages on disk

First, let's count the number of pages on disk:

$ ls *.php | wc -l
65
Number of categories

There is a page for each category:

# select count(*) from categories;
 count
-------
   120
(1 row)
Number of ports

There are ports, and there are deleted ports. I'll show both:

# select count(*) from ports_all where status = 'A';
 count
-------
 39449

# select count(*) from ports_all where status = 'D';
 count
-------
 19231
(1 row)
Number of files in the ports tree

There is a page for each file in the ports tree:

[dan@ngaio:/usr/ports] $ find . | wc -l
176248
[dan@ngaio:/usr/ports] $
Count last performed at Tue, 26 May 2020 03:01:27 GMT
Number of commits

There is a page for each commit:

# select count(*) from commit_log;
 count
-------
479874
(1 row)
Number of ports for each commit

For each commit, you can view the files modified by that commit for a particular port:

# select count(*) from commit_log_ports;
 count
-------
1044961
(1 row)
How many days?

For each day, there is a page showing the commits for that day. How many days do we have?

# select count(distinct commit_date) from commit_log;
 count
-------
  7295
(1 row)
How many users?

Each user has a page:

# select count(*) from users;
 count
-------
 13186
(1 row)
How many watch lists?

For each watch list, there is a page:

# select count(*) from watch_list;
 count
-------
 13629
(1 row)
Estimated total

That gives a grand total of 1,794,058 pages. On my last count, that's about 0.022264% of the web pages on Google1

Notes

  • These statistics are updated daily.
  • 1The number of Google pages used in this calculation is 8,058,044,651.
How much diskspace?

The total space used by the FreshPorts database is:

# select pg_database_size('freshports.org');
 pg_database_size
------------------
54,207,677,295
(1 row)

That's bytes...

This value might be easier to parse: 50 GB

Login
User Login
Create account

Servers and bandwidth provided by
New York Internet, iXsystems, and RootBSD

This site
What is FreshPorts?
About the authors
Issues
FAQ
How big is it?
The latest upgrade!
Privacy
Blog
Contact

Search
Enter Keywords:
 
more...

Latest Vulnerabilities
chromiumMay 24
piwigoMay 23
drupal7May 22
tomcat-develMay 22
tomcat7May 22
tomcat85May 22
tomcat9May 22
unboundMay 22
zabbix3-proxyMay 20
zabbix3-serverMay 20
rubygem-actionpack52May 19
rubygem-actionpack60May 19
rubygem-actionview52May 19
rubygem-actionview60May 19
rubygem-activestorage52May 19

14 vulnerabilities affecting 165 ports have been reported in the past 14 days

* - modified, not new

All vulnerabilities

Last updated:
2020-05-25 19:05:09


Ports
Home
Categories
Deleted ports
Sanity Test Failures
Newsfeeds

Statistics
Graphs
NEW Graphs (Javascript)
Traffic

Calculated hourly:
Port count 39457
Broken 80
Deprecated 625
Ignore 370
Forbidden 5
Restricted 147
No CDROM 76
Vulnerable 23
Expired 19
Set to expire 580
Interactive 0
new 24 hours 6
new 48 hours9
new 7 days42
new fortnight85
new month245

Servers and bandwidth provided by
New York Internet, iXsystems, and RootBSD
Valid HTML, CSS, and RSS.
Copyright © 2000-2020 Dan Langille. All rights reserved.