Firewall Reference Manual
Firewall Reference Manual
Firewall Reference Manual
1 Introduction ....................................................................................................................... 5
1.1 What is URL Filtering? ....................................................................................................... 5
1.2 Why is Tunnel Detection a Must-Have ? ............................................................................ 5
1.3 About This Document ......................................................................................................... 6
1.4 Latest Major Enhancements ................................................................................................ 6
1.5 Copyright ............................................................................................................................ 6
1.6 Support and Feedback ......................................................................................................... 7
2 Prerequisites ...................................................................................................................... 8
2.1 System Resources ............................................................................................................... 8
2.2 Prerequisites for Installation from RPM Package ................................................................ 8
2.3 Prerequisites for Installation from Source ........................................................................... 8
3 Architecture ....................................................................................................................... 9
3.1 How URLs are blocked ..................................................................................................... 10
3.1.1 Blocking HTTP URLs .......................................................................................... 10
3.1.2 Blocking HTTPS URLs ........................................................................................ 10
3.1.3 Recommended Configuration ............................................................................... 12
3.2 HTTP URL Redirection .................................................................................................... 12
3.3 HTTPS URL Redirection .................................................................................................. 12
3.3.1 Squid Example Configuration, No SSL-Bump ..................................................... 13
3.3.2 Squid Example Configuration, SSL-Bump peek+splice ....................................... 13
3.3.3 Squid Example Configuration, SSL-Bump peek+bump ....................................... 14
3.4 How ufdbGuard Deals with URLs .................................................................................... 15
3.5 Dynamic Proxy Tunnel Detection ..................................................................................... 15
3.6 Enhanced HTTPS Security ................................................................................................ 15
4 Software Installation ....................................................................................................... 17
4.1 Installation from RPM Package ......................................................................................... 17
4.1.1 File Layout on Redhat and CentOS ...................................................................... 17
4.1.2 RPM Package Installation ..................................................................................... 17
4.2 Installation from Sources .................................................................................................. 18
4.2.1 Upgrading from a Previous Version ..................................................................... 18
4.2.2 User Account ........................................................................................................ 18
4.2.3 Installation Directory ............................................................................................ 18
4.2.4 Unpack Software .................................................................................................. 18
4.2.5 Configure the Software Build ............................................................................... 18
4.2.6 Configure Software for Multiple Systems ............................................................ 19
4.2.7 Compile Software ................................................................................................. 19
4.2.8 Install Software ..................................................................................................... 19
4.3 Configure ufdbGuard ........................................................................................................ 20
4.4 Get Daily Updates ............................................................................................................. 21
4.4.1 Daily Updates from URLfilterDB ........................................................................ 21
4.4.2 Exit Codes of ufdbUpdate .................................................................................... 22
4.4.3 Database Updates from Other Sources ................................................................. 22
4.4.4 Converting a URL Database ................................................................................. 23
4.5 Firewall and Proxy Settings .............................................................................................. 23
ufdbGuard 1.32 supports a new database format (2.2) which supports the ZLIB compression algorithm.
The URL database from URLfilterDB uses ZLIB compression since it is approximately 5 times faster
decompressing URL tables although the space saving is a few percent less than with the BZIP
compression algorithm. BZIP was used by ufdbGuard before version 1.32.
ufdbGuard 1.33 supports the new keywords ipv6 and ipv6list which can be used in source
definitions. ufdbgclient has a new -m option to enable multithreading which performs much better than
the previous queueing mechanism that can be enabled with the -C option. ufdbGuard now fully
supports URLs with %-encoded and native UTF8 characters.
ufdbGuard 1.33.2 has a fix for a bug that appears on overloaded or slow systems as well as inside virtual
machines where ufdbguardd exits with the error “HUP signal received but could not acquire a lock on
the configuration”.
ufdbGuard 1.34 supports OpenSSL 1.1.x, has a new execiplist feature and to support complex
bump/splice scenarios, one can define now per category whether connections are bumped or spliced. To
further support bump-or-splice decisions by Squid, the URL database comes with a new plain text file
.../blacklists/finance/domains.squidsplice which enables admins to use Squid acls
to enforce that sites of banks are always spliced (not decrypted).
A detailed list of all changes can be found at the top level of the source tree in the file CHANGELOG.
1.5 Copyright
The ufdbGuard software suite version 1.x is Open Source Software and free to use. To protect the
ownership and the freedom of use of ufdbGuard, there is a copyright and you have a license to use and
modify the software freely, known as the GPL version 2 license. The full text of the license is here:
www.gnu.org/licenses/old-licenses/gpl-2.0.html.
redirect URL, with ssl-bump ufdbguardd may redirect to an other HTTPS URL.
peek+splice caveat: the browser displays an error message to the end user like
“the certificate does not match the domainname” or equivalent.
second caveat: ufdbguardd only sees only the FQDN and not the
full URL to make blocking decisions.
redirect URL, with ssl-bump ufdbguardd sees the full URL and may redirect to an other HTTPS
peek+bump URL. The user sees an understandable message like “this site is
blocked ...”.
caveat: the browser needs to trust the Squid bump by importing the
Squid CA certificate.
second caveat: there is a list of sites that malfunctions or refuses
operation with Squid peek+bump mode and must be explicitly
configured to use peek+splice.
terminate the connection Squid can terminate a connection by acls. caveat: the browser
displays an error message to the end user like “server terminated
the connection” or equivalent.
second caveat: ufdbguardd is not used.
Note that ssl-bump features client-first and server-first of older Squid versions are not supported.
peek+splice is an operating mode of Squid where it peeks at the SSL/TLS hello message of the browser,
extracts the SNI, the Server Name Indication, and then splices the connection. Splicing in this context
means that the connection between the browser and the webserver is untouched.
peek+bump is an operating mode of Squid where it peeks at the SSL/TLS hello message of the browser,
extracts the SNI, the Server Name Indication, and then
The SNI that Squid peeked at, must be passed on to ufdbGuard for it to know to which webserver the
browser connects to. See section 3.3.2 for more details.
A mix of peek+splice and peek+bump is also possible by means of acls of Squid. The mix can be used
for example when one prefers to use peek+bump for the majority of HTTPS sites and use peek+splice
for a small set of sites where bumping is not desired (e.g. sites of banks, sites with pinned certificates,
etc.).
NOTE: the database includes .../blacklists/finance/domains.squidsplice which is a
plaintext file with all domains of the finance category which can be used in a Squid acl to ensure that the
URLs of financial institutions will always be spliced.
sslproxy_options NO_SSLv2,NO_SSLv3,No_Compression
sslproxy_cipher ALL:!ADH:!DSS:!MD5:!EXP:!DES:!PSK:!SRP:!RC4:!IDEA:!SEED:!
aNULL:!eNULL
sslcrtd_program /local/squid/libexec/ssl_crtd -s \
/local/squid/var/lib/ssl_db -M 4MB
sslcrtd_children 5
sslproxy_options NO_SSLv2,NO_SSLv3,No_Compression
sslproxy_cipher ALL:!SSLv2:!SSLv3:!ADH:!DSS:!MD5:!EXP:!DES:!PSK:!SRP:!RC4:\
!IDEA:!SEED:!aNULL:!eNULL
The URL database and log files may need up to 2 GB disk space. This amount depends mostly on the
configuration of the maximum size of the log files. So make sure that /var/ufdbguard has
sufficient space or configure ufdbGuard to use an other location to store the URL database and log files.
1
depending on the OS, the system configuration file may be in an other directory.
To install the cron job for ufdbUpdate, edit the crontab table of the user ufdb and add the
appropriate lines:
$ crontab –e
To run the URL database update each day at 6:15 AM, add the following line:
15 6 * * * /usr/local/ufdbguard/bin/ufdbUpdate
At this time, you may want to verify that the Squid housekeeping is also executed. Verify that the
crontab for user squid has an entry to run “squid –k rotate” (see the squid manuals for more
details). It is a common misconception that this command only rotates the logfile of Squid but it also
does necessary housekeeping. Note that the Squid housekeeping must be done when there is almost no
load (e.g. at 03:00 AM).
code explanation
0 all OK
11 configuration error
13 download OK but cannot signal ufdbguardd to load the new URL database
21-40 exit code of ufdbUpdate is exit code of wget + 20. wget is the command that downloads the
new URL database from the servers of URLfilterDB.
41-60 exit code of ufdbUpdate is exit code of gunzip + 40. gunzip uncompresses the downloaded
URL database. There may be an issue with file system space.
61-80 exit code of ufdbUpdate is exit code of tar + 60. tar unpacks the downloaded URL database.
There may be an issue with file system space.
In case of an error, it is advised to run ufdbUpdate -v from the command line to have more
feedback about what is going wrong. License expiration warnings are also issued by ufdbguardd.
4.5.2 ufdbUpdate
ufdbUpdate downloads the URL database and obviously needs access to the servers of URLfilterDB.
Firewall rules may need to be modified to provide access to updates.urlfilterdb.com.
A proxy can be used to download the URL database: edit the /etc/sysconfig/ufdbguard
configuration file and assign the appropriate values to the variables http_proxy, PROXY_USER and
PROXY_PASSWORD.
3
ufdbGuard and URLfilterDB B.V. respects the privacy of all persons and statistics do not include, directly or
indirectly, any information about any person. See also the privacy policy of URLfilterDB B.V. at
http://www.urlfilterdb.com/privacystatement.html.
# Microsoft collects vast amounts of user and system data from workstations,
# browsers and apps. Define the Microsoft Data Collection category
category ms-data-collection {
domainlist “ms-data-collection/domains”
redirect “http://cgibin.urlfilterdb.com/cgi-bin/URLblocked.cgi?
category=%t&url=%u”
}
5 Configuration
This chapter and chapter 7 and 8 describe the features and configuration syntax for ufdbGuard in detail.
5.1 Infrastructure
Using a URL filter only contributes to a safer internet experience when the infrastructure is safe. The
configuration of the infrastructure is out of scope of this document but it is highly recommended to use a
DNS server that validates DNSSEC records, use a firewall that blocks incoming and outgoing traffic and
to use antivirus solutions.
Google has invented the QUIC protocol that is used by some browsers and some apps on smartphones.
Squid cannot proxy QUIC and it is recommended to configure the firewall to reject the QUIC protocol
on UDP ports 80 and 443 with an ICMP icmp-port-unreachable packet to signal applications that try to
use QUIC to abandon it immediately and revert to HTTP(S).
5.3.1 Recommendations
URL filters are used for various and often good reasons. We like to make clear that a URL filter is not
the solution for everything and that education of users of what is allowed and what is suspicious is
always necessary.
4
URLfilterDB reserves the right to block access to cgibin.urlfilterdb.com to any site that uses it
excessively.
The second section defines the IP address range of your local network. The section looks like this:
source allSystems {
ip 10.0.0.0/8
}
The appropriate network subnet must be entered in this section. 10.0.0.0/8 and 192.168.0.0/16
are the most common values for this. Consult your network administrator for assistance.
The third section defines the usage and settings of ufdbhttpd, the lightweight HTTP daemon:
http-server { port= 8080, interface= all, images=”/usr/local/ufdbguard/images” }
If you don not use ufdbhttpd, then transform the line above into a comment (with a #).
The fourth and fifth sections are close to each other and define the list of categories to be blocked (one
list for the systems with IP address defined in allSystems and one list for all other systems). Change
the list of categories to be blocked. The default list of blocked categories contains the categories
security, adult, p2p, proxies, gambling, violence and warez. The fourth section looks like this:
acl {
allSystems {
# EDIT THE NEXT LINE
pass !adult !p2p !proxies !gambling !violence !warez !security … any
}
To block a category, it needs to be present with an exclamation mark (!) that is used as a blocking
indicator. So to block the adult category, !adult must be present in the line that starts with pass. If
you prefer to allow gambling, the definition “!gambling“ must be removed.
NOTE: because of the large set of URLs used by chat applications, configuration of the chat category is
a little more complex. See section 8.8 for a detailed explanation.
At a site that only blocks security, adult, p2p and proxies, the section looks like this:
acl {
allSystems {
# EDIT THE NEXT LINE
pass !adult !p2p !proxies !security any
}
The fifth section is very similar to the fourth section and defines which categories to block for computer
systems that are not part of allSystems.
The configuration for the use of Skype and other chat applications is more complex. Read section 8.8.3
on how to allow or block Skype.
parameter value
mode default
noads
square
cross
simple-red
color white
black
grey
orange
red
size small
normal
large
redirection style parameters
6
See for example http://en.wikipedia.org/wiki/Web_Proxy_Autodiscovery_Protocol for more information on
WPAD.
7 Exception Rules
In cases where exceptions to the categories of URLfilterDB are desired, an administrator can define
2 extra categories that are managed by the administrator and never by URLfilterDB.
ufdbGuard only uses proprietary database files, so generate an .ufdb file from the ASCII file with
ufdbGenTable:
$ cd /var/ufdbguard
$ ufdbGenTable -W -n -t alwaysdeny -d blacklists/alwaysdeny/domains
The -W option removes the initial www. from all URLs. The above command generates the file
blacklists/alwaysdeny/domains.ufdb and should be invoked each time when the domains
file is changed. Then activate the category by editing the ufdbGuard.conf file and uncomment the
category definition for alwaysdeny. The configuration file should have the following lines:
category alwaysdeny
{
domainlist alwaysdeny/domains
redirect ...
}
Also, add the category alwaysdeny to the ACL allSystems. The ACL should then start with
pass alwaysallow !alwaysdeny !adult ...
7
Note that there is no leading www. since the URL filter works internally with stripped URLs that do not contain a
www. part. Use the -W option for files with URLs that have a www. prefix.
WARNING: although the performance of parameter matching is very good, performance can degrade
when 2 or more parameters are used inside URL tables and URLs have many parameters.
Due to technical limitations, individual parameter names and values cannot exceed 244 characters. If
exceeded, the parameter match fails for the particular parameter and ufdbguardd continues processing.
8 Advanced Options
acl {
# more privileged users first
dealingroom {
pass !adult !warez !security any
}
allSystems {
pass !adult !p2p !proxies !dating !entertain !warez !security any
}
acl {
# more privileged users first
dealingroom {
pass !adult !warez !security any
}
allSystems {
pass !adult !p2p !proxies !dating !entertain !warez !security any
}
default {
pass none
# the following redirect is for the pseudo category 'none'
redirect http://cgibin.urlfiterdb.com/cgi-bin/URLblocked.pl?...
}
}
Alternatively, one can also use a single IPv6 address or a list of IPv6 addresses.
Single IPv6 address:
ipv6 2001:db8:2::12:1001
IPv6 subnet:
ipv6 2001:db8::/32
List of IPv6 addresses:
ipv6list “<filename>”
CAVEAT: additional configuration is required to make Squid able to find out which users are using
Squid. You may configure Squid to use user authentication or you have to install identd on all PCs to
support this feature. Please read the Squid documentation for more information and do not forget to use
acl foo ident REQUIRED.
9
the option strip-domain-from-username was introduced in version 1.29.
CAVEAT: additional configuration is required to make Squid able to find out which users are using
Squid. You may configure squid to use user authentication or you have to install identd on all PCs to
support this feature. Please read the Squid documentation for more information and do not forget to use
acl foo ident REQUIRED.
CAVEAT: additional configuration is required to make Squid able to find out which users are using
Squid. You may configure Squid to use user authentication or you have to install identd on all PCs to
support this feature. Please read the Squid documentation for more information and do not forget to use
acl foo ident REQUIRED.
The list of usernames retrieved by execuserlist is retrieved again every 15 minutes to automatically pick
up changes in user groups. The refresh time (default 15 minutes) can be set using the
refreshuserlist keyword followed by a number representing the number of minutes for the
refresh interval. The interval must be between 5 and 1440 minutes. For example:
refreshuserlist 5
Debugging scripts which generates a list of usernames may be difficult and therefore ufdbguardd has an
option to debug and log the output generated by shell scripts. For example:
ufdb-debug-external-scripts on
The logfile of ufdbguardd will contain lines with execuserlist: received “foo” when the
debug option is on.
8.7 Whitelisting
Whitelisting is used in case that users are only allowed to visit a predefined set of websites. In this case,
the ACL for allSystems contains the categories alwaysallow and none. The ACL in the configuration
file looks like this:
acl {
allSystems {
pass alwaysallow none
# the following redirect is for the pseudo category 'none'
redirect “http://cgibin.urlfiterdb.com/cgi-bin/URLblocked.pl?...”
}
default {
pass none
redirect “http://cgibin.urlfiterdb.com/cgi-bin/URLblocked.pl?...”
}
}
Note that the extra redirect statement is required because the pseudo-category none is used.
8.8.2 Ebuddy
Ebuddy is a web-based chat application that provides chat, voice and video calls. Ebuddy does not have
additional features like file sharing that may violate a local Internet Usage Policy and is therefore
considered safer than other chat and VOIP applications.
Blocking Ebuddy is easy because one only needs to block a small set of URLs to block the whole
application. To block Ebuddy, the subcategory chat-ebuddy must be defined and the relevant ACL
must contain !chat-ebuddy to block Ebuddy.
The URL database has a subcategory for ebuddy called chat-ebuddy. The default definition is as
follows:
category chat-ebuddy {
domainlist “chat/ebuddy/domains”
expressionlist “chat/ebuddy/expressions”
redirect ...
}
To allow ebuddy chat, the ACL must include the chat-ebuddy category in front of the security
and proxies category. The ACL must be similar to the following:
acl {
allSystems {
pass alwaysallow !alwaysblock
chat-ebuddy
!proxies !security ...
8.8.3 Skype
Skype is a VOIP application that also can be used to transfer files and give remote access using screen
sharing. These additional features may violate the local Internet Usage Policy.
Blocking Skype is easy because one only needs to block a small set of URLs to block the whole
application. To block Skype, the subcategory chat-skype must be defined and the relevant ACL
must contain !chat-skype to block Skype.
Skype is a popular VOIP application that requires a more complex configuration because it uses the
HTTPS port 443 for its proprietary protocol, which would be blocked if the options for safer HTTPS are
used. Skype also uses IP addresses instead of hostnames, so Skype conflicts with the security options
enforce-https-with-hostname and enforce-https-official-certificate. The
URL database of URLfilterDB contains a subcategory for Skype with many URLs that Skype uses.
Since Skype connects to IP addresses of Skype users, it is not feasible to have all these IP addresses in
the URL database and therefore ufdbGuard dynamically detects the use of Skype.
An extra subcategory needs to be configured:
category chat-skype {
domainlist “chat/skype/domains”
expressionlist “chat/skype/expressions”
redirect ...
}
For those who want to use Skype and have a safer use of HTTPS, the option allow-skype-over-
https must be used. The category security must have this option set to ON:
category security {
option allow-skype-over-https on
...
}
Note: URLs are matched against categories in the order that they appear in an ACL and therefore the
category chat-skype must be placed before the categories security and proxies in the ACLs to
prevent that the security category blocks access to Skype URLs, e.g.:
acl {
allSystems {
pass alwaysallow !alwaysblock
chat-skype
!proxies !security ...
...
Skype must be configured to use Squid as its proxy: menu Options, tab Advanced, tab Connection,
HTTPS proxy.
And the security category must have the following option settings:
category security {
option allow-yahoomsg-over-https on
...
}
Note: URLs are matched against categories in the order that they appear in an ACL and therefore the
category chat-yahoo must be placed before the categories security and proxies in the ACLs to
prevent that the security category blocks access to Yahoo IM URLs, e.g.:
acl {
allSystems {
pass alwaysallow !alwaysblock
chat-yahoo chat-allowed
!proxies !security ...
...
Yahoo IM must be configured to use Squid as its proxy: menu Messenger, item Preferences, category
Connection, option connect via a proxy server, option HTTP proxy.
And the security category must have the following option settings:
category security {
option allow-fb-chat-over-https on
...
}
Note: URLs are matched against categories in the order that they appear in an ACL and therefore the
category chat-facebook must be placed before the categories security and proxies in the
ACLs to prevent that the security category blocks access to Facebook Chat URLs, e.g.:
acl {
allSystems {
pass alwaysallow !alwaysblock
chat-facebook chat-allowed
!proxies !security ...
...
And the security category must have the following option settings:
category security {
option allow-aim-over-https on
...
}
Note: URLs are matched against categories in the order that they appear in an ACL and therefore the
category chat-aim must be placed before the categories security and proxies in the ACLs to
prevent that the security category blocks access to AIM URLs, e.g.:
acl {
allSystems {
pass alwaysallow !alwaysblock
chat-aim chat-allowed
!proxies !security ...
...
AIM must be configured to use Squid as its proxy: menu Menu, item Settings, category Connection,
option connect using proxy and protocol HTTP/HTTPS.
And the security category must have the following option settings:
category security {
option allow-gtalk-over-https on
...
}
Note: URLs are matched against categories in the order that they appear in an ACL and therefore the
category chat-google must be placed before the categories security and proxies in the ACLs
to prevent that the security category blocks access to Google Talk URLs, e.g.:
acl {
allSystems {
pass alwaysallow !alwaysblock
chat-google chat-allowed
!proxies !security ...
...
Google Talk must be configured to use Squid as its proxy: menu Settings, category Connection, option
Use the following proxy.
Note: URLs are matched against categories in the order that they appear in an ACL and therefore the
category chat-msn must be placed before the categories security and proxies in the ACLs to
prevent that the security category blocks access to MSN URLs, e.g.:
acl {
allSystems {
pass alwaysallow !alwaysblock
chat-msn chat-allowed
!proxies !security ...
...
MSN must be configured to use Squid as its proxy: menu Tools, item Options, category Connection,
Advanced Settings, HTTP proxy server.
To allow all chat applications, the ACL must have the chat and chat-allowed categories in front
of the security and proxies categories and the security category must have the correct settings
for options that control the behaviour of ufdbguardd for HTTPS ports (port 443).
The default configuration file of ufdbguardd version 1.25 and higher have the following URL categories
defined.
category chat {
domainlist “chat/domains”
expressionlist “chat/expressions”
redirect ...
}
Various chat applications use the HTTPS port (port 443) but do not use URLs with a FQDN and do not
use SSL. Therefore the security category must have the following options set.
category security {
option allow-aim-over-https on
option allow-gtalk-over-https on
option allow-skype-over-https on
option allow-yahoomsg-over-https on
option allow-unknown-protocol-over-https on
...
}
8.10.2 ufdbgclient
ufdbgclient is the glue between Squid and ufdbguardd. By default, when ufdbgclient cannot
communicate with ufdbguardd, it allows all URLs, i.e. all URL filter requests of Squid are answered
with a message indicating that the URL is allowed. The only possible alternative to allowing all URLs
is to block all URLs. To block all URLs when ufdbguardd is not running or there is a fatal
communication error, ufdbgclient can be started with the “-e deny” option.
In case of a fatal error, ufdbgclient returns by default the URL http://cgibin.urlfilterdb.com/
cgi-bin/URLblocked.cgi?category=fatal-error which displays an appropriate message. To
overrule the default redirection URL for fatal errors, the -E option can be used which takes a valid URL
as its argument, e.g. -E http://example.com/error.html
The -e and -E options must be specified in the configuration file of Squid, squid.conf. E.g.:
url_rewrite_program /var/sbin/ufdbgclient -e deny
source “unix” {
Also file names, URL redirection strings, ACLs and parameters dbhome and logdir accept quoted
parameters. E.g.
logdir “/usr/local/ufdbguard”
time “working-hours” {
category “aggressive” {
domainlist “aggressive/domains”
expressionlist “aggressive/expressions”
redirect “...”
source “unix” {
8.16 Monitoring
ufdbGuard has two monitoring options: monitoring by email and monitoring by execution of an external
command. It is recommended to use these features since an error always occurs unexpectedly and often
needs a quick response.
ufdbGuard maintains a status of itself and whenever the status changes, it may send an email message to
a configurable email address and/or execute a configurable external command. The external command
can be any program or script. Usually it is a script that can send an appropriate command for a
monitoring tool. By default, ufdbguardd does not send emails and does not execute external commands.
The status values used in the emails and scripts are started, reloading, reloaded, fatal error, crash
report uploaded, crash report NOT uploaded and terminated.
In the rare event that ufdbGuard crashes, it calls the debugger gdb and produces a crash report which is
very useful in finding and resolving bugs. By default ufdbGuard uploads any crash report that it finds
when it is restarted. Crash reports reside in /tmp and the filename starts with “urlfilterdb.crashreport”.
Uploaded crash reports are renamed and the filename starts with “uploaded.urlfilterdb.crashreport”. To
prohibit the automatic upload of a crash report, use the following in the configuration file:
upload-crash-reports off
9 Performance Tuning
9.1 Web Proxy Infrastructure
For those organisations that use internal websites, it is recommended to configure and use PAC files that
instruct browsers to connect directly to internal servers bypassing Squid and a URL filter. This offloads
Squid and usually makes browsing more responsive.
Browsers itself have a cache. The default value for the size of the cache can be very large and it is
recommended to set it to a reasonable value, e.g. 100 MB. Note that smaller browser caches usually
make the browser faster due to less cache management. A very small cache may make the browser
inefficient.
NOTE: the memory allocator in the standard library on Linux has been reimplemented and has an
impact on the above recommendations. Starting with glibc 2.10, the memory allocator (malloc) uses on
64bit systems many memory segments of 64 MB. There is nothing wrong with the implementation
The unique hexdigit string has 4 substrings of 8 hexadecimal digits, separated by a minus sign, e.g.
f7792304-bbf0895f-46f4a2e2-dbd23e58.
There are many other tuning parameters for TCP connections and many websites contain advise on how
to change them, so they are not described here.
Category #URLs % KB %
Adult 1935 4.0 78204 15.7
Audio & video 128 0.3 58207 11.7
Advertisements 2449 5.0 2296 0.5
Chat 18 0.1 56 0.1
Dating & Personals 190 0.4 4425 0.9
Drugs 0 0.0 0 0.0
Entertainment 6379 13.0 59174 11.9
External Applications 0 0.0 0 0.0
Finance & Investment 1623 3.3 12780 2.6
Forums 152 0.3 817 0.2
Gambling 119 0.3 482 0.1
Games 413 0.9 2565 0.6
Illegal 0 0.0 0 0.0
Jobs 357 0.8 3042 0.6
News 2055 4.2 22224 4.5
Weblogs & private sites 882 1.8 5902 1.2
Peer-to-peer 0 0.0 0 0.0
Web Proxies 148 0.3 812 0.2
Religion 119 0.3 816 0.2
Security violations 0 0.0 0 0.0
Shops 3733 7.6 32542 6.5
Social Networks 0 0.0 0 0.0
Sports 813 1.7 6938 1.4
Toolbars 23 0.1 290 0.1
Travel 2586 5.3 21847 4.4
Hacking & warez 9 0.1 89 0.1
Violence & hate 0 0.0 0 0.0
Web-based email 105 0.3 809 0.2
The ufdbAnalyse command uses options to specify the logfile, the own domain, your email address
and your full name. The -l option may occur up to 64 times to produce a single aggregate report:
$ cd /usr/local/ufdbguard/bin
$ ufdbAnalyse –l .../access.log –d example.com –e “[email protected]” –n “John Doe”
10.4 Statistics
UfdbGuard prints in its log file total numbers of URL lookups, number of blocked URLs, numbers of
tunnels detected, number of enforced SafeSearches, number of times Youtube edufilter was applied, and
the total number of clients, and prints basic statistics about its rules. The statistics about the rules
contain for each category and for each source the number of times it was matched. The statistics are
printed every time that ufdbGuard receives a signal to reload the configuration (ufdbUpdate sends
such signal) or every 48 hours, whichever event comes first.
Example of statistics:
$ grep “ statistics: “ ufdbguardd.log
2016-06-20 22:01:09 [11845] statistics: 2407 URL lookups (0 https). 58 URLs blocked.
0 tunnels detected. 0 safe searches. 0 Youtube edufilter. 0 uncategorised URLs.
6 clients.
2016-06-20 22:01:09 [11845] statistics: category security was blocked 0 times
2016-06-20 22:01:09 [11845] statistics: category entertainment was blocked 31 times
2016-06-20 22:01:09 [11845] statistics: category proxies was blocked 2 times
2016-06-20 22:01:09 [11845] statistics: category adult was blocked 5 times
2016-06-20 22:01:09 [11845] statistics: category games was blocked 18 times
2016-06-20 22:01:09 [11845] statistics: category p2p was blocked 2 times
2016-06-20 22:01:09 [11845] statistics: source privileged_users was blocked 0 times
2016-06-20 22:01:09 [11845] statistics: source allSystems was blocked 58 times
2016-06-20 22:01:09 [11845] statistics: acl privileged_users-within-one-hour:
!security:0 !entertainment:0 !adult:0 !games:0 all:0
2016-06-20 22:01:09 [11845] statistics: acl privileged_users-else: !security:0
!proxies:0 all:0
2016-06-20 22:01:09 [11845] statistics: acl allSystems-within-working-hours:
!security:0 !proxies:2 !adult:5 games:18 p2p:2 !entertainment:31 !safesearch:0
all:0
2016-06-20 22:01:09 [11845] statistics: acl allSystems-else: !security:0 all:0
2016-06-20 22:01:09 [11845] statistics: acl default: none:0
proxy
API
Internet
URLfilterDB
The ufdbGuard API was developed for Telco's and system integrators and has a different license fee
table. Contact the support desk for additional information and licensing for 3 rd parties using the URL
database.
Ads
Websites with advertisements, user behavior monitors, traffic trackers and web page counters.
P2P
P2P stands for point-to-point file sharing. The P2P category contains websites that can be used directly
or indirectly to upload, download and share files.
Qmovies
Qmovies is short for Questionable Movies and contains websites that link to or contain movies and
series with questionable legal status.
Proxies
Sites that can be used to download content of other sites, URL rewriting sites and VPNs. Proxies are
commonly used in an attempt to circumvent a URL filter and should always be blocked. A subcategory
exists for teamviewer. Teamviewer is a popular remote access tool that gives full control to any
Adult
Websites suitable for adults only, including but not limited to pornographic content and obscene content.
Warez
Websites with illegal software, illegal software codes, hacker’s sites, warez and cracks.
Malware
Websites with malicious software. This category includes domains and IP addresses supplied by
abuse.ch.
Toolbars
Websites for toolbars of browsers. A toolbar is an extension to a web browser that may violate your
privacy or make private files public.
Illegal
Websites explaining how to perform Illegal activities.
Arms
Websites that sell, display or demostrate arms and weapons. Toy weapons that do not resemble real
weapons are excluded.
Violence
Websites about violent behavior.
Gambling
Websites offering gambling opportunities.
Drugs
Websites about hard drugs and soft drugs.
Webmail
Email accessible with a web browser. Webmail of business sites is not included. Webmail of ISPs is
included in this category.
Dating
Websites about love, dating, romantic poetry, and friendship.
Chat
Websites to use IRC and chat. Subcategories exist for AIM, Ebuddy, ICQ, Facebook Chat, Google
Talk, MSN Messenger, Oovoo, Skype, Telegram, Whatsapp and Yahoo Chat.
Forum
Websites where people exchange non-business information in a forum.
Private
Blogs and sites of private persons.
Web TV
TV and collections of video including Youtube.
Dailymotion
Dailymotion videos.
Vimeo
Vimeo videos.
Youtube
Youtube videos.
Audio-Video
Audio and video streams. Many entertainment sites are both in the category entertainment and in audio-
video (e.g. www.youtube.com).
Sports
Websites related to sports including sports sections of news sites, fans of sports, sites about actively
doing a sport.
Finance
Websites of banks and insurance companies.
Trading
Websites about stock markets and trading systems.
Jobs
Websites about and for job applications.
Games
Websites to play games and information about gaming.
Entertainment
Entertainment, lifestyle, hobby, arts, museums, fashion, electronic cards, magazines, horoscopes,
desktop wallpapers, clip art, photos, portals, events, fan sites, baby-related, child sites, picture sharing
and other sites for interest of private persons that are not related to business.
Food
Websites about restaurants and recipes. Fast food sites are in the category shops.
Health
Websites about medicine and health.
Housing
Websites about construction houses and sites where houses and apartments are offered for sale or rent.
Shops
Websites with shops, price comparisons, and auctions aimed at consumers (b2b is excluded).
Travel
Websites about travel agencies, airliners, tourism sites, hotels, holiday resorts.
News
Websites providing news and opinions.
External Applications
Free web-based document editors, spreadsheet applications, desktops, groupware, etc. where “internal”
documents can be stored on external servers.
Subcategories exist for CitrixOnline, Dropbox, iCloud and Telegram.
Search Engines
Websites that are a Web Search Engine.
Security
This is an administrative category to be able to enforce strict HTTPS usage options (see section 5.5).
Safe
Websites with content that is considered safe to access, including but not limited to antivirus sites, SSL/
TLS certificate verification, and some popular webfonts.
Social Networks
Sites that focuses on building and reflecting of social networks or social relations among people.
Subcategories exist for Badoo, Facebook, Google+ and Twitter.
Dynamic domains
Computer systems without a static address use dynamic addresses which are usually managed by
dynamic DNS servers (DDNS servers). DDNS is often used to gain remote access to computer systems
at home and can also be used as proxies.
Mozilla
Mozilla Firefox uses a range of IP addresses to connect to using the HTTPS protocol and may need to be
whitelisted with this URL category.
Chrome
The Chrome browser does at startup 3 queries to random non-existent webservers. This URL category
matches those queries.
Checked
URLs that are verified by URLfilterDB not to be part of any other category and hence always are
allowed by the URL filter. This category contains business sites, governmental sites and useful sites for
the general public. This administrative category is used by the URL filter to track uncategorized URLs
and users do not have to configure this category.
13 Privacy Policy
The privacy policy of URLfilterDB is stated on the website:
www.urlfilterdb.com/privacystatement.html.
14 More Information
More information can be found on the internet at the following addresses.
URLfilterDB www.urlfilterdb.com
Squid www.squid-cache.org