SRX UTM: Web Filtering (Local)

Time to test the UTM Web Filtering feature.

The first thing to know is that there are 4 different ways to do this on the SRX.

1) Surfcontrol - This is know as Integrated Web Filtering and uses a cloud based Surfcontrol server to categorise URLs. This option requires a license and of course you can test with the 30 day trial license. The cloud based Surfcontrol server doesn't permit or deny anything. When queried by the SRX it provides a category that is permitted or denied by the policy you create on the SRX. The URL-to-category site results from the Surfcontrol server are cached locally which means subsequent requests for the same site don't require a lookup on the Surfcontrol server. This is a sensible approach as it means you dont have to download the whole database from Surfcontrol every time you start the SRX - it just requests the info from Surfcontrol on as as needed realtime basis.

Note: Surfcontrol is now owned by Websense though Juniper still name this option as Surfcontrol.

2) Enhanced Web Filtering - This is another type of Integrated solution which uses the Websense ThreatSeeker Cloud (TSC). Similar to Surfcontrol option above it provides a cattergory for the site to be checked but in addition also provides a reputation for the site.
A license is required for this option.

Juniper consider this option "next-generation URL filtering solution, building upon the existing SurfControl solution".

3) Websense - This is known Redirect Web Filtering and uses your local Websense server. This option does not require a licence. The main difference here to the Surfcontrol option is the the Websense server itself, through its own polices is doing the denying or permitting.

4) Local lists. This option uses local black and white lists to permit or deny access to URLs. No license required.

In this post I will configure and test the Local lists 

CONFIGURATION

a) Configure UTM URL Pattern Custom Objects

Lets start by trying to block access to Cisco and permitting access to Juniper.
So we stick Juniper and Cisco URLs in good and bad lists respectively

blogger@LEFTY# set security utm custom-objects url-pattern good-list value http://www.juniper.net
blogger@LEFTY# set security utm custom-objects url-pattern bad-list value http://www.cisco.com

Note: If you know the IPs of the sites you can add them too. Also wildcards * and ? can be used as well with some limitations on use.

b) Put the Custom Objects in Custom URL Categories

blogger@LEFTY# set security utm custom-objects custom-url-category good-category value good-list
blogger@LEFTY# set security utm custom-objects custom-url-category bad-category value bad-list

Lets commit that and see what it looks like...

blogger@LEFTY# run show configuration security utm 

custom-objects {
    url-pattern {
        good-list {
            value http://www.juniper.net;
        }
        bad-list {
            value http://www.cisco.com;
        }
    }
    custom-url-category {
        good-category {
            value good-list;
        }
        bad-category {
            value bad-list;
        }
    }

c) Configure the Web Filtering Feature Profile

blogger@LEFTY# set security utm feature-profile ?

Possible completions:
> anti-spam            Configure anti-spam feature
> anti-virus           Configure anti-virus feature
+ apply-groups         Groups from which to inherit configuration data
+ apply-groups-except  Don't inherit configuration data from these groups
> content-filtering    Configure content filtering feature
> web-filtering        Configure web-filtering feature
[edit]
blogger@LEFTY# set security utm feature-profile web-filtering ?
Possible completions:
+ apply-groups         Groups from which to inherit configuration data
+ apply-groups-except  Don't inherit configuration data from these groups
> juniper-enhanced     Configure web-filtering juniper enhanced engine
> juniper-local        Configure web-filtering juniper local engine
> surf-control-integrated  Configure web-filtering surf-control integrated engine
> traceoptions         Trace options for web-filtering feature
  type                 Configure web-filtering engine type
  url-blacklist        Configure custom URL for blacklist category
  url-whitelist        Configure custom URL for whitelist category
> websense-redirect    Configure web-filtering websense redirect engine

blogger@LEFTY# set security utm feature-profile web-filtering url-blacklist bad-category 

blogger@LEFTY# set security utm feature-profile web-filtering url-whitelist good-category   

blogger@LEFTY# set security utm feature-profile web-filtering type ?                             
Possible completions:
  juniper-enhanced     
  juniper-local        
  surf-control-integrated  
  websense-redirect    

blogger@LEFTY# set security utm feature-profile web-filtering type juniper-local

blogger@LEFTY# set security utm feature-profile web-filtering juniper-local ?

Possible completions:
+ apply-groups         Groups from which to inherit configuration data
+ apply-groups-except  Don't inherit configuration data from these groups
> profile              Juniper local profile

blogger@LEFTY# set security utm feature-profile web-filtering juniper-local profile wf-profile ?
Possible completions:
+ apply-groups         Groups from which to inherit configuration data
+ apply-groups-except  Don't inherit configuration data from these groups
  custom-block-message  Juniper local custom block message
  default              Juniper local profile default
> fallback-settings    Juniper local fallback settings
  timeout              Juniper local timeout (1..1800 seconds)

blogger@LEFTY# set security utm feature-profile web-filtering juniper-local profile wf-profile custom-block-message ">>NOT PERMITTED<<"      

blogger@LEFTY# set security utm feature-profile web-filtering juniper-local profile wf-profile fallback-settings ?         
Possible completions:
  <[Enter]>            Execute this command
+ apply-groups         Groups from which to inherit configuration data
+ apply-groups-except  Don't inherit configuration data from these groups
  default              Fallback default settings
  server-connectivity  Fallback action when device cannot connect to server
  timeout              Fallback action when connection to server timeout
  too-many-requests    Fallback action when requests exceed the limit of engine
  |                    Pipe through a command

blogger@LEFTY# set security utm feature-profile web-filtering juniper-local profile wf-profile fallback-settings too-many-requests ?                    
Possible completions:
  block              
  log-and-permit    

blogger@LEFTY# set security utm feature-profile web-filtering juniper-local profile wf-profile fallback-settings too-many-requests block 

blogger@LEFTY# set security utm feature-profile web-filtering juniper-local profile wf-profile fallback-settings default ? 

Possible completions:
  block                
  log-and-permit

blogger@LEFTY# set security utm feature-profile web-filtering juniper-local profile wf-profile fallback-settings default block

Lets commit that and see what it looks like...

blogger@LEFTY# run show configuration security utm feature-profile web-filtering 
url-whitelist good-category;
url-blacklist bad-category;
type juniper-local;
juniper-local {
    profile wf-profile {
        default log-and-permit;
        custom-block-message ">>NOT PERMITTED<<";
        fallback-settings {
            default block;
            too-many-requests block;
        }
    }
}

d) Apply the Web Filtering profile to the UTM Policy

blogger@LEFTY# set security utm utm-policy ?
Possible completions:
  <name>               UTM policy name
[edit]
blogger@LEFTY# set security utm utm-policy utm-protect ?
Possible completions:
> anti-spam            Anti-spam profile
> anti-virus           UTM policy anti-virus profile
+ apply-groups         Groups from which to inherit configuration data
+ apply-groups-except  Don't inherit configuration data from these groups
> content-filtering    Content-filtering profile
> traffic-options      Traffic options
> web-filtering        Web-filtering profile

blogger@LEFTY# set security utm utm-policy utm-protect web-filtering http-profile wf-profile

e) Add the UTM Policy to the Security Policy

blogger@LEFTY> show configuration security policies from-zone trust to-zone untrust 
policy trust-to-untrust {
    match {
        source-address any;
        destination-address any;
        application any;
    }
    then {
        permit {
            application-services {
                idp;
                utm-policy utm-protect;
            }
        }
    }
}

Note: we still have IDP running on the policy so these features ca co-exist.

Quite a few steps there to get this all going In summary:
* Create the custom objects - the URLs you want to permit or deny
* Apply the custom objects to custom URL categories
* Configure the web filtering feature profile which uses the custom URL categories
* Apply the web filtering feature profile to the UTM policy
* Apply the UTM policy to the security policy 

TESTING

First lets consider the order checking of the lists involved
1) Check black list - drop if a match is found
2) Check white list  - permit if a match is found
3) Check the default fallback action and apply as appropriate - If there is no fallback action then permit.

Test setup is as follows:
Trust side  - an XP PC host  
Untrust side - the Internet.

First check web filtering status

blogger@LEFTY> show security utm web-filtering status 
 UTM web-filtering status: 
    Server status: Juniper local URL filtering

Now check stats before doing anything.

blogger@LEFTY> show security utm web-filtering statistics 
 UTM web-filtering statistics: 
    Total requests:                  0
    white list hit:                  0
    Black list hit:                  0
    Web-filtering sessions in total: 4000
    Web-filtering sessions in use:   0
    Fallback:                       log-and-permit           block 
          Default                                 0               0
          Timeout                                 0               0
     Connectivity                                 0               0
Too-many-requests                                 0               0

Ok, now we will access both Cisco and Juniper sites and see what happens.

Juniper worked and we got this response in our browser when trying to access the Cisco site...



Lets check the stats...

blogger@LEFTY> show security utm web-filtering statistics     
 UTM web-filtering statistics: 
    Total requests:                  31
    white list hit:                  7
    Black list hit:                  2
    Web-filtering sessions in total: 4000
    Web-filtering sessions in use:   0
    Fallback:                       log-and-permit           block 
          Default                                 0               0
          Timeout                                 0               0
     Connectivity                                 0               0
Too-many-requests                                 0               0

Hits on the black and white lists as expected.

Now lets try the fallback option by accessing a site not in either list - www.checkpoint.com
It worked as the default option is log-and-permit.

Lets find that log then..

First I setup a log file with below config and then try again,

blogger@LEFTY> show configuration system syslog file WEB 
any any;
match WEBFILTER;
archive size 500k files 3 world-readable;

Well that just generated 19 lines of logging. Looks like its logging every single element as its downloaded from the page. Here is a few lines form the top...

blogger@LEFTY> show log WEB                                  
Aug 26 15:36:34 LEFTY clear-log[5796]: logfile cleared
Aug 26 15:36:54  LEFTY RT_UTM: WEBFILTER_URL_PERMITTED: WebFilter: ACTION="URL Permitted" 10.10.10.50(1398)->216.200.241.66(80) CATEGORY="N/A" REASON="BY_OTHER" PROFILE="wf-profile" URL=www.checkpoint.com OBJ=/ USERNAME=N/A ROLES=N/A
Aug 26 15:36:55  LEFTY RT_UTM: WEBFILTER_URL_PERMITTED: WebFilter: ACTION="URL Permitted" 10.10.10.50(1403)->184.87.21.68(443) CATEGORY="N/A" REASON="BY_OTHER" PROFILE="wf-profile" URL=184.87.21.68 OBJ=/ USERNAME=N/A ROLES=N/A
Aug 26 15:36:55  LEFTY RT_UTM: WEBFILTER_URL_PERMITTED: WebFilter: ACTION="URL Permitted" 10.10.10.50(1402)->216.200.241.66(80) CATEGORY="N/A" REASON="BY_OTHER" PROFILE="wf-profile" URL=www.checkpoint.com OBJ=/css/homepage2012.css USERNAME=N/A ROLES=N/A
Aug 26 15:36:57  LEFTY RT_UTM: WEBFILTER_URL_PERMITTED: WebFilter: ACTION="URL Permitted" 10.10.10.50(1404)->203.206.129.66(80) CATEGORY="N/A" REASON="BY_OTHER" PROFILE="wf-profile" URL=download.zonealarm.com OBJ=/bin/js/lib-plugins/jquery.cycle.all.2.72.js USERNAME=N/A ROLES=N/A

Ok,so know that logging permits is going to generate a hell of a lot of logs. So I am changing the default to just permit, clear the log and try Checkpoint again...

blogger@LEFTY# set security utm feature-profile web-filtering juniper-local profile wf-profile default permit 

And we check the log and the log is empty.

Lets try now changing the default to block and try Checkpoint again.

Here is what we see...



And here is the log...

blogger@LEFTY> show log WEB     
Aug 26 15:55:43 LEFTY clear-log[5872]: logfile cleared
Aug 26 15:55:50  LEFTY RT_UTM: WEBFILTER_URL_BLOCKED: WebFilter: ACTION="URL Blocked" 10.10.10.50(1473)->216.200.241.66(80) CATEGORY="N/A" REASON="BY_OTHER" PROFILE="wf-profile" URL=www.checkpoint.com OBJ=/ USERNAME=N/A ROLES=N/A

For comparison lets access Cisco (Still blocked by the Black list) again and see what the log shows...

blogger@LEFTY> show log WEB    
Aug 26 16:51:01 LEFTY clear-log[5931]: logfile cleared
Aug 26 16:51:47  LEFTY RT_UTM: WEBFILTER_URL_BLOCKED: WebFilter: ACTION="URL Blocked" 10.10.10.50(1479)->184.87.16.170(80) CATEGORY="bad-category" REASON="BY_BLACK_LIST" PROFILE="wf-profile" URL=www.cisco.com OBJ=/ USERNAME=N/A ROLES=N/A

So a couple of points here..
* Sites that get blocked seem to log even though there is no action specified in the config to log.
* Reason "BY_OTHER" refers to the default action taking place. Whereas if a site is blocked by the actual back list then the reason in the log is "BY_BLACK_LIST"

Lets now remove the white list from the feature profile and reset the default action to permit.

blogger@LEFTY# delete security utm feature-profile web-filtering url-whitelist 
blogger@LEFTY# set security utm feature-profile web-filtering juniper-local profile wf-profile default permit 

So the web filtering feature profile looks like this

blogger@LEFTY# run show configuration security utm feature-profile web-filtering                                 

url-blacklist bad-category;
type juniper-local;
juniper-local {
    profile wf-profile {
        default permit;
        custom-block-message ">>NOT PERMITTED<<";
        fallback-settings {
            default block;
            too-many-requests block;
        }
    }
}

Result - Checkpoint and Juniper (Which are now not in any  specific list) work and Cisco fails.
At the end of the day what combination of lists you choose to use depends on whether you are trying to explicitly permit or block.

The fallback settings come into play when the SRX can not process any more web filtering traffic.

Here, to finish up this post is the complete UTM Web Filtering Local config with colour highlights showing how all the different elements "glue" together.

blogger@LEFTY> show configuration security utm 

custom-objects {
    url-pattern {
        good-list {
            value http://www.juniper.net;
        }
        bad-list {
            value http://www.cisco.com;
        }
    }
    custom-url-category {
        good-category {
            value good-list;
        }
        bad-category {
            value bad-list;
        }
    }
}
feature-profile {
    web-filtering {
        url-blacklist bad-category;
        type juniper-local;
        juniper-local {
            profile wf-profile {
                default permit;
                custom-block-message ">>NOT PERMITTED<<";
                fallback-settings {
                    default block;
                    too-many-requests block;
                }
            }
        }
    }
}
utm-policy utm-protect {
    web-filtering {
        http-profile wf-profile;
    }

}

blogger@LEFTY> show configuration security policies 

from-zone trust to-zone untrust {
    policy trust-to-untrust {
        match {
            source-address any;
            destination-address any;
            application any;
        }
        then {
            permit {
                application-services {
                    idp;
                    utm-policy utm-protect;
                }
            }
        }
    }
}


Model: srx100h
JUNOS Software Release [12.1R3.5]

1 comment:

  1. I have tired following Juniper's configuration guides but they make it confusing. Your instructions were clear and it worked! thanks!

    ReplyDelete