Chapter 8. IPsec

Table of Contents

8.1. Preface
8.1.1. Features
8.1.2. Site to Site VPN Explained
8.1.3. Remote Access IPsec VPN
8.1.4. Tunnel Mode
8.1.5. Perfect Forward Secrecy
8.1.6. IPsec Software Clients
8.2. Special Features
8.2.1. Dead Peer Detection
8.2.2. Dynamic DNS Support
8.2.3. NAT Traversal
8.2.4. IPsec Traffic Filtering
8.3. Prerequisites
8.4. Configuring the VPN Tunnel
8.5. Possible Issues
8.5.1. What if your SmallWall is not the main Internet Firewall?
8.5.2. Additional Questions
8.6. Quick Start for RSA Signature Authentication

This chapter will go over configuring a site to site Virtual Private Network (VPN) links between two SmallWalls, discuss how to configure site to site links with third party IPsec-compliant devices and discuss VPN to remote IPSec client software. Once you have IPSec properly configured you will be able to take advantage of the following capabilities:

The Example VPN Configurations chapter goes over, in detail, how to configure site to site IPsec links with some third party IPsec devices. Although it might seem confusing, in most cases you just need to assure that all of the parameters match on both sides (except of course the definition of who is the remote network). Some routing issues might come up depending on your situation but reading the rest of this chapter should be enough to successfully use IPsec encryption.

If you have gotten SmallWall working in a site to site IPsec configuration with some third party IPsec device that is not already listed, we would appreciate if you could put together a short write up of how you got it configured, preferably with screen shots where applicable.

8.1. Preface

IPsec (IP security) is an international standard for providing security to IP protocols via encryption and/or authentication, typically employing both. Its use in SmallWall is for Virtual Private Networks (VPN's). After two or more points securely authenticate each other's identification, access rights, and how to encrypt data (phase 1), they will be able to communicate using encrypted data packets (phase 2). The two points can be on a local network, a wireless network or even on the Internet.

There are two general types of IPsec VPN capabilities in SmallWall, site to site and remote access. Site to site will connect entire networks while remote access allows mobile users access to secured networks.

8.1.1. Features

The IPsec specification includes many features and services. Below is a list of IPsec features, including features not currently supported by selected SmallWall versions.

Table 8.1. IPSec Feature List

Feature     1.8
Site to site     x
Mobile user to site     x
Tunnel mode     x
Transport mode      
Perfect Forward Security (PFS)     x
Main Mode     x
Aggressive Mode     x
Remote gateway hostname/domain support     x
IKEv2 support      
Phase 1 local IP, Domain, FQDN Identifier     x
Phase 1 local RSA Cert Subject Identifier     x
Phase 1 Authentication Hashes md5, sha1 support     x
Phase 1 Authentication Hashes tiger192, ripemd160 support      
Phase 1 Authentication Preshared Key support     x
Phase 1 Authentication RSA / PKI X.509 Certificate support     x
Phase 1 Authentication DSA Certificate support      
XAUTH Authentication      
Phase 2 Diffie-Hellman Key support 768, 1024, 1536 bit (also Modp)     x
Phase 2 Diffie-Hellman Key support 2048, 3072, 4096 bit (also Modp)      
Encryption Ciphers DES,3DES, Blowfish, CAST128     x
Encryption Cipher AES (Rijndael)     x
Encryption Ciphers Twofish, Serpent, IDEA      
NAT-T Traversal     x
Dead Peer Detection     x
IPSec diagnostic logs     x
Dynamic DNS remote site support     x
IPSec Traffic filtering      
DHCP over IPSec      
L2TP Authentication     x
Manual Key support     x
Certificate Revocation List      

8.1.2. Site to Site VPN Explained

Site to site VPN's connect two locations with static public IP addresses and allow traffic to be routed between the two networks. This is most commonly used to connect an organization's branch offices back to its main office, so branch users can access network resources in the main office. Prior to VPN's, much more expensive private Wide Area Network (WAN) links like frame relay, point to point T1 lines, etc. were commonly used for this functionality. Some organizations are moving towards VPN links between sites to take advantage of reduced costs.

Site to site VPN's can also be used to link your home network to a friend's home network, to provide access to each other's network resources without opening holes in your firewalls.

While site to site VPN's are a good solution in many cases, private WAN links also have their benefits. IPsec adds processing overhead, and the Internet has far greater latency than a private network, so VPN connections are typically slower (while maybe not throughput-wise, they at least have much higher latency). A point to point T1 typically has latency of around 4-8 ms, while a typical VPN connection will be 30-80+ ms depending on the number of hops on the Internet between the two VPN endpoints.


When deploying VPN's, you should stay with the same ISP for all sites if possible, or at a minimum, stay with ISP's that use the same backbone provider. Geographic proximity usually has no relation to Internet proximity. A server in the same city as you but on a different Internet-backbone provider could be as far away from you in Internet distance (hops) as a server on the other side of the continent. This difference in Internet proximity can make the difference between a VPN with 30 ms latency and one with 80+ ms latency.

8.1.3. Remote Access IPsec VPN

SmallWall provides three means of remote access VPN, PPTP, L2TPP and Mobile IPsec. SmallWall's mobile IPsec functionality has some serious limitations that hinder its practicality for many deployments.

For most situations, PPTP or L2TP is probably the best remote access VPN option in SmallWall right now. See the PPTP or L2TP chapter for more information.

8.1.4. Tunnel Mode

IPsec's Tunnel mode is supported on SmallWall devices. This mode allows secured communication between entire subnets. When the packet leaves the subnet it will be encrypted, when it gets to the remote IPSec device the packets are decrypted and routed/ sent into the remote network.

The IPsec Specification supports a 2nd mode of operation called Transport mode. Transport mode limits encrypted communication to the 2 devices that are encrypting the information. If this was supported it would only allow secured communication to the SmallWall device itself and not to its connected networks. Transport mode is not supported.

8.1.5. Perfect Forward Secrecy

This option increases security during authentication by assuring that new keys (which are generated on a regular basis to ensure security) are not based on previous keys. When activated, this means that if someone obtains or discovers 1 encryption key that they cannot use it to discover previous or future keys. This can be disabled to allow faster key negotiation.

8.1.6. IPsec Software Clients

Most operating systems include IPsec clients. Windows 2000 and above includes a free IPsec client but it is also difficult to configure. MacOSX 10.3 and later also includes a free IPsec client but the free configuration tool is only for L2TP/IPsec. Free configuration tools exist for both operating systems but commercial solutions, at least for Windows, are more evolved and easier to use than the built-in free version.

Below is a list of IPsec software clients.


In some versions of Microsoft Windows, you must deactivate the built-in IPsec client before installing a commercial 3rd party IPsec client. Be sure to read the installation instructions.