Rev. Tig posted the following information on connecting Smoothwall and m0n0wall via IPsec VPN in a post on the mailing list on September 30, 2004. It also applies to SmallWall.
I could not find a working solution in the mailing list archives but here is how I have managed to create a VPN between Smoothwall Corporate with Smoothtunnel and m0n0wall and I thought I would share it here to same people going through the same headbashing experience I did :) This will be far to much of a teaching granny to suck eggs for most people on the list but it might help someone get up and running quickly. Variety is the spice of life and just to confuse matters the m0n0wall box was stuck behind NAT :) The office I was linking to was in a serviced building and hence the connection was a shared one with a private IP and public one port forwarded to it. I had never done this before so corrections are welcome :) I am not saying these are the best settings all I know is my VPN is up and running and it seems to be happy :) What I have created is a VPN between one subnet at one site running Smoothwall Corporate Server 3.0 with Smoothtunnel and a m0n0wall v1 box sitting behind NAT with a private IP at the other site. Any other versions of the software may need slightly different settings but hopefully this should put you in the right ballpark. First off IPSEC over NAT, if at all possible don't :) If you have to or for some perverse reason you fancy a crack at this then read on, if you are just here for the Smoothwall bit scroll down :) IPSEC over NAT does work but it can be a case of sacrificing the odd network card to the deity of your choice, what I did in the end was ask their network guy to just send everything and I will let m0n0 do the firewalling, this is what I would recommend as then you don't have to hassle them every time you want a port opening, but from what I have gathered is that all you need are port 500 forwarding and IP protocols 50 and 51 to be routed but the firewall. Apparently your IPSEC traffic goes through port 500 but IP protocols 50 and 51 are needed for phase 1 (authentication) and phase 2 (key exchange). If I am wrong (this is quite possible there will be a load of mails below correcting me :) If m0n0 is behind NAT and you are certain the other end is right but there appears to be no attempts to authenticate then check here first. Now onto Smoothwall Corporate, now I know Rich Morrell posts on here so I have to be careful about what I say about the interface but that is just a personal taste thing :) Right here are the Smoothwall settings : Local IP : your RED IP address (if you are using Smoothhost then put the IP of your firewall in) Local ID type: Local IP Remote IP : the external IP of your NATted m0n0wall box. Remote ID type : Remote IP Authenticate by : Preshared Key Preshared Key : put your shared key here Use Compression : Off Enabled : On Local network : in this case it was 192.168.0.0/255.255.255.0 Local ID value : same as your Local IP Remote network: in this case it was 192.168.1.0/255.255.255.0 Remote ID value : the same as your Remote IP Initiate the connection : Yes I will use these networks in this example as it shows you a little gotcha in m0n0wall that threw me because I was not thinking :) Next block : Local Certificate : (your local certificate) Perfect Forward Secrecy : Yes Authentication type: ESP (it has to be AH will NOT work over NAT) Phase 1 crypto algo: 3DES Phase 1 hash algo : MD5 Key life : 480 (mins) Key tries : 0 (never give up) Right now the m0n0wall settings : Phase 1: Mode : tunnel (well you can't change it and why would you want to :) Interface : WAN Local Subnet : 192.168.1.0 / 24 (don't do what I did and select LAN :) Remote Subnet : 192.168.0.0 / 24 Remote IP : The RED IP of your Smoothwall box Negotiation Mode : Main My Identifier : IP Address : Your public IP (non NATed) for your m0n0wall box Encryption Algo: 3DES Hash Algo : MD5 DH Key Group : 5 Lifetime : (blank) Preshared Key : put your shared key here. Phase 2: Protocol : ESP Encryption Algo: 3DES (only! untick the others) Hash Algo: MD5 (again only) PFS Key Group : 5 Lifetime : (blank) That is it, your can now bring the link up from Smoothwall by going into the VPN control tab and clicking UP!