I could not find a working solution in the mailing list archives but
here is how I have managed to create a VPN between Smoothwall Corporate
with Smoothtunnel and m0n0wall and I thought I would share it here to
same people going through the same headbashing experience I did :) This
will be far to much of a teaching granny to suck eggs for most people on
the list but it might help someone get up and running quickly.

Variety is the spice of life and just to confuse matters the m0n0wall
box was stuck behind NAT :) The office I was linking to was in a
serviced building and hence the connection was a shared one with a
private IP and public one port forwarded to it.

I had never done this before so corrections are welcome :) I am not
saying these are the best settings all I know is my VPN is up and
running and it seems to be happy :)

What I have created is a VPN between one subnet at one site running
Smoothwall Corporate Server 3.0 with Smoothtunnel and a m0n0wall v1
box sitting behind NAT with a private IP at the other site. Any other
versions of the software may need slightly different settings but
hopefully this should put you in the right ballpark.

First off IPSEC over NAT, if at all possible don't :) If you have to
or for some perverse reason you fancy a crack at this then read on, if
you are just here for the Smoothwall bit scroll down :)

IPSEC over NAT does work but it can be a case of sacrificing the odd
network card to the deity of your choice, what I did in the end was ask
their network guy to just send everything and I will let m0n0 do the
firewalling, this is what I would recommend as then you don't have to
hassle them every time you want a port opening, but from what I have
gathered is that all you need are port 500 forwarding and IP protocols
50 and 51 to be routed but the firewall. Apparently your IPSEC traffic
goes through port 500 but IP protocols 50 and 51 are needed for phase 1
(authentication) and phase 2 (key exchange). If I am wrong (this is
quite possible there will be a load of mails below correcting me :) If
m0n0 is behind NAT and you are certain the other end is right but there
appears to be no attempts to authenticate then check here first.

Now onto Smoothwall Corporate, now I know Rich Morrell posts on here so
I have to be careful about what I say about the interface but that is
just a personal taste thing :)

Right here are the Smoothwall settings :

Local IP : your RED IP address (if you are using Smoothhost then put
the IP of your firewall in)
Local ID type: Local IP
Remote IP : the external IP of your NATted m0n0wall box.
Remote ID type : Remote IP
Authenticate by : Preshared Key
Preshared Key : put your shared key here
Use Compression : Off
Enabled : On
Local network : in this case it was 192.168.0.0/255.255.255.0
Local ID value : same as your Local IP
Remote network: in this case it was 192.168.1.0/255.255.255.0
Remote ID value : the same as your Remote IP
Initiate the connection : Yes

I will use these networks in this example as it shows you a little
gotcha in m0n0wall that threw me because I was not thinking :)

Next block :
Local Certificate : (your local certificate)
Perfect Forward Secrecy : Yes
Authentication type: ESP (it has to be AH will NOT work over NAT)
Phase 1 crypto algo: 3DES
Phase 1 hash algo : MD5
Key life : 480 (mins)
Key tries : 0 (never give up)

Right now the m0n0wall settings :

Phase 1:
Mode : tunnel (well you can't change it and why would you want to :)
Interface : WAN
Local Subnet : 192.168.1.0 / 24 (don't do what I did and select LAN :)
Remote Subnet : 192.168.0.0 / 24
Remote IP : The RED IP of your Smoothwall box
Negotiation Mode : Main
My Identifier : IP Address : Your public IP (non NATed) for your
m0n0wall box
Encryption Algo: 3DES
Hash Algo : MD5
DH Key Group : 5
Lifetime : (blank)
Preshared Key : put your shared key here.

Phase 2:
Protocol : ESP
Encryption Algo: 3DES (only! untick the others)
Hash Algo: MD5 (again only)
PFS Key Group : 5
Lifetime : (blank)

That is it, your can now bring the link up from Smoothwall by going
into the VPN control tab and clicking UP!