I've been trying to figure this out but all the documentation I can find seems to be very old. The feeling seems to be that once Dell acquired Sonicwall, documentation became sparse and mostly useless. Ended up here at StackExchange due to a few Google searches bringing me to conversations where it looked like people were doing this sort of thing recently and knew what they were doing.
This was the main piece of documentation that I was using for reference, but as you can see it dates to 2006 and is for SonicOS Enhanced 3. SonicwallOS Enhanced to Openswan Linux Using GroupVPN with XAUTH.
Here's the environment
Sonicwall (SonicOS Enhanced 5.9.0.2) - internet - Netgear router - Cisco integrated business router, series 881 (In the DMZ for the Netgear router) - Ubuntu 14.04 running Openswan.
LAN subnet on Sonicwall: 192.168.0.0/23
Lan behind Cisco: 192.168.100.0/24
Ubuntu private IP: 192.168.100.25
So where do I even start? I configured ipsec.conf file as described in that PDF:
conn TSBackup
left=192.168.100.25
leftsubnet=192.168.100.25/32
leftnexthop=%defaultroute
right=12.69.113.194
rightsubnet=192.168.0.0/23
rightnexthop=192.168.0.1
rightid=@0017C568BE38
auto=add
authby=secret
keyingtries=3
pfs=no
ike=3des-sha1;modp1024!
esp=3des-sha1
On the Sonicwall, there's a Site-to-Site defined with the following settings:
Policy type: Site to site
Auth method: IKE using Preshared Secret
IPsec Primary gateway: (public IP of Ubuntu server)
Local IKE ID: (Sonicwall's ID)
Peer IKE ID: ubuntuid
Local network: Lan Subnets
Remote Network: (Address object, pointing to Ubuntu's private address 192.168.100.25)
Phase 1: Main Mode, Group 2, 3DES-Sha1 28800
Phase 2: ESP, 3DES, SHA1, perfect forward secret not checked,28800
Enable Keep alive
Dead peer detection: 180, 3
VPN policy bound to Zone WAN
I don't even know where to look for logs. I've started over too, thinking maybe I goofed somewhere. Any help or advice is greatly appreciated, and my apologies for being green in all of this.
Results when initiating the connection:
104 "TSBackup" #1: STATE_MAIN_I1: initiate
003 "TSBackup" #1: ignoring unknown Vendor ID payload [5b362bc820f60008]
003 "TSBackup" #1: received Vendor ID payload [RFC 3947] method set to=115
106 "TSBackup" #1: STATE_MAIN_I2: sent MI2, expecting MR2
003 "TSBackup" #1: ignoring Vendor ID payload [Sonicwall 1 (TZ 170 Standard?)]
003 "TSBackup" #1: received Vendor ID payload [XAUTH]
003 "TSBackup" #1: received Vendor ID payload [Dead Peer Detection]
003 "TSBackup" #1: NAT-Traversal: Result using draft-ietf-ipsec-nat-t-ike (MacOS X): i am NATed
108 "TSBackup" #1: STATE_MAIN_I3: sent MI3, expecting MR3
003 "TSBackup" #1: next payload type of ISAKMP Hash Payload has an unknown value: 175
003 "TSBackup" #1: malformed payload in packet
010 "TSBackup" #1: STATE_MAIN_I3: retransmission; will wait 20s for response
003 "TSBackup" #1: ignoring informational payload, type INVALID_COOKIE msgid=00000000
003 "TSBackup" #1: received and ignored informational message
ipsec barf @ Pastebin
Edit: Updated ipsec.conf and SonicWall settings. Added results from opening connection.
Edit 2: addded link to ipsec barf.