Stunnel

[cah]

Verizon is implementing sendmail over ssl and they are changing from outgoing.verizon.net:25 to smtp.verizon.net:465.

Since sendmail doesn't have the capability of encrypting with SSL, stunnel is required.

I went to pkg.oracle.com and found the following stunnel package:
http://pkg.oracle.com/solaris/release/i ... 27T113739Z

Code: Select all

           Name: service/security/stunnel
        Summary: An SSL-encrypting socket wrapper
      Publisher: solaris
        Version: 4.29
  Build Release: 5.11
         Branch: 0.175.0.0.0.0.0
 Packaging Date: Tue Sep 27 11:37:39 2011
           Size: 460.41 kB
Compressed Size: 187.37 kB
           FMRI: pkg://solaris/service/security/stunnel@4.29,5.11-0.175.0.0.0.0.0:20110927T113739Z

License:
Copyright (c) 2011, Oracle and/or its affiliates.  All rights reserved.

For the avoidance of doubt, except that if any license choice other
than GPL or LGPL is available it will apply instead, Oracle elects to
use only the General Public License version 2 (GPLv2) at this time
for any software where a choice of GPL license versions is made
available with the language indicating that GPLv2 or any later
version may be used, or where a choice of which version of the GPL
is applied is otherwise unspecified.

--------------------------------------------------------------------

Licensing and copyright information for stunnel:

 License: GPLv2 License

 You can find more information in the following directory:
 /usr/share/doc/stunnel

 COPYRIGHT: GPLv2 
 AUTHORS: List of contributors of stunnel
 ChangeLog: List of all changes and authors of stunnel
 README: More information about stunnel

 The sources of stunnel-4.29 were downloaded
 from http://stunnel.mirt.net/

After downloading it, it didn't seem to work. I was trying to compile from source code but it didn't work so I went back to troubleshoot the stunnel issue. I finally found somewhere on the internet to put "debug = 7" and "foreground = yes" in /etc/stunnel/stunnel.conf. After starting it, I got to see what was the issue:

Code: Select all

2013.11.08 19:29:02 LOG7[18902:1]:  smtps  accepted FD=13 from 127.0.0.1:43240
2013.11.08 19:29:02 LOG7[18902:4]:  smtps  started
2013.11.08 19:29:02 LOG7[18902:4]: FD 13 in non-blocking mode
2013.11.08 19:29:02 LOG7[18902:4]: Waiting for a libwrap process
2013.11.08 19:29:02 LOG7[18902:4]: Acquired libwrap process #0
2013.11.08 19:29:02 LOG7[18902:4]: Releasing libwrap process #0
2013.11.08 19:29:02 LOG7[18902:4]: Released libwrap process #0
2013.11.08 19:29:02 LOG4[18902:4]:  smtps  REFUSED by libwrap from 127.0.0.1:43240
2013.11.08 19:29:02 LOG7[18902:4]: See hosts_access(5) manual for details
2013.11.08 19:29:02 LOG5[18902:4]: Connection reset: 0 bytes sent to SSL, 0 bytes sent to socket
2013.11.08 19:29:02 LOG7[18902:4]:  smtps  finished (0 left)

It was TCP Wrapper that is blocking it. Notice, it has extra space in front of and after smtps. That's because [ smtps ] was in stunnel.conf. I had to remove the space and make it [smtps] in stunnel.conf and add "smtps: localhost, 127.0.0.1" to /etc/hosts.allow (TCP Wrapper) and then it worked!

After I ran the following command, it connected to verizon!!

Code: Select all

%telnet relay 55555
Trying 127.0.0.1...
Connected to relay.
Escape character is '^]'.
220 vms173017pub.verizon.net -- Server ESMTP (Sun Java(tm) System Messaging Server 7u2-7.02 32bit (built Apr 16 2009))
...
Connection to relay closed by foreign host.

After 60 seconds idle time, the connection timed out and got closed.

From a separate terminal, I saw:

Code: Select all

2013.11.08 19:39:34 LOG7[18949:1]: smtps accepted FD=13 from 127.0.0.1:60269
2013.11.08 19:39:34 LOG7[18949:3]: smtps started
2013.11.08 19:39:34 LOG7[18949:3]: FD 13 in non-blocking mode
2013.11.08 19:39:34 LOG7[18949:3]: Waiting for a libwrap process
2013.11.08 19:39:34 LOG7[18949:3]: Acquired libwrap process #0
2013.11.08 19:39:34 LOG7[18949:3]: Releasing libwrap process #0
2013.11.08 19:39:34 LOG7[18949:3]: Released libwrap process #0
2013.11.08 19:39:34 LOG7[18949:3]: smtps permitted by libwrap from 127.0.0.1:60269
2013.11.08 19:39:34 LOG5[18949:3]: smtps accepted connection from 127.0.0.1:60269
2013.11.08 19:39:34 LOG7[18949:3]: FD 14 in non-blocking mode
2013.11.08 19:39:34 LOG6[18949:3]: connect_blocking: connecting 206.46.232.100:465
2013.11.08 19:39:34 LOG7[18949:3]: connect_blocking: s_poll_wait 206.46.232.100:465: waiting 10 seconds
2013.11.08 19:39:34 LOG5[18949:3]: connect_blocking: connected 206.46.232.100:465
2013.11.08 19:39:34 LOG5[18949:3]: smtps connected remote server from 192.168.1.225:51826
2013.11.08 19:39:34 LOG7[18949:3]: Remote FD=14 initialized
2013.11.08 19:39:34 LOG7[18949:3]: SSL state (connect): before/connect initialization
2013.11.08 19:39:34 LOG7[18949:3]: SSL state (connect): SSLv3 write client hello A
2013.11.08 19:39:34 LOG7[18949:3]: SSL state (connect): SSLv3 read server hello A
2013.11.08 19:39:34 LOG7[18949:3]: SSL state (connect): SSLv3 read server certificate A
2013.11.08 19:39:34 LOG7[18949:3]: SSL state (connect): SSLv3 read server key exchange A
2013.11.08 19:39:34 LOG7[18949:3]: SSL state (connect): SSLv3 read server done A
2013.11.08 19:39:34 LOG7[18949:3]: SSL state (connect): SSLv3 write client key exchange A
2013.11.08 19:39:34 LOG7[18949:3]: SSL state (connect): SSLv3 write change cipher spec A
2013.11.08 19:39:34 LOG7[18949:3]: SSL state (connect): SSLv3 write finished A
2013.11.08 19:39:34 LOG7[18949:3]: SSL state (connect): SSLv3 flush data
2013.11.08 19:39:34 LOG7[18949:3]: SSL state (connect): SSLv3 read finished A
2013.11.08 19:39:34 LOG7[18949:3]:    2 items in the session cache
2013.11.08 19:39:34 LOG7[18949:3]:    2 client connects (SSL_connect())
2013.11.08 19:39:34 LOG7[18949:3]:    2 client connects that finished
2013.11.08 19:39:34 LOG7[18949:3]:    0 client renegotiations requested
2013.11.08 19:39:34 LOG7[18949:3]:    0 server connects (SSL_accept())
2013.11.08 19:39:34 LOG7[18949:3]:    0 server connects that finished
2013.11.08 19:39:34 LOG7[18949:3]:    0 server renegotiations requested
2013.11.08 19:39:34 LOG7[18949:3]:    0 session cache hits
2013.11.08 19:39:34 LOG7[18949:3]:    0 external session cache hits
2013.11.08 19:39:34 LOG7[18949:3]:    0 session cache misses
2013.11.08 19:39:34 LOG7[18949:3]:    0 session cache timeouts
2013.11.08 19:39:34 LOG6[18949:3]: SSL connected: new session negotiated
2013.11.08 19:39:34 LOG6[18949:3]: Negotiated ciphers: DHE-RSA-AES256-SHA SSLv3 Kx=DH Au=RSA Enc=AES(256) Mac=SHA1
...
2013.11.08 19:40:34 LOG7[18949:3]: SSL alert (read): warning: close notify
2013.11.08 19:40:34 LOG7[18949:3]: SSL closed on SSL_read
2013.11.08 19:40:34 LOG7[18949:3]: Socket write shutdown
2013.11.08 19:40:34 LOG7[18949:3]: Socket closed on read
2013.11.08 19:40:34 LOG7[18949:3]: SSL write shutdown
2013.11.08 19:40:34 LOG7[18949:3]: SSL alert (write): warning: close notify
2013.11.08 19:40:34 LOG6[18949:3]: SSL_shutdown successfully sent close_notify
2013.11.08 19:40:34 LOG5[18949:3]: Connection closed: 0 bytes sent to SSL, 166 bytes sent to socket
2013.11.08 19:40:34 LOG7[18949:3]: smtps finished (0 left)

Now that stunnel is working, sendmail should not be a big problem.

/etc/stunnel/stunnel.conf:

Code: Select all

client = yes
# The following parameters are used for troubleshooting
#debug = 7
#foreground = yes

[smtps]
#accept=relay.hsiao.net:55555
accept=55555
connect=smtp.verizon.net:465

[cah]

Last Friday (02/07/2014), the server rebooted itself after OS detected disk issue and the server was down about 2 hours. I am not sure how it came back on its own.

Code: Select all

cahtoh02:/root%fmadm faulty   
--------------- ------------------------------------  -------------- ---------
TIME            EVENT-ID                              MSG-ID         SEVERITY
--------------- ------------------------------------  -------------- ---------
Feb 07 15:21:59 a08fb650-e301-c48c-d289-a8d0a420cb85  SUNOS-8000-KL  Major     

Host        : cahtoh02
Platform    : KQ496AA-ABA-a6530f        Chassis_id  : 3CR8201F63
Product_sn  : 

Fault class : defect.sunos.kernel.panic
Affects     : sw:///:path=/var/crash/.a08fb650-e301-c48c-d289-a8d0a420cb85
                  faulted but still in service
Problem in  : sw:///:path=/var/crash/.a08fb650-e301-c48c-d289-a8d0a420cb85
                  faulted but still in service

Description : The system has rebooted after a kernel panic.

Response    : The failed system image was dumped to the dump device.  If
              savecore is enabled (see dumpadm(1M)) a copy of the dump will be
              written to the savecore directory /var/crash.

Impact      : There may be some performance impact while the panic is copied to
              the savecore directory.  Disk space usage by panics can be
              substantial.

Action      : Use 'fmadm faulty' to provide a more detailed view of this event.
              If savecore is not enabled then please take steps to preserve the
              crash image. Use 'fmdump -Vp -u
              a08fb650-e301-c48c-d289-a8d0a420cb85' to view more panic detail.
              Please refer to the associated reference document at
              http://sun.com/msg/SUNOS-8000-KL for the latest service
              procedures and policies regarding this diagnosis.

--------------- ------------------------------------  -------------- ---------
TIME            EVENT-ID                              MSG-ID         SEVERITY
--------------- ------------------------------------  -------------- ---------
Apr 04 2012     3fc7fede-4705-cafe-f38f-83a5ee12e08e  ZFS-8000-D3    Major     

Host        : cahtoh02
Platform    : KQ496AA-ABA-a6530f        Chassis_id  : 3CR8201F63
Product_sn  : 

Fault class : fault.fs.zfs.device
Affects     : zfs://pool=rpool/vdev=ae0e345559e1e690
                  faulted but still in service
Problem in  : zfs://pool=rpool/vdev=ae0e345559e1e690
                  faulted but still in service

Description : A ZFS device failed.  Refer to http://sun.com/msg/ZFS-8000-D3 for
              more information.

Response    : No automated response will occur.

Impact      : Fault tolerance of the pool may be compromised.

Action      : Run 'zpool status -x' and replace the bad device.

--------------- ------------------------------------  -------------- ---------
TIME            EVENT-ID                              MSG-ID         SEVERITY
--------------- ------------------------------------  -------------- ---------
Feb 07 15:21:12 0bb9003a-4a4d-462f-b74a-886849b91a55  SMF-8000-YX    major     

Host        : cahtoh02
Platform    : KQ496AA-ABA-a6530f        Chassis_id  : 3CR8201F63
Product_sn  : 

Fault class : defect.sunos.smf.svc.maintenance
Affects     : svc:///application/pkg/server:default
                  faulted and taken out of service
Problem in  : svc:///application/pkg/server:default
                  faulted and taken out of service

Description : A service failed - a method is failing in a retryable manner but
              too often.

Response    : The service has been placed into the maintenance state.

Impact      : svc:/application/pkg/server:default is unavailable.

Action      : Run 'svcs -xv svc:/application/pkg/server:default' to determine
              the generic reason why the service failed, the location of any
              logfiles, and a list of other services impacted. Please refer to
              the associated reference document at
              http://sun.com/msg/SMF-8000-YX for the latest service procedures
              and policies regarding this diagnosis.

stunnel is a service and svcadm can be used to start, restart and stop this service. It should come up when the server boots up but it didn't.
Port 55555 was not listening and the service was disabled/

Code: Select all

%svcs -a | grep stunnel
disabled       Feb_07   svc:/network/ssl/stunnel:default

I had to manually start it:

Code: Select all

%svcadm enable svc:/network/ssl/stunnel:default

Then, port 55555 is listening:

Code: Select all

%portn 55555
      *.55555              *.*                0      0 128000      0 LISTEN

And mail was sent out.

/var/log/syslog entries before stunnel was turned on:

Code: Select all

Feb  9 09:21:23 cahtoh02 sendmail[8770]: [ID 801593 mail.info] s17MoGsh002026: to=sharon1133@hotmail.com,stephaniepeng@hotmail.com,sunnys@seed.net.tw,tingting54514@yahoo.com.tw,vampirestatic@yahoo.com.tw,yendow4411@kimo.com, ctladdr=<cah@hsiao.net> (1001/4), delay=1+15:31:07, xdelay=00:00:00, mailer=relay, pri=15570609, relay=relay.hsiao.net., dsn=4.0.0, stat=Deferred: Connection refused by relay.hsiao.net.

/var/log/syslog entries after stunnel was turned on:

Code: Select all

Feb  9 09:36:18 cahtoh02 sendmail[8901]: [ID 801593 mail.info] s17MoGsh002026: to=schueh1218@yahoo.com.tw,sd110531@yahoo.com.tw,sharon1133@hotmail.com,stephaniepeng@hotmail.com,sunnys@seed.net.tw,tingting54514@yahoo.com.tw,vampirestatic@yahoo.com.tw,yendow4411@kimo.com, ctladdr=<cah@hsiao.net> (1001/4), delay=1+15:46:02, xdelay=00:00:01, mailer=relay, pri=15660609, relay=relay.hsiao.net. [127.0.0.1], dsn=2.0.0, stat=Sent (Ok, envelope id 0N0Q00E66GKH3MF0@vms173025.mailsrvcs.net)

and mail queue was cleared.

I just need to figure out why it didn't start at boot time.

[cah]

I used svcprop to check the current property setting for stunnel:

Code: Select all

%svcprop stunnel                   
general/complete astring 
general/enabled boolean false
general/action_authorization astring solaris.smf.manage.stunnel
general/entity_stability astring Unstable
general/single_instance boolean true
general/value_authorization astring solaris.smf.value.stunnel
config/config_file astring /etc/stunnel/stunnel.conf
config/value_authorization astring solaris.smf.value.stunnel
filesystem/entities fmri svc:/system/filesystem/local
filesystem/grouping astring require_all
filesystem/restart_on astring none
filesystem/type astring service
loopback/entities fmri svc:/system/filesystem/local
loopback/grouping astring require_all
loopback/restart_on astring none
loopback/type astring service
net-physical/entities fmri svc:/network/physical
net-physical/grouping astring require_all
net-physical/restart_on astring none
net-physical/type astring service
cryptosvc/entities fmri svc:/system/cryptosvc
cryptosvc/grouping astring require_all
cryptosvc/restart_on astring none
cryptosvc/type astring service
manifestfiles/lib_svc_manifest_network_ssl_stunnel_xml astring /lib/svc/manifest/network/ssl/stunnel.xml
startd/duration astring contract
start/exec astring /usr/bin/stunnel\ %{config/config_file}
start/timeout_seconds count 60
start/type astring method
stop/exec astring :kill
stop/timeout_seconds count 60
stop/type astring method
tm_common_name/C ustring stunnel\ as\ a\ \ daemon
tm_man_stunnel8/manpath astring /usr/share/man
tm_man_stunnel8/section astring 8
tm_man_stunnel8/title astring stunnel
tm_doc_stunnel_org_website/name astring stunnel.org\ website
tm_doc_stunnel_org_website/uri astring http://www.stunnel.org
restarter/contract count 5466
restarter/start_pid count 8861
restarter/start_method_timestamp time 1391956516.844574000
restarter/start_method_waitstatus integer 0
restarter/logfile astring /var/svc/log/network-ssl-stunnel:default.log
restarter/auxiliary_state astring none
restarter/next_state astring none
restarter/state astring online
restarter/state_timestamp time 1391957657.561933000
restarter_actions/auxiliary_tty boolean true
restarter_actions/auxiliary_fmri astring svc:/network/ssh:default
restarter_actions/refresh integer

"general/enabled boolean false" caught my attention.
I checked on sendmail and it has 'true" value in this parameter.
I think I just need to make the change to this parameter.

Code: Select all

%svccfg -s svc:/network/ssl/stunnel:default setprop general/enabled=true

At this moment, the property has been updated but the running configuration still shows 'false'.

Code: Select all

%svcprop -p general/enabled stunnel                             
false

The service configuration has to be re-read before the running configuration will show the new value.

Code: Select all

%svcadm refresh svc:/network/ssl/stunnel:default 

After re-reading the service configuration, it shows the new value.

Code: Select all

%svcprop -p general/enabled stunnel                             
true

A reboot confirms that change works.