Page History

...

...

Install required (and used, throughout this documentation) packages, possibly replacing vim with your $EDITOR of choice (e.g. emacs-nox or nano, both of which also support syntax highlighting, which helps when editing XML files) and stop the automatically started tomcat until we've completed more configuration performed further below:

apt install --no-install-recommends default-jdk-headless tomcat9 \
  vim less openssl unzip curl expat multitail gnupg net-tools

systemctl stop tomcat9

...

On the IDP server create an RSA private key and CSR for the web server's TLS certificate, e.g.:First, create and note down a random passphrase to use for protecting/encrypting the private key at rest. (Use this passphrase in all the steps below when asked for a key passphrase or import/export password.)

openssl reqrand -new -newkey rsa:2048 -nodes -out webserver.csr -keyout webserver.key -new

Request a TLS certificate based on the CSR generated, e.g. from the ACOnet TCS. In the resulting email with the certificate there's also the CA chain file to use (e.g. called DigiCertCA.crt). Copy the certificate and CA file to the server you're installing the IDP on (where the private key should already be, having been generated there).

Convert the TLS/SSL keypair into PKCS12

hex 16

On the IDP server create an RSA private key of at least 2048 bits size and the CSR for the web server's TLS certificate, supplying the necessary data (at least the subject) on the command line or by entering any data interactively when being prompted for it (when not adding -subj to the command)Create and note down a random password for your PKCS#12 keystore that will hold the above created TLS/SSL keypair plus the CA chain file:

openssl randreq -hex 16

Convert the TLS certificate you recieved from your CA (i.e., from DigiCert, if using ACOnet TCS), the locally generated private key and the certificate chain file into one password-protected PKCS#12 keystore file. When being asked for an "export password" set the previously generated (and noted down) password. Below you'll also add that password to the Tomcat server configuration:

openssl pkcs12 -export -in webserver.crt -inkey webserver.key -certfile DigiCertCA.crt -name "webserver" -out webserver.p12

Move the newly created keystore to its final location (we're chosing Tomcat's config directory) and set strict file system permissions on it:

new -newkey rsa:2048 -out webserver.csr -keyout webserver.key -subj "/CN=WEBSERVER-FQDN"

When asked to "Enter pass phrase for webserver.key" provide the passphrase from the previous step.

openssl pkcs12 -in

 /etc/tomcat9/webserver.p12

Configure Tomcat Connector

...

 -nocerts | tail +5 > webserver.key

...

openssl req -new -key webserver.key -out webserver.csr -subj "/CN=WEBSERVER-FQDN"

Equipped with the CSR you can now request a TLS certificate based from your CA, e.g. using the ACOnet TCS supplier. Once the certificate has been issued copy it to the IDP server as webserver.crt.
You'll also need to copy any intermediate Certificate Authority (CA) certificates to the IDP server.

Convert the TLS/SSL keypair into PKCS12

Copy the private key, new TLS certificate and any intermediate CA certificates into a PKCS#12 keystore file:

openssl pkcs12 -export -in webserver.crt -inkey webserver.key -certfile GEANT-OV-RSA-CA-4.crt -name "webserver" -out webserver.p12

When asked to "Enter pass phrase for webserver.key" provide the passphrase generated earlier.
Again when asked to "Enter Export Password".
And yet again, when asked to "Verifying - Enter Export Password".

Move the newly created keystore to its final location (we're chosing Tomcat's config directory) and set strict file system permissions on it:

[[ -f /etc/tomcat9/webserver.p12 ]] && cp -a /etc/tomcat9/webserver.p12 /etc/tomcat9/webserver.p12.`date -u +%Y%m%d`
mv webserver.p12 /etc/tomcat9/
chown root:tomcat /etc/tomcat9/webserver.p12
chmod 640 /etc/tomcat9/webserver.p12

Configure Tomcat Connector

Remove or comment out all other Connectors in /etc/tomcat9/server.xml, then add the two Connectors as per below, replacing keystorePass with the password generated earlier:

<!-- Localhost-only connector for IDP command line tools -->
<Connector address="127.0.0.1" port="80" />
 
<!-- https://tomcat.apache.org/tomcat-9.0-doc/ssl-howto.html -->
<Connector
  port="443"
  protocol="org.apache.coyote.http11.Http11NioProtocol"
  maxThreads="150"
  maxPostSize="100000"
  SSLEnabled="true"
  scheme="https"
  secure="true"
  clientAuth="false"
  sslProtocol="TLS"
  sslEnabledProtocols="TLSv1.2,TLSv1.3"
  keystoreType="pkcs12"
  keystoreFile="/etc/tomcat9/webserver.p12"
  keystorePass="see above">
  <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
</Connector>

Start Tomcat, check for listening ports, and access https://webserver-fqdn/foo which should result in an HTTP Status 404 error (since /foo won't exist) but allows you to confirm a hopefully valid TLS/SSL webserver configuration:

systemctl restart tomcat9
netstat -lntp | fgrep java  # should show 443, and 80 only on the loopback interface

Verify TLS/SSL

Next validate the TLS/SSL configuration on the system itself with openssl (and quit again with ctrl-c or by typing QUIT into the prompt):

openssl s_client -CApath /etc/ssl/certs/ -connect webserver-fqdn:443 </dev/null

Look for "Certificate chain" in the output from that command, e.g.

openssl s_client -CApath /etc/ssl/certs/ -connect webserver-fqdn:443 2>&1 </dev/null | grep -A8 "^Certificate chain"

and verify that it looks something like the "Certificate chain" presented below. The Subject of cert 0 will obviously differ, and depending on your choice of CA or certificate product the CA or certificate chain may also be different. A correct chain (and therfore PKCS#12 keystore) for TLS usage should contain all the certificates up until but excluding the root CA certificate. I.e, in the example below the certificate with CN=USERTrust RSA Certification Authority is not included in the chain sent from the server (but must be known by the web browser):

Certificate chain
 0 s:C = AT, postalCode = 1010, ST = Wien, L = Wien, street = Universitaetsstrasse 7, O = ACOnet, CN = idp.aco.net
   i:C = NL, O = GEANT Vereniging, CN = GEANT OV RSA CA 4
 1 s:C = NL, O = GEANT Vereniging, CN = GEANT OV RSA CA 4
   i:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority

In case of errors check the output of "journalctl -u tomcat9 -ef".

If everything works fine and the certificate chain looks as expected you can remove the private key and certificate again (as both can be extracted from the PKCS#12 keystore if needed), keeping the CSR in file webserver.csr around for next time you need to renew that certificate (as long as you still consider the matching private key secure):

rm webserver.{key,crt}

Tune log file creation

IDP logs

You might prefer to have the IDP application write its logs to a more standard file system location in the file system, specifically one outside the application's own directory and on a file system that where data usage is expected to grow dynamically (e.g. on /var). To do that simply set the idp.logfiles property in any of the property files read by the IDP, e.g. within conf/idp.properties:

idp.logfiles=/var/log/shibboleth

We also have to create that directory. And in order for the example commands in this documentation to work with either log directory we'll remove the (still empty) log dir created by the IDP installer and replace it with a symlink to the actual log directory:

install -o tomcat -g root -m 0750 -d /var/log/shibboleth/
cd /opt/shibboleth-idp/ && rmdir logs && ln -s /var/log/shibboleth logs

Tomcat logs

By default Tomcat logs everything multiple times, including

<!-- Localhost-only connector for IDP command line tools -->
<Connector address="127.0.0.1" port="80" />
 
<!-- https://tomcat.apache.org/tomcat-9.0-doc/ssl-howto.html -->
<Connector
  port="443"
  protocol="org.apache.coyote.http11.Http11NioProtocol"
  maxThreads="150"
  maxPostSize="100000"
  SSLEnabled="true"
  scheme="https"
  secure="true"
  clientAuth="false"
  sslProtocol="TLS"
  keystoreType="pkcs12"
  keystoreFile="/etc/tomcat9/webserver.p12"
  keystorePass="see above" />

Start Tomcat, check for listening ports, and access https://webserver-fqdn/foo which should result in an HTTP Status 404 error (since /foo won't exist) but allows you to confirm a hopefully valid TLS/SSL webserver configuration:

systemctl restart tomcat9
netstat -lntp | fgrep java  # should show 443, and 80 only on the loopback interface

Verify TLS/SSL

Next validate the TLS/SSL configuration on the system itself with openssl (and quit again with ctrl-c or by typing QUIT into the prompt):

openssl s_client -CApath /etc/ssl/certs/ -connect webserver-fqdn:443

Look for "Certificate chain" in the output from that command, e.g.

openssl s_client -CApath /etc/ssl/certs/ -connect webserver-fqdn:443 2>&1 </dev/null | grep -A8 "^Certificate chain"

and verify that it looks something like the "Certificate chain" presented below. The Subject of cert 0 will obviously differ, and depending on your choice of CA or certificate product ("SSL Plus", "Unified Communications", "EV" etc.) the CA may also be different. A correct chain (and therfore PKCS#12 keystore) for TLS usage should contain all the certificates up until but excluding the root CA certificate. I.e, in the example below the certificate with CN=DigiCert Assured ID Root CA is not included in the chain sent from the server:

Certificate chain
 0 s:/C=AT/ST=Vienna/L=Wien/O=ACOnet/CN=webserver-fqdn
   i:/C=NL/ST=Noord-Holland/L=Amsterdam/O=TERENA/CN=TERENA SSL CA 3
 1 s:/C=NL/ST=Noord-Holland/L=Amsterdam/O=TERENA/CN=TERENA SSL CA 3
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Assured ID Root CA

In case of errors check the output of "journalctl -u tomcat9 -ef".

Tune log file creation

By default Tomcat logs additional (and duplicate) events to /var/log/tomcat9/catalina.$date.logout and /var/log/tomcat9/localhost.$date.log*, which we don't care for. So let's create a backup copy of Tomcat's logging.properties and replace its content with the minumum needed to get an access log comparable to Apache httpd in /var/log/tomcat9/access.log. .properties and replace its content with the minumum needed to getTomcat's stdout/stderr will go to the systemd journal to the console (which ends up in the systemd journal in our configuration). To prevent catalina.out from being created we deacticate it further below (in our "Systemd service" override) by setting the CATALINA_OUT=/dev/null environment variable for the java process.

systemctl stop tomcat9
cp -a /etc/tomcat9/logging.properties /etc/tomcat9/logging.properties.orig
 
echo -n 'handlers = org.apache.juli.FileHandler, java.util.logging.ConsoleHandler
org.apache.juli.FileHandler.level = SEVERE
org.apache.juli.FileHandler.rotatable = false
org.apache.juli.FileHandler.directory = /dev
org.apache.juli.FileHandler.prefix = null
org.apache.juli.FileHandler.suffix =/logging.properties /etc/tomcat9/logging.properties.`date -u +%Y%m%d`
 
echo -n 'handlers = java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level = INFO
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatterorg.apache.juli.SystemdFormatter
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].level = INFO
.[Catalina].[localhost].handlers = java.util.logging.ConsoleHandler
' > /etc/tomcat9/logging.properties

Then comment out or delete the whole Valve element at the end of your /etc/tomcat9/server.xml, and replace it with the following one:

<Valve className="org.apache.catalina.core.ContainerBase.[Catalina].[localhost].handlers = org.apache.juli.FileHandler
' > /etc/tomcat9/logging.properties

Then comment out or delete the whole Valve element at the end of your /etc/tomcat9/server.xml, and replace it with the following one:

<Valve className="org.apache.catalina.valves.AccessLogValve" maxLogMessageBufferSize="320"
       prefix="access" suffix=".log" renameOnRotate="true" pattern="combined" />

Then delete all Tomcat logs now and after a bit only access.log should have been generated:

rm -f /var/log/tomcat9/*
systemctl restart tomcat9
ls -l /var/log/tomcat9/
multitail /var/log/tomcat9/* -l 'journalctl -u tomcat9.service -f'  # exit with 'q'
systemctl stop tomcat9

Systemd service

valves.AccessLogValve" maxLogMessageBufferSize="320"
       prefix="access" suffix=".log" renameOnRotate="true" pattern="combined" />

After deleting all Tomcat logs (only) access.log should be generated in Tomcat's log directory going forward:

rm -f /var/log/tomcat9/*
systemctl restart tomcat9
ls -l /var/log/tomcat9/
multitail /var/log/tomcat9/* -l 'journalctl -u tomcat9.service -f'  # exit with 'q'
systemctl stop tomcat9

If you're certain there's no catalina.log file being generated anymore you can also disable the default logrotate config snippet for it:

sed -i 's/^/#/' /etc/logrotate.d/tomcat9

Systemd service

Debian's Tomcat comes with an almost-usable systemd service that needs to be amended in order to

1. Avoid the systemd-house-of-horror that's still all too common with Tomcat/Java packaging
2. Avoid slow startup times due to use of a blocking /dev/random (cf. Myths about urandom also linked from the Shib wiki).
3. Allow the IDP application to write logs and metadata to the filesystem as needed
4. Try avoiding the creation of catalina.out (we already have its content in journald using this configuration)

And since Debian 10's Tomcat comes with an almost-usable systemd service that needs to be amended in order to (1) avoid the systemd-house-of-horror that's still all too common with Tomcat packaging, and (b) allow the IDP application to write logs and metadata to the filesystem. Since we're creating an override for the system-supplied systemd service unit anyway we'll also set the maximum memory usage there (to 3GB in the example "-Xmx3g" in the example below) – adjust as needed (, i.e., 3GB).
Adjust as needed, but 3-4GB should be sufficient ), also leaving even for large metadata aggregates (as are common with Interfederation). Also leave a bit of RAM for the OS. (Not that you should be running anything else on an IDP server.)

install -o root -g root -m 0755 -d /etc/systemd/system/tomcat9.service.d

cat <<'EOF' > /etc/systemd/system/tomcat9.service.d/override.conf
[Service]
Environment="CATALINA_OUT=/dev/null"
Environment="JAVA_OPTS=-Djava.security.egd=file:/dev/urandom -Djava.awt.headless=true -Xmx3g"
Environment="JSSE_OPTS=-Djdk.tls.ephemeralDHKeySize=2048"
ExecStart=
ExecStart=/usr/bin/java \
  $JAVA_OPTS $JSSE_OPTS \
  -classpath ${CATALINA_HOME}/bin/bootstrap.jar:${CATALINA_HOME}/bin/tomcat-juli.jar \
  -Dcatalina.base=${CATALINA_BASE} \
  -Dcatalina.home=${CATALINA_HOME} \
  -Djava.util.logging.config.file=${CATALINA_BASE}/conf/logging.properties \
  -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager \
  -Djava.io.tmpdir=${CATALINA_TMPDIR} \
  org.apache.catalina.startup.Bootstrap
ReadWritePaths=/opt/shibboleth-idp/logs/
.startup.Bootstrap
ReadWritePaths=/var/log/shibboleth/
ReadWritePaths=/opt/shibboleth-idp/logs/
ReadWritePaths=/opt/shibboleth-idp/metadata/
EOF

(If you've set "idp.logfiles=/var/log/shibboleth" via the IDP's property files as described in section "IDP logs" above you can remove the line ReadWritePaths=/opt/shibboleth-idp/

...

logs/. The above example config just makes sure IDP logs can get written using either log location.)

Activate the override with systemctl daemon-reload, maybe also verify with systemd-delta | fgrep tomcat

...