<div><br></div><div dir="auto">(almost verbatim copy of my previous email — in case anybody else needs it, or has ideas for improvements :-)</div><div dir="auto"><br></div><div dir="auto"><font size="2" face="Arial" style="font-family:Arial;color:rgb(0,0,0)">The way I would do this is to install "haproxy" on all these nodes, and have haproxy terminate SSL and balance incoming requests over the 3 CES-addresses. For S3 -- we only need to provide access to the swift port at 8080.</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Arial" style="font-family:Arial;color:rgb(0,0,0)"><br><br>First install haproxy:</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br><br># yum install haproxy</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Arial" style="font-family:Arial;color:rgb(0,0,0)"><br><br>Put your cert and key into /etc/haproxy/ssl.pem:</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br><br><br># cat server.key server.crt cert-chain.crt > /etc/haproxy/ssl.pem</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br># chmod 400 /etc/haproxy/ssl.pem<br></font><font size="2" face="Arial" style="font-family:Arial;color:rgb(0,0,0)"><br>Then create a /etc/haproxy/haproxy.cfg:</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br><br># cat <<'EOF' > /etc/haproxy/haproxy,cfg</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>#---------------------------------------------------------------------</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br># Global settings</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>#---------------------------------------------------------------------</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>global</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>log 127.0.0.1 local2</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>chroot /var/lib/haproxy</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>pidfile /var/run/haproxy.pid</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>maxconn 4000</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>user haproxy</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>group haproxy</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>daemon</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>tune.ssl.default-dh-param 2048</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br><br># turn on stats unix socket</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>stats socket /var/lib/haproxy/stats</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>#--------------------------------------------------------------------- <br># common defaults that all the 'listen' and 'backend' sections will</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br># use if not designated in their block</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>#---------------------------------------------------------------------</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>defaults</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>mode http</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>log global</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>option httplog</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>option dontlognull</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>option http-server-close</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>option forwardfor except <a href="http://127.0.0.0/8">127.0.0.0/8</a></font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>option redispatch</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>retries 3</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>timeout http-request 10s</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>timeout queue 1m</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>timeout connect 10s</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>timeout client 1m</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>timeout server 1m</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>timeout http-keep-alive 10s</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>timeout check 10s</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>maxconn 3000</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br><br>listen stats *:80</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>maxconn 10</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>timeout client 100s</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>timeout server 100s</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>timeout connect 100s</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>timeout queue 100s</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br><br>stats enable</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>stats hide-version</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>stats refresh 30s</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>stats show-node</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>stats auth admin:password</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>stats uri /haproxy?stats</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br><br>frontend s3-frontend</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>bind *:443 ssl crt /etc/haproxy/ssl.pem</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>default_backend s3-backend</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br><br>backend s3-backend</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>balance roundrobin</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>server ces1 <a href="http://10.33.23.167:8080">10.33.23.167:8080</a> check</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>server ces2 <a href="http://10.33.23.168:8080">10.33.23.168:8080</a> check</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>server ces3 <a href="http://10.33.23.169:8080">10.33.23.169:8080</a> check</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br>EOF</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br><br># systemctl enable haproxy</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Courier New" style="font-family:"Courier New";color:rgb(0,0,0)"><br># systemctl start haproxy</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Arial" style="font-family:Arial;color:rgb(0,0,0)"><br><br><br>You only need to modify the IP-addresses in the s3-backend (make sure they point at your floating CES addresses, not the static ones), and maybe make a better username/password for the stats page at "</font><a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__hostname_haproxy-3Fstats&d=DwMGaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=C3SCgx_FF-wJZebP7c5CEG8K2j7pCg2LdZP4SFMyCQI&m=ZCneR00GhPGRUqmpEvPdSEl4EHMUA6j_5WdnrnYvqXk&s=0UY3Rjy-1AOANEsyvosHlOCpzys6e6UL8PTBpQeJ-Rg&e=" style="font-family:UICTFontTextStyleBody;font-size:17px"><font size="2" face="Arial" style="font-family:Arial;color:rgb(0,0,255)"><u style="font-family:Arial">http://hostname/haproxy?stats</u></font></a><font size="2" face="Arial" style="font-family:Arial;color:rgb(0,0,0)">".</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><font size="2" face="Arial" style="font-family:Arial;color:rgb(0,0,0)"><br><br><br>This setup does two layers of loadbalancing. First DNS round-robin, then haproxy roundrobin -- I don't think this is a negative thing, and it does make it possible to tune the loadbalancing further with more advanced algorithms for selecting backends. F.ex. we can do weighting, if some are more powerful than others. Or "leastconn", to send new requests to backends with the least number of active connections.</font><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> </font><br></div><div dir="auto"><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"><br></font></div><div dir="auto"><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"><br></font></div><div dir="auto"><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"><br></font></div><div dir="auto"><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"> -jf</font></div><div dir="auto"><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"><br></font></div><div dir="auto"><font size="3" face="Calibri" style="font-family:Calibri;color:rgb(0,0,0)"><br></font></div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">tor. 7. mai 2020 kl. 14:59 skrev Andi Christiansen <andi@christiansen.xxx>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><div dir="auto">Hi Christian,<div><br></div><div>Thanks for answering!</div><div><br></div><div>We solved this with lab services a while back now, and ended up setting up haproxys I front of the ces nodes and then they handle the ssl encryption to the S3 API</div><div><br></div><div>Thanks</div><div>Andi Christiansen<br><br><div dir="ltr">Sendt fra min iPhone</div><div dir="ltr"><br><blockquote type="cite">Den 7. maj 2020 kl. 12.08 skrev Christian Vieser <<a href="mailto:christian.vieser@1und1.de" target="_blank">christian.vieser@1und1.de</a>>:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr">
</div></blockquote></div></div><div dir="auto"><div><blockquote type="cite"><div dir="ltr">
<pre style="font-family:monospace">Hi Andi,</pre>
<pre style="font-family:monospace">up to now there are no instructions available on how to enable SSL on the Swift/S3 endpoints.</pre>
<pre style="font-family:monospace">The only thing is that you can enable SSL on the authentication path. So your connection to Swift authentication on port 35357 will be secured and the S3 authentication arriving at http port 8080 will internally take the SSL path, if configured properly. We have successfully done that in a test environment. Be sure to use the --pwd-file option with the "mmuserauth service create ..." and verify the proxy settings afterwards. It should look like this:</pre>
<pre style="font-family:monospace"># mmobj config list --ccrfile proxy-server.conf --section filter:s3token
[filter:s3token]
auth_uri = https:<span style="font-family:monospace">//<a href="http://127.0.0.1:35357/" target="_blank" style="font-family:monospace">127.0.0.1:35357/</a>
</span>use = egg:swift3#s3token
insecure = <span style="font-family:monospace">true
You can correct wrong settings with
</span># mmobj config change --ccrfile proxy-server.conf --section filter:s3token --property insecure --value <span style="font-family:monospace">true</span>
# mmobj config change --ccrfile proxy-server.conf --section filter:s3token --property auth_uri --value <span style="font-family:monospace">'https:<span style="font-family:monospace">//<a href="http://127.0.0.1:35357/" target="_blank" style="font-family:monospace">127.0.0.1:35357/</a>'
Regards,
Christian
</span></span><span style="font-family:monospace"></span></pre>
<pre style="font-family:monospace">> i have tried what you suggested. mmobj swift base ran fine. but after i have
> deleted the userauth and try to set it up again with ks-ssl enabled it just
> hangs:
>
> # mmuserauth service create --data-access-method object --type local
> --enable-ks-ssl
>
> still waiting for it to finish, 15 mins now.. :)
>> Basically all i need is this:
>>
>> <a rel="nofollow" href="https://s3.something.com:8080" target="_blank" style="font-family:monospace">https://s3.something.com:8080</a> <a rel="nofollow" href="https://s3.something.com:8080" target="_blank" style="font-family:monospace">https://s3.something.com:8080</a> which points
>> to the WAN ip of the CES cluster (already configured and ready)
>>
>> and endpoints like this:
>>
>> None | keystone | identity | True | public | <a rel="nofollow" href="https://cluster_domain:5000/" target="_blank" style="font-family:monospace">https://cluster_domain:5000/</a>
>> <a rel="nofollow" href="https://cluster_domain:5000/" target="_blank" style="font-family:monospace">https://cluster_domain:5000/</a>
>> RegionOne | swift | object-store | True | public |
>> <a rel="nofollow" href="https://cluster_domain:443/v1/AUTH_%" target="_blank" style="font-family:monospace">https://cluster_domain:443/v1/AUTH_%</a>(tenant_id)s
>> RegionOne | swift | object-store | True | public |
>> <a rel="nofollow" href="https://cluster_domain:8080/v1/AUTH_%" target="_blank" style="font-family:monospace">https://cluster_domain:8080/v1/AUTH_%</a>(tenant_id)s</pre>
<span>_______________________________________________</span><br><span>gpfsug-discuss mailing list</span><br><span>gpfsug-discuss at <a href="http://spectrumscale.org" target="_blank">spectrumscale.org</a></span><br><span><a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a></span><br></div></blockquote></div></div>_______________________________________________<br>
gpfsug-discuss mailing list<br>
gpfsug-discuss at <a href="http://spectrumscale.org" rel="noreferrer" target="_blank">spectrumscale.org</a><br>
<a href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" rel="noreferrer" target="_blank">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</a><br>
</blockquote></div></div>