Request #173200 "Machine stuck in provisioni..." created

Welcome to Joyent Cloud Customer Support

Stay updated with announcements, get answers from the community and share your feature suggestions with us.
You can also submit a request or send us an email at support@joyent.com.

 

Joyent Support

Jason S. December 30, 2014 Announcements / Announcements

This advisory only applies to customers with on-prem SmartDataCenter 7 installations.

Joyent has identified a potential issue with the rabbitmq messaging zone; in extremely rare cases rabbit can lose it's state across restart, resulting in older messages being re-sent. This patch ensures that rabbit starts with a clean state database in the event it is restarted.

This bug only affects rabbitmq images older than 15-Dec-2014; newer images contain this fix already.

To check if you are vulnerable:

  1. Log into your HN.

  2. Log into the rabbit zone: sdc-login rabbit

  3. Check for HEAD-2187 in the ouput from grep HEAD-2187 /opt/local/sbin/rabbitmq-server

  4. If you return a line reading # Ensure we don't keep state across restarts, per HEAD-2187, your installation has been patched. If not, you will need to follow the procedure below to patch rabbit:

Patch procedure:

  1. Log into the HN.

  2. Download the patch file: curl -k https://us-east.manta.joyent.com/joyentsup/public/SWSUP-69.patch -o /zones/$(sdc-vmname rabbit)/root/root/SWSUP-69.patch

  3. Log into the rabbit zone: sdc-login rabbit

  4. Verify the patch file exists in root's home directory: ls -l /root/SWSUP-69.patch

  5. Make a backup copy of the rabbit configuration: cp /opt/local/sbin/rabbitmq-server /opt/local/sbin/rabbitmq-server.SWSUP-69

  6. Patch the rabbit configuration file: patch /opt/local/sbin/rabbitmq-server < /root/SWSUP-69.patch

    [root@251f8aee-09f7-4a49-b30d-9d7adae2bb8d (mxpa:rabbitmq0) ~]# patch /opt/local/sbin/rabbitmq-server < /root/SWSUP-69.patch patching file /opt/local/sbin/rabbitmq-server

  7. Diff the files to ensure the change is included:

    [root@251f8aee-09f7-4a49-b30d-9d7adae2bb8d (mxpa:rabbitmq0) ~]# diff /opt/local/sbin/rabbitmq-server /opt/local/sbin/rabbitmq-server.SWSUP-69 26,28d25 < # Ensure we don't keep state across restarts, per HEAD-2187 < rm -rf /var/db/rabbitmq/* <

  8. The patch is now applied.

  9. Important You do not need to and should not restart rabbit (either the zone or the service) after applying this patch.

Please contact Joyent Support via your normal support channels if you have any questions, comments, or concerns regarding this. Thank you.

Joyent

Elizabeth November 21, 2014 Announcements / Current System Status

This notice is to advise Joyent Public Cloud and Smart Data Center customers of the recently identified Kerberos Checksum Vulnerability (CVE-2014-6324) for anyone using Microsoft Windows Server 2003 SP2, Windows Vista SP2, Windows Server 2008 SP2 and R2 SP1, Windows 7 SP1, Windows 8, Windows 8.1, and Windows Server 2012 Gold and R2.

If you are running a Windows VM, your environment may allow remote authenticated domain users to obtain domain administrator privileges via a forged signature in a ticket. For more information on this vulnerability as well as steps you can take to mitigate CVE-2014-6324, please see: 

https://technet.microsoft.com/library/security/ms14-068

https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-6324

If you have any questions or concerns about  CVE-2014-6324 please raise a ticket at https://help.joyent.com or by email to support@joyent.com.

Richard October 15, 2014 Announcements / Current System Status

This notice is to advise Joyent Public Cloud and Smart Data Center customers of the recently identified SSL 3.0 security issue CVE-2014-3566 (POODLE).

Your environment can be vulnerable to the unauthorized capture of data when using SSL 3.0. This article explains POODLE in detail.

Eliminating the Risk

We recommend customers disable SSL 3.0 in order to completely protect themselves against this vulnerability. Customers should ensure TLS 1.0 or greater is enabled if it is not already. Please review ZMap's server page for details on how to disable SSLv3 for common servers. For disabling SSL 3.0 in Chrome, FireFox, and IE please refer to ZMap's browsers page.

What Steps is Joyent Taking?

Joyent has analyzed this vulnerability and have decided that with effect from Friday, October 17, 2014 we will be disabling SSL 3.0 on all our web sites and API's. These sites include the following:

How can you check if you are vulnerable?

You can test for this vulnerability using either this client test link or Qualys Labs' server test link.

 

If you have any questions or concerns about POODLE please raise a ticket at https://help.joyent.com or by email to support@joyent.com.

Sean G. October 14, 2014 Announcements / Current System Status

Joyent Cloud Firewall Advisory

Joyent engineers have discovered an issue with Cloud Firewall and KVM.

If you are not running KVM (Linux or Windows) instances or if you have not enabled Cloud Firewall, this vulnerability will not affect you at all.

If you are running one or more KVM instances with Cloud Firewall enabled, please note that Cloud Firewall does not currently handle IPv6. Both Linux and Windows enable IPv6 by default, so if you are using Linux or Windows VMs and have blocked ports using Cloud Firewall, those ports may still be open to other Linux or Windows VMs in the same data center, over the IPv6 link-local address. These ports are not accessible across Joyent Cloud data centers, and they are not accessible over the Internet.

Determining Whether a Service is Affected

Linux

Use the command ip -6 addr show to see if any interfaces are configured with IPv6 addresses. If there are no addresses listed, then the instance is not affected by this issue.

Use the netstat -l6 command to check for services that may be exposed:

root@3ed4c538-dadc-477b-9892-ddfa68433ca4:~# netstat -l6
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp6       0      0 [::]:ssh                [::]:*                  LISTEN
udp6       0      0 [::]:9434               [::]:*
udp6       0      0 2001:db8::2:ntp         [::]:*
udp6       0      0 localhost:ntp           [::]:*
udp6       0      0 fe80::92b8:d0ff:fe4:ntp [::]:*
udp6       0      0 [::]:ntp                [::]:*
udp6       0      0 [::]:33207              [::]:*

Any services that are listed on [::] or addresses beginning with fe80:: are not adequately protected by Cloud Firewall alone. Services listening on localhost are not affected by this issue.

Windows

Use the command ipconfig /all | findstr IPv6 to see if any interfaces are configured with IPv6 addresses.

From command prompt or power shell, use the netstat -nap IPv6 command. Any services that are listed on [::] or addresses beginning with fe80:: (link-local), 2002 (6to4 tunnel) or 2001 (Global address or Toredo tunnel).

Mitigating Exposure of Services

Linux

Linux can be protected by one or more of the following methods:

  • Configure services to only listen on IPv4 addresses.
  • Disable IPv6 completely.
  • Use ip6tables to filter all IPv6 traffic.
  • Use ip6tables to filter individual ports.

Configure services to only listen on IPv4 addresses

In general, applications can be configured to listen to specific IPv4 addresses, or to 0.0.0.0. This will force the application to exclude IPv6. Consult the application documentation for specific details, or contact the application vendor and/or authors.

Disable IPv6 Completely

Add the following line to /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6=1

And execute the following command:

sysctl -w net.ipv6.conf.all.disable_ipv6=1

Use the following command to verify that IPv6 has been disabled:

ip -6 addr show

If IPv6 has been properly disabled then no addresses will be listed.

Use ip6tables to Filter all IPv6 Traffic

Note that ip6tables rules do not persist reboots. You will need to execute the rules after every boot. The iptables-persistent package can help to ensure that packet filter rules are automatically loaded at boot time. Consult the iptables-persistent documentation for further information.

Execute the following:

ip6tables -P INPUT DROP

Use ip6tables to Filter Individual Ports

Note that ip6tables rules do not persist reboots. You will need to execute the rules after every boot. The iptables-persistent package can help to ensure that packet filter rules are automatically loaded at boot time. Consult the iptables-persistent documentation for further information.

To filter TCP port 80, execute the following. Substitute port 80 for the port you wish to filter:

ip6tables -I INPUT -p tcp --dport 80 -j DROP

To filter UDP port 123, execute the following. Substitute port 123 for the port you wish to filter:

ip6tables -I INPUT -p udp --dport 123 -j DROP

 

Windows

Note: Windows Firewall is enabled by default and does not allow traffic to IPv6 by default. Nevertheless, you may wish to verify Windows Firewall settings or disable IPv6.

Questions or Concerns

If any unanswered questions or concerns happen to arise while following the instructions above, please contact Joyent Support by emailing support@joyent.com or by submitting a request via https://help.joyent.com/home. If any additional necessary information comes to light, we will update this announcement accordingly.

Elizabeth September 24, 2014 Announcements / Current System Status

This notice is to advise all Joyent Public Cloud (JPC) and SmartDataCenter (SDC) customers of the recently-identified bash security vulnerability CVE-2014-6271 (http://seclists.org/oss-sec/2014/q3/649) and the follow-on CVE-2014-7169 (https://access.redhat.com/security/cve/CVE-2014-7169), collectively known as Shellshock.

Note that CVE-2014-7169 has arisen due to incomplete fixes created for the CVE-2014-6271 vulnerability. (These fixes are created by the upstream maintainers of bash, not by Joyent.)

 

[UPDATED Tues Oct 2 2:54am UTC 2014]

AT THIS TIME, JOYENT has patched the platform bash addressing CVE-2014-6271 as well as CVE-2014-7169 in the Joyent Public Cloud. Updates to pkgsrc bash are also now available in SmartOS pkgsrc repositories (please read details below under "Joyent Public Cloud"). 

SmartDataCenter customers should have received a notification by ZenDesk ticket with further instructions.

 

The impact of these vulnerabilities is as follows:

Joyent Public Cloud

  1. Joyent has applied a patch to the underlying platform of all our servers to address this bug. Users of many SmartOS VMs will NOT need to take any action. If the command: which bash returns "/usr/bin/bash", no action is required on your part. 

  2. Joyent has updated bash in pkgsrc. Please note some older pkgrsc repositories either do not contain bash or will not be patched due the their age, see the table below for details. The package that includes the fix to both CVE-2014-6271 & CVE-2014-7169 is called "bash-4.3.025nb2". If the command: which bash returns "/opt/local/bin/bash" you will need to either update the pkgsrc provided bash by running: "pkgin -f up && pkgin in bash" or remove the pkgsrc version via "pkgin rm bash". The latter command will remove the pkgsrc version and the machine will fall back to the patched platform provided version.
  3. Users of Linux VMs on Joyent's Public Cloud will need to apply the necessary updates, based on the distribution they are using. Please follow the appropriate link below:

Ubuntu: http://askubuntu.com/questions/528101/what-is-the-cve-2014-6271-bash-vulnerability-and-how-do-i-fix-it

Fedora: http://fedoramagazine.org/flaw-discovered-in-the-bash-shell-update-your-fedora-systems/

Debian: https://www.debian.org/security/2014/dsa-3032

CentOS: http://centosnow.blogspot.com/2014/09/critical-bash-updates-for-centos-5.html

Pkgsrc Repo Status User Action Required
2010Q4  Bash not in Repo  None
2011Q1  Wont be patched  Remove bash. See below
2011Q2  Wont be patched  Remove bash. See below
2011Q3  Wont be patched  Remove bash. See below
2011Q4  Wont be patched  Remove bash. See below
2012Q2  Wont be patched  Remove bash. See below
2012Q3  Wont be patched  Remove bash. See below
2012Q4 and later  Patch applied  Re-install Bash. See below

Removing bash

1. Check if any users are set to use pkgsrc bash

getent passwd | grep /opt/local/bin/bash

2. If you have users with pkgsrc bash as their shell, change it to /usr/bin/bash

usermod -s /usr/bin/bash <login>

3. Remove pkgsrc bash

pkgin rm bash

 

Re-installing pkgsrc

pkgin -f up && pkgin in bash

 

SmartDataCenter Customers

  • Both a hot-fix service and new platform image (for SDC 7 only) are now available to enable customers to update their SDC installations. Action will be required by customers to apply these fixes. Instructions for applying the hot-fix (for both SDC 6 and SDC 7) as well as applying the latest platform image for SDC 7 have been sent by ZenDesk to SDC customers. If you have NOT received instructions, you can open a support ticket at https://help.joyent.com or by email to support@joyent.com to request the link and instructions.
  • SmartDataCenter customers will need to advise their end users of this vulnerability, and advise them of any actions their end users will need to take. We will follow up with additional details as they are available.

At any time, please do not hesitate to contact our Support team by raising a ticket at https://help.joyent.com or by email to support@joyent.com if you have any questions or concerns.

 

Overview | Recent