Recently, we discovered an issue with Polycom CX500, CX600, and CX3000 model phones in our environment. New phones and phones that were hard reset while troubleshooting issues would no longer register with Lync.
As we were troubleshooting through log files on the phone, we noticed the following error – 0x2F0D/0 on phone = ERROR_WINHTTP_SECURE_INVALID_CA. The timing of this pointed to when we recently updated both our internal and external Lync certificates to Sha2. Digging in to articles online, this seemed to point to Entrust using new intermediate certificates in the chain that the phone did not have as part of its firmware.
We have been deploying new firmware to Polycom CX phone models through Lync using the ucupdates package from Microsoft. This method kept connected phones up-to-date automatically and included the new intermediate certificates. However, any phone that we tried to deploy out of the box was using an older firmware that did not have these certificates. Also, whenever we performed a hard reset of the phone, it reverted back to an older version of the firmware that then prevented the phone from registering with Lync.
This reminded me of the old adage, “Which came first? The chicken or the egg?”. The phone needs a new firmware in order to register to Lync, but we cannot receive the update unless the phone is registered to Lync. This presents a bit of a problem.
Luckily, we discovered there is a little known feature on the phones. During the boot up and connection process, the phones check for a record in your Lync environment called ucupdates-r2 based off your domain’s FQDN. For example, if your internal domain is ragnar.lothbrok.local, the phone searches for both ucupdates-r2 and ucupdates-r2.ragnar.lothbrok.local. If it finds DNS records that match, the phone will attempt to connect to this address to perform a firmware update. Part of the verification process requires that these DNS records also be found on the internal Lync front end certificate.
Next, the required DNS records were created and during off hours the Lync front end certificates were updated to include the new records in the additional SAN list. The update seemed to go well and Lync soft clients on Windows and Apple registered normally. I grabbed a Polycom phone out of the box, booted it up, and waited for a few minutes. The phone rebooted on its own and updated itself with the new firmware. I was able to register the phone with Lync using both the pin authentication method, as well as tethering a CX600 model phone to my PC and registering using the Lync soft client.
The next day I took some phones to our satellite office and attempted to register them as well. I booted up the phone and waited a few minutes, but nothing happened. Rebooted again and waited a few more minutes. The phones would not register with Lync. Digging in to the logs on the phone, I found an error stating that it could not find the SIP server FQDN. Looking at DHCP options, they all seemed to be set correctly. On my computer I ran the DHCPUtil.exe command with the –emulateclient parameter. This returned everything as expected, except the SIP FQDN was missing! I reset the DHCP options several times, but it would not return the FQDN. During evening hours, I rebooted the domain controller running DHCP. The next morning, I returned to our satellite office and booted up one of the phones. After a few minutes the phone updated and I was able to register it with Lync. DHCP options also returned the correct SIP FQDN setting.
All seemed to be going well until we started receiving reports of trouble with some Lync response groups. Phone calls to these response groups did not ring through to clients, but instead went directly to the voice mail set up for the response group. I tested with my response group and some others. They all worked normally.
Searching through log files on both the Lync front end server and edge servers, I found errors in the edge server logs pointing to certificates. I then realized that when I updated the Lync front end certificates, they were issued using a new CA. All other domain bound servers and computers in our environment trusted this new CA, as it had been pushed out through group policy. Because the edge servers are not domain bound, they never received the new CA certificate. I installed the new CA chain on the edge servers, rebooted the entire Lync environment overnight, and the response groups began working again.
As a side note, if there are other child domains that are part of the Lync environment using Polycom CX phones, DNS records must be set up for the ucupdates-r2 record, as well as added to the list of SANs on the Lync front end certificate. For example, ucupdates-r2.bjorn.lothbrok.local.
Tech Talk Live is the only conference of its kind in the region specifically designed for IT pros in education.
1020 New Holland Avenue, Lancaster, PA 17601