Summary
An improperly secured ElasticSearch database was recently discovered containing a huge volume of VOIP call logs, SMS/MMS message logs, and plaintext internal system credentials. This database was discovered using Shodan. Following a brief investigation, it was determined this database was controlled by VOIPO, a VOIP provider based in California. This database was promptly secured after I notified the company. I would like to thank VOIPO for their quick assistance in securing this data.
Incident Timeline:
| Date | Event |
|---|---|
| January 8, 2019 12:15 PM PST | Open ElasticSearch database discovered. |
| January 8, 2019 2:45 PM PST | Contacted CEO of VOIPO via LinkedIn, requesting that he email me. |
| January 8, 2019 3:19 PM PST | Email received from VOIPO asking for more information. Sometime shortly after this email, the ElasticSearch database went offline. |
| January 8, 2019 3:53 PM PST | Security issue reported to VOIPO via email. |
| January 8, 2019 3:53 PM PST | VOIPO acknowledged the data leak, and confirmed the server had been taken offline. |
VOIPO's response:
VOIPO indicated this was a development server that had accidentally been left publicly accessible. However, they also confirmed in their reply to me that the database also contained "valid data" -- i.e. real production data. They did not specify which data was allegedly development data and which was production data though. I would speculate that the leaked credentials (discussed below) were likely production credentials. It also appeared that the SMS/MMS and VOIP phone logs appeared likely to be production data.
What was found:
- Call logs (partial originating #, partial destination #, timestamp, duration of the call) going back to July 2017. 6.7M documents
- SMS/MMS logs (timestamp and content of the message sent) going back to December 2015. 6M documents
- Some of the logs contained references to internal hostnames. Some of the logs also included plaintext usernames and passwords for those systems. 1M documents
- Some of the logs contained API key for internal systems. 1M documents
Below is a sampling the indices that were exposed:
Index: events-2017-07 - events-2017-12, events-2018.08 - events-2019.01 (10 months, roughly 6.7M documents)
This index included the partial originating #, partial destination #, timestamp, and duration of VOIP calls. Based on the user agent of some of the calls observed, specifically sipcli/v1.8, it seemed likely that this VOIP provider was targetted (as many of them) by scanners attempting to make fraudulent 9xx premium calls.
"event-name": "CHANNEL_HANGUP_COMPLETE",
"core-uuid": "b36e2f18-d376-41be-bd70-3db52934e2bc",
"freeswitch-hostname": "pbx-core02.XXXXXXXXX",
"freeswitch-switchname": "pbx-core02.XXXXXXXXX",
"freeswitch-ipv4": "67.XX.XX.XX",
"freeswitch-ipv6": "::1",
"event-date-local": "2019-01-01 00:03:36",
"event-date-gmt": "Tue, 01 Jan 2019 00:03:36 GMT",
"event-date-timestamp": "1546301016",
"event-calling-file": "switch_core_state_machine.c",
"event-calling-function": "switch_core_session_reporting_state",
"event-calling-line-number": "908",
"event-sequence": "56207542",
"hangup-cause": "NORMAL_CLEARING",
"channel-state": "CS_REPORTING",
"channel-call-state": "HANGUP",
"channel-state-number": "11",
"channel-name": "sofia/public/[email protected]",
"unique-id": "8f3c77ce-dd08-4603-8523-1476517537ae",
"call-direction": "inbound",
"presence-call-direction": "inbound",
"channel-hit-dialplan": "true",
"channel-presence-id": "[email protected]",
"channel-call-uuid": "8f3c77ce-dd08-4603-8523-1476517537ae",
"domain": "pbx-core02.XXXXXXXXX",
"domain-id": "1",
"talk-time": 1.718999,
"phone-ring-time": 0.132995,
"duration": "2",
"other-leg-destination-number": "",
"lag-seconds": 0,
"direction": "inbound
Index: messages-2015.12 - messages-2019.01 (roughly four years, roughly 6M documents)
This index included a log of SMS and MMS messages sent over four years. This log included the full content of these messages as well.
"message":
"Fresh Drop \u0000CannablyssLakeStevens! Phat Panda Platinum series has arrived!!
Perfect way to bring in the New Year!
Remember, spend \u000245 or more and receive 15% off your purchase, when you whisper password: \"BLOWOUT\".
Ends Today 12/31!
*Excludes Sale Items* #MakeGoodChoices Text STOP to end"
Index: pbxlogs-2018.06.03 - pbxlogs-2019.01.08 (roughly 7 months, roughly 1M documents)
This index included what appeared to be internal hostnames, usernames, passwords, and API keys. These sensitive values were logged in plaintext, and these values had been exposed since June 3, 2018. Thank you to a colleague at my $dayjob -- Matthew Gall -- for his assistance decoding this garbled data into something I could easily make sense of.
[broadvox] => Broadvox_API Object
[uri] => https://XXXXXXX/EnterpriseServices/EnterpriseServicesWS/DIDOrder.asmx?WSDL
[username] => XXXXXXX
[password] => XXXXXXX
[telephonic] => Telephonic_Api Object
[uri] => https://XXXXXXX:8443/v1
[key] => XXXXXXX
[mysql] =>
[uri] => https://billing.XXXXXXX/includes/api.php
[username] => XXXXXXX
[password] => XXXXXXX
[kayako] => Kayako_Api Object
[uri] => https://helpdesk.XXXXXXX/api/index.php
[apiKey] => XXXXXXX
[secretKey] => XXXXXXX
[salt] => XXXXXXX
[e911] => e911_Api Object
[uri] => https://XXXXXXX/dash-api/xml/emergencyprovisioning/v1/
[username] => XXXXXXX
[password] => XXXXXXX
[broadvoxE911] => Broadvox_E911_API Object
[uri] => https://XXXXXXX/EnterpriseServices/EnterpriseServicesWS/E911Service.asmx?WSD
[username] => XXXXXXX
[password] => XXXXXXX
"file ": "XXXXXXX/htdocs/include/config/database/freeswitch.ini",
"section ": "freeswitch",
"data ": {
"freeswitch ": {
"hostname ": "XXXXXXX",
"username ": "XXXXXXX",
"password ": "XXXXXXX",
"database ": "freeswitch",
"file ": "XXXXXXX/htdocs/include/config/database/kamailio.ini",
"section ": "kamailio ",
"data ": {
"kamailio ": {
"hostname ": "XXXXXXX",
"username ": "XXXXXXX",
"password ": "XXXXXXX",
"database ": "openser",
"file ": "XXXXXXX/htdocs/include/config/database/mydns.ini",
"section ": "mydns ",
"data ": {
"mydns ": {
"hostname ": "XXXXXXX",
"username ": "XXXXXXX",
"password ": "XXXXXXX",
"database ": "mydns",
"file ": "XXXXXXX/htdocs/include/config/database/mydnsw.ini",
"section ": "mydnsw ",
"data ": {
"mydnsw ": {
"#hostname ": "XXXXXXX",
"hostname ": "XXXXXXX ",
"username ": "XXXXXXX",
"password ": "XXXXXXX",
"file ": "XXXXXXX/htdocs/include/config/database/voipo.ini",
"section ": "voipo ",
"data ": {
"voipo ": {
"hostname ": "XXXXXXX",
"username ": "XXXXXXX",
"password ": "XXXXXXX",
"database ": "voipo ",
"file ": "XXXXXXX/htdocs/include/config/database/voipopbx.ini",
"section ": "voipopbx ",
"data ": {
"voipopbx ": {
"hostname ": "XXXXXXX",
"username ": "XXXXXXX",
"password ": "XXXXXXX",
"database ": "voipoPBX",
"file ": "XXXXXXX/htdocs/include/config/database/whmcs.ini",
"section ": "whmcs ",
"data ": {
"whmcs ": {
"hostname ": "XXXXXXX",
"username ": "XXXXXXX",
"password ": "XXXXXXX",
"database ": "whmcs",
Index: siptracetests-2018.12.05 - siptracetests-2019.01.08 (34 days, roughly 25M)
This index appeared to include verbose logging for VOIP calls placed during this time period. I cannot speculate as to why this additional logging would be needed for some calls. The number of documents is quite significant.
"trace":
"U 2019/01/08 00:26:03.623800 67.XX.XX.XX:5060 -> 209.XX.XX.XX:5060\\n
INVITE sip:[email protected]:5060 SIP/2.0.\\n
Record-Route: <sip:67.XX.XX.XX;lr=on;ftag=gK0813a627;did=dab.9522>.\\n
Record-Route: <sip:67.XX.XX.XX;r2=on;lr=on;ftag=gK0813a627>.\\n
Record-Route: <sip:192.XX.XX.XX;r2=on;lr=on;ftag=gK0813a627>.\\n
Via: SIP/2.0/UDP 67.XX.XX.XX;branch=z9hG4bKb5e3.0308cc943b0a16fbc4ddf7afdfb2ef71.6.\\n
Via: SIP/2.0/UDP 67.XX.XX.XX;rport=5060;branch=z9hG4bKb5e3.1d952213.0.\\n
Via: SIP/2.0/UDP 192.XX.XX.XX:5060;branch=z9hG4bK08Bc0de0a9cfb7c2cce.\\n
From: \"WIRELESS CALLER\" <sip:[email protected]>;tag=gK0813a627.\\n
To: <sip:[email protected]>.\\n
Call-ID: [email protected].\\n
CSeq: 805091 INVITE.\\n
Max-Forwards: 67.\\n
Contact: \"WIRELESS CALLER\" <sip:[email protected]:5060>.\\n
Request-Disposition: fork, parallel.\\n
Content-Length: 239.\\n
Content-Type: application/sdp.\\n
P-Asserted-Identity: \"WIRELESS CALLER\" <sip:856266XXXX@pstn>.\\n
\\n
v=0.\\n
o=Sonus_UAC 556425 510008 IN IP4 192.XX.XX.XX.\\n
s=SIP Media Capabilities.\\n
c=IN IP4 67.XX.XX.XX.\\nt=0 0.\\n
m=audio 21922 RTP/AVP 0 101.\\n
a=rtpmap:0 PCMU/8000.\\n
a=rtpmap:101 telephone-event/8000.\\n
a=fmtp:101 0-15.\\n
a=sendrecv.\\n
a=maxptime:30.\\n
\\n"
Index: voipo-application-logs-2017.07.03 - voipo-application-logs-2019.01.08 (roughly 18 months, roughly 100 documents)
This index contained primarily a listing of polycom (conferencing) devices. The server client IP addresses of these devices has been logged, the MAC address, the timestamp, and the useragent of the device. In some cases the useragent value also appeared to include the MAC address of the device.
"inet": {
"server": 1160XXXXXX,
"client": 1282XXXXXX
},
"userAgent": "Htek UC840 2.0.4.4.33 00:1f:c1:XX:XX:XX",
"hash": "b8306b8c",
"tag": "f44178c8",
"@timestamp": "2019-01-08T13:23:42.000Z",
Potential implications of this data leak:
Call logs: In terms of the implications of the leaked VOIP call data, luckily this leaked data only contains partial phone numbers. It appeared some level of scrubbing/redacting has been applied before logging the phone numbers. This is good.
SMS/MMS logs: In terms of the leaked SMS/MMS messages going back roughly 4 years -- while I did not observe any 2FA values, it is not outside the realm of reason to speculate that some 2FA values could have been logged and leaked here. If an attacker had access to this data as part of a targeted attack on an individual or organization they knew to be using this service they would have been able to observe in real time the SMS being sent that contained the 2FA code. Hypothetically this could have then allowed the attacker to bypass 2FA on the user's account. Using SMS for 2FA continues to be a terrible idea, regardless of this particular data leak.
Internal hostnames, usernames, passwords, and API keys: It is difficult to overstate the severity of this part of the leak. Unless VOIPO had deployed adequate firewall protections (which this researcher did not test) to limit access to internal systems to a specific whitelist of IPs and/or a corporate VPN then leaked internal hostnames in combination with the leaked usernames and passwords could have resulted in a near total compromise of all leaked production systems. Just a small list of hypothetical scenarios that could have resulted from the leaked information:
- Billing -- This researcher did not confirm what was visible/accessible in the internal billing system, but hypothetically it could have contained full unredacted billing information for all of VOIPO's customers. This could have then lead to credit fraud and identity theft.
- Broadvox_API and Telephonic_Api -- Customers could have been phished and/or had their accounts deactivated which would have interrupted their service.
- Helpdesk -- Customers could have been sent fake/phishing support emails asking for account credentials.
- DNS -- DNS could have been updated, and then customers could have been silently phished into providing credentials.
- e911 -- e911 services could have been disabled for all customers, rendering those users unable to use e911 in an emergency situation.